Abstract
In industrial human-robot collaboration, variability commonly exists in the operation environment and the components, which induces uncertainty and error that require frequent manual intervention for rectification. Conventional teach pendants can be physically demanding to use and require user training prior to operation. Thus, a more effective control interface is required. In this paper, the design and evaluation of a contactless gesture control system using Leap Motion is described. The design process involves the use of RULA human factor analysis tool. Separately, an exploratory usability test was conducted to compare three usability aspects between the developed gesture control system and an off-the-shelf conventional touchscreen teach pendant. This paper focuses on the user-centred design methodology of the gesture control system. The novelties of this research are the use of human factor analysis tools in the human-centred development process, as well as the gesture control design that enable users to control industrial robot’s motion by its joints and tool centre point position. The system has potential to use as an input device for industrial robot control in a human-robot collaboration scene. The developed gesture control system was targeting applications in system recovery and error correction in flexible manufacturing environment shared between humans and robots. The system allows operators to control an industrial robot without the requirement of significant training.
1. Introduction
Many high-value low-volume manufacturing tasks are labour intensive, and workers are often required to work in poor conditions and uncomfortable positions. Due to the physical nature of medium and large-scale manufacturing, workers spend considerable amount of time traveling around the factory floor and sizable work pieces to retrieve tools and to carry out work on various locations, which prolong manufacturing cycle time. On the other hand, the use of full automation on these applications is impeded by the complex nature of the processes and the environments. However, by combining the strengths of humans and robots, human-robot collaborative systems could be viable manufacturing solutions for these applications [1]. Collaborative robots could operate as assistants in shared workspaces to reduce manual handling and improve ergonomic of manual tasks by minimising the requirements for human to work in awkward positions and to lift heavy parts, which also improve productivity [2, 3].
Interaction is an integral part of human-robot collaboration that requires effective communication for seamless performance [4]. Industrial robots are often programmed and controlled using robot teach pendants, which require trained users who have knowledge of the user interface and its proprietary programming languages. These training requirements have negative implications on the overall implementation cost and the complexity of the controls could reduce the seamlessness of the interaction. On the other hand, the investment cost of effective human-machine interfaces could be recovered by savings generated by a more efficient working system which will turn into profit once the cost is broke even [5].
In human work teams, verbal and gesture communications are practically inseparable. Hand gestures are often used to reinforce the speaker’s ideas, and people sometimes communicate only through hand gestures [6]. Thus, it is logical to consider using similar methods for communication in human-robot cooperative systems. Gesture communication is sometimes preferred in industrial settings due to the restrictions of working environments; a prime example is hand signals for crane operation where communication between operators could subject to high background noise and over a long distance. It shows that gesture communication could be valuable in human-robot cooperative manufacturing systems, especially those in unstructured open workspaces.
In large-scale manufacturing, assistant industrial robots could manipulate large components but the position of components may require frequent adjustment due to misalignments. In this case, gesture control systems enable workers to guide robots into correct positions using simple hand gestures to continue the tasks. Furthermore, gesture control could be used anywhere within a work cell. In a large scale manufacturing environment the robot controller can be separated from operator by a significant distance; the ability to command the robot without the need to relocate will have positive impact on process cycle time.
The feasibility of performing robot teleoperation using gestures interfaces has been studied before. These studies have used both contact and noncontact digital devices. Handheld accelerometer-based input device [7, 8] and photogrammetry based data gloves [9–11] are examples of handheld and wearable interfaces. These devices can track position and orientation of hands but they can hinder human-limb movements and cause discomfort for prolonged use [12]. Furthermore, the use of wearable devices is less preferable in industrial contexts due to the time required for wearing, removing and calibrating the devices, and wear and tear will increase maintenance cost. Contactless gesture user interfaces enable human-machine interaction using natural hand movements in a noninvasive manner to users’ motions and the surroundings which makes it a preferable option for this research [13].
Existing research on robot teleoperation using contactless gesture control could be classified into two main types and these are imitation of human movements [12, 14–18] and gesture recognition [19–26]. These researches have focused on technological development with a lack of application and ergonomic considerations; these could affect users’ health and experience for practical long-term use. Some research has shown that gesture interfaces designed without human factors considerations are prone to cause negative health implication such as muscle fatigue after prolonged usage [27–29]; apart from intuitiveness of the system, biomechanical constraints of humans are also important design elements. In terms of intuitiveness and control precision, Norman (2010) has pointed out that a pure gestural system makes it difficult to discover the accurate dynamics of execution, but the problem could be overcome by adding conventional interface elements such as menus, operations, tutorials, and other forms of feedback and guides. It was also suggested that because gesturing is a natural, automatic behaviour, the system must be adjusted to avoid false responses to movements that were not intended to be system inputs [30]. It is particularly important when industrial robots could potentially cause damage to people and surrounding objects by false triggers.
The novelty of the research presented in this paper is the human-centred approach used in the development of a hand gesture robot control system which enables the user to perform motion control and simple programming of an industrial robot. The user can control the robot via joint movement as well as linear movement in Cartesian space. The system was designed using a user-centred design approach with the use of human factor analysis tool in the design of gesture commands. The system has undergone functionality test and usability test with participants to compare physical workload associated with using the system and to gather suggestion for future improvements.
The rest of the paper is organised as follows. First, the characteristics of the system and its functionality are described in System architecture. In this chapter the system is described in detail as well as the design process of the gesture and design constraints. In the method section the system testing method and details are described. The findings of the experiment are explained in results and discussion. Finally the paper is concluded with future work suggestions.
2. System Architecture
The system consists of a small industrial robot with six degrees of freedom (DoF), control system with software written in C#, and a hand-tracking sensor. A Leap Motion sensor is used to capture the user’s hands positions and orientations and the control software processes the input and sends signals to robot controller. A decoder programme is embedded in the robot controller which receives signals from the control software through TCP server and actuate robot to perform actions. The system architecture is illustrated in Figure 1.
2.1. Hands and Fingers Tracking
The Leap Motion Controller is a low cost consumer hand tracking device designed as a human-computer interface. The controller is a small device with surface area of 24 cm2 and it is based on stereo vision which has three Infrared Light emitters and two Infrared cameras. The field of view of the controller is approximately 150° and the effective range is approximately 25 to 600 mm above the device. The controller is capable of tracking position of hands, fingers, and finger-like tools with submillimetre accuracy. The manufacturer claimed that the accuracy of fingertip detection is approximately 0.01 mm and the frame rate is up to 300 fps. However, Weichert et al., 2013, have performed a series of tests to measure Leap Motion’s accuracy with an industrial robot and they have discovered different results as the manufacturer has stated. It is summarised that it is not possible to achieve the claimed accuracy of 0.01 mm under real condition, but the overall average accuracy of 0.7 mm from the experiments is still relatively good for gesture-based user interfaces [31]. On the other hand, Guna et al. (2014) have performed a series of test on the Leap Motion with the aid of a professional fast and high-accuracy motion tracking system, and by measuring a plastic arm with pointing finger they have measured accuracy with standard deviation of 0.5 mm and best case at less than 0.01 mm. However, they found a significant increase in the standard deviation when moving towards the edge of the controller’s working area [32]. Furthermore, the set of measurements in the dynamic scenario have revealed some inconsistent performance of the controller which is limited by its inconsistent sampling frequency. The accuracy of tracking data can be improved by applying filtering algorithms in the programme. Du and Zhang (2015) have incorporated both particle filters and Kalman filters to improve the tracking accuracy of Leap Motion. However, the recorded tracking error is still significant compared to typical manufacturing tolerance in high-value production [33]. The developed system utilise a gesture control method based on hand positional thresholds which is not affected by the reported tracking error. Furthermore, the Leap Motion can measure hands positional data which cannot be performed with other markerless and noncontact off-the-shelf device so it is a suitable sensor to be used in this system.
2.2. Gesture Control for Robotic Applications
Hand and body gestures are broadly recognised as a natural way of communication, but when designing gestures as machine inputs or control it is important to consider the constraints of both the technology and the human. The main design objectives are to create a natural human-machine interface for users to perform robot motion control without significant training and to allow operators to work in comfortable postures. One of the many causes of work-related musculoskeletal disorders is the adoption of static or constrained postures. Using a control system may involve placing a load on the musculoskeletal system and discomfort, pain, and fatigue will be influenced by the amount, duration, and distribution of this load [34]. Thus, a gesture control system used in prolonged periods should be designed with ergonomic considerations that reduce risks to these injuries.
Gesture control should be responsive where the user can alter robot motion with immediate intended response. This provides instant feedbacks which reassures users that the robot is operating according to the their input. In this case, dynamic gestures are less preferred due to their completion time of typically over half a second that cause delayed system responses; therefore, it cannot provide a continuous input to control the robot motion. The actuation method used in the developed system is based on relative hand positions so the robot responses when user’s hands reach the actuation positions.
The BS ISO/IEC 30113-1 and BS ISO 9241-960 provide guidelines for the design of gesture-based interfaces. The standard highlighted a number of requirements and recommendations and these include activating/finishing a gesture, performing a gesture, feedback for confirming a gesture, feedforward, cancelling a gesture, criteria of gesture size, controlling the criteria, changing correspondence of a gesture command, and descriptions of individual gestures within part. A number of these recommendations have been considered during the design of the developed system [35, 36].
2.3. Technological Constraints and Solutions
Machines perceive human gestures through sensors by using algorithms to recognise and to track a human body and its components, so it is important to know the limitation of the technology to design a gesture control system that is reliable and robust to use. The developed system uses the leap motion as an input device. The device API provides tracking information of the user’s hands which include the pitch, roll, yaw, and the , , positions relative to the sensor’s centre point. The API can also recognise pinch or grab motions and return values between zero to one where one is a full pinch or grab, and zero is none. As explained previously, the Leap Motion device has tracking errors which can be compensated by incorporating appropriate filters as demonstrated in [12]. However, the errors have remained relatively high compared to typical high-value production manufacturing tolerance [33]. For this reason, real-time robot teleoperation involving mimicking human hand movement is not feasible for critical tasks that require precise positioning because precise hand positional data is one of the key enablers for this method. However, for gesture recognition the Leap Motion has sufficient accuracy when applied with appropriate filtering and a well-designed gesture set. The developed system uses a mean filter to remove noises from the sensor data stream prior to gesture recognition to stabilise the hand tracking. For recognition of the hand gestures, the user’s hands must be arranged within a set of positional thresholds which are relative to a set of calibrated hands position values recorded prior to motion control. The thresholds could be adjusted to minimise the ambiguity between different gestures and to reduce sensitivity to tracking uncertainties which improve the system robustness.
2.4. Robot Control
The robot used is a Universal Robots UR5 as shown in Figure 2. It has a common layout with six joints: base, shoulder, elbow, wrist 1, wrist 2, and wrist 3. This is a common layout for industrial robots for its flexibility, spherical working envelop, and adaptability to numerous applications. This robot is controlled similarly to most industrial robots using a touch screen teach pendant as shown in Figure 3.
The purpose of the developed system described in this paper is to enable users to control robot movements precisely without the use of a teach pendant. This novel gesture control system enables users to move the robot by joint and linear movements. These different control strategies are integrated into one control programme by segregating different control methods into modes where the users can switch between modes with a simple pinch motion of their left hand.
The developed software consists of two joint modes and a Cartesian mode. In joint mode 1, the user can take control of the robot’s base, shoulder, and elbow. Joint mode 2 enables control of wrist 1, wrist 2, and wrist 3. The linear mode enables user to move the robot in , , and directions of the base frame (Figure 2 and Table 1). The list of modes, functions, and triggers are shown in Table 1. The level of control is expandable to include rotational movements and actuation of auxiliary systems.
The hands recognition and tracking modules process data from the sensor, passing the tracking data on to the data enhancing module before forwarding to the gesture recognition module. If the identifier recognises a hand gesture, the system sends a signal to the robot controller. Each gesture has a threshold which could be adjusted to determine the sensitivity for recognition. For example, the pitch of the right hand has to reach a certain degree to trigger a signal as illustrated in Figure 4. The base value of these thresholds is defined using the results of the ergonomic analysis.
The control PC communicates with the robot controller via TCP/IP network. The control software sends encoded signals to the robot controller and a decoder within the robot programme interprets the signals and drives the robot according to the input. Each encoded signal is an array which contains variables representing the movement mode, joints speeds, velocity vectors, and rotation angles. The robot controller decoder interprets this array and sets the speed of the robot depending on the mode of movement and directional values. For movements in Cartesian space, the controller sets the speed in the format of () where () represent the linear movement and (α, β, γ) represent the rotation movement about the -, -, -axes. Joints movements are assigned by rotational speed in the format of (J1, J2, J3, J4, J5, J6). The robot movement is configured according to the acceleration, speed, and response time of the robot. The design of this control system is to ensure the user interface is comfortable and intuitive to use without the requirement of significant training and the user can produce input with simple hand movements.
Due to individual differences, people have varied neutral positions of the wrists, which affects radial deviation and flexion of the wrists. For adapting to different users, the neutral hands positions are stored in temporary memory during initiation and this calibration occurs at the beginning of every mode change cycle. Furthermore, some system characteristics were included purposely to enhance robustness and safety. For example, the robot movement is progressive and the speed is limited to less than 250 mm/s so the user should notice any abnormal movement of the robot and react to it. The system only operates when both hands are present above the sensor, if the user lifts one or both hands away from the sensor; then the robot will stop immediately. The system also features an “Off mode” for when the system is not being used and this feature avoid false trigger by movements near the sensor.
A simple graphical user interface has been designed to provide information about robot mode and the hands tracking state of the programme as shown in Figure 5. An image of a pair of hands appears when both hands are detected above the sensor. Directional arrows become visible when user’s hands go above or below any positional thresholds; this reassures the user that the robot is engaged in the appropriate mode and provides the status of the hand tracking module. The interface is shown on a screen in front of the sensor and the user.
2.5. Human Constraints
The developed system receives input from the users in the form of hand gestures which detects the arrangements of arms, hands, and fingers. Thus, the posture analysis focuses on musculoskeletal movements involved in changing the hands’ positions and orientations. The control hand movements are broken down into potential risks of musculoskeletal injury. For instance, changing the hands’ pitch, roll, and yaw requires wrist flexion, radial deviation, and ulnar deviation, respectively. Changing the , , position of the hand requires upper arm movement or lower arm movement and sometimes both. When designing a control system, these required body movements should be identified and the implication on health should be assessed.
The Rapid Upper Limb Assessment (RULA) [37] postural analysis is used to aid the design of gestures and assess the risk of musculoskeletal injury associate with system usage. It is an effective method for assessing the risk level of task requires movement of the upper limbs. The tool is chosen based on its simplicity and reliability compared to other assessment methods [38].
The RULA assessment method is used in the assessment of human constraint during the system design phase. The assessment has a score system with two parts which include part A, arm and wrist analysis, and part B, neck, trunk, and leg analysis. The scores accumulate at each step and the final score is calculated using a combination of three score tables.
The arm and wrist analysis indicates that the upper arm position should stay within ±20 degrees from the vertical axis or the score increases. A raised shoulder or abducted arm also increases the score which should be avoided. The lower arm position should stay within 60–100 degrees from the vertical axis; otherwise, the risk increases. Any deviation of the lower arms from the midline of the body will increase the risk, but it is inevitable if one of the hand positions is required to change in the -axis of the sensor. Wrist flexion within ±15 degrees will add two points, but any movement over the range will add three points. Any ulnar deviation will add another one point to the score, but it is avoidable by using lower arm movement instead of changing hand yaw in sensor values. Wrist twist within midrange will only add one point to the wrist and arm score. Finally, highly repeatable posture or prolonged static posture will increase the risk as well as added force or load larger than 4.4 lbs on the hand. The developed system has received minimum risk score from the neck, trunk, and leg analysis as it is design to be used in comfortable standing posture as described in BS EN ISO 9241-5:1999 [39].
The RULA assessment is the core of the human-centred design process of this development. The human constraints highlighted from the assessment are considered at the initial gesture design stage as well as each design iterations. It was established at the beginning that in order to achieve intuitive gesture control the user’s hands will be moving within a Cartesian space and some wrist flexion movement will be required. In this case, the threshold for low risk hands movements can be calculated using the joint angles provided in RULA.
The Leap Motion sensor was positioned flat on a solid surface and in front of the user as originally intended. The , , positions of one of the user’s hands will be used to actually control input which require shoulder and elbow joint movements. It is known from the assessment that neither joints should be required to move over ±20 degrees from a default position. Using this information combined with anthropomorphic data the ideal sensor height, working area, and threshold for actuation are calculated. Figure 6 is a model in the operating posture; the parameters used in the calculation for the system setup are labelled. Maximum joints movement angles for low risk classification are obtained from the RULA assessment. Limb lengths can be obtained from national anthropomorphic data according for use in different region.
The sensor height Hs is calculated using the elbow height He, the neutral lower position angle , the elbow to fingertip length EF, and the ideal hand to sensor height Hhs which can be expressed asThe lower arm movement threshold Mla is calculated using the elbow to fingertip length EF and the maximum low risk elbow joint angle which can be expressed asThe upper arm movement threshold Mua is calculated using the shoulder to elbow length SE and maximum low risk shoulder joint angle which can be expressed as
3. Method
The evaluation of the contactless robot control system is described in this section. Traditionally the motion of a robot manipulator is controlled using a teach pendant. However, in a shared human-robot collaborative workspace a more flexible and intuitive way of communicating with robots is required. Using the omnipresent gesture control, the user can control a robot without having to hold any devices, which reduces physical workload and enables seamless cooperation. Evaluations were carried out during the design process and after the development to address potential ergonomic issues at design phase and to assess the user experience of the developed system by comparing it with a standard robot teach pendant.
3.1. RULA Study
A comparison study was carried out at the development stage to compare the risk of using the developed system against a touch screen teach pendant as the benchmark using RULA. RULA has a final score scale from one to seven where a final score of one to two means the posture is acceptable, three to four means change may be needed, five and six require further investigation and change required soon, and a score of seven leads to investigation and implements change.
3.2. Exploratory Usability Study
In addition to the RULA assessment, an exploratory ergonomic study was carried out.
3.2.1. Participants
Eight people from the general population of Cranfield University have participated in the usability experiment. Four of them were males and four of them were females, and all participants were right-handed. Their age ranged from 23 to 51 years with mean of 30 (SD = 9.27). Five of the participants have some experience in robot control and three of the participants are inexperienced robot operator.
3.2.2. Design
This experiment followed a within-subject design. The two independent variables are interface type and usability attribute, and the dependent variable is the usability score. Participants had to complete one time in each of the two conditions, which are the gesture control and teach pendant. The testing orders for the two control interfaces were counterbalanced to mitigate the risk of carryover effects. Each experiment took around 35 minutes.
3.2.3. Experiment Setup
The experiment was conducted in a 4 m × 4 m laboratory area surrounded by four sides of wall. The gesture control system was tested with a Universal Robot UR5 robot positioned on top of a stand at a height of 1 m. The robot has 850 mm reach with a circular working envelop and features a touch-screen teach pendant, which was used in the test. The Leap Motion device and a laptop were positioned next to the edge of the robot’s working envelop on a table with adjustable height. The robot was equipped with a two-finger gripper that was deactivated. Three black plastic pipes were suspended within the working envelop of the robot on a worktop which were used as obstacles during the test.
3.2.4. Procedure
Informed consent was sought from participants prior to participating in the experiment. Participants were informed regarding their right to withdraw at any time, confidentially and anonymously. They were asked to provide their demographic information (age, sex, and job title) in a questionnaire. A scripted experiment briefing was given to each participant to explain about the purpose of the study and the expectations of him or her.
Participants were given a five-minute brief about the basic of industrial articulated robot with 6DoF which covers motion control techniques. Before the test of each control interface, participants were given a five-minute training and practicing period. The actual testing time of each system lasted 10 minutes, participants were given the option to take a five minutes’ break after the first test before continuing to the second test. Each participant was asked to fill out a system usability questionnaire (SUS) [40] immediately after each test. Participants were asked to score the system against three ergonomic criterions which are physical workload, intuitiveness, and enjoyableness. These criteria are similar to those described by Bhuiyan and Picking in [41], but the questionnaire used in this test was simplified. Each criterion is assessed based on a seven-point Likert scale where seven is positive and zero is negative. The purpose of the questionnaire is to measure their experience of using the system.
The experiment focuses on evaluating ergonomic aspects of the systems and participants were not given specific task to perform. They were instructed to first try all possible movements with each system and then manipulating the robot gripper around the suspended plastic pipes without hitting them. They were told that each test will end in 10 minutes, but they can stop at any time if they are too tired to continue.
4. Discussion
In the RULA test a higher score indicates an increased risk of musculoskeletal injury after prolonged usage. In this particular assessment gesture control scored three points and teach pendants scored four points. The score of the teach pendant can further increases if the device weighs over two kilograms. It was identified that the wrist movement in Leap Motion Robot Control poses the biggest problem in terms of posture, but the risk is relatively insignificant. The analysis highlighted potential issue with the neck when using a teach pendant, because its operating angle suggests a high load which could lead to musculoskeletal discomfort when used for prolonged periods. Furthermore, the lower arms have to be raised to a significant angle when using a teach pendant which may cause tiredness after a long period of usage. It is also found that prolonged usage of a teach pendant causes tiredness in the wrist of the hand holding the teach pendant. The RULA results are summarised in Table 2.
The ergonomic test results are summarised in Table 3 and presented in Figure 7. Two participants verbally reported it was tiring to hold the robot teach pendant during the test which lead them to leave the device on the worktop in majority of the test. However, in the questionnaire they have provided a positive score for the level of tiredness which were contradictory. Thus, their results were deemed as inaccurate and discarded.
The mean scores for physical workload were 3.83 (SD = 0.75) for gesture control and 3.33 (SD = 1.03) for teach pendant. For intuitiveness, gesture control received a mean score of 4.5 (SD = 0.84) and teach pendant received a mean score of 4.33 (SD = 1.03). The enjoyability mean scores were 5.67 (SD = 0.52) for gesture control and 5.33 (SD = 0.52) for teach pendant. A Wilcoxon Signed-Rank test showed that there are no significant difference between the usability scores in all categories: physical workload (, ), intuitiveness (, ), and enjoyability (, ). Nonetheless, the developed gesture control system is comparable with the off-the-shelf teach pendant in terms of usability with some addition advantages in human-robot collaboration scenarios.
Some participants have reported the visual feedback of the GUI and the tactile feedback of the touchscreen have made it easier to use, which indicates the potential benefit of incorporating visual feedback and tactile feedback into gesture control system. Half of the participants reported that the teach pendant is heavy to hold.
5. Conclusion and Future Work
The purpose of the developed system is not to completely replace a traditional robot teach pendant, but to operate as a complementary input device in an interactive environment. The system is designed to be a secondary input device to reduce travel distance of operator to location of robot teach pendant as well as to reduce risk of musculoskeletal injury from frequent usage. The developed gesture control system was targeting applications in system recovery and error correction in large-scale manufacturing environment. The system allows operators to control an industrial robot without the requirement of significant training. The experiment shows that the developed gesture control system has potential to be used as an input device for industrial robot control in a human-robot collaboration scene. However, addition features could be incorporated in the future to improve the ease of use and intuitiveness. The usability of gesture control for robot can be enhanced when complemented by a graphical interface with conventional elements which provide user with visual feedback. It is also suggested that tactile feedback can improve user experience. Visual feedback can be achieved by using augmented reality or virtual reality technology to provide real-time information and instructions. Contactless tactile feedback can be provided by integrating ultrasonic force fields devices around the control area. Such implementations are in the scope of future work.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
The authors would like to thank Teegan Johnson who provided practical advice on the use of RULA assessment that greatly helped the research. This work was funded by the EPSRC Centre for Innovative Manufacturing in Intelligent Automation under the Grant Reference no. EP/1033467/1.