Journal of Robotics

Journal of Robotics / 2010 / Article

Research Article | Open Access

Volume 2010 |Article ID 967379 | 13 pages |

Skill Evaluation from Observation of Discrete Hand Movements during Console Operation

Academic Editor: Suguru Arimoto
Received17 Jul 2009
Revised10 Dec 2009
Accepted15 Feb 2010
Published16 May 2010


This paper focused on discrete movements of hand in reaching actions, which necessarily occur during machine operation. The relationship between the performance of a console operation and the operator's reaching actions was investigated by applying Fitts' law and by examining a state transition of the operation. A remote operation experimental system was built using two radio controlled construction equipments, and the operator training process was analyzed empirically with those methods. The results showed the covariance of the fitting error to Fitts' law decreased as the operators' skill improved, although the fitting itself to the law was not sufficiently good. And it was confirmed that the covariance of difficulty index of the reaching action increased. These facts indicated that skill level of the discrete motion during operation can be estimated by investigating two types of the covariance.

1. Introduction

However sophisticated a machine is, training is necessary to familiarize a human user with its operation. To reduce user learning efforts, system developers must design an adequate human-machine interface that will be acceptable to a majority of people [1]. However, differences in individuals make it difficult to design such a versatile human-machine system. In addition, the human user has to adapt to the machine because the machine functions are fixed at the design stage. This situation, however, appears to contradict the very reason for the existence of a machine that was created to help humans and to make them comfortable.

To resolve this paradoxical situation, it is necessary for advanced machines of the future to change its functions by analyzing differences in individual human skills. Therefore, the concept of Human Adaptive Mechatronics (HAM) was proposed [2, 3], and studies of such intelligent mechatronics, that not only support a human operator but also enhance his/her abilities, have been promoted globally [4, 5]. Although other research activities that consider human interactions, such as the COG project [6], the intelligent, and the Oxygen Project [7], have lead to various knowledge and fruitful results, they do not deal with individual skill sufficiently. The first reason for this is that the concept of skill itself is ambiguous. In some instance, the term skill indicates a primitive action, such as the skill-based behavior in Rasmussen's model [8]; in other instances, this term expresses a knowledge-based optimization, such as task planning and scheduling; therefore, the interpretations of skill are quite broad. In practical applications, the concept of skill has been modified on a case-by-case basis and used depending on the contents of each task. An example of an established skill analysis method is Therblig, which is a practical classification method for analyzing manual operations [9]. Therblig is, however, a type of a coding procedure through video observation and does not define the general concept of skill. Furthermore, it is difficult to use this method as an algorithm that can be implemented in a computer.

The second difficulty is that skilled human actions arise from an excellent combination that involves perception, cognition, and motor control [10], as shown in Figure 1. This process has been studied in various fields, such as medicine, anatomy, brain science, cognitive science, psychology, control engineering, system science, and medical electronics; however, there are few studies from the human skill aspect. The ultimate elucidation of skill requires advanced study of the interaction among every process. Conversely, if it is the skill level of a user that is desired to be evaluated, then the detection of unnatural motions may be effective, because it can be expected that certain unskilled factor may induce such unnatural motion. The representative example is a jerky action in a hand reaching action, which is known as microslip. A microslip is a type of action hesitation we experience in everyday life, and this behavior includes tiny change in direction of hand reaching motion. Four types of microslip, trajectory change, hesitation, change of the hand shape, and touching objects, were defined [11]. Suzuki et al. reported that the microslip appears more frequently under high cognitive loads and complex circumstances [12]. In a previous study by the present authors, the relationship between task performance and microslips was investigated through a coffee-making test, and a strong correlation was confirmed statistically [13]. This result indicated that human skill in a manual operation can be estimated by observing the hand motions. Since the microslip is, however, found by a coding procedure through video observation by trained specialists, the result of the analysis depends on the skill level of the specialist and on individual differences. For the solution of this problem, the present authors tried to present an automatic detection method for microslip using visual processing, and the method could detect half of microslips that were found by human analysts [14]. However, the detection accuracy was insufficient. The reason appeared to come from low performance of the measurement system because the resolution of a standard camera ( pixel, 30 fps) used in the experiment was not enough to detect tiny behavior of the microslip. A high-performance camera will improve the detection accuracy; however, it would be inadvisable due to the high cost.

Existing studies concerning human characteristics of hand operation have mainly dealt with continuous time operation, including voluntary motion (e.g., [15]). Continuous time operation means that when a human operates a machine, he/she manipulates the machine by connecting the hand (or other body parts) to the machine's interface devices, as a result the human and the machine form a closed-loop system. The human is a nonlinear system, but can be treated as a continuous time dynamic system through a transfer function model such as “first-order” + “time-delay” when the operating point is limited [16]. The feedback-error-learning model [17, 18], which claims that a mechanism of human learning is the acquisition of an inner model of the controlled object's dynamics, is widely accepted. The crossover model, which expresses human flexibility and wide ranging adaptability, is also well known; it suggests that a human changes control characteristics so that the whole closed-loop transfer function of the human-machine system becomes a first-order system in a wide frequency band [19]. These models are often useful for designing the human-machine system. Actually, the present authors studied a hand positioning and its assisting method through a positioning task using a haptic interface device. A method for identifying operator control characteristics and evaluating the skill level was presented based on a similar transfer function model to the abovementioned human model, and an assistant control with a tiny force addition was proposed. Experimental analysis of participant data demonstrated that the assistant control could enhance operator learning [20].

On the other hand, in the general machine operation, discrete movements including reaching actions inevitably occur because machines are typically equipped with several switches. Discrete movements are operator hand motions that change the contact conditions between the operator and the machine interfaces. An interesting discovery concerning discrete movement is Fitts' law [21]. Fitts' law is an experimental formula of the relationship between the time and distance of voluntary hand motions. This law is valid for a reaching action of other body parts, of a hand in the water, and of nonstraight line courses [22]. Fitts' law is used for evaluation of computer interface designs, such as a GUI [2325] or a stylus [26]. As examples of application of Fitts' law to a machine operation, an evaluation of control laws for a haptic device [27] and an investigation of human cognitive processing [28] were reported. These studies, however, evaluated the computer system and not the human characteristics, because a hand reaching action is considered as a general movement already mastered by adult operators. To the authors' knowledge, evaluation of discrete movements in console operation, using Fitts' law, has not been reported so far, possibly because of the fact that the reaching actions during console operation are not always visual voluntary motions. The operator tends to look mainly at the controlled object or the environment, and moves the hand without looking at the target of the reaching action. The motion under such situation is not a visual voluntary motion, which is an assumption for applying Fitts' law.

As mentioned above, an observation of a hand motion behavior or an identification of operator's control characteristics on continuous-time operation can give a way to estimate operator skill. Such skill estimation methods, however, require more complex and high-cost devices than an original operation console system unit; hence, their method is not always applicable for any type of operation system. Therefore, paying attention to simplification of additional measurement systems for the skill estimation, the present study was initiated to establish a method of quantifying the skill of a machine operator by focusing on the discrete motion of hand. Since an approach presented in this study utilizes operational timing of switches of a console and the hand trajectories are not measured, the measurement system is very simple. This paper investigates the following questions experimentally.

Question 1. Is Fitts' law valid for a reaching action that is not pure visual voluntary motion during console operation?

Question 2. How do differences in skill level emerge on matching to Fitts' law?

Question 3. Is the skill estimation of a console operation possible by using the discrete motions of hands?

For experimental verification, a remote operation system was built that simulated the operation of actual construction equipment, and the process of learning its operation was analyzed based on Fitts' law. In order to validate a defined performance index for the experimental task, inching operations were investigated using a state transition matrix of operation commands. It is generally accepted that the frequency of inching operations indirectly reflects the performance of continuous time operation, since inching operations are interpreted as corrective actions for insufficient continuous time operations [29]. Then, a fitting to Fitts' law was investigated statistically, and aforementioned questions were verified.

This paper is organized as follows: Section 2 explains the preliminary computations for analyzing discrete hand movements; Section 3 explains the remote operation experimental system and the contents of the test; Section 4 explains several analysis methods of discrete movements that involve Fitts' law and the state transition matrix; Sections 5 and 6 present the results of the experiment and the analyses, respectively; and Section 7 provides the conclusions.

2. Data Treatment for Discrete Movement

Normal hand reaching actions have geometrically distinguishable starting and ending points. For console operations, it would be appropriate to separate discrete motions by timing the touching and releasing of the switches. Such segmentation is, however, inadequate, because there are switches that geometrically occupy the same position but have more than one function, such as a rotary or multidirectional switches. That is, reaching actions change depending on the intended operating direction, even for the same target switch. Therefore, reaching actions are classified based on differences in their functions and not on the switch layout.

2.1. Event and Sampling Counters

For automatic processing to find discrete movements from the measured data, two types of counters were introduced: a sampling counter and an event counter. The sampling counter is obtained by dividing an actual time by a sampling interval . The event counter is incremented by one whenever a specific event occurs. The sequence of the sampling counter when the th event occurs is defined as where is the total number of switching operation types, and indicates the th switching mode. It is assumed that when the operation is OFF or STOP. The sampling counts at which the switch status was changed were extracted using (1) from the logging data.

2.2. Extraction of Discrete Movements

One “reaching action” is a hand motion moving through the air to switch the console switches. Since the hand passes above the console and does not touch any switches during the reaching action, the ending point of an interval during which the operation mode is “non-STOP” is a candidate for the start timing of a reaching action. Using this property, both the starting and ending times, named and , are extracted from the sequence . Because all operation modes at must be zero, and an operation by a nonzero command must be executed at the previous sampling time, can be found by Similarly, the ending points are found by Therefore, the time of a reaching action is obtained as Applying (2) and (3) alternately and sequentially to the measured data, multiple data that are the starting and ending points and the times of their reaching actions are obtained.

3. Experimental Setup

3.1. Remote Operation of Radio Controlled Construction Equipment

A remote operation experimental system with radio controlled model construction equipment was devised as a human operation test system that requires discrete movements of the hands [30]. The purpose of the operation is a basic soil excavation work, as shown in Figure 2(a). The operator manipulated both an excavator and truck and was required to perform and to optimize task scheduling. Wireless cameras on the excavator and the truck-captured video images, displayed on monitors for the operator, as shown in Figure 2(b).

3.2. Task

Figure 3 shows a top view of the work area. The field size is and consists of motorable road and restricted areas. As the restricted area was 25 mm higher than the motorable road area, the operator could notice the restricted area when the machinery run on the restricted area. The field had three drilling sites and one unloading site. The excavator and truck were put at their starting points at the beginning of trial. The operator used console switches to move the machines to the drilling site, collected sample pieces with the excavator, loaded the pieces on the truck bed, and carried them to the unload site by the truck. The operator was instructed to collect different samples from the three drilling sites and to execute digging operations at each site only once. Only one type of sample was permitted to be loaded into the truck at a time. Unloading of the samples from the truck bed was done by vacuuming them after the truck arrived at the unloading site. One trial consisted of a set of all these operations. The total trial time was measured from the start to the time when the truck carrying its third samples reached the unloading site. Participants were asked to execute the tasks while maintaining the following conditions: (i)the total trial time as short as possible, (ii)the number of samples as large as possible, (iii)the number of “difficulty” as small as possible.

Here, “difficulty” indicates a limited access to the restricted area or the machines colliding. The comprehensive performance index given below was used for later analysis where and are the total time of one trial and the total number of collected sample pieces, respectively. Considering that a maximum of was about  sec and a minimum of was 12 for some participant who cooperated at the preliminary experiment, the constant values in (5) were decided.

3.3. Measurement System

To determine the positions of the excavator and truck in the field, the two machines were observed by a camera attached to the ceiling. The camera consists of a wide angle lens with an infrared filter. Eight infrared LED markers were placed on the work area for calibration of the optical distortion of the camera. The excavator and truck carried three and two infrared LED markers, respectively, and their positions and directions were computed using the detected positions of the LEDs. Data acquisition was performed by the Labview system with an image processing module and an AD/DA interface. After capturing the video image from the ceiling camera, the images of LED markers were extracted through the video processing, and their coordinate values were obtained by centroid computation. Compensation for optical distortion was performed offline using MATLAB. The operation of switches on the console was also recorded via the A/D and the DIO interfaces in the Labview system. The sampling frequencies of the video images and analog signals were 30 fps and 1 kHz, respectively.

4. Analysis Methods

Initially, the operation commands given through the levers and switches were discretized. In general, a machine console system is equipped with exclusive switches (e.g., the first and reverse gears of a shift lever) and combination switches (e.g., simultaneous operation using both the handle and acceleration pedal); hence, the operation commands were classified into independent and combination types. Analog commands, such as those of the handle and pedal angles, were translated into an integer numbered command. The classification details are explained in the following section.

4.1. Classification of Crawler Operation

Both the excavator and truck had crawler transporter systems in the present experiment. The right and left crawlers could be controlled independently based on the velocity commands, which are controlled by the angle of two sliders operated by hand, although the velocities of a real construction machine are controlled with foot pedals. Forward and backward motion, rotation, and pinwheeling were available. Because the velocity commands were transmitted by an analog signal, the analog data was converted into discrete values expressing a mode number in the present experiment. The crawler operation mode, which is termed , was obtained by checking inequality conditions concerning the velocities of the crawlers on both sides. Considering the effect of noise and drift in the analog signal, the mode was derived through the following algorithm, where and are velocity commands to the right and left crawlers, respectively, and both were normalized to the range .

Step 1. Classification of translation and rotation

Step 2. In case of translation,

Step 3. In case of rotation,

Step 4. In case of normal steering, where indicates threshold parameters, and “” and “” are abbreviations for “forward” and “backward,” respectively.

4.2. Classification of Bucket Arm Operation

A bucket arm with a three-link mechanism is mounted on the superstructure of an excavator. The bucket arm is manipulated by one cross lever and one slide lever. Although there are several possible types of combinations of the cross lever layout and the bucket arm movements, the JIS (Japanese Industrial Standards) type was chosen. In the console of the present experiment system, commands to the bucket were given by the on-off switching of the cross lever. The operation modes were determined as follows: where , and are action modes of the superstructure's rotation, the arm, and the bucket, respectively. As all 27 (=) combinations of the superstructure operation do not actually occur, an index for the bucket manipulation was assigned in order from the generated combinations. The first step is the computation of the combination index by The second step is to make the following new set by extracting different combination indexes after every is computed from experimental data Finally, describing for th element of the set , the operation mode at the time is obtained by

4.3. Fitts' Law for Reaching Actions

Using the truck crawler modes , the excavator crawler modes , and the excavator shovel operation mode , equation (1) yields the event counter sequence as The starting time and ending time were computed from the time sequences of , and by checking the status as follows: Then, the time of a th reaching action was computed as , where describes an index of the reaching actions.

The present study used the following formula from Fitts' law: where and are constant coefficients, is the distance to the target of a reaching action, and is the target width. is called an index of difficulty, and depends only on geometric data of the distance and width. and reportedly relate to the weight of an object carried by hand and the speed of the human information processing, respectively. is a fitting error by uncertainty against Fitts' law, and was newly introduced in the present study. appears to change for several reasons, such as additional cognitive loading and the effects of prediction. Figure 4 shows the layout of the console switches and the coordinate system. Each planar coordinate value of the switch position was registered as a table, and the distance was computed as a Euclid norm between the starting point and the ending point , obtained through a lookup table from the operation commands and , respectively. The target width is the size of lever grips, and it was obtained from a preregistered lookup table by checking the command . The coefficients and were estimated by a least square method with (16) using the multiple paired values from all the reaching actions that were obtained from the experimental logging data. Finally, the fitting error of the th reaching action was computed as To remove abnormal values from the measured data, reaching actions with larger than 3 sec were ignored, because normal actions were completed in less than about 2 sec. Discriminant equations (15) might detect fake reaching actions of which movement distance is zero; hence, such timing data was removed for the later analysis of reaching actions. On the other hand, the inching was investigated by other procedure that counted the repeating manipulation of the console switches, as explained in the next section.

Essentially, Fitts' law concerns the reaching action of one hand; however, reaching actions with both hands were sometimes required in console operation, and the law cannot be strictly applied to such cases. In the present study, however, to apply Fitts' law for the analysis, an operation requiring both hands was simplified to a virtual one hand operation using a virtual point, which is a midpoint of two positions of switches grasped by both hands.

4.4. Skill Confirmation by Checking Inching Actions

Although the performance index described by (5) was defined based on a result of the preliminary experiment so as to be adequate for skill evaluation, an appropriateness of the index was not proved yet. Hence, an inching operation, that is recognized empirically as a criterion to estimate skill level on a machine operation, was used for the validation of the proposed performance index. Inching is a technique for slowly moving machinery a small amount by applying short pulses of commands. Unskilled operator tends to execute inching when he/she does not finish learning dynamics of the machinery. Hence, if correlation between an incidence of inching and value of index is strong, reliability of the index would be enhanced. Hence, a state transition of the operation was computed from the logging data, and the inching was investigated by checking the state transition. The procedure is explained below.

First, all operation modes were summarized into one comprehensive variable . Considering and for the stop operation, was defined as Secondly, assuming the Markov process for the operation, the counting frequency of the transition from operation mode to operation mode , and putting this frequency on the element in the state transition matrix, . The element of matrix is computed by the following accumulation. as is increased from to , where is the maximum event counter. As the matrix contains transitions involving stop commands, the modified state transition matrix is computed by skipping the stop status () in (20). Finally, the transition probability matrix is obtained as where is -element of . Inching can be interpreted as a transition from -mode to the same -mode as it is a repetition of the same action; hence, on-diagonal elements in the matrix indicate ratio of the inching. The transition ratio of inching is computed as , where is an element of .

5. Experiment Results

Participants were ten males from 21 to 23 years old. Written consent and ethical approval from them were obtained before the experiment. 24 trials a day were conducted for each participant with each of them repeating 10 trials for several days. Participants were instructed to improve their operational performance while considering the criterion described in (5).

5.1. Total Task Time and Performance Index

Figure 5 shows the improvement in total time . Nearly all of participants exhibited a monotonic decrease in time and their performances in shortening the total time were enhanced.

The transition of the performance index computed by (5) is shown in Figure 6. The tendency to increase was confirmed. Approximating each curve to the regression line, the coefficient of gradient , the -intercept , and the correlation factor are summarized in the left three columns in Table 1. Since for participant E (labeled “(1a)”) appeared to be exceptionally small, the Smirnov-Grubbs test was applied to judge whether the value was an outlier or not. The test statistic was and the rejection region at significant level is ; hence, was satisfied, then was an outlier. Therefore, except for participant E, the approximation with the regression line is valid since the correlation factors are sufficiently large (0.75 to 0.89). The coefficients of gradient are all positive and large, at 0.44 to 0.84; hence, the data prove that all participants could enhance their performance in increasing performance index .


A 0.60 2.520.89 0.69 −0.26 0.73
B 0.84 4.050.87 1.65 0.32 0.89
C 0.54 3.540.87 0.52 1.33 0.68
D 0.68 4.760.86 1.20 1.93 0.89
E 0.13 6.870.31(1a) 0.20 5.450.22(1b)
F 0.63 3.710.88 1.07 0.51 0.87
G 0.72 4.530.79 1.35 1.54 0.79
H 0.67 3.110.87 1.18 −0.99 0.82
I 0.51 3.120.82 0.79 0.33 0.80
J 0.44 5.900.75 0.79 3.75 0.79

Figure 7 shows the transition of the soil excavation efficiency , which is the number of samples carried per minute to the final unloading site. From the first to sixth trial, nearly all of participants demonstrated an upward trend, and the curves for some participants were saturated after the sixth trial. The parameters of the regression line were computed similarly, and they are shown in the right three columns in Table 1. Because the correlation factors were sufficiently large (0.68 to 0.89), except for participant E (, labeled “(1b)”), this approximation was also valid. Their gradient coefficients were positive and large (0.52 to 1.65); hence, the work efficiency was also successfully improved.

As an example, trajectories of the operated machines on the field for participant A, whose correlation coefficient of was largest (), are shown in Figure 8. The (a) and (b) graphs show the trajectories for the first and tenth trials, respectively. As the trajectories of tenth trial are smooth, with fewer zigzags than in the first trial, it can be confirmed intuitively that the operational skill increased.

5.2. Validation of Proposed Performance Index

Figure 9 shows the transition ratio of inching . Although most curves appear to decrease, it is difficult to observe an obvious tendency from the figure; hence, correlations between and were investigated. Table 2 shows the parameters obtained by the correlation analysis; the coefficients and denote the gradient coefficients and correlation factors, respectively. The data of participant E who was judged as an outlier by aforementioned analysis summarized in Table 1 was not shown in Table 2. Most of and were larger than 0.5. For all participants (other than E), the correlations between inching and total task time were strong ranging from 0.30–0.88. (labeled “()”) for participant D was, however, solely positive. Applying the Smirnov-Grubbs test to this value, was an outlier because the test statistic satisfied . Hence, gradients for all valid participants (other than D) were all negative. This result indicated that the incidence of inching (that tends to decrease as the skill improved) and the performance index defined by (5) had negative correlation. Therefore, the performance index was adequate for evaluation of the skill level. In what follows, is treated as an index showing skill level.


A 62.60.88 −0.60 0.65
B 38.20.85 −0.23 0.46
C 26.70.75 −0.31 0.70
D 7.3 0.30  (2) 0.25 0.48
F 12.00.80 −0.29 0.84
G 5.0 0.42 −0.24 0.66
H 15.70.69 −0.36 0.63
I 22.60.55 −0.33 0.76
J 7.80.56 −0.23 0.63

6. Analysis Utilizing Fitts' Law

6.1. Fitting Results and Trend Analysis

First, results of the fitting to Fitts' law (16) were summarized in Figure 10. One graph shows a case of one participant. On each graph, transition of the identified coefficients and (solid lines) and the correlation factor (dashed line) are shown. Averages of their values are also written in each graph. Since their correlation factors were not so large at 0.24–0.53, the fitting result to Fitts' law was not good. The bias term , as drawn using blue solid line, changes widely. The transition of gradient coefficients , as drawn using red solid line, was flat compared with . Monotonic tendencies that synchronize with the number of trials in transition of or were not confirmed. Since the performance index indicated an increasing tendency but , and did not show specific tendencies, it was surmised that the correlation between and was weak.

Second, individual differences concerning their coefficients were investigated through a correlation test between the Fitts' parameters and the performance index. That is, correlation between an average , say , and the gradient of , , was analyzed since an index that represents characteristics of total ten trials should be chosen for comparison of the individual difference. (Because, it appeared that indicated characteristics of each individual since this value was almost constant at any trial, as shown in Figure 10.) Assuming that a data size is and that a sample correlation coefficient is , an uncorrelated test statistic matches the distribution with degrees of freedom; hence, the correlation between and was tested using this property. In this case, and was not satisfied; hence, the correlation was not significant at the 0.05 level. Concerning , and ; hence, the correlation was not also confirmed because was not satisfied. As a conclusion, there was no individual difference concerning estimated coefficients of Fitts' law.

Thirdly, the authors investigated whether there was any common tendency to all participants in some statistical values. The survey was performed mainly by using average values of each participant since no individual differences concerning the Fitts' fitting were confirmed as demonstrated in an aforementioned correlation analysis. As a result, characteristic tendencies were found in changes of the fitting error and the index of difficulty. Figure 11 shows the trends of the mean and covariance of the fitting error for all eight participants. The transition of the average of all participants' mean values is indicated by a bold solid line in Figure 11(a). While the transition of the average was almost flat, the average of the covariance decreased as shown in Figure 11(b). Figure 12 shows the mean value and covariance of the index of difficulty computed by (17). Similarly, each dotted line denotes the data of each participant, and the bold solid line is the average for all eight participants. It was confirmed that both the average and variance showed an upward tendency. These results suggest that the participants repeated simple and similar type of reaching actions at the beginning and were able to execute various different types of reaching actions as their skill improved.

6.2. Detail Analysis

Correlation analysis was applied to find clear relationship between the skill level and reaching actions because some tendencies in covariance values of the index of difficulty and the fitting error were found. Below, these covariance values are described as and , respectively. and were chosen as task performance factors at the correlation analysis. The computed correlation coefficients were summarized in Table 3. In the table, a description expresses the correlation coefficient between and . The table shows that nearly all of the coefficients had the same sign except for participant G in (labeled “(3a)”). Seeing the original transition for this participant, the covariance of did not decrease uniformly. Applying the Smirnov-Grubbs test to this with significance level , the test value for participant G, , was not outlier; however, the test value is close to the significant point . Considering a result of this test with the sole positive value in , participant G was judged as an exceptional case. Therefore, the tendencies were almost identical among the participants.


A−0.77 0.83 0.86 −0.73
B−0.47 0.42 0.24 −0.56
C−0.71 0.35 0.79 −0.24
F−0.85 0.24 0.54 −0.11
G−0.59 0.07 0.71   (3a)0.22
H−0.54 0.71 0.63 −0.41
I−0.78 0.55 0.80 −0.59
J−0.40 0.42 0.34 −0.50

As discussed above, because the learning processes of participants D, E, and G were exceptional compared with the other seven participants, the following conclusions about the majority of the participants are inferred. (i)Covariance of the index of difficulty increases as the number of trials increases. In other words, variation in reaching actions increases with the operator skill.(ii) Covariance of the fitting error to Fitts' law decreases, although an accuracy of the fitting to the law was not sufficiently large.

7. Conclusion

The potential of a skill evaluation method based on observation of human hand motion was verified experimentally as an early phase study to establish a method for a machine to evaluate human skill ability. Focusing on reaching actions during console operation, the discrete movements of hands were investigated based on Fitts' law. Using a remote operation system for a radio controlled model of construction equipment, the training processes of operators were observed and analyzed with the proposed methods. First, in order to validate a defined performance index for the experimental task, inching operations, which is considered to reflect performance of machinery manipulation, were investigated. As a result, validity of the defined performance index was verified since the correlation between the performance index and other three indexes (the incidence of inching, the total task time, and the soil excavation efficiency) was confirmed, and the performance index was declared a skill index for the present task.

Secondly, results of the fitting to Fitts' law were analyzed. The correlation factors showed that the accuracy of the fitting was not sufficiently large for reaching actions during a console operation (answer of Question 1), and there were no individual differences among all participants concerning the identified Fitts' parameters. It was, however, confirmed that the covariance of the fitting error to Fitts' law decreased and that the covariance of the index of difficulty of reaching actions increased as the skill improved (answer of Question 2). These facts indicated that the aforementioned two types of covariance were utilizable to evaluate operator skill. In conclusion, an investigation of perturbations from Fitts' law of discrete hand movements could evaluate the skill level for machine console operation (answer of Question 3).

Next step in future study is an establishment of a formula or a method computing skill level value from variables that concerns a discrete motion. Based on the conclusion obtained in this paper, if subjective discussion is permitted, a formula such as is a candidate to quantify the skill, where and are covariance of the fitting error to Fitts' law and of the difficulty index of reaching action, respectively. To determine adequate parameters and structure of the formula, sufficiently many evaluation data will be required for the statistical verification.

Moreover, the environmental conditions and machines have to be considered to enhance the skill estimation, because an operator chooses the operation commands according to circumstances. With respect to such statuses, the present authors have been studying human command switching through a self-organizing map that includes environmental conditions [31]. In addition, human task scheduling with a discrete state transition model, such as a timed automaton, has been studied [32]. The network structures and timing of the state transition appear to be able to incorporate differences in the skill level. The authors are considering conducting studies on these topics in the future.


Research topics mentioned in this paper were supported by a Grant-in-Aid for the Scientific Research (A) 20246071 of the Japanese Ministry of Education, Culture, Sports, Science and Technology. Data from the participants were measured with the support of Yorito Maeda and Akihiro Makita. Authors appreciate their cooperation.


  1. D. A. Norman, The Design of Everyday Things, Basic Books, New York, NY, USA, 1988.
  2. K. Furuta, “Control of pendulum: from super mechano-system to human adaptive mechatronics,” in Proceedings of the 42nd IEEE Conference on Decision and Control, vol. 2, pp. 1498–1507, Maui, Hawaii, USA, December 2003. View at: Google Scholar
  3. F. Harashima, “Human adaptive mechatronics,” in Proceedings of the IEEE Industrial Electronics Society, Keynote Speech, CD-ROM, Raleigh, NC, USA, 2005. View at: Google Scholar
  4. “Special section papers on human-adaptive mechatronics,” IEEE Transactions on Industrial Electronics, vol. 52, no. 5, 2005. View at: Google Scholar
  5. “Special issue on human adaptive mechatronics,” International Journal of Modeling, Identification and Control, vol. 4, no. 4, 2008. View at: Google Scholar
  6. R. A. Brooks and L. A. Stein, “Building brain for bodies,” Tech. Rep. MIT AI Lab Memo 1439, pp. 1–15, 1993. View at: Google Scholar
  8. J. Rasmussen, A. M. Pejtersen, and L. P. Goodstein, Cognitive System Engineering, John Wiley & Sons, New York, NY, USA, 1994.
  9. B. Price, “Frank and Lillian Gilbreth and the motion study controversy, 1907–1930,” in A Mental Revolution: Scientific Management since Taylor, D. Nelson, Ed., The Ohio State University Press, 1990. View at: Google Scholar
  10. S. K. Card, T. Moran, and A. Newell, The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, Hillsdale, NJ, USA, 1983.
  11. M. Schwartz, E. S. Reed, M. Montgomery, C. Palmer, and N. H. Mayer, “The quantitative description of action disorganisation after brain damage: A case study,” Cognitive Neuropsychology, vol. 8, no. 5, pp. 381–414, 1991. View at: Google Scholar
  12. K. Suzuki, H. Mishima, M. Sasaki et al., “Affordance and variability of action-environment , action and microslips,” Journal of Japan Society for Fuzzy Theory and Systems, vol. 9, no. 6, pp. 826–837, 1997. View at: Google Scholar
  13. H. Kobayashi, T. Yasuda, and S. Suzuki, “The relations between development of human's manipulative skills and physiological signals of brain and hand,” International Journal of Assistive Robotics and Mechatronics, vol. 7, no. 1, pp. 49–61, 2006. View at: Google Scholar
  14. S. Takeuchi, S. Suzuki, H. Kobayashi, T. Yasuda, and M. Kakikura, “Quantification of skill from a hand's operation and micro slip,” in Proceedings of the International Conference on Instrumentation, Control and Information Technology (SICE '05), pp. 1558–1563, Okayama, Japan, August 2005. View at: Google Scholar
  15. T. Tsuji and Y. Tanaka, “Tracking control properties of human-robotic systems based on impedance control,” IEEE Transactions on Systems, Man, and Cybernetics A, vol. 35, no. 4, pp. 523–535, 2005. View at: Publisher Site | Google Scholar
  16. A. Tustin, “The nature of the operator's response in manual control and its implications for controller design,” Journal of the Institution of Electrical Engineers A, vol. 94, part 2, pp. 190–202, 1947. View at: Google Scholar
  17. M. Ito, “Internal model visualized,” Nature, vol. 403, no. 6766, pp. 153–154, 2000. View at: Publisher Site | Google Scholar
  18. M. Kawato and H. Gomi, “A computational model of four regions of the cerebellum based on feedback-error learning,” Biological Cybernetics, vol. 68, no. 2, pp. 95–103, 1992. View at: Publisher Site | Google Scholar
  19. D. T. McRuer and E. S. Krendel, “Mathematical models of human pilot behavior,” Tech. Rep. AGARDograph 188, NATO Advisory Group for Aerospace Research and Development, 1974. View at: Google Scholar
  20. S. Suzuki, K. Kurihara, K. Furuta, and F. Harashima, “Assistance control on a haptic system for human adaptive mechatronics,” Advanced Robotics, vol. 20, no. 3, pp. 323–348, 2006. View at: Publisher Site | Google Scholar
  21. P. M. Fitts, “The information capacity of the human motor system in controlling the amplitude of movement,” Journal of Experimental Psychology, vol. 47, no. 6, pp. 381–391, 1954. View at: Publisher Site | Google Scholar
  22. J. Accot and S. Zhai, “Beyond Fitts' law: models for trajectory-based HCI tasks,” in Proceedings of the Conference on Human Factors in Computing Systems (CHI '97), pp. 295–302, Atlanta, Ga, USA, March 1997. View at: Google Scholar
  23. M. J. Mcguffin and R. Balakrishnan, “Fitts' law and expanding targets: experimental studies and designs for user interfaces,” ACM Transactions on Computer-Human Interaction, vol. 12, no. 4, pp. 388–422, 2005. View at: Google Scholar
  24. S. Zhai, S. Conversy, M. Beaudouin-Lafon, and Y. Guiard, “Human on-line response to target expansion,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '03), vol. 5, pp. 177–184, Fort Lauderdale, Fla, USA, April 2003. View at: Google Scholar
  25. A. Cockburn and P. Brock, “Human on-line response to visual and motor target expansion,” in Proceedings of the 32nd Graphics Interface, pp. 81–87, Quebec, Canada, June 2006. View at: Google Scholar
  26. X. Ren and S. Mizobuchi, “Investigating the usability of the stylus pen on handheld devices,” in Proceedings of the 4th Annual Workshop on HCI Research in MIS, pp. 30–34, Las Vegas, Nev, USA, 2005. View at: Google Scholar
  27. I. Emeagwali, P. Marayong, J. J. Abbott, and A. M. Okamura, “Performance analysis of steady-hand teleoperation versus cooperative manipulation,” in Proceedings of the 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS '04), pp. 316–322, Chicago, Ill, USA, March 2004. View at: Google Scholar
  28. A. Leganchuk, S. Zhai, and W. Buxton, “Manual and cognitive benefits of two-handed input: an experimental study,” Transactions on Human-Computer Interaction, vol. 5, no. 4, pp. 326–359, 1998. View at: Google Scholar
  29. T. Yukimachi and Y. Okada, “A study of manual control with inching operation,” The Japanese Journal of Ergonomics, vol. 25, no. 1, pp. 1–9, 1989 (Japanese). View at: Google Scholar
  30. S. Suzuki and F. Harashima, “Skill analysis focused on the hand discrete movement for machine manipulation,” in Proceedings of the 13th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA '08), pp. 156–163, Hamburg, Germany, September 2008. View at: Google Scholar
  31. S. Suzuki and F. Harashima, “Segmentation and analysis of console operation using self-organizing map with cluster growing method,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '09), pp. 4875–4880, St. Louis, Mo, USA, October 2009. View at: Publisher Site | Google Scholar
  32. F. Miyawaki, K. Masamune, S. Suzuki, K. Yoshimitsu, and J. Vain, “Scrub nurse robot system—intraoperative motion analysis of a scrub nurse and timed-automata-based model for surgery,” IEEE Transactions on Industrial Electronics, vol. 52, no. 5, pp. 1227–1235, 2005. View at: Publisher Site | Google Scholar

Copyright © 2010 Satoshi Suzuki and Fumio Harashima. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

More related articles

950 Views | 521 Downloads | 6 Citations
 PDF  Download Citation  Citation
 Download other formatsMore
 Order printed copiesOrder

Related articles

We are committed to sharing findings related to COVID-19 as quickly and safely as possible. Any author submitting a COVID-19 paper should notify us at to ensure their research is fast-tracked and made available on a preprint server as soon as possible. We will be providing unlimited waivers of publication charges for accepted articles related to COVID-19. Sign up here as a reviewer to help fast-track new submissions.