Abstract
Human characteristics concerning voluntary motion control are investigated, because this motion is fundamental for the machine operation and human-computer system. Using a force feedback haptic device and a balancing task of a virtual inverted pendulum, participants were trained in the task, and hand motion/force was measured, and brain activity was monitored. First, through brain analysis by near-infrared spectroscopy (NIRS) and motion analysis of the pendulum, we identified a participant who was the most expert. Next, control characteristics of the most expert were investigated by considering the operational force and delay factor of a human. As a result, it was found that predictive control based on velocity information was used predominantly although a perception feedback control against the pendulum posture worked. And it was shown that an on-off intermittency control, which was a strategy for the skilled balancing, can be described well by a liner model involving two types of time shifts for the position and velocity. In addition, it was confirmed that the cortex activity for observation in an ocular motor control area and visual processing area was strong to enhance above-mentioned control strategies.
1. Introduction
Modern life is surrounded and enhanced by gadgets based on mechatronics and computer products. However, in some cases of machine manipulations, such as driving of a vehicle, it appears to be implicitly required that humans must train themselves to use the machines. Hence, control theories on a human-machine system and the development of assistive robots and computer systems have become an active area of research. On this issue, many international research projects have been launched: Human Adaptive Mechatronics (HAM) project [1, 2] whose main concerns are analyses of a human skill and the establishment of the assistive methods, COGNIRON [3], which is concerned with robot companions for human-centered environments, and the MORPHA [4] project, which studies interactions with intelligent system assistants.
Because knowledge of human control characteristics is useful in designing an operational assistance system, human modeling is a significant theme in such research projects, and this theme has been studied in the field of control engineering since its early beginnings. Regarding basic models, a linear servo control model [5], a PID-based time-variant model having randomness [6], and an optimal control model [7] had been proposed. A crossover model that expresses human flexibility and wide-range adaptability is frequently utilized to tune a controller in a human-machine system [8].
Skilled motion of human body parts requires an adequate cooperation between perception from sensory receptors, cognition in the brain, and motor control, as in the model human processor (MHP) [9], and voluntary motor control and visuomotor control are especially significant. Points of discussion about the visuomotor control appear as (a) time-delay compensation, (b) learning, and (c) estimation. Since delays lie between about 30 ms for a spinal reflex up to 200–300 ms for a visually guided response [10], the visuomotor control cannot work well without some compensation for delay. This is why the discussion (a) occurs. The Smith-predictor is often used as a human model to compensate for time-delays of the nerve system [11]; however, a forward model on the sensory preprocessing loop of control is another good model because the estimation error in the Smith-predictor does not converge theoretically if the controlled system is unstable [12]. Regarding the discussion (b), learning, feedback-error-learning model is widely accepted [13]. After learning is finished sufficiently, a forward model-based controller gives the most suitable explanation for the visuomotor control [14]. And, the forward or internal model is considered a basic mechanism to estimate the next state of the body parts given the current state and action [12]. Mechanism of the visuomotor control has been studied through various experimental tasks: a hand reaching test under dissociated visual information against proprioception [15], a tracking task using an impedance-controlled robot [16], and a crank turning task [17]. Many studies on the stick balancing or manual control of a pendulum are also reported [18–25], and the following facts about the skilled stick balancer are known. (F1) The distribution of change in the hand velocity is a truncated Lévy distribution [21]. (F2) The power spectrum of fluctuations in the stick's height shows two scaling regions with two different power laws [20, 24, 26]. (F3) About 98% of corrective movement occurred faster when compared to the time delay of human perception [22].
Concerning (F1), it was reported that increases in stick balancing skill over were mirrored by a broad of the tails of the distribution of the changes in speed of the hand. The reason of the truncation is the decrease of the role of closed-loop feedback [21]. The fact described in (F2) is interpreted as a sign for on-off intermittency. Existence of the on-off intermittency control means that human is not a simple continuous-time controller but a complex of controllers that are switched depending on the circumstances. Since the two different power laws are also found in both finger-tip direct stick balancing and virtual indirect stick balancing, this tendency appears an intrinsic characteristic of a skilled human who controls unstable object. Fact (F3) is very fascinating and suggests that an effect of direct visual feedback is small. The phenomenon described in (F3) is interpreted as a result of parametric noise control or drift-and-act control [25]. So-called noise control increases stability of slow (passive) phase in the on-off intermittency control (the drift-and-act control). It has already been verified that such random fluctuations can improve balance control [23]. The noise control appears to be governed in the motor system in the brain due to faster processing than delays in the nerve system for visual perception; however, visual attention to the controlled object is also required to switch the on/off phases. The above discussion is summed up by the fact that the human control in stick balancing is a complex of the delay compensator in the sensory loop, a parametric noise control in the motor system, and a feedback-like switcher for on-off intermittency. Then how is the human brain activated to execute such complex processing?
On the other hand, although force was often measured to investigate operational characteristics of a human, the force information was rarely considered in previous studies on the stick balancing task, where alternative information such as the hand position, the cart velocity, or the acceleration computed from the positional data was used. In a precise sense, movement of the stick is a result that occurs after the human affects it; the movement is not direct information from the human. For analysis of skill for the machine operation, a relation between a human and a machine should be considered carefully in order to avoid misconstruing the machine characteristics as human properties.
In addition, two delays should be considered in the position and velocity variables for description of a human controller, because the velocity information cannot be recognized using only measurement of the present time . That is, the velocity is computed as a difference between the present position and past position , where is some delay on perception [27]. In most of the studies, however, one type of time delay was only treated (Although two delays were considered in the proportional-minus-delay-controller in [28], the discussion there was based on the theoretical analysis using control engineering and not on the human response data. In [29], two delays associated with the proprioception and vision control loops were introduced; however, human postural sway control, which was not pure voluntary motion control, was analyzed. Moreover, both studies assumed solely pure time-delay and did not consider prediction effect that can be expressed by minus value of time delay.).
From the above-mentioned discussions, in the present study, the following were performed: (O1) development of a virtual stick balancing task system using an input device that enables to generate precise reaction force, (O2) wide-area monitoring of the brain cortex in the process of learning during a machine operation, and(O3) identification of the human control characteristics using the force information by considering two delays for the position and velocity.
For (O1), an haptic interface, that had a movable unit controlled with specified dynamics and equipped with a force sensor, was developed. Using this interface system, data for the identification denoted in (O3) were obtained.
Regarding (O2), several noninvasive brain function measurement methods capable of normal activity are known, as summarized in Figure 1. The functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) have made a significant contribution to elucidation of the functions of the nervous system. These methods, however, request to rest completely supine or prone inside the tunnel and the subject cannot move the body. That is, it is difficult for the fMRI and PET to investigate the brain of an operator who manipulates the machine in a natural attitude. Other popular method capable to monitor brain activity concerning voluntary motion is the electroencephalogram; however, the measured signal is contaminated by electric noise from the device that was operated by the subject. Hence, in the present study, a near-infrared spectroscopy (NIRS) system was used to resolve above-mentioned issues. The NIRS system measures changes in the concentration of oxy- and deoxy-hemoglobin using different spectra lasers in the near-infrared range. Since “the hemodynamic response is partly related to neuronal activity [30],” the activation strength at each local brain area can be estimated by detecting changes in the concentration. The NIRS is robust against electrical noise because of laser measurement and allows it to measure the brain activation of natural behavior in a nonrestrictive environment, such as speaking [31], reading [32], and language recognition [33]. In addition, there are several reports showing robustness of the NIRS against body heady motion: cyclic coupled movements of the hand and foot [34], multijoint discrete motor task (i.e., bilboquet) [35], and medical rehabilitation by walking [36]. Of course, the NIRS is not a perfect method and there are some objections due to the limitation of the spacial and temporal resolution. Alternative approach to investigate voluntary motion might be the motor imagery (MI) using fMRI [37]. However, in practice, it can be very difficult for subjects, especially novices, to perform MI at all [38]. Considering situations mentioned above, for the wide-area monitoring that permits the subject to manipulate the interface device, there is no adequate measurement system except the NIRS.
In the present study, actions described in (O1)–(O3) were performed, and properties of the most expert were investigated. As previous studies using NIRS did not pay attention to a decomposition of the multiple activation patterns, the measured NIRS signals were analyzed by principal component analysis (PCA).
The latter sections are organized as follows: Section 2 explains details of an experimental system and PCA. The experimental results are shown in Section 3. Section 4 describes the preliminary analysis of brain activation and manipulation in the accreditation of the most expert in the balancing task. Main analyses of the brain and the control characteristics of the certified expert are outlined in Sections 5 and 6, respectively. Section 7 presents the conclusion and discussion.
2. Experimental System Setup
The left side of Figure 2 shows a photograph of the experimental setup. The experimental system consists of three units: a real-time computer graphics (CG) generator of a virtual pendulum, a haptic interface device, and an NIRS system. The participant sat in a chair to suppress excessive artifact caused by unnatural body movement. The arm and elbow were not fixed to the chair since we intentionally allowed the participant to posture themselves. In order to give the participant a sense of supporting the virtual pendulum on the palm, the participant manipulated a grip fixed to the slider of the interface device by placing the back of the right hand on the grip. The force added by the participant was detected by the sensor embedded in the slider. The motion of the virtual pendulum was computed in real time using the detected force. Details of each unit are explained below.
(a)
(b)
2.1. Virtual Pendulum
Although many studies have been conducted on manual stabilization of an (virtual) inverted pendulum (-like motion) using an input device [12, 24], there are few studies utilizing such device that can generate precise reaction force. Commercially available joysticks with force-feedback function were used in some researches, but such joystick cannot measure the human force and cannot generate precise reaction force due to insufficient linearity, large hysteresis, and backlash. Therefore, in the present study, a linear stage controlled using a virtual-internal-model control (VIM) was utilized in order to make the operator feel precise reaction force.
In what follows, computation of virtual pendulum motion with force input is mentioned briefly, because a numerical model is required to explain the later analysis in Section 6. Motion equations of a standard linear-type pendulum adopted in this research are given as where , , , , , and are the inertia of the pendulum around the center of gravity, masses of the pendulum-link and the cart, viscous coefficients of the pendulum and the cart, total length of the link, and gravity acceleration, respectively. Variables , and are the inclination angle of the pendulum link, the position of the cart, and the virtual exogenous force computed from the measured force . The notations of and are abbreviations for and , respectively. For a real-time computation, (1) and (2) are modified as Since (3) yields , values of the variables are obtained as follows by Euler integration: where is a sampling interval. Using computed by (5), the posture of the CG-pendulum was drawn using the Direct-3D library, and the motion picture was displayed via a projector. In the experiment, the parameters of the pendulum were chosen as , and so as to simulate a light pendulum. As the scale factor was chosen as , the participant could virtually feel the reaction force of the g pendulum.
2.2. Haptic Interface Controlled by VIM
The haptic device is a one degree-of-freedom slider driven by a linear direct drive motor, produced by NSK corporation, and the maximum force is and the accuracy of positioning is 0.3 . The VIM control was used to move the slider like the cart of the virtual pendulum. The dynamic equation of the slider is expressed as where , and are the mass, viscous coefficient, position of the slider, direct force added by the participant, and force generated by the motor, respectively. Because the movable range of the slider was limited to mm, scale relations between the actual slider and the virtual cart were converted using the scale factor as
Defining the error between the actual slider position and the virtual cart position as , (2) and (6) yield where Using matrices and , the adequate positive definite weighting matrix , and semidefinite , the feedback gain is computed by the popular LQR method, and the final form of the control law is obtained as follows: The block diagram of the control scheme is summarized in Figure 3. The force was measured using a 6 DoF force/torque sensor (IFS-70M35A25-M50B, NITTA Corporation, Osaka, Japan. The maximum measuring load is 98 N.) and the one horizontal direction force was used. The measurement noise was eliminated by LPF whose cut-off frequency was 31.25 Hz. Control interval of the VIM was ms.
2.3. Brain Monitoring and Analysis for Voluntary Motion
Functions of the brain are differentiated roughly into local areas. The brain map shown in Figure 4(a) explains this localization. The primary motor cortex (MsI) and the primary somatosensory cortex (SmI) are important for voluntary motion, and the movements of most muscles in the body are controlled by local regions in these cortices. The correspondence relationship is known as the motor and sensor homunculus (see Figure 4(b)). The fold that separates the parietal lobe from the frontal lobe is called the central sulcus. Figure 4(b) shows a cross-section of the brain at the central sulcus. In neuroscience, five hierarchical structures describe the mechanisms of the motor control system [39], as shown in Figure 5. An intention of motion is generated from the cerebral limbic system and from the rear side of the parietal cortex. The generated intention is transferred to the basal ganglion and to the corresponding projected area in the frontal cortex. The latter area includes the supplementary motor area (SMA) and the premotor cortex (PMC). The third-level layer is a descending motor pathway involving two paths of a pyramidal tract and an extrapyramidal tract. The former pyramidal tract has its origin in the PMC and the SmI.
Since various areas relate to the voluntary motion as mentioned above, ideally all areas should be monitored for the analysis. The NIRS can, however, measure only cortices that lie on the surface of the brain; hence, SMP, PMC MsI, and SmI, that lie in the cortex and relate to voluntary motion, were monitored simultaneously in the present study. To monitor the large area covering these regions, a long sideways probe cap was used in the NIRS measurement, as shown on the right in Figure 2. The International 10–20 measurement system, which is an application method that uses the electrodes of an electroencephalogram and is based on the distance between the nasion and the inion of the scalp, was used to determine the position of the probe. Changes in the concentration of total hemoglobin (sum of oxy- and deoxy-hemoglobin) were measured using an ETG-4000 system (Hitachi Medical Corporation, Tokyo, Japan). Reflections of lasers in the near-infrared were measured 10 times during each sampling interval, and the measured data was output every 100 milliseconds by averaging these reflections to attenuate noise effect.
By means of PCA, 48ch data of total-Hb measured by NIRS were analyzed. PCA is a kind of analysis of multivariate data, and it can identify subsets of variables that contain the main features of all the data [40]. PCA transforms a number of correlated signals into a smaller number of uncorrelated variables. Because this method yields a correlation pattern for the signals, the brain activation patterns that indicate mutual relations of the brain's local areas can be investigated [41]. Another advantage for PCA is its robustness against measurement noise, because the method is a statistical computation based on covariance of the signals. PCA is explained in more detail below.
From an multivariable sampled data set , an interval covariance matrix at time of is computed by where and are the total length of the sampled data and the length of interval for PCA analysis, respectively. Singular Value Decomposition (SVD) of is computed as where . If unitary matrix is denoted as , can be decomposed as Computing next SVD for gives Here, is a second principal component. Repeating the computation in the same manner, the strength sequence for the decomposed mode , and the corresponding distribution vectors are obtained. Visualization of these components in in a 2D-geometric map of the brain yields a topographic map of the th decomposed mode.
2.4. Methods of Examination
Analysis for novices in the stick balancing can be very difficult [38], and a skill level for the same person dynamically changes [42]. In short, it is difficult in the middle of training to find an expert; hence, all participants had been monitored during all training. Nine right-handed volunteers were tested. They comprised five males aged 22 to 23 years and four females who were 21, 22, 52, and 55 years of age; none had a history of neurological deficits. Written consent and ethical approval were obtained before the examinations. At least 10 trials a day were organized for each participant basically. The training period was selected such that they were consecutive weekdays. For the brain monitoring, 90-second rest periods were allowed between trials. During the rest period, the participant was instructed to close his/her eyes so as not to receive any visual stimulation. Because of the necessity of the rest time, the total time of the training became long even if continuance of successful balancing was short; hence, the training was stopped so as to avoid any influence of fatigue of the participant if the total time exceeded 30 minutes. As the primary objective of the experiment was to obtain data regarding the skilled operator, training for more than four days was conducted for those participants who showed potential as skilled balancers. For this reason, the number of days of training differs between the participants
3. Experimental Results
Three days after the start of the training, one participant could not continue the training because of discomfort caused by the head gear used for the brain monitoring, and valid data were obtained from total eight participants. Each participant began manipulating the haptic device for stabilizing the virtual pendulum from the moment that the pendulum was inverted vertically. Continuance of the stabilization was counted from the starting moment to the time when the pendulum-link slanted more than 45 degrees. Figure 6 shows the improvement of the continuance of all participants. The -axis is the number of the trial days. The maximum, average, and raw data of all continuances of each day were drawn using a solid line, dotted line, and dots, respectively. It was found that participants E and G succeeded in stabilizing the pendulum for more than 200 seconds on the final day; they were thus categorized into a group named the high-performance (HP) group. Participants B, D, F, and H could not stabilize the pendulum for more than 30 seconds, and the average was less than 10 seconds. Although participant A could stabilize it over 90 seconds only once on the fourth and fifth days, the averages were low as 20 seconds. As the progress of participants A, B, D, F, and H were comprehensively low, they grouped into the low-performance (LP) category. The performance of participant C was not as good as that in the HP-group and was therefore the sole member of the moderate-performance (MP) group.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
4. Accreditation of Expert
In this section, first, the satisfactoriness of analysis conditions for PCA is discussed, and the most expert in the virtual stick balancing is then identified by utilizing the PCA results. Finally, a highly skilled operator is certificated by checking the characteristics that were discussed in Section 1.
4.1. Confirmation of Validity of PCA
Because it is preferable that PCA is applied to nontransition signals because of the necessity of covariance computation, the data for the analysis needs to be extracted from the whole of the measured data to obtain the steady-state situation period. Investigating the transition of the topographic map using the raw measured data as a preliminary analysis, it was found that the data of successful stabilization was comparatively steady state. (Conversely, other data of the trial-and-error phase was nonsteady.) By considering the time-constant of change in the hemoglobin concentration, the period of moving computation for PCA was specified as 10 seconds (the data points were ) so as to prevent loss of the PCA computational results. To verify whether the PCA works well under these conditions, the strength of the decomposed modes was investigated by applying it to the data obtained from participants in HP, MP, and LP groups. Concerning the LP group, participant A was chosen since only this participant could stabilize the pendulum for more than 10 seconds, which is a requirement of the above-mentioned PCA data condition. Participant C was the sole member of the MP group and was therefore automatically selected as this group's representative. Concerning the HP group, participant G was chosen as the representative through consideration of comparisons with participant G and participant E. (The reason for this will be explained in a later section.) Figure 7 shows the change of the decomposed modes for the first 400 seconds of data on each final day for HP, MP, and LP group. In all three cases, the first modes were strongest, and the 2nd–48th modes were almost zero. This shows that investigating the first mode alone is sufficient for checking brain status; hence only the first mode was investigated. In checking the strength of the modes of the three participants, HP was the strongest and MP was the second strongest. Further, the strength of HP increased monotonically, although that of the other group was almost constant after about 200 seconds. Because the data measured by NIRS does not yield an absolute value, this comparison between participants is not strictly valid; however, it appears that this continuously strong activation is characteristic of a skilled operator.
(a) HP
(b) MP
(c) LP
Next, potential for expertise was checked for participants E and G of the HP group. The data of the first trial on the final day and of the longest successful trial were chosen for PCA, because the participant had mastered stabilization and was still not fatigued. In short, the second (185 s) and third trials (311 s) on the fifth day were chosen for participant E, and the second (75 s) and fifth trials (903 s) on the sixth day were chosen for participant G.
Topographical images obtained by PCA are shown in Figure 8. The grayscale pattern of all images shows the relative strength of each decomposed distribution of topographic patterns. The black areas show strongly activated areas, and the white areas weakly activated areas. In these images, the brain functional areas were labeled by referring to the homunculus of Figure 4. In Figure 8, the patterns of the upper and lower maps in each participant did not change, in short, they showed steady state. Comparison of the two participants, however, shows that the map for participant E is covered with large white areas and the other map of participant G has large dark areas that indicate strong activation. Concerning the strength of their decomposed modes, participant G maintained a level as high as 50 after 400 seconds, as shown on the upper graph in Figure 7. On the other hand, participant E was as low as 20 after 100 seconds (the graph is omitted due to limitations of space). This tendency resembles cases of MP and LP indicated in Figure 7. From these results, the participant G was assumed to be a potential expert. To investigate participant G more closely, additional analysis was applied, as described in the next section.
(a)
(b)
(c)
(d)
4.2. Skill Analysis of the Motion of the Controlled Pendulum
In the previous section, participant G was elected as a candidate for expert from the viewpoint of the length of stabilizing time and the strength of the brain activity. In this section, possibility of a true expert for this participant is investigated based on the findings of Cabrera and Milton [20–22]. Since truncation of the Lévy distribution of change in the hand velocity is a hallmark of a skilled stick balancer, the distributions of the velocity changes were computed. Next, the power spectrum of height perturbation of the stick was computed to know whether an intermittency control was performed or not. Since the slider used in the present experiment has one DoF, the change in the hand velocity is the same as an acceleration of the slider except the difference of the scale factor . From the data of the longest successful trial on the final day, a distribution probability of the acceleration was computed, where the bin width of the histogram to make the distribution was set as . The probability distributions for the HP (participant G), MP (participant C), and LP (participant A) groups were drawn in same graph, as shown in Figure 9. It was confirmed that the tails of the distribution for the HP (and MP) became broader than those for the LP. Hence, it was found that the participant G had characteristics of a skilled balancer similar to that reported by Cabrera and Milton regarding a skilled stick balancer [21].
The next investigation was a power spectrum analysis of height perturbation () of the stabilized pendulum. For all groups, the number of data points was specified as (about 90 seconds) by considering the shortest survival time of the LP among them, and -point DFTs were computed. The power spectrums for the three participants are shown in Figure 10. Two power laws were found in Figure 10(a) for the HP. The power law in the range of Hz shows an exponent of , and the other law higher than Hz shows an exponent of . The cut-off frequency Hz is almost the same as the case of the direct stick balancing (1 Hz) presented in [21]. These values of exponents, however, differ from finding known as 0.5 and 2.4 [22]. In other case of a virtual stick-balancing task on a computer screen using a mouse-pointer, it was reported that the power laws had 0.7 and 2.1 [43]. The reason of the difference between our case and other reports appears to be due to the use of the haptic interface device, that is, the dynamics controlled by local compensator using the VIM control might influence motion of the virtual pendulum beyond the machine interface. Therefore, an existence of the on-off intermittency control cannot be proven directly from results of the present analysis. On-off intermittency-like control, however, can be admitted in the present case since two power laws in the spectrum were considered as a hallmark of the on-off intermittency control [26]. Figure 10 shows that the spectrum for the LP is governed by a single power law, unlike the case of HS; hence, it can be confirmed that the HP had particular property which the LP did not have.
(a) HS
(b) MS
(c) LS
Figure 11 shows the power spectrums of the force for the three participants. Same conditions above were used for this DFT computation. Figure 11(a) for the HS indicates two peaks at and Hz, and a notch at Hz. Monotonic tendency for the LS shown in Figure 11(c) is similar to Figure 10(c). This frequency property can be interpreted as an appearance of the on-off intermittency control, that is, these peaks at and Hz correspond to slow-drift movement and fast-correct action, respectively.
(a) HS
(b) MS
(c) LS
Analyses mentioned above support the opinion that participant G has the potential to be an expert stick-balancer. Both truncation in the Lévy distribution and an appearance of two power laws in the power spectrum confirmed an existence of the on-off intermittency-like control in case of the use of an input device. Considering the long continuance of the stabilization and the PCA results, it can be concluded that the participant G did indeed become the most skilled operator. To make explanation simple in later sections, the participant G is called simply “expert”.
5. Brain Function Analysis on an Expert
Treating participant G as the most expert, further analysis is described in this section. First, investigating the PCA topographical image of the expert by comparing it with others for the MP (participant C) and LP (participant A), characteristic activation in the cortex area for the HP was found. Next, functions corresponding to the found areas are identified by referring a relation drawn in Figure 4, and interpretation of the brain activation was derived. To satisfy the PCA condition, the trial with longest continuance on the final day was chosen. Because the brain activity pattern changed rapidly after a participant failed to stabilize the pendulum, the last 10-second data before failure of the stabilization were removed from the master data for PCA. That is, periods of for HP, of for MP, and of seconds for LP were analyzed by PCA, and these topographical images were then obtained, as shown in Figure 12.
(a) HP (G)
(b) MP (C)
(c) LP (A)
As mentioned in the previous section, the NIRS cannot show the absolute strength of brain activation. However, high-activated areas of HP participant were sufficiently stronger than the other areas of the LP and the MP participants, because an intensity level for the HP group was about five times larger than that of the LP and the MP groups, as shown in the first graph in Figure 7. Therefore, this fact and a result shown in Figure 12 lead to the following suppositions about the expert by considering locations of the brain functions shown in Figure 4. (a1) Activation over a wide range involving torso, hip, elbow, and arm in the primary somatosensory cortex (SmI) was strong. (a2) The left and right of the premotor cortices (PMC) were activated strongly. (a3) Activation of the right and left regions corresponding to the eyeball in the primary motor cortex (MsI) could be recognized. (a4) Activation in the right arm and hand regions (that are located at MsI in the left hemisphere (the left (right) hemisphere cortex receives signals from the right (left) side of body) was not strong.
Areas concerning the above-mentioned suppositions (a1)–(a4) were labeled using the same characteristic in the topographical images, as shown in Figure 12(a). Supposition (a1) suggests that the expert utilized sensory information from over a wide areas of the body, although the participant moved the hand mainly to manipulate the slider. This fact is also supported by considering that strong activations for the MP and LP participants were found in the narrow areas of the torso and hip (the corresponding areas are labeled using (b1) and (c1) on images (b) and (c) in Figure 12). Concerning supposition (a2), it appears that the expert enhanced observation of the pendulum motion more than controlling of the arm, because the PMC that is related to ocular motor movement [39] was strongly activated. This is also supported by the fact that the strength of the PMC activations in the case of the LP and MP groups was weaker than in the case of the HP group. There is, however, an opinion that the monitoring of PMC by NIRS is difficult because the bone of the parietal region is thick. In view of this, (a2) may not be reliable; however, supposition (a3) is fairly reliable with no problems of the measurement. In particular, participant E, who was the other member of the HP group, showed comparatively strong activation around PMC, as shown in the right of Figure 8. For these reasons, it would be appropriate to think that activation of the ocular motor area is a common characteristic of the HP group. This phenomenon appears to have relation with the delay-compensation in visual sensory system and/or enhancement of attention for on-off intermittent switching.
Eye movement on manipulation by watching a computer screen predominantly consists of smooth-pursuit since vergence eye movement is not required for watching the screen located at a constant distance from participants. The smooth-pursuit eye movement is relevant with wide area of the brain cortex. Specifically, velocity information of the visual-target on the retina is transmitted to two areas, and one of them is the back part of the frontal eye fields (FEF) in frontal cortex [44]. It was reported that the smooth-pursuit eye movement region in the frontal pursuit area (that corresponds to human FEF) of a monkey brain enhanced the response to visual target motion [45]; hence, it can be expected that activation of the FEF is relevant with the present task of the virtual stick balancing. Unfortunately, since the probe used in the present NIRS experiment did not cover FEF, it was impossible to investigate FEF directly from the measured data. If priori argument is permitted here, strong activation in the SMA and PMC shown in Figure 12(a) might indicate other strong activation of FEF because the back part of FEF neighbors the SMA and PMC by their front sides. Also this hypothesis appears to be associated with supposition (a3), and it might become additional evidence for an enhancement of observation in the skilled operator.
On the other hand, it was expected that the arm-hand area in MsI would be activated strongly since the operators used the hand to manipulate the haptic device. This phenomenon was observed in the LP and MP, but was not observed in the HP group. (Compare the weak area (a4) for the HP with the same areas of the MP and LP in Figure 12) As pointed out by the supposition (a4), motor cortex concerning the arm motion had small involvement in case of the expert.
6. Analysis of Control Law for the Expert
Time-delay and nonlinearity are indispensable for the consideration of human control characteristics, as mentioned in the introduction. In case of the manual stick balancing, it was reported that a linear model with time-delay could express human behavior [12]. Therefore, the present authors tried to identify the human control model using a simple liner formation including the time shifts. Generally, for the stabilization of a pendulum, an integral compensation is not important [1]; hence, the state variables of the pendulum were chosen from , and for the input of the human controller. The data of the fifth trial on the sixth day for the expert was used for subsequent analyses, because the period was sufficiently long at 230 seconds.
As the simplest strategy is a feedback control using the angle information of the pendulum link, the following control law was assumed at first: where corresponds to a proportional gain, is a time shift, and is a constant for the bias existing in the actual system. The notation of is a simplified description of discrete time and expresses a discrete counter value that is the closest to actual time . Here, , , and were estimated from actual logging data , by the least-squares method. To find suitable parameters, multiple pairs of the estimated coefficients were computed by changing from to at second intervals. Next, the best pair was found by searching for the minimum error that was computed by averaging the estimation error , where was computed as by using the identified coefficients and . The results are shown in Figure 13. The -axis is the time shift parameter for the search, and the -axes are the identified parameters , the error index , and the correlation coefficient , respectively. The third and fourth graphs indicate that the minimum of was given with the largest correlation coefficient when s. With this , was the largest, and the bias was sufficiently small compared with the value of . The value of is reasonable because it was close to that of the delay in visual-voluntary motion (in general, the delay is known to be [9]). Therefore, it appears that the basic equation (15) is comparatively adequate for expressing the human control law. The maximum correlation coefficient is, however, not sufficiently close to . Therefore, it was surmised that proportional control was not a main factor.
(a)
(b)
(c)
(d)
Next, similar analysis was applied to the following formation based on the angular velocity information where is a derivative gain, is the time shift, and is a bias constant. Figure 14 shows the result of the identification. The third graph indicates that the error index is smallest when . This negative sign of is interpreted as the prediction based on the measured velocity information. Future information cannot be obtained solely by observation of an object by reason of causality; hence, the expert appears to predict the velocity of the pendulum by using an internal dynamics model mastered through training. Moreover, the maximum correlation coefficient 0.93 was larger than the maximum of 0.53, which proves that the model (16) is more suitable for describing the expert. In other words, the expert utilized velocity information more strongly than the static posture information.
(a)
(b)
(c)
(d)
As it is naturally expected that the expert uses both posture and velocity information, the following combined formulation is considered where and are constant gains to be estimated. Here, and are time shifts, and is the drift term. To search for the best values of the time shifts, the error index computed by averaging the identification error was checked, where is the estimated force, which was similarly computed using the identified parameters and . The best combination of and was determined by changing them to and in similar way to the former analyses. Figure 15 shows the strength distribution of the identified coefficients , and . The vertical and horizontal axes on each graph are and , respectively. The white (black) area in each graph indicates the large (small) value.
As the drift term is sufficiently smaller than any of or , (17) appears to be appropriate as the form of the control law of the expert. Moreover, the magnitude of was larger than that of the other terms; hence, it transpired that the expert was paying attention to the change in velocity more than posture information. Moreover, Figure 15 shows that the error index in the vertical axis is the smallest when and that the error index in the horizontal axis is the smallest when . These values are almost the same as the former results that were obtained separately. In conclusion, the expert recognized the posture of the controlled object about seconds after observation and predicted the velocity about seconds before by using one's internal model.
The above results are summarized as follows. (i)It is surmised that the delay ( seconds), appeared in the angular control term, occurred from visual processing, and that the lead ( seconds) in the angular velocity term came from prediction based on an internal model of the controlled object. (ii)The predictive control was dominant because the fitting ratio to the velocity control including the time-lead was larger than the other ratio to the positional control with time-delay.
7. Conclusion and Discussion
Using a force-feedback haptic device with the balancing task of a virtual inverted pendulum, participants were asked to master a stabilization task, and the learning process was monitored. Participants were classified into three groups according to the length of continuance of the stabilization. The most expert was elected as the best performing operator by comparing the members of the three groups using the analysis of the distribution probability of change in the slider velocity and the frequency analysis of the pendulum movement. Moreover, the brain activation and the control law for the most expert were investigated, and the following results were deduced. (R0) Similar hallmark of skills to the direct stick balancing was confirmed on virtual balancing through the interface device operation. And, on-off intermittency-like control appeared in the skilled operator who manipulated the haptic device. (R1) System identification analysis using the operational force illustrated that the visuomotor control was performed based on the delayed posture perception and the predictive velocity control. (R2) It was found that consideration of two types of time-shifts for the perception delay and prediction lead enabled precise identification of the skilled operator's control characteristics with a linear model. (R3) In the cerebral cortex of the skilled operator, there were strong activations in areas associated with the ocular motor control and the visual processing. (R4) On the other hand, participation of the motor control area in the cortex for the arm was small.
Result (R0) showing an existence of the intermittency control may seemingly contradict result (R2) ensuring the high fitting to the linear control law, because the on-off intermittency control belongs to nonlinear control class and it seems difficult to be approximated by a linear model. However, as mentioned before, a previous study also showed that a linear model with one type of time-delay could approximate human control law adequately (e.g., the correlation factor was ); hence, there is a strong possibility that result (R2) is general. And, result (R3) that mentions strong activation concerning the ocular motor control in the cortex and result (R1) that indicates a velocity-based prediction control conclude that an enhancement of observation sensitive to the velocity information is a strategy for skilled operation. This strategy might be effective to increase attention for adequate switching in the on-off intermittency control.
These results suggest several ideas or hints for design of better man-machine systems involving visuomotor control. For instance, from results (R1) and (R2), system performance would be improved if its controller is redesigned using a new human model with two types of time-shift. In past research by the present authors, an assistive control based on an on-line identification of the voluntary motion was proposed [46], and a human control model that was treated in the present study would be effective to enhance the assist control. With the deduction from results (R2) and (R3), a visual-interface that enables a user to perceive the velocity of the controlled object is desirable. These ideas remain a matter of speculation, and further study is required.
In addition, various kinds of switching control like the on-off intermittency control have been studied in the control engineering field. It is surmised that minimum attention control [47], sliding sector variable structure control [48], and event-based control [49] have a relation with human on-off intermittency control. Such systems engineering approach will be useful for a practical stage of controller designs in human-machine systems. The present authors would like to realize these ideas for an advanced mechatronics and human-computer systems in the future.
Acknowledgments
This research was supported by a Grant-in-Aid for 21st Century COE (Center of Excellence) Program and a Grant-in-Aid for Scientific Research (A) from the Ministry of Education, Culture, Sports, Science and Technology, Japan. The present authors wish to thank the Ministry for their generous financial assistance. Further, the building of a haptic experimental system and several forms of analyses were performed by Keiichi Kurihara. Long and arduous training measurements were investigated by Yukihito Suzuki and Yasutaka Baba. The data could not be obtained without the many participants who embraced our requests kindly. Finally, the present authors appreciate reviewers giving us helpful comments and meaningful informations. The authors appreciate their cooperation.