Abstract
Background. The surface electromyogram (sEMG) is strongly related to human motion and is useful as a human interface in robotics and rehabilitation. The purpose of this study was to establish a new system for estimating finger joint angles using few sEMG channels. Methods. To deal with a dynamic system, the proposed method adopts time delay factors and a feedback stream into a neural network (NN) with 6 system parameters. The 2 target motion patterns were each tested with 5 subjects. 1000 combinations of system parameter sets were tested. Results. A system with only 4 channels can estimate angles with 7.1–11.8% root mean square (RMS) error, which is approximately the same level of accuracy achieved by other systems using 15 channels. Conclusions. The use of so few channels is a great advantage in an sEMG system because it provides a convenient interface system. This advantage is conferred by the proposed NN system.
1. Background
It is hoped that biological signals may be used as a new type of human-machine interface. The electroencephalogram (EEG) and surface electromyogram (sEMG) have been used in many studies. The EEG provides a great variety of information on humans and can be used as a brain-machine interface (BMI); applications currently being studied include interface systems for computers [1] and robot hands [2].
While the EEG examines internal human information (i.e., emotion, thinking), the sEMG is used to study human motion [3–6] because it is generated while voluntary muscle movements. It is highly useful as a machine interface, especially for prosthetics and robotic devices that assist rehabilitation. The human hand has many joints and performs important functions; therefore, injury to or loss of fingers is a serious problem. Prosthetic hands [7–12] and robotic devices that assist rehabilitation [13–26] are being developed in many institutes, and some prosthetic hands are available in the world market [7–9]. Robotic devices that assist rehabilitation are helpful in achieving an early recovery for injured patients. Devices that target the upper limb (shoulder or elbow, without hand) are currently being researched [13–19], while other work focuses on hand and finger function [20–26]. While research on prosthetic hardware has achieved a certain level, the next step must focus on how to control these devices. One solution is to use sEMG, which in fact has already been applied to controlling a device that assists rehabilitation [19, 27] and to estimating the degree of recovery [28].
In research that targets estimating hand state from sEMG, many studies have focused on discriminating hand patterns [29–34]; that is, they deal with discrete clustering problems using sEMG as input information. The resultant systems can recognize hand shape (e.g., “open”, “fist,” or “peace sign”), and the main aim now is to apply such systems to the problem of controlling a prosthetic hand. Some studies have achieved high estimating accuracy; however, this type of system can estimate only predetermined patterns. Thus, if such a system was trained to recognize only “open hand” and “fist,” it will be unable to recognize “peace sign.” A system that estimates continuous finger joint angles, however, would be able to recognize every hand state and therefore has high versatility. Some studies have begun to examine the estimation of finger joint angle [35–37].
Shrirao et al. [35] establish an index finger joint angle estimation system that uses neural network (NN) committees and that can interpret 3 flexion-extension speeds (low-, middle- and high-speed flexion and extension). It uses 1 sEMG channel at the extensor digitorum superficialis muscle. Kitamura et al. [36] report on an estimation system for base flexion/extension of the joint of the index finger. Their system adopts an analysis of the autoregressive relationship. It applies 4 channels at forearm, but no strict attach locations are determined. Their target is only a single joint of a single finger; however, since human hand skill involves multiple fingers and joints, this system is inadequate as a human-machine interface for understanding hand state.
Smith et al. [37] estimate multiple finger joint angles from many sEMGs. They used an NN to estimate the metacarpophalangeal (MP) joint angles of five fingers during the individual movement of each joint. Their system uses 15 channels with indefinite attach locations. They report an estimating precision of ~11%. However, certain estimated joint angles vary even if the joints are not moving. For example, even when the thumb joint is not moving, the joint angle is estimated as a moving joint. The reason for this is that the dynamics of the estimation system cannot be trained because the system uses a simple feedforward NN. Furthermore, in order to apply sEMG to a prosthetic hand system or a device to assist rehabilitation, the number of sEMG electrodes should be small. Fifteen channels are too many for this purpose.
Our goal was to establish a new type of finger joint angle estimation system, specifically, one that can effectively interpret a dynamic system using very few sEMG electrodes. Such a system would be useful not only for operating robot hands but also for controlling a prosthetic hand by teleoperation.
In the present study, a perceptron with three layers (one type of NN) was adopted as a multi-input and multioutput system that can estimate finger joint angles. In order to deal with the complicated relationship between sEMG and multifinger joint angles, a recurrent structure using a feedback stream and time delay factors was integrated into the perceptron system. In this way, the input vector into the perceptron includes information about current and past sEMG and feedback, allowing the estimate system to train the dynamics of the target system. Note that while the NN has a recurrent structure, the network can train a method for a feedforward network, but not for a recurrent one. This system requires many system parameters, and the present paper shows the influence that such parameters exert on the estimated results.
The target finger motion groups in the present study were the following:(1)individual finger flexion and extension,(2)finger flexion and extension both individually and together.
After training the network with the appropriate data set, finger joint angles were estimated using an untrained data set. The estimation system was then evaluated by comparing the results with measured finger joint angles.
2. Methods
2.1. Data Acquisition
2.1.1. sEMG
The sEMG signals were measured at the right forearm of each subject at the 4 locations shown in Figure 1 and Table 1. In order to configure the sEMG system with the lowest possible number of electrodes, all electrodes in the proposed system are placed on the flexor side of the arm and none are placed on the extensor side. Therefore, for the purposes of the present study, we focused on the flexor muscles, using 2 channels for thumb motion and 2 for the 4 fingers (a total of 4 channels).
The locations were determined by palpation for each subject such that channels 1 and 2 would record thumb flexion, and channels 3 and 4 would record finger flexion. Because displacement of the electrode affects the precision of the estimate [3], all data must be remeasured if for any reason the electrodes must be reattached.
The sEMG sampling was performed at above 4 ch, 1 ms sampling frequency, 12 bit resolution of the AD converter, and −5~+5 V dynamic range with the appropriate equipment (sEMG amplifier system: PTS-137, DKH Co., Ltd., Tokyo, Japan).
2.1.2. Finger Joint Angles
The present system adopts a three-layered perceptron whose core is an NN that can train the desired input-output relation; a teacher signal vector is necessary to train the NN. Because the system aims to estimate finger joint angles, the teacher vector should be extracted from the data glove measurements. Finger joint angles were sampled at 30 Hz using a CyberGlove (Immersion Co., Ltd., San Jose, California, USA) on the right hand at the time of sEMG measurement. The measured finger joints were the metacarpophalangeal (MP) joint, the proximal interphalangeal (PIP) joint, the distal interphalangeal (DIP) joint, and the abduction-adduction (Abd) joint. Therefore, the total number of measured finger joints per subject was 20.
2.2. Signal Processing
2.2.1. Feature Vector from sEMG
Measured raw sEMG data was processed using the segmentation technique [4] after processing with a band-pass filter (first order, pass band: 10–350 Hz). This signal processing is useful and powerful for extracting the feature vector from raw sEMG data. The present system adopted the following segmentation parameters: segment length, 1024; shift length, 32. Every 32 ms, waveform length [3, 4] (WL) was adopted as the feature value for each segmentation of raw sEMG. This type of feature value includes both a time domain feature and a frequency domain. The raw sEMG vector had 4 elements, as did the feature vector.
After the feature vector was extracted, it was normalized in the range between 0 and 1. The parameters necessary for normalizing (maximum and minimum values) were obtained and calculated from advanced measuring data, which takes the relaxed state of the muscle as the minimum value and the contracted state as the maximum value.
2.2.2. Teacher Vector from Finger Joint Angles
After the measurement of the 20 finger joint angles is available, however, there will be redundant information. For example, because of the structure of the human muscle-tendon-joint system, it is difficult to flex only the DIP joint during single-finger flexion. Thus, the number of joints tested can be reduced to streamline the estimation system. With respect to single-finger flexion, the PIP joint has a large range of motion and it is possible to flex only the PIP without also flexing the MP or DIP. And PIP can move in the range of around 120 degrees, while MP can in around 90 degrees. It can therefore be said that the PIP joint shows finger flexion. In processing for joints, we therefore extracted 5 finger joints as the teacher vector, that is, the dimension is reduced from 20 to 5.
Similarly, the parameters necessary for normalization were obtained and calculated from advanced measurement data. The value of the teacher vector indicates the flexion rate of the joint (0, full extension; 1, full flexion). In other words, this value also means flexion degree, so this can be represented with unit of percentage.
2.3. Estimation System with an NN
The present estimation system for finger joint angles was developed from feature values obtained from sEMG by integration with a three-layered perceptron. This is a feedforward type of NN with the following useful characteristics that are effective in estimating finger joint angles from sEMG.(i)The network does not need a strict system model. With training, it can approximate the input-output nonlinear function for training data. This is important for dealing with personal differences.(ii)It is possible to apply this network to a multi-input multioutput system. Modifying the dimensions of the input or output vectors is therefore quite easy.(iii)By using the sigmoid function as a unit output function in the output layer of the NN, it is possible to limit the output values from the network to , which is an easy range to relate to finger joint angles. Additionally, the limitation of output from the network system can also contribute to system stability.
Furthermore, in order to train the dynamics of the estimation system, we introduced to the three-layered perceptron a recurrent structure using a feedback stream of past output signals from the network and a time-delay factor to input to the network. This makes it possible for the estimation system to train the system dynamics. The estimation system is shown in Figure 2. In this system, the time delay parameters for feature vector and feedback vector indicate the extent to which previous feature vectors and precious feedback vectors, respectively, are considered.
In this figure, (·) is a three-layered perceptron. This type of NN can train any input-output relation if the number of the middle layer unit is adequate. The present system adopts the standard sigmoid function as the activating function of each unit in the NN, and the steepest descent and backpropagation methods as the training method. This training method changes the coefficients of the internal unit and it can be shown simply as follows: where is the amount of change of a certain coefficient of the unit, is the training rate and adequate small value, is a backpropagation term, and is the input into this unit. Under this method, a coefficient that takes large input values will change largely. During network training, modifying the value of the input vector into the network is also important for highly precise functioning.
The gain is a constant value matrix whose value is 1 or more than 1. The makes the value of the feature vector components rather than the feedback vector components large because the NN trains the system depending on the sEMG information rather than on feedback information. The forgetting rates for the feature vectors and for the feedback vectors have a value of 1 or less than 1. These parameters make the estimation system weigh newer information as more important than older information.
The system can be shown as follows (without gain or forgetting rates): The input vector into the NN is composed of the following factors:(1)newest feature vector: ,(2)past feature vectors: , (3)past output vectors from the NN, that is, feedback components: .
Thus, the estimation system can train the target system dynamics. If parameters and are both zero, the estimation system becomes ordinary [22], and it is clear that the proposed system is the expansion of the ordinary
As described above and shown in Figure 2, the number of units at the input layer of NN will change depending on the values of and because the dimensions of and are constant at 4 and 5, respectively. The number of units at the input layer can be represented as follows: On the right-hand side of this equation, the first term shows the dimension of the newest feature vector (), the second term shows the dimensions of past feature vectors () and the third term shows the dimensions of feedback components (). For example, if is 0 and are both 0 (i.e., an ordinary NN system), then is 4 (). Similarly, if is 2 and is 6, then is 42 .
About the other information of NN structure, the number of middle layer is dealt with as system parameter and the number of output layer is five (constant) because the proposal system uses one unit of output layer for estimating one joint.
2.4. Training Phase and Estimation Phase
The execution of this estimation system is divided into 2 phases: the training phase and the estimation phase (Figure 3). The system parameters show the parameters discussed in the previous section (n, h, G, etc.).
2.4.1. Training Phase
The primary purpose of the training phase is to obtain optimized NN coefficient data oW that can represent the input-output mapping from sEMG data to the finger joint angle dynamics system with high precision (Figure 3). During this phase, the estimation system uses the feature and teacher vector sets for training, but does not use feedback.
The training is complete when it reaches the number of the maximum training epoch. After training, the coefficient data should be optimal. The initial values of the elements contributed by the time delay factor are 0, and the initial values of the NN coefficients are initialized by random number.
2.4.2. Estimation Phase
During the estimation phase, the estimated angle vector is calculated from feature vectors that are not used for training or the optimized NN coefficient oW (Figure 3). The system must use the same system parameter set as that used in the training phase.
The NN is not trained and calculates the estimated vector from the input vector. While system training requires a considerable amount of time, estimation is rapid and can be realized in real-time processing.
3. Experiment
3.1. Outline
Through the experiment, the effectiveness of the estimating system and the contribution of the system parameters to the estimation result will be shown. Although all training and estimation were done offline in this experiment, the estimation can be performed in real time. This experiment was carried out on 5 able-bodied subjects (all are male, age is in 21–26, height is in 165–172 cm, weight is in 50−65 kg). This experiment was based on their approval after informed consent with Gifu University ethical committee.
3.1.1. Target Finger Exercise
The sEMG data reflect voluntary muscle contractions and strongly relate to human body motion. At the forearm in particular, the relationship between muscles and their performance is very complex. For example, the flexor digitorum profundus contributes to the DIP flexion of the 2nd–5th fingers. Note that muscles and joints may not necessarily have a one-to-one relationship. In the present study, the following types of motion set are shown.Motion Set 1: finger individual flexion and extension; Motion Set 2: finger flexion and extension individually and together.
Motion Set 1 reveals the independence of finger motion with this estimation system. Single-finger flexion and extension are basic finger motions. Motion Set 2 includes the complexities of both single and multifinger motion. In daily life both single-finger and multifinger motions are important. Sample graphs are shown in Figure 4.
(a) Motion Set 1
(b) Motion Set 2
3.1.2. Combination of System Parameters
The proposed estimation system requires 6 system parameters. These parameters are independent of each other, and the total number of combinations is enormous because each parameter may have any value.
In order to limit the number of combinations of system parameters, the available values for each parameter were predetermined (Table 2). There are 8 values available for each parameter. Parameters n, h, and are integer values, and r, q, and are decimal values. These available values were determined by trial and error.
With this limitation on allowable values, the number of possible combinations is 262,144 (=86). Because this number is too large to analyze, we adopted the regression method with random numbers.
3.2. Experimental Procedure
The experimental procedure was divided into the following phases.(i)Data acquisition: sEMG data set and finger joint angles for extracting normalizing parameters, training and estimating are needed in this system.(ii)Determination of system parameters: the 6 system parameter values are determined by random number.(iii)Training: the estimating system with the determined system parameters is trained with a data set.(iv)Estimating: after training, the system estimates finger joint angles with data not used for training (i.e., unknown data). The results and actual data from the data glove are compared and system performance is estimated.
In this experiment, phases (ii)–(iv) were repeated 1,000 times for each subject and motion set. Note that because new random number is generated at every parameter decision, the selected system parameters are not always same between the tasks and among the subjects. Since we had 5 subjects and 2 motion sets, the total number of trials (phases (ii)–(iv)) was 10,000.
3.2.1. Data Acquisition
In order to normalize the feature vectors extracted from sEMG and the teacher vector from the data glove, a data set for normalizing is needed. This data set is obtained from each subject. To normalize the sEMG information, it is necessary to perform sEMG during both relaxation and contraction of the muscles. Similarly, to normalize finger joint angles, the data obtained during full flexion and extension of each joint is necessary.
With respect to data sets for training and estimating, the measured terms are shown in Table 3.
3.2.2. System Training for Each Subjects and Motion Set Individually
First, the set of system parameters was determined by random number. Second, one estimation system trained with the feature-teacher data set and the set of system parameters until training epoch 200 was reached. This value of epoch limitation was determined by preexperiments, of which an example is shown in Figure 5. Almost all training trials had finished by the 50th epoch. Therefore, 200 epochs can be considered a sufficient value for this system.
This procedure was repeated for each subject and motion set, and 1000 combination parameter sets were tested. Global suboptimal parameter sets were then obtained using the above random search.
3.2.3. Estimation with the Trained System and Evaluation of Results
The 1000 parameter sets were tested with the estimation system for each subject and motion set. For each estimation system, estimation errors were calculated with data sets for estimation that were not used in training. The results were evaluated using the root means square (RMS) of the estimation error.
4. Results
4.1. Best Performance Parameters Sets
Table 4 shows the results of the above experiments. For each subject and motion set, 1000 trials were performed. In 1000 trials, the system parameters set with the smallest RMS of estimate error can be considered the optimal solutions. Examples of the estimated graphs for Subject A are shown in Figures 6 and 7.
(a) The feature values of sEMG (normalized WL)
(b) Actual finger joint angles from data glove
(c) Estimated finger joint angles from sEMG information
(d) RMS of error
(a) The feature values of sEMG (normalized WL)
(b) Actual finger joint angles from data glove
(c) Estimated finger joint angles from sEMG information
(d) RMS of error
4.2. Local Search Near the Optimal Solution
On the assumption that the above random search can find a global optimal solution, system result behaviors near the solution were examined in greater detail. Experiments were performed in which 1 parameter varied while the remaining 5 remained fixed (the set of system parameters consists of a total of 6 parameters).
Figures 8 and 9 show the system behavior near the optimal solution for Subject A’s motion sets. Note that the units of the vertical axis of each figure show the RMS error and are in different ranges. For example, in Figure 8, the range of (b) is from 7.0 to 8.2, while the range of (c) is from 0.0 to 50.0.
(a) Time delay parameter for feature vector
(b) Time delay parameter for feedback vector
(c) Number of middle layer units of NN
(d) Gain for feature vector
(e) Forgetting rate for feature vector
(f) Forgetting rate for feedback vector
(a) Time delay parameter for feature vector
(b) Time delay parameter for feedback vector
(c) Number of middle layer units of NN
(d) Gain for feature vector
(e) Forgetting rate for feature vector
(f) Forgetting rate for feedback vector
5. Discussion
For each trial results, finger joint angles can be estimated by a trained system with appropriate value system parameters. These parameters sets are determined by the above random search method, which functions as a global search in multidimensional space. Note again that the range of the vertical axis of each figure (Figures 8 and 9) is different, which means that the contribution of the parameter to the accuracy of the estimation is also different.
Parameters and relate to the number of input vector elements (or dimensions). Because the number of output vector elements remains constant at 5, n and can change the ratio of input items to output items. As a result, these parameters should have values that are neither very small nor very large. Parameter in particular should be determined more carefully than others, as should Parameter n (Table 4, Figures 8(a), 8(b), 9(a) and 9(b)). Figures 8(b) and 9(b) show the estimation accuracy for each value of . It appears that feedback components may make more of an impact on estimation than time-delayed sEMG components. In either case, the combination of and must be determined carefully (Table 4).
Parameter represents the level of complexity at which NN can train input-output mapping. Generally, an NN with a large that can train a complex input-output relationship is also able to train an unexpected relationship. An NN system with a small cannot carry out high-precision estimation. Based on the data shown in Table 4 and Figures 8(c) and 9(c), it is clear that should be 20 or more to achieve a highly accurate estimation, and that there is no significant difference among large m values. In the local search cases shown in Figures 8 and 9, the dimensions of the input vector into the NN and of the output vector remain constant. In other words, in the local search for Parameter m, the complexity of the output-input relationship is constant as long as the value of is adequate. As described above, too large a value of is not desirable from the view point of avoiding training unexpected relationships and the cost of calculation of training. The present results suggest the use of a sufficiently small value of m (in this experiments, the value is 20 from Figures 8 and 9).
Parameters G, r, and are the gain constant for the feature vector, the forgetting rate for older feature vectors, and the forgetting rate for feedback vectors, respectively. These parameters determine the relative amplitude of the input vector elements and therefore relate to training the NN.
According to the data shown in Table 4, and Figures 8 and 9, the larger the value of G, the more precise the estimation. Thus, based on the function of this parameter presented in previous section, the components that relate to the sEMG are more important than those relating to feedback. This is a natural result because the sEMG relates directly to muscle contraction and finger joints are moved by muscles.
With respect to Parameter r, an estimation system in which is near 1.0 can estimate with great precision. That is, among the sEMG components in the input vector into NN, both newer and older data are helpful for highly accurate estimation. With respect to Parameter q, a system in which has a small value can estimate with great precision. Older feedback components are not more important than newer elements in this case. Comparing these 2 parameters, the contribution of to the estimation result is larger than that of r, and parameter should be determined carefully.
As discussed in the previous section and (1), each unit training depend on the input value of the unit with training method of the NN that is used in the present system. Additionally, the coefficients of the NN change in small steps at every training step. Thus, we believe that Parameters G, r, and may be used to modify NN training appropriately.
The system described by Smith et al. [37] targets finger flexion/extension individually with an estimating precision of ~11% using 15 sEMG channels. In contrast, the present system can train a dynamical system at a similar level of precision (7.1~11.8%) in the case of Motion Set 1 (which is equivalent to Smith’s motion set), but using only 4 sEMG channels, due to the proposed NN system structure with time delay, feedback, and some coefficients. This is a large advantage with respect to the sEMG system and provides a convenient interface system. With this structure, the system can train the essential sEMG-finger joint angle system and the system can thus accurately estimate angles with little sEMG information.
6. Conclusions
This paper presents a finger joint angle estimation system based on sEMG. The type of information provided by sEMG is very useful not only for creating and using robot hands but also as a rehabilitation interface; however, the relationship between sEMG and finger joint angles is complex. In order to train the sEMG-finger joint angle dynamic system, the estimation system adopts time delay and a feedback stream with specific system parameters.
We tested the proposed system with 5 subjects using 2 types of finger motion sets for each subject: individual flexion and extension of all fingers and flexion and extension of all fingers individually and together. For each subject and motion set, 1000 system parameter sets were used to establish the estimation system by random search and to set the parameters such that they produce results with the smallest error (i.e, the optimal solution). Furthermore, in order to analyze solution behavior around an optimal point, local analysis was carried out for single subjects.
From the results of the above, certain characteristics of the 6 system parameters were defined. While time delay parameters and must be selected for the subject or motion set, the value of is sufficient at a certain level (in the present experimental results, 20). Gain parameters G, r, and should be large, large, and small, respectively.
The proposal system achieved the continuous multifinger joint angle estimation from sEMG. The angle estimation system has high versatility rather than pattern recognition system. In addition, multijoints estimation is important for controlling robotic device comparing with single-joint estimation [35, 36]. Furthermore, in case of Motion Set 1, the proposed system achieved a 7.1–11.8% RMS error in its estimation. This accuracy is approximately same level achieved by the system described by Smith et al. [37]; however, while Smith’s system uses 15 sEMG channels, the proposed system is able to use only 4 due to the proposed NN system structure with time delay, feedback, and certain coefficients. This is a large advantage of our sEMG system, which provides a convenient interface system.
The results of the present experiments (Table 4) indicate that the proposed system can be applied to many subjects, that is, it is universal. The system introduced in this paper can be used in other applications for controlling humanoid robot hands, especially with regard to fingertip position, fingertip force, and wrist posture. It is useful and helpful for controlling devices used to assist rehabilitation or for estimating the degree of recovery in patients undergoing rehabilitation.
Abbreviations
| Abd: | Abduction-adduction |
| BMI: | Brain machine interface |
| DIP: | Distal interphalangeal |
| EEG: | Electroencephalogram |
| MP: | Metacarpophalangeal |
| NN: | Neural network |
| PIP: | Proximal interphalangeal |
| RMS: | Root means square |
| sEMG: | Surface electromyogram |
| WL: | Waveform length. |
Conflict of Interests
The authors declare that they have no competing interests.
Authors’ Contribution
M. Hioki designed the system and experiments, conducted data collection and analysis, and wrote the paper. H. Kawasaki supervised the study. Both authors have read and approved the final paper.