Abstract
In this study, seven-channel electromyography signal-based two-dimensional wrist joint movement estimation with and without handgrip motions was carried out. Electromyography signals were analyzed using the synergy-based linear regression model and musculoskeletal model; they were subsequently compared with respect to single and combined wrist joint movements and handgrip. Using each one of wrist motion and grip trial as a training set, the synergy-based linear regression model exhibited a statistically significant performance with 0.7891 ± 0.0844 Pearson correlation coefficient () value in two-dimensional wrist motion estimation compared with 0.7608 ± 0.1037 value of the musculoskeletal model. Estimates on the grip force produced 0.8463 ± 0.0503 value with 0.2559 ± 0.1397 normalized root-mean-square error of the wrist motion range. This continuous wrist and handgrip estimation can be considered when electromyography-based multi-dimensional input signals in the prosthesis, virtual interface, and rehabilitation are needed.
1. Introduction
Owing to advances in surface electromyography (EMG) signal-based models and algorithms, numerous techniques have been proposed for prosthesis controls and clinical controllers. Though in a discreet fashion, several studies had attempted to convert hand motion into input signals to control prosthetic machines [1], virtual hands [2], and exoskeletons [3], with the aim of estimating both the wrist motion and hand gesture. Nishikawa et al. [1], Sebelius et al. [2], and Kita et al. [4] classified several gestures, such as hand gestures and wrist motions, using algorithms like machine learning, Gaussian mixture models (GMMs), and other linear classifiers (e.g., k-NN and Bayes).
Continuous estimations are applied in response to feedback from real users who require various movements suitable for daily life [5]. Vogel et al. [6] used standard supervised machine learning algorithms to create a mapping between arm/forearm muscle activities and 6-dimensional (6D) position/orientation; this has extended the four rotational degree-of-freedom (DOF) models for the joints of the shoulder and elbow [7]. An algorithm for simultaneous estimation of the three DOFs of the wrist was also proposed [8]; it showed promise of applicability to unilateral amputees by employing a bilateral mirror-training strategy [9]. However, these continuous estimations did not consider the combined motions of the wrist and fingers.
Under the flexor muscles are the multiple finger muscles that lie deep inside the forearm [10]. Many researchers use their expertise to minimize the interference of surface EMG (sEMG), without fully solving crosstalk. For this reason, the sEMG electrode is not free from the inclusion of both external and internal muscle signals. These mixed signals can influence the estimates obtained via the other.
Muscle synergy is defined as a set of muscles recruited by a neural command [11]. A muscle synergy generates a primitive motion, and complex motions are produced by the combination of several synergies [12, 13]. Real-time classification for upper limb motion was conducted using a machine learning technique [14]. However, synergies were differentially weighted according to task constraints [15]; therefore, in this study, two different synergy calculations were attempts: deriving wrist and grip synergies simultaneously and deriving each synergy separately. Besides, synergy model performance with the change of synergy was analyzed, and the choice of number of wrist synergy was checked.
The musculoskeletal model (MSM) is a second-order computational motor control model with nonlinear dynamics. It estimates a one-degree-of-freedom joint angle for flexion and extension considering muscle elasticity and viscosity [16]. Kawase et al. [17] developed a simplified computational model that investigated the estimation of three different joint angles (i.e., elbow, wrist, and finger) with a little influence between finger joint and wrist position estimation.
To realize the prosthetic hand for daily use, wrist motion and grip motion have to be controlled simultaneously; however, few papers treat this problem [17] because of the crosstalk of muscle activation measurement.
This study aims to estimate wrist motion with and without grip motion and compared the estimation performance between optimized MSM and synergy models.
2. Materials and Methods
2.1. Subjects
Ten healthy subjects (males, aged 28.0 ± 5.7, 9 right-handed, 1 left-handed, none ambidextrous) participated in the experiment. They did not have a history of any form of neurological disorder. They used their dominant hand (either left or right hand) during the conduct of this experiment.
2.2. Experimental Protocol
The study protocol was approved by the ethics committee of the Tokyo Institute of Technology (2014042) and was carried out in accordance with the Declaration of Helsinki. Written consent was obtained from each subject before the experiment.
Table 1 indicates the muscle groups chosen to estimate the wrist and grip movements. Five muscles are associated with wrist motion (i.e., ECR, ECU, FCU, FCR, and APL) and two with grip action (i.e., FDS and FDP). Previous wrist-based experiments analyzed ECR, ECU, FCU, and FCR [17, 18], which are the flexor and extensor muscles of the wrist with different deviations (radial and ulnar). In addition to these muscles, in particular APL, an extensor of the thumb was included to trace the radial movement of the wrist. The FDS and FDP—the flexor muscles of the finger—were included to estimate the grip force with a synergy-based model.
Figure 1 shows the placement of the EMG sensor on the forearm. The EMG signals were measured using Trigno™ EMG system.
In the experiment, two tasks were conducted. The first task was a wrist motion, which measured motion in different movement conditions. For the second task, isometric grip force was measured in different grip force levels. Thus, the trials were divided into two tasks to check the wrist movement at a certain grip condition and grip force at a certain posture.
In the first task, EMG was measured using wireless Trigno™ EMG system sensors, and wrist joint angles were measured using the IM sensors of the system. Figure 2 shows the placement of the IM sensors, which were attached to the back of the hand and the back of the forearm; they were attached to detect the relative wrist joint angle from the forearm.
Subjects placed their forearms on the table fastened by a wrist binder. Thereafter, they performed four wrist motions: flexion, extension, radial deviation, and ulnar deviation. These motions were conducted while the hand was free (no gripping action) and in gripping mode (normal strength). Wrist motions were conducted under three conditions based on the subject’s comfort: comfortable maximum limit (with and without grip), half of comfortable maximum limit (without grip only), and stiffened movement with force exertion (without grip only). The subjects performed each motion three times per trial. Three trials were conducted for each condition. Thereafter, the gripping action (without wrist motion) was conducted in the center position (Figure 2) for which the subjects performed strong grips and weak grips. Three trials were conducted for the gripping experiment.
In the second task, EMG and grip force were measured, with the latter done using ReachMAN robot [19]. The subjects adjusted the angle of the grip to their best fit while maintaining a center position posture, as demonstrated in Figure 2, after which the grip force was measured. Three levels of grip strength were performed: strongest, half, and a quarter of gripping power. The strongest grip force (in newton N) varied for every subject with an average of 16.2 ± 3.2 N.
2.3. Data Acquisition
The data were sampled separately per signal category using lab streaming layer (LSL) in MATLAB 2018b program base [20]. The EMG signals were sampled at 2000 Hz, IMU sensors at 74 Hz, and ReachMAN force sensor at 100 Hz.
The seven EMG signal channels were filtered and normalized before computing the synergy set. The EMG signals were rectified and filtered using a second-order Butterworth low-pass filtering with 5 Hz cutoff frequency [21]. Figure 3 shows the conversion of an EMG signal. The filtered EMG signals are called “quasi-tension” because it showed a high correlation between the joint torques of its muscle [21].
(a)
(b)
(c)
A recurring issue during experiments and analyses was the fact that the magnitude of EMG signals for each channel had to be changed every time the sensor was detached then attached again. To resolve this, all signals were normalized by the peak activation level of the whole task, including a range of joint angles and maximum effort of trials [22]. Normalization was performed after quasi-tension signal filtering. In this experiment, the combined hand motion tasks, co-activating both grip and wrist motions, were chosen; hence, the normalized quasi-tension signals, which were obtained by filtering and normalizing the EMG signals, were resampled into the other sampling rates and measured together.
2.4. Wrist Angle Derivation
The Madgwick IMU algorithm was implemented to estimate the two-dimensional wrist joint angle [23]. IMU sensors were placed on the back of the hand and forearm to track the orientational difference between the hand and forearm.
Subjects performed self-paced movements without visual feedback; consequently, most of them performed diagonal movements even if only vertical and horizontal movements had been requested. To compensate for this, the two angles obtained by the IMU algorithm were normalized by each angle’s absolute maximum value; the sum and difference obtained can be seen in Figure 4. In each model, these calculations were estimated and the summation was recalculated to estimate the angle; furthermore, considering the EMG crosstalk error and wide range of wrist angle movements, a comfortable maximum limit trial was mainly used as the train data.
2.5. Synergy-Based Linear Regression Model
A synergy-based linear regression model was used to estimate wrist and grip values. To reduce computational costs in a model calculation, a simplified version of the nonnegative matrix method, i.e., the hierarchical alternating least square (HALS) method, was used [24]. Apart from the computational cost, HALS also has a wide capability: it can work with a large number of components [24], in contrast to the canonical NMF method [25], which is only applicable if the number of the sources is greater than the number of components; it can work in conditions where the number of components is large [24]. This feature of HALS is appropriate when multiple hand gestures need to be applied. Hence, this computation method is valid even when the number of combined synergy set exceeds the number of measured EMG signals. The HALS decomposes the normalized quasi-tension as follows:where E is the normalized quasi-tension signals in an m by n matrix with m being the number of time series and n the number of EMG channel inputs; is the synergy set, where j is the number of synergies and representing a single set of synergy, where is the coactivation coefficient of EMG n; and furthermore, M is the coactivation coefficients of the synergy in m by j matrix:
When the synergy model is derived, learning algorithm procedures are used to iterate (2) and (3) several times, where k (1, 2, …, j) denotes the label of synergies. The matrices S and M were computed using one set of single wrist motion data and single grip motion data.
In the analysis, wrist synergies with varying numbers from one to six were calculated from a wrist movement trial to confirm the validity number of synergies; thereafter, a single grip synergy was taken from a grip trial. More often than note, the variance account for (VAF) became the standard means of choosing the muscle synergy number [26–28]. In the same context, this study applies the number of synergies that matches over 0.9 VAF to all the subjects to ensure the synergy model consistency. The gains of the wrist motion synergy for the angles were derived using linear regression to compute the normalized sum and difference of the wrist angle (flexion-extension, radial-ulnar deviation) from the following equation:where denotes the angle bias, s are the regression coefficients for each synergy coefficient , and denotes the random noise error. A combined synergy set with regression coefficients were used to estimate both grip motion and wrist motion task; hence, the synergies and gains from a combination of two trials were applied to all other tasks.
The computation of the wrist and grip synergies was conducted in two different ways. A facial image study showed that NMF learns the object in part-based representation [29]. In the case of grip motion, the muscles of all channels work together; therefore, multitrial-based muscle synergy was calculated in two ways: simultaneously from jointed wrist and grip trials (SLRM1) and separately per trial (SLRM2). The synergy sets were derived from comfortable maximum limit trials and grip trials.
2.6. Musculoskeletal Model (MSM)
The musculoskeletal model was used to compare the angle estimates of the synergy-based model. The MSM succeeded in estimating the joint angles of the elbow, wrist, and index finger with little influence from a change in wrist position [17]. The performances of SLRM and MSM were compared with each other to ascertain how good that of SLRM is; furthermore, the train set of MSM was taken from a comfortable maximum limit, which are the same trials used in SLRM. To optimize the MSM performance, MSM was derived from two different muscle numbers, namely, MSMS1 and MSMS2; the former used all measured muscles, while the latter used five wrist muscles. Kawase et al. constructed a one-degree-of-freedom model per joint [17]; to fit the model into this experiment, two wrist joint angles were converted as depicted in Figure 4.
2.6.1. Statistical Analysis
An exhaustive cross-validation was used to test the performance of each model per subject, with indices used to estimate performance. The Pearson correlation coefficient (r) and normalized root-mean-square error (nRMSE) are defined as follows:where n is the number of samples, y is a reference, x is an estimate, and a is defined as the normalization coefficient. nRMSE chooses a to be 90, the limit of the wrist angle range. All statistical analyses were conducted using t test2 function of MATLAB 2018b.
3. Results
3.1. Synergy Number Optimization
The reproducibility check of SLRM1 and SLRM2 in the different number of synergies was tested in VAF. The number of grip synergies was fixed to one, both in SLRM1 and SLRM2 to ensure that SLRM1 computed the synergy one more from the joint trials. SLRM1 with two wrist synergies had over 0.9 VAF on average (0.9342 ± 0.0245), and the minimum VAF of three wrist synergies was 0.9529. For SLRM2, three wrist synergies had over 0.9 VAF on average (0.9442 ± 0.0340), and the minimum VAF of four wrist synergies was 0.9647. To include all subjects, four wrist motion synergy numbers were chosen and used throughout the study.
3.2. Task 1: Wrist Motion Test
Figures 5(a) and 5(c) show the r and nRMSE values of SLRMs in wrist motion task. The values of SLRM1 and SLRM2 were 0.7523 ± 0.1466 and 0.7891 ± 0.0844, respectively. For nRMSE, SLRM1 had 0.1864 ± 0.0835 in wrist motion and 0.2471 ± 0.1387 in grip motion, while SLRM2 had 0.1564 ± 0.0388 in wrist motion and 0.1458 ± 0.0251 in grip motion. The differences are statistically significant in both cases (, Student’s t-test). From the results, SLRM2 was chosen as the representative SLRM model.
(a)
(b)
(c)
(d)
Similarly, Figures 5(b) and 5(d) show the values and nRMSE values of the MSMs in wrist motion task. The values of MSM1 and MSM2 were 0.7691 ± 0.1056 and 0.7608 ± 0.1037, respectively, which exhibits no statistical significance. For nRMSE, MSM1 had 0.1695 ± 0.0505 in wrist motion and 0.2368 ± 0.1107 in grip motion, while MSM2 had 0.1718 ± 0.0608 in wrist motion and 0.1864 ± 0.0770 in grip motion. The nRMSE of the grip motion implies a statistical significance (, Student’s t-test). From the results, MSM2 was chosen as the representative MSM model.
The time series of the wrist angle in two dimensions is shown in Figure 6. Subjects were asked to rotate their wrists in four directions. They were able to move freely at their own pace, moving in an inclined diagonal direction at different angles. Because of this tendency, both models appear to have the underlying assumption that a subject moved in a diagonal direction even if they performed a gradual movement, as shown in Figure 6(c).
The exact performances of SLRM and MSM in r and nRMSE are shown in Tables 2 and 3. Wrist motion performances r on average are 0.7891 ± 0.0844 in SLRM and 0.7608 ± 0.1037 in MSM, implying a statistically significant difference (, Student’s t-test). Similarly, nRMSE also shows a significant difference between SLRM and MSM (, Student’s t-test). This trend continued during wrist motion trials without a grip (comfortable maximum limit trial, comfortable half limit trial, and stiffened movement trial).
However, when grip motion was added, there was no statistically significant difference in r; however, differences were still apparent in nRMSE. The r of the comfortable maximum with grip trials were 0.7562 ± 0.0631 in SLRM and 0.7579 ± 0.0877 in MSM, and nRMSEs were 0.1654 ± 0.0.271 in SLRM and 0.1804 ± 0.0511 in MSM (, Student’s t-test). Finally, in the grip trial, where r measurement was inappropriate because the wrist motion in the trial is just an indication of a perturbation, here, SLRM had 0.1458 ± 0.0251 and MSM had 0.1864 ± 0.0770 in nRMSE, implying a statistically significant difference (, Student’s t-test).
In detail, the r values of the comfortable maximum limit trials were 0.8256 ± 0.0270 in SLRM and 0.7994 ± 0.0528 in MSM (, Student’s t-test); the nRMSEs were 0.1539 ± 0.0178 in SLRM and 0.1713 ± 0.0343 in MSM (, Student’s t-test). In comfortable half limit trials, the corresponding values were 0.7460 ± 0.0731 in SLRM and 0.6844 ± 0.1022 in MSM when measuring r values (, Student’s t-test), and the nRMSEs were 0.1204 ± 0.0106 in SLRM and 0.1256 ± 0.0219 in MSM (, Student’s t-test). The stiffened movement trials also had the same trend in values, being 0.8406 ± 0.0344 in SLRM and 0.8142 ± 0.0589 in MSM (, Student’s t-test); and, for nRMSEs, 0.1852 ± 0.0216 in SLRM and 0.2098 ± 0.0558 in MSM (, Student’s t-test).
3.3. Task 2: Grip Motion Test
Figure 7 shows the SLRM-based time series grip force estimate and resulting angle estimate perturbation. Subjects were constrained to a grip device during the task to ensure that there was no actual wrist motion during the entirety of task. Hence, wrist motion estimation during grip motion was checked for strong distortion in angle estimate. The results showed that during the gripping task, instability of the wrist angle estimation occurred in the presence of a strong force activation, as may be seen in Figure 7(d). For the half and quarter grip force task, angle estimation was less than 30°, as shown in Figures 7(e) and 7(f). The value for grip force estimate and nRMSE of the X-Y angle estimate, compared with zero angle (no movement), were computed as indicated in Table 4. Sub8 data were omitted in this task because the EMG signal was saturated during the ADC converting process using NIDAQ (±5 voltage). The SLRM-based grip-force estimate from nine subjects was 0.8463 ± 0.0503 in r with 0.2559 ± 0.1397 nRMSE in wrist movement estimation.
(a)
(b)
(c)
(d)
(e)
(f)
(a)
(b)
4. Discussion
This study tested both MSM and SLRM in two different conditions to optimize each model. Figure 9 shows the VAF from SLRM1 and SLRM2 with the different number of wrist motion synergies. The VAF of SLRM1 converged in three wrist motion synergies and overfitted thereafter. This trend was the same as obtained in other studies whether in patients or healthy subjects [26–28]. For SLRM2, the VAF converged in four wrist motion synergies and overfitted up to five. There was also a decrease in VAF in SLRM2. d’ Avella et al. suggested that when the number of extracted synergies is greater than the generator synergies combination, each additional synergy captures an equal amount of noise-generated variation [30]. In that sense, fifth and sixth muscle synergies are noise-derived synergies. This study measured five wrist motion muscles and two finger motion muscles; thus, the sixth muscle synergy in SLRM2 formulated synergy set with finger motion muscles, which made reproducibility impossible. Such noise-derived synergies contaminated the estimation performance. The wrist joint angle estimation performance of SLRM2 with varying numbers of wrist synergies are shown in Figure 10. The highest performance was obtained in four wrist motion synergies; the additional number of synergies deteriorates the estimation performance.
(a)
(b)
In the SLRM, wrist movement estimation performance showed statistical significance depending on the synergy extraction method both in and nRMSE. This study aims to use NMF for prosthetic and interface purposes; therefore, synergies were modulated per subject and trial. Separate synergy sets preserve multi-EMG coactivation in the grip synergy, which enables synergy to cluster movement type. In a joint-trial-based synergy set, the NMF divided multi-EMG signals into several part-based groups, resulting in tying the grip synergy to be a combination of EMG signals not used in other synergies. Therefore, the joint-trial-based synergy set could not discriminate movement type. This tendency can be seen in Figure 11. Synergy 5 had the most distinctive shape, representing grip synergy. In a time-domain reaction, SLRM1 responded to grip activation regardless of grip force. In SLRM2, it also increased other synergy coefficients albeit, relatively, by a small amount when a strong grip was assumed. This result could support extracting a unique synergy for each motion [31].
The FDS and FDP are flexor muscles of the fingers from an anatomical point of view. Those two muscles are separated by synergy analysis, which can be observed from Synergy 5, as shown in Figure 10. For the wrist motion with and without grip task, the other five muscles are appropriate to estimate joint angles. For the MSM, the minimum number of muscles exhibited good performance.
The estimation performance between SLRM and MSM had a statistically significant difference in the entire wrist-only movement task both in r and nRMSE values; however, there were no other values in wrist motion with grip. The results show the robustness of the wrist movements with respect to finger movement of the MSM, which is the same as those obtained by Kawase et al.’s experiment [17]. The estimation performance of SLRM also showed comparable performance with previous results in the literature on trajectory [6] and joint force [9] estimation. The advantage of SLRM is that it estimates not only continuous wrist movements but also complex movements in addition to grip motion. In this experiment, we confirmed that the estimation performance of SLRM for complex motion was equivalent to MSM performance.
Within the wrist-only motion trials, both SLRM and MSM had the lowest performance in comfortable half performance. This is most likely due to the nonlinearity between EMG signals and arm motion or contamination of movement artifact and baseline noise to the EMG (representatively, Sub3). Therefore, the nonlinear regression techniques used in the previous studies [6, 9] could be comparable with the synergy-based model having an alternative to using linear regression.
The linear envelope filtering used in the EMG signal analysis was proven to have highly correlated signals with joint torque induced by the target muscle [21]. Grip synergy, which is a coactivation of these filtered EMG signals, also showed a high grip force estimation performance of 0.8463 ± 0.0503 without resort to further conversion or regression techniques. However, in the strong grip trial, it was confirmed that the wrist angle estimation was distorted by gripping EMG signals. This strong grip distortion indicates the necessity to investigate the limits of the SLRM in grip force estimation. Within the current experiment results, it is difficult to determine whether each subject distorted the wrist angle estimation with a similar absolute force level or with a specific ratio of the maximum force.
5. Conclusions
In this study, we explored a model for estimating wrist joint angle with and without grip action. In the first task, in which we examined wrist angle estimation of SLRM and MSM, the SLRM exhibited a relatively higher performance in wrist motion. In the grip task, SLRM showed robustness in angle estimation when the grip force is half or a quarter of its maximum force level. In addition, SLRM can provide the extent of grip force exerted in the center position with little perturbation. These characteristics of SLRM are useful for combined wrist and grip action; however, obtaining limiting grip force required to crash the wrist angle estimation, and vice versa, was beyond the scope of this study. Further studies are required to obtain the simultaneous estimation of both parameters necessary for daily usage.
Data Availability
The experimental results from the individual participants are provided as a table within the article. Data of the raw EMG, IMU, and force sensors are available upon request to the corresponding author.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work was supported by JSPS under KAKENHI grants (grant no. 17H05903), Tateishi Science and Technology Foundation (grant no. 2188001), JST PRESTO (Precursory Research for Embryonic Science and Technology) (grant no. JPMJPR17JA), and JST MIRAI (grant no. JY300171).