Abstract
Hand-force prediction is an important technology for hand-oriented user interface systems. Specifically, surface electromyography (sEMG) is a promising technique for hand-force prediction, which requires a sensor with a small design space and low hardware costs. In this study, we applied several artificial neural-network (ANN) regression models with different numbers of neurons and hidden layers and evaluated handgrip forces by using a dynamometer. A handwear with dry electrodes on the dorsal interosseous muscles was used for our evaluation. Eleven healthy subjects participated in our experiments. sEMG signals with six different levels of forces from 0 N to 200 N and maximum voluntary contraction (MVC) are measured to train and test our ANN regression models. We evaluated three different methods (intrasession, intrasubject, and intersubject evaluation), and our experimental results show a high correlation (0.840, 0.770, and 0.789 each) between the predicted forces and observed forces, which are normalized by the MVC for each subject. Our results also reveal that ANNs with deeper layers of up to four hidden layers show fewer errors in intrasession and intrasubject evaluations.
1. Introduction
Owing to the downsizing of transistors and batteries, wearable devices, including high-performance processors, have recently been expanding into various markets. Wearable devices (e.g., smart watches, smart glasses, and head-mounted displays) release users from locational restrictions and create a hands-free lifestyle. However, the strict requirements of portability and usability require developers to design a user interface (UI) with limited space and low power consumption [1, 2].
The hand has an important role for communication, and hand-gesture recognition has been researched as key technology of hand-oriented user interface [3–5]. Accelerometer and gyroscope are generally used for wearable devices to detect specific motion of the hand. However, these sensors could not be applied to activities accompanied with the isometric contraction, which generates external forces with the static pose of the limb. To measure the forces generated by the isometric contraction, mechanical gloves with pneumatic piston (RMII) [6, 7], strain-gauge sensor (CyberGlove) [7], and cable-actuator [8], for instance, have been used. Ma et al. [9] classified the gloves as body-based (i.e., portable) gloves and ground-based (i.e., stationary) ones, and proposed handgrip-force measurement using force sensors at the fingertips and Gaussian Mixture Model (GMM) for the training. Even though the weight of the portable globe is relatively light (430 g), the weight still has much potential to exhaust the user with long-term use and seems to be a cause to prevent going out from the early stage of development for the practical use [9, 10].
Therefore, the development of small, low-power, and light-weight sensors has been expected for wearable devices. Surface electromyography (sEMG) sensors are often used to detect the hand motion of users when controlling robot arms and virtual hands in VR rehabilitation systems [11–13]. sEMG is also used to measure a quantitative value representing the force of muscular flexion or extension [13, 14]. sEMG sensors have three advantages over other sensors. First, sEMG sensors are noninvasive and safe to use. Second, they have the capability to measure the state of an inner-body function, that is, the state of muscle contraction that appears as external forces. Compared with accelerometers and piezoelectric sensors, sEMG sensors can obtain signals without motion (i.e., isometric contraction) and without any restrictions on the positions of the points of effort. Finally, sEMG sensors are potentially inexpensive because their hardware comprises a simple cascaded structure of electrodes, amplifiers, A/D converters, and band-pass filters. These advantages can contribute to the improvement of the general UI of wearable devices, including buttons, dials, and touch-panel displays.
Gesture recognition by sEMG sensors is a popular topic, and many studies have applied a machine learning-based classification [3, 16, 17]. However, force prediction using sEMG signals is still a challenging task because the signals are weak, noisy, and contaminated with signals from other muscles. Different from the gesture recognition which classifies binary or several gestures from the signal set, accurate regression models are required to predict consecutive values of the force.
Some researches utilized sEMG signals as complemental information of the other inputs. Mobasser and Hashtrudi-Zaad [18] predicted the force of elbow by 2-layer ANN with the inputs of sEMG signals, elbow-joint angle, and the angular velocity obtained from a sensory motor system. Kasuya et al. [19] estimated the hand-force by a system consisting of a muscle stiffness sensor and a sEMG sensor during a tray-holding task.
SEMG-specific approaches have also been studied, even though the amount of the studies to estimate finger-related forces is not much [20]. Srinivasan et al. [21] proposed an idea to estimate the wrist-flexion force from sEMG signals obtained from electrodes on the flexor carpi ulnaris muscle using ANN during an action. Choi et al. [20] evaluated prediction of pinch-forces with the thumb and the index finger. A single-layer ANN with the input of the signals from electrodes on three intrinsic muscles is used as a nonlinear regression model of the force. Hoozemans and Van Dieën [15] predicted handgrip forces by using linear regression models with sEMG inputs of six forearm muscles among eight healthy subjects. Calibrations were performed before the measurement, and the Mean Absolute Errors (MAEs) of the predicted forces were between 27 and 41 N. Leonardis et al. [14] applied a grasping prediction by using an artificial neural-network (ANN) with a single hidden layer for stroke rehabilitation. The average correlation coefficient to predict approximately 50 N was 0.77 among six healthy subjects and two stroke patients. Both the training and test data included the same subjects.
Two unsolved problems exist in the application of sEMG sensors to wearable devices. One problem is that the electrode positions are mainly set for the upper forearms, thus decreasing wearability. Only the abovementioned work by Choi et al. [20] has studied the pinch-force prediction from three intrinsic hand muscles. They assumed the phenomenon “muscle synergies” [22], in which it is known that a behavior produced by contraction of many muscles can be represented by the small subset. The other problem is that the abovementioned studies used paste electrodes, thus severely limiting portability and usability. Dry electrodes are supposed to be used for nonclinical wearable devices, but the contact impedance to the skin of dry electrodes is higher than that of paste electrodes. Hence, the impedance and the variation of the contact area during motion cause noise and artifacts [23, 24].
In this study, we focused on handgrip-force prediction and evaluated the accuracy of ANN regression models with different numbers of neurons and hidden layers. We assumed that the device will be used for nonclinical applications which require portability and usability, and an elastic handwear with dry electrodes was used in our experiment. The dry electrodes are located only on the back of the hand to limit obstructions to hand motions. To compensate for miniaturization of the device by reducing the number of electrodes and concentrating the location on the back of the hand, we evaluated ANNs with multiple hidden layers to represent complex nonlinear regression models. One of the great benefits of adopting ANN is that a lot of variety of nonlinear regression models can be represented only with the change of the number of hidden layers and neurons, without changing kernel functions such as polynomial and support vector regression models for instance. A dynamometer was used to measure handgrip forces, and the result shows a high correlation between the predicted data and observed data (0.77–0.84). The result indicates that the placement of sEMG electrodes only on the back of the hand enables the acquisition of input signals for predicting handgrip forces in handwear devices.
2. Materials and Methods
2.1. Muscle Structure for Finger Activities
Many muscles are involved in the movement of the hands and fingers. In the case of gripping, forces are exerted by the phalanges of the fingers [25]. Approximately 20 types of muscles are related to finger activities (Table 1). Approximately half of these muscles are located in the hand itself (i.e., intrinsic hand muscles), and the other half is located in the forearm (i.e., extrinsic hand muscles) [26]. Extrinsic muscles are commonly used for measuring finger conditions by sEMG sensors [13, 14, 27] because the forearm has enough space to place electrodes for sensing the sEMG signals of superficial muscles. However, the placement of electrodes on the forearm presents problems for wearable devices (e.g., when the user has long sleeves).
By contrast, signals of intrinsic hand muscles have less contamination with other muscles because the intrinsic muscles are located much closer to the fingers, and higher signal-to-noise ratio (SNR) can be obtained because the intrinsic hand muscles are typically covered with less subcutaneous fat than extrinsic hand muscles [13, 28]. In the present study, we focused on sensing the dorsal interosseous (DI) hand muscles to evaluate the accuracy of handgrip-force prediction. The dorsal interosseous (DI) hand muscles are located in the superficies of the back of the hand, where the electrodes and the cables do not prevent general hand activities.
2.2. Processing Flow and System Components
Figure 1 shows a diagram of our process flow. The acquired signals are amplified, quantized, and band-pass filtered before being stored in memory. We used wireless sEMG sensors (OE-WES1224, Osaka Electronic Equipment Ltd.) with bipolar nonpasted Ag/AgCl electrodes to sense sEMG signals. Figure 2 shows the elastic handwear with dry electrodes used in our experiment. The electrodes are supposed to be placed approximately on the four DI hand muscles. Signals were sampled at 1 kHz and converted to digital by a 16-bit A/D converter and filtered by a 4th-order band-pass filter from 30 Hz to 700 Hz. Data acquisition from each electrode was triggered and synchronized by a LabVIEW application for the sensors. The collected data were analyzed, studied, and evaluated by MATLAB on a PC.
2.3. Experimental Procedure
Eleven healthy subjects in their early 20s participated in our experiment. The experiment aims to measure sEMG signals while gripping a dynamometer with six different force levels: 0, 50, 100, 150, and 200 N and the maximum force of each subject (i.e., MVC: maximum voluntary contraction). The task of each force level alternated between gripping the dynamometer for 3 s and releasing for 10 s, and the total duration of a session is 60 s (i.e., totally, gripping four times for 3 s each in a single session). Some time-dependent features (i.e., velocity and acceleration of a time-series of the force) may differ among each trial according to many slight factors including psychological conditions, so we adopted a static measurement method [15, 20] to obtain temporally stable data to train and test ANNs, instead of the dynamically fluctuated measurement [14, 15, 20]. The subjects were told to sit down on a chair and to grip a dynamometer on a desk (Figure 4). The angle of the elbow was almost perpendicular with a natural and relaxed posture. Subjects wore an elastic handwear with electrodes. sEMG sensors were connected to each electrode, and the signals were temporarily saved in the memory of the sensors. The saved signals were sent to a PC after the experiment for each subject was finished. The study was approved by the Human Ethics Committee of the Waseda University.
2.4. ANN Regression Model of Hand-Force
Figure 5 shows our ANN model for the regression of the handgrip forces. We applied a feedforward network with gradient descent backpropagation. The network consisted of an input layer, hidden layers, and an output layer. A sigmoidal logistic function was used as the transfer function. We added a preprocessing layer before the input layer to compute the root mean square (RMS) at certain intervals of the sEMG signals.
Some previous works applied one or two hidden layers for the regression [14, 18, 20, 21, 29–32]. In the present study, the number of hidden layers varied from one to four according to variations in the number of neurons of each layer. To reduce the variety of the number of neurons, the ratio between a previous hidden layer and the later hidden layer was fixed at 2 : 1. The number of neurons of the last hidden layer varied from 1 to 20 in our evaluation. For example, the number of neurons was fixed to eight, four, two, and one from the first hidden layer to the last hidden layer; that is, the number of neurons of the last hidden layer is one when the number of the hidden layers of the example is four.
2.5. Evaluation Methods of ANN Regression Models
The acquired sEMG signals were separated to train data and test data according to the following three different evaluation methods of generalizability [29, 30]:(1)Intrasession evaluation: signals of a single gripping task out of four in a session were chosen as the test data, and the signals of the other three gripping tasks and releasing tasks of the same session were used as training data. Therefore, 25% of all gripping data was used for the test, and the other data including releasing tasks were used for the training in each evaluation. The stability and repeatability of a task were evaluated by this method.(2)Intrasubject evaluation: signals of a whole session of a single force level out of five except for 0 N (i.e., releasing tasks) were chosen as test data, and the signals of the other force levels including 0 N were used as training data. Therefore, 20% of all gripping data was used for the test, and the other data including releasing tasks were used for the training in each evaluation. The errors of the test data among all force levels were averaged. We applied this method to evaluate the difficulty of the prediction of each force level.(3)Intersubject evaluation: all signals of one subject out of 11 were chosen as the test data, and the signals of the other 10 subjects were used as the training data. Therefore, 9% of all data was used for the test, and the other data were used for the training in each evaluation. The errors of the test data among all subjects were averaged. This method was applied to evaluate the public use of the device by unspecified users.
Figure 6 shows an example to collect sets of sEMG signals from a waveform of a single gripping task. Multiple samples can be obtained from a single task by a sliding window method. We set the parameter to T = 500 ms, and the sliding interval was set to . Thereafter, approximately 11 samples were obtained from a waveform of a single gripping task for 3 s. Samples of the force 0 N were acquired by the same sliding window method from the waveforms of intervals between the gripping tasks for 10 s each. Totally 8,302 pieces of data which consist of 2,087 pieces of gripping data and 6215 pieces of releasing data were obtained by the sliding window method.
3. Experimental Results
Figures 7–9 show the relationship between the number of neurons of the last hidden layer and the RMS error (RMSE) of each evaluation method. The results of the ANNs with different numbers of hidden layers (L1–L4) are compared in each graph. The graphs on (a) show the results trained by the observed data with the unit N; the graphs on (b) show the results trained by the observed data with the unit % MVC, that is, the ratio of the forces normalized by the maximum force of the subject. Both the intrasession and intrasubject evaluation results show some correlations between the RMSE and the number of neurons (Figures 7 and 8). The errors decrease according to the increment of neurons, and no obvious overfitting could be seen. Furthermore, these evaluation results show that the accuracy improves with the increasing number of hidden layers. The results indicate the effectiveness of deep and wide ANN regression models for intrasession and intrasubject evaluations. By contrast, no correlation between the RMSE and both the number of neurons and hidden layers could be found in the intersubject evaluation (Figure 9). The results imply that a certain amount of individual differences exist for the regression models of handgrip forces.
Table 2 shows the results of the best ANNs, which have the least RMSEs in each evaluation. Both the results of the networks trained by the data with the unit N and the unit % MVC are described. Regardless of the units, the results of the intrasession evaluation show the lowest errors, whereas the results of the intersubject evaluation show the highest errors. The correlation coefficients () of the results of the network trained by the data with the unit % MVC are higher than the ones with the unit N for the case of the intrasubject and intersubject evaluations (the coefficients of the unit % MVC are 0.770 and 0.789 in the intrasubject evaluation and intersubject evaluation, resp.; by contrast, the coefficients of the unit N are 0.613 and 0.561 in the intrasubject evaluation and intersubject evaluation, resp.).
4. Results and Discussion
Our study showed the availability of sEMG signals on four DI hand muscles and the prediction of handgrip forces by using unpasted dry electrodes. The lowest RMSEs in the intrasession, intrasubject, and intersubject evaluations are 16.2%, 21.4%, and 26.5% of the MVC, respectively; the RMSEs with absolute values in the intrasession, intrasubject, and intersubject evaluations are 45, 67, and 73 N, respectively. The average maximum force is 283 N among the 11 subjects, and all maximum forces of the subjects exceeded 200 N. A comparison with other methods is shown in Table 5. Our study shows less errors (45 N as RMSE) than the error in [15] (52–80 N as MAE) which was also measured statically. Besides we preferred RMSEs in our work to MAEs, which include less weight of outliers than RMSEs. The correlation coefficient () of our study shows higher value than the result in [14], even though the highest force level of ours (200 N) is much higher than the one in [14] (approximately 50 N). Our study also has much advantage of wearability than the previous studies. Our study maintains the wearability in our evaluation by using dry electrodes which are located only on the back of the hand. It would be critical disadvantage for wearable devices to insist that users attach paste electrodes on the correct positions on the forearm without a guide of clinicians.
We also showed the designability of ANN for prediction in terms of the number of neurons and hidden layers. As shown in Figures 7 and 8, the errors consistently decrease with increasing number of neurons and hidden layers in the intrasession and intrasubject evaluations without any critical overfittings. This tendency indicates the facility of the design of the network. By contrast, the results in the intersubject evaluation shown in Figure 9 did not show the same tendency. The results imply the difficulty of designing this type of network, which can be overfitted or fall into a local optimum.
The relationships between the observed data and predicted data obtained in each evaluation are shown in Figure 10. The results trained by the data with the unit N are shown from (a) to (c), and the results trained by the data with the unit % MVC are shown from (d) to (f). The blue and bold line in each graph represent the linear regression between the predicted data and observed data. The gray and fine lines represent the line in which the predicted data are equal to the observed data (i.e., the correlation coefficient is equal to one). The expression of the linear function is described along the -axis, and the correlation coefficient, which is also shown in Table 2, is shown on the top of the graph. The results trained by the data with the unit N show less correlation (0.613 in the intrasubject evaluation and 0.561 in the intersubject evaluation) than the results for the unit % MVC. The predicted data with the unit N in the intrasubject evaluation tends to be lower than the observed data when the values are the maximum forces (Figure 10(b)). This result implies that the sEMG signals obtained from the DI hand muscles with maximum force tend to be saturated at some level. One possible reason for the saturation is that the other muscles related to hand-gripping tasks, such as FDS, APB, and FDM (Table 1), start bearing the loads of the forces instead of the DI hand muscles according to the increase in the loads.
Figure 11 shows the scatter plots of the observed data and the corresponding RMS values of the sEMG signals obtained from each electrode (DI(I)–DI(IV) in Figure 3). The plot of DI(I) is shown as the first column ((a) and (e)), and the plot of DI(IV) is shown as the last column ((d) and (h)). The unit of the observed data on the upper row ((a)–(d)) is N, and the unit of the observed data on the lower row ((e)–(h)) is % MVC. An apparent saturation of the amplitude of the RMS sEMG signals over 200 N can be seen at the electrode DI(II) in Figure 11(b) even though the amplitude of the signals obtained from the electrode DI(I) in Figure 11(a) is not saturated. The first DI hand muscle is inserted into the radial side of the proximal phalanx of the index finger, and the second DI hand muscle is inserted into the radial side of the proximal phalanx of the middle finger [33]. Therefore, the difference in the saturation of the forces over 200 N is supposed to represent the difference of the contribution ratio of the index finger and middle finger when generating the handgrip-force.
(a)
(b)
(a)
(b)
(a)
(b)
(a)
(b)
(a)
(b)
(c)
(d)
(e)
(f)
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Figure 11 also shows the instability of the signals obtained from dry electrodes. The amplitude of the signals obtained from the electrodes DI(III) (shown in (c) and (g)) and DI(IV) (shown in (d) and (h)) has a high variance, thus causing the fluctuation of the impedance of the contact area between the skin and electrode during the gripping activities. This phenomenon is caused by the peeling of the electrodes derived from the difference of the height of the bulges of the muscles. Our experimental results showed a fact that even the fluctuated signals contribute to raising up the accuracy of ANN regression models as shown in Tables 3 and 4. The Tables include the RMSEs of the predicted data obtained from the ANNs trained by the partial inputs of the electrodes. Compared with the errors of the prediction trained by the whole inputs DI(I–IV), which are shown at the last column, most of the prediction errors derived from the partial inputs are higher than the errors derived from the whole inputs.
5. Conclusions
This paper evaluated the accuracy of ANN regression models to predict handgrip forces from the sEMG signals obtained from dry electrodes on the four DI hand muscles by using an elastic handwear. Our experimental results show low prediction errors (16.2%, 21.4%, and 26.5% of the maximum forces) and high correlation (0.840, 0.770, and 0.789) in the intrasession, intrasubject, and intersubject evaluations, respectively. We also showed the relationship between the number of neurons and hidden layers of ANNs and the prediction errors, which consistently decrease with increasing number of neurons and hidden layers in the intrasession and intrasubject evaluations. Our ANN regression models predicted forces from signals even with the saturation of amplitude at some force levels and fluctuations caused by the peeling of the dry electrodes. Our work shows that hand forces can be predicted accurately by using sEMG signals obtained from dry electrodes of handwear devices.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
The authors would like to thank Professor N. Togawa and Professor Y. Shi for their contributions to this work.