Abstract
Analyzing human muscle states has attracted extensive attention. EMG (electromyography) pattern recognition methods based on these works have been proposed for many years. However, uncomfortable wearing and high prices make it inconvenient for motion tracking and muscle analysis by using robotic arms and inertial sensors in daily life. In this study, we propose to use smart clothes integrated with flexible sensors to collect arm motion data, estimate the kinematic information of continuous arm motion, and predict the EMG signal of each arm muscle. Firstly, the neural network regression model integrated with the LSTM (long short-term memory) module is used to continuously estimate the sensor resistances collected by the smart clothes and the angles collected by Kinect. Then, six types of shoulder and elbow movements’ angles and the corresponding EMG signals of 5 subjects are preprocessed and aligned. The stacked regression model based on extremely randomized trees (extra trees) is used for regression. Our experimental results show that the average estimation absolute error from the sensor resistances to the joint angle is 3.45 degrees, and the absolute percentage error from the joint angle to the EMG signal is only 1.82%.
1. Introduction
Human motion posture tracking and muscle analysis capture the continuous motion of bone joints by various types of sensors or multiple types of cameras to continuously estimate the value of muscle activity at the moving parts [1]. It is of great significance in medical rehabilitation, military and national defense, animated cartoon, and other fields. Take the postoperative rehabilitation of patients with cerebral stroke as an example, there are about 2 million new stroke patients in China every year, and up to 80 percent of stroke patients have the sequelae of dyskinesia [2]. The existing rehabilitation treatment is mainly concentrated in the hospital environment [3], under the guidance of professional rehabilitation therapists. In contrast, the rehabilitation training of patients in the home environment lacks effective and accurate monitoring methods. Unscientific rehabilitation training may lead to an abnormal movement, leading to poor recovery and even secondary physical impairment. So, it is of great value to accurately track the movement posture of these people and analyze the muscle exercise situation for the development of personalized rehabilitation programs.
A lot of early work is based on mechanical arm [4–6] mechanical inertial sensor (MEMS) [7–9] to capture human motion posture. The human motion posture tracking based on an inertial sensor unit (IMU) is the current mainstream, wearable method [10, 11]. IMU has the advantages of low cost, small size, no interference, etc. It is suitable for human motion posture tracking outdoors. Nevertheless, there are three main problems: foreign-body sensation when wearing, requiring calibration, and data drift. One reason for the widespread use of inertial sensor units (IMUs) is that wearing foreign body sensations is severely constrained, and researchers are looking for ways to collect physiological data in ways that make subjects more comfortable [12, 13]. In recent years, the use of intelligent wearable sensors with integrated flexible sensors to monitor human health state has aroused widespread interest [14, 15]. They provide good user experience and comfortable wearing [16] with cheap price [17].
Our work proposes a method of human posture tracking based on a flexible sensor. The captured posture is calculated to obtain the joint angle of the upper limb, which is used to estimate the electromyographic signals of the upper limb movement continuously.
Our work is aimed at solving the problem of muscle analysis on a smart clothing platform. Muscle analysis using a smart jacket is defined as two key subproblems: (1)Using flexible sensors to track human body posture so as to calculate upper limb joint angles(2)Using the joint angle of the upper limb movement to estimate the myoelectric signal continuously
There are two main contributions to this study: (1)We provide a complete human posture tracking and joint angle calculation solution. We use flexible sensors to minimize interference with the users’ daily activities and ensure the best comfort possible in their experience. And the novelty of the method is that it is not affected by the light, and the user does not need to consider the illumination effect of the environment(2)We introduce a stacked regression model based on the extreme random tree for continuous estimating joint angles and EMG signals. Compared with other traditional machine learning methods and regression networks with short and long memory modules, this regression model has a better regression effect
2. Related Work
2.1. Motion Tracking Based on Flexible Sensor
The main scope of monitoring human movement is divided into two kinds. One is a large range of individual position movement; the other is the individual joint movement, temperature, and other physiological indicators. Because of its physical characteristics, flexible sensors are mainly used to detect human body states. And flexible sensors used for human body state monitoring mainly convert the signals to be monitored into the stretching and bending of the sensor through mechanical deformation [18] or use the physicochemical mechanism driven by temperature, humidity, and chemical reaction to measure the changes in resistance or capacitance to achieve monitoring purposes [19, 20]. Its functions mainly include motion posture tracking and physical skin deformation. Representative works are listed as follows. The Massachusetts Institute of Technology (MIT) group uses high-density array-type pressure sensor gloves for object recognition [21]; Northwestern University research uses a wireless passive flexible vibrator to realize tactile feedback in virtual reality scene [22]; the Swiss Federal Institute of Technology in Zurich uses the flexible gloves on finger movement tracking [23]; the United States Dartmouth College team uses the flexible sheath on elbow motion tracking [24]; the National University of Singapore research uses the sensor fusion of leap motion controller and flexible sensor to track human finger using Kalman filter [25]; Tsinghua University researchers use the array-type pressure membrane to identify the interaction between the human body and the object, etc. [26]; Northwestern Polytechnical University research uses a small set of wearable sensors to estimate whole-body pose in human bicycle riding [27]; Chinese Academy of Sciences research uses micro flow on the skin surface tactile sensing [28]; Institute of electronics, Chinese Academy of Sciences research uses flexible pressure sensor to 3-dimension force recognition [29]; Shenzhen University research group realized noncontact human-computer interaction by using short-range capacitance sensor [30]; etc.
The human body motion posture tracking method based on flexible (nonfabric) sensors achieves the target of finger motion posture tracking [31] and arm deformation reconstruction [32]. In the above work, the researchers collected training data and established a deep neural network and then realized the prediction of joint posture and skin deformation. The human motion posture tracking method based on the fabric sensor realizes the tracking of the whole body [33], elbow [34], and back and shoulder [33]. The researchers used a neural network of short- and long-duration memory to interpret stretch sensor signals as body posture but only did error analysis for specific movements such as squatting and bending. Another work used stretch sensors to track the posture of upper body movements (back and shoulders). The researchers also used stretch sensors to track elbow motion and analyzed motion tracking errors for different arm circumferences and sensor position offset (up to 1 cm). The pliable and flexible characteristics of the flexible sensor provide a convenient way of wearing and a comfortable user experience and provide more possibilities for human motion posture tracking [17].
2.2. Muscle Analysis
According to the International Classification of Functioning, Disability, and Health [35], muscle power is the maximum power that can be released by a test muscle under certain limits [36, 37]. And muscle power controls the movement of our limbs. These forces must be estimated by indirect means since the direct measurement is usually neither possible nor practical. Therefore, many studies measure muscle strength through inverse kinematics [38, 39], including the fingers [40], upper limbs [41], and lower limbs [42–44]. In experimental studies of human movement, muscle strength tests are useful in assessing the recovery of stroke patients [45]. And EMG signals are measured to determine the electrical current generated by muscle contractions in neuromuscular activity [46]. Therefore, electromyography (EMG) is often used as a tool to determine muscle activity [47–50]. Bogey et al. [44, 51] recently developed a method to estimate force from EMG signals and based on normalization of activation during maximum voluntary contraction to record maximum muscle force. Heintz and Gutierrez-Farewik [52] adopted the numerical algorithm established based on the traditional optimization technology, that is, the constraint minimization technology using the Lagrange multiplier method to solve constraints. Lloyd and Besier [53] used EMG to predict knee torques through inverse dynamic calculations under different dynamic contraction conditions. They used a modified Hill-type muscle model to better estimate living muscle strength during exercise tasks. Amarantini et al. [54] used two-step EMG and optimization methods to estimate muscle forces under dynamic conditions. This method has the ability to propose a method to account for agonist/antagonist cocontraction properly. In addition, the method can improve the confidence of muscle force estimation. Hashemi et al. [55] combined angle-based EMG amplitude calibration and parallel cascade identification (PCI) modeling for EMG-based force estimation in dynamic contractions, including concentric and eccentric contractions of the biceps and triceps, in order to enhance dynamic EMG-force estimation. Hsu et al. [47] used EMG sensors to study the sequence of muscle contractions in patients from sitting to standing (STS) after stroke. Kim et al. [56] estimated the muscle strength of nine muscle groups of the lower limbs using a static optimization method with inverse dynamics based on motion data and compared it with EMG signals. It is proved that establishing the relationship between EMG signal and muscle force calculated by inverse kinematics is a practical method to measure muscle strength in vivo.
3. Method
We built a sparse sensor network on the smart jacket to capture the sensor resistance value of human right upper limb movement to estimate the EMG signal continuously. The workflow of our algorithm is shown in Figure 1.
We split the task into two subtasks: (1)Prediction from sensor signal to joint angle(2)Continuous estimation from joint angle to EMG signal. We use a neural network with long short-term memory (LSTM) [57–59] module to regress the sensor data and joint angle data to solve the nonlinear and hysteretic problems of the sensor itself. Joint angle data and EMG data were fitted by stacking a regression model based on extreme random tree. Thus, a new method of continuous estimation from sensor signal to EMG signal is provided
3.1. Hardware Preparation
We prepared a smart jacket prototype (Figure 2(a)) with integrated, flexible sensors to complete our study. The smart jacket cloth has five flexible, stretchable sensors, four around the shoulder and one under the elbow, as shown in Figure 2(b). And the above parts contain 20 cm sensors, respectively, as the total cost is about 15.1488 dollars. Sensors are fixed to the garment by hand sewing. The clothes are tight tracksuits, ensuring the sensors fit snuggly and better capture shoulder and elbow movement. The fabric is made of 80% polyester fiber and 20% polyurethane fiber. The sensor we use is a conductive rubber wire stretch sensor manufactured by Adafruit [60, 61]. The sensor is 2 mm in diameter and made of carbon-black impregnated rubber. As for the traditional sensor, multipoint instruments need to be used; the production and use are complex; the experimental conditions are not convenient; also users cannot use them directly at home. In addition, traditional sensors are more expensive compared to flexible, stretchable sensors. As a result of fact, the conductive rubber wire stretch sensor is more suitable and practical, and for the sensor, in the relaxed state, the resistance is about 350 ohms per inch. The human body posture tracking is realized by monitoring the resistance changes of the five flexible sensors. At the elbow, for example, when the user bends the elbow, the sensor is stretched and its resistance increases. The resistance sampling frequency of the sensor is 32 Hz. The server receives sensor data through the Arduino UNO3 development board and is responsible for all subsequent calculations. The server comes with Intel Core I7 (6 cores), 16 GB of RAM, and NVIDIA GTX 1080Ti.
(a)
(b)
3.2. Sensor Resistance and Joint Angle Data Collection
For the participant, the experiment involved a 30-year-old doctoral student in the lab. The participant is familiar with the whole experimental design and the experimental process before the investigation. For the collection process, before the experiment, the participant got to know the purpose of the experiment, put on the intelligent jacket with integrated, flexible sensors; and took part in some guiding activities to get familiar with the collection system and experimental process. The participant started and stopped the data collection experiment by listening to voice instructions. He adjusted his position towards a Kinect camera that could collect depth information, allowing the camera to capture the full movements of his right upper limbs. If the participant gets tired during the collection process, he can terminate the collection process at any time.
The data collected each time is saved to the server. The sensor resistance data includes the collection time and resistance values of five sensors. The data collected by Kinect includes the collection time and three-dimensional position of each node of the right upper limb. The joint angles of each part were calculated according to ISB (International Society of Biomechanics) convention [62]. These angles are calculated by the following methods. First, create local coordinate systems based on marker points and then, decompose the rotation matrix into Euler sequences proposed by ISB for a particular skeleton. The data are in the -axis upward direction, which is consistent with the joint angle data introduced in the next section.
3.3. Joint Angle and EMG Data Alignment Preprocessing
We used the data set published by Bolsterlee et al. [63], Shoulder Database, for continuous estimation from shoulder elbow angle to EMG signal. The data set contained data on six movements of five healthy young adults. No one has uncomfortable shoulder joints or has been specially trained. Kinematic data were collected using a four-device Optotrak system (Northern Digital, Inc., Waterloo, Ontario, Canada) to collect marker location information for six groups of markers in the subjects’ chest, scapula, humerus, forearm, and hands. The acquisition frequency is 100 Hz. EMG data were collected using surface electrodes (Ambu N-00-SECG) and a 16-channel Porti system (TMS International, Enschede, the Netherlands, Sampling frequency: 1000 Hz). EMG data were collected from 14 muscles, including the triceps and biceps (Figure 3). The electrodes are placed as recommended by SENIAM [64]. Since the frequency of kinematic data and EMG data are not the same, we need to preprocess the data for alignment. Due to the fast-sampling frequency of EMG data, we first performed linear interpolation on the joint angle data to obtain 1000 pieces of data per second and then aligned the joint angle data with EMG data through the acquisition time.
3.4. Regress from Sensor Signal to Joint Angle
From the description of the sensor manufacturer and our experiments, it is found that the relationship between the resistances of the sensors is nonlinear when subjected to tension. Moreover, the sensor has a lag problem, and it needs to be stationary for a period of time to return to the initial state. Therefore, we propose to use the LSTM model to regress sensor signals and joint angles in order to obtain more accurate angle prediction values.
LSTM is an artificial recursive neural network (RNN), which can effectively process temporal data. The network model we designed has an LSTM layer with five hidden cells, followed by a full connection layer with ten cells, and finally, a full connection layer with only one cell as output. The input to the network is a vector: where is the sensor resistance at a specific time , is the time step read by the sensor signal, and is the number of sampling points. In our experiment, and were set to 0.286 and 50, respectively. The output of the LSTM network is the estimated joint angle. LSTM regression model is used to solve the nonlinear and hysteresis characteristics of the sensor, which realizes the precise prediction of the joint angle.
3.5. Regress from Joint Angle to EMG Signal
We chose to use a stacked regression model based on extra trees [65] to process from joint angle to EMG signal. The extreme random tree algorithm builds a set of nonrunning decision trees or regression trees based on the classical top-down process. The two main differences between it and other tree-based integration approaches are that it splits nodes by randomly selecting complete pointcuts and uses the entire learning sample to grow the tree. From the perspective of the bias difference, the basic principle behind the extreme random tree approach is that explicit randomization of pointcuts and attributes combined with integrated average should be able to reduce variance more effectively than the weaker randomization schemes used by other methods. Using the complete original learning sample rather than the boot stringing copy is aimed at minimizing bias. From a computational point of view, suppose there is a balanced tree, the complexity of the tree growth is order with respect to the learning sample size, just like most other tree growth processes. Our stacked model (Figure 4) integrates extreme random tree, random forest [66–68], and -nearest neighbor [69] algorithm as the basic model and uses a linear regression model in the metalearner [70]. The metamodel is trained based on the prediction results of the training samples output by the basic model. The stacked model can solve the problem of the prediction errors made by different models which are not correlated or have low correlations.
4. Result Analysis
4.1. Evaluation Criteria
The following six evaluation criteria were used to evaluate our experimental results: mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), square (), root mean square logarithmic error (RMSLE), and mean absolute percentage error (MAPE). The values of MAE, MSE, RMSE, and MAPE are in the range of ; when the predicted value is exactly the same as the true value, the value is 0; the larger the error, the larger the value. MAE and RMSE can roughly estimate the difference between the predicted value and the true value; when the predicted value is entirely consistent with the true value, the value is 1; when each predicted value of the sample is equal to the mean value, the value is 0; it may also be negative, and the regression effect is poor.
4.2. Regression Results of Short- and Long-Term Memory Network Regression Model
Sensor data is composed of resistance signals from five sensors numbered 1-5 (Figure 2(b)). Joint angle data is calculated by the spine points, shoulder points, elbow points, and wrist points collected by Kinect, including shoulder angle and elbow angle. We defined that the elbow angle has only one degree of freedom, and the shoulder angle has three degrees of freedom. Then, the LSTM regression model was used to perform training regression on sensor data and joint angle data. 70% of the collected data were trained in the front segment, and the rest was used for predictive testing.
We compare the LSTM regression model with the traditional machine learning regression model and polynomial regression model, and the true value is also involved. In this paper, the gradient lifting decision tree (GBR) with the best regression effect in the traditional machine learning regression model and the regression results of the fourth-order polynomial regression are selected for comparative display, as shown in Figure 5.
It can be seen from the experimental results that the LSTM regression model (red curve) fits the real value curve (black curve) well, and there is a smooth transition between each point. While the regression results of the gradient lifting decision tree (blue curve) are consistent with the real curve at some points, it fluctuates greatly at many points. Although the fitting result of the fourth-order polynomial (green curve) is relatively smooth, the error between some values and the real values is huge.
We also conducted absolute error statistics for these three regression models, as shown in Figure 6. The mean absolute errors of the LSTM regression model, GBR regression model, and polynomial regression method are 3.45, 7.48, and 7.98, respectively. The LSTM regression model has the smallest overall regression error range.
Table 1 shows the statistics of LSTM regression model prediction results. It can be seen from the table that all error evaluation criteria are quite small, value reaches 0.9839, the absolute error value is only 3.45 degrees, and the average absolute error percentage is only 3.17%. Therefore, the LSTM regression model can get accurate angle prediction results.
4.3. Comparison of Regression Results between Extreme Random Tree Regression Model and Other Traditional Machine Learning Regression Models
In this section, we show the results of comparing the first subject’s elbow’s slow-motion data and EMG data when using extreme random tree regression models with other traditional machine learning regression models in the Regression Shoulder Database. The objective of regression was the EMG signal of the coracobrachialis. There are 18,729 pieces of data set shown in Figure 7(a). We select 70% of the data as the training set and the rest as the test set. The number of folds in cross-validation is 10. Data is normalized using -score.
(a)
(b)
The regression results are shown in Table 2. We show the three models, extreme random tree (ET), random forest (RF), and -nearest neighbor (KNN) regression models, with the best regression effect among the traditional machine learning regression methods; the extreme random tree takes 4.06 seconds in training time. However, the final value of extreme random tree regression is the highest, and other error evaluation items are the smallest. We can intuitively see the estimated difference between the predicted value and the actual value through the mean absolute error (MAE) and root mean square error (RMSE). The extreme random tree regression error is the smallest, and it is almost 54% of the regression error of the random forest model, which has the second-best regression effect. Finally, Figure 7(b) shows the residual prediction results and residual distribution of the extreme random tree regression model. The residual is concentrated below the absolute value of 1000 (about 3% of the original EMG data).
Table 3 shows the prediction results of the extreme random tree regression model on the test set. The data are even better than the training results. Therefore, in the experiment, the extreme random tree regression model has the best regression effect and is much better than any other traditional machine learning regression model.
4.4. Stacked Regression Model
In order to further reduce the regression error, we designed a stacking model based on the extreme random tree (Figure 4), stacked the three regression models in Table 2 (including extreme random tree, random forest, and -nearest neighbor regression model), and then trained and predicted the data.
The first row (beginning with T) of Table 4 shows the training results of the stacking model. The average absolute error and root mean square error are more than 100 smaller than the extreme random tree regression method in the traditional machine learning method (see Table 2), indicating that our stacking model effectively reduces the regression error. Moreover, from the second row of Table 4 (beginning with P), we can see that the stacked regression model also achieved 312 as the results of average absolute error and 457 as the root mean square error, which is about 22% less than the regression model with only extreme random tree. The average absolute percentage error is only 0.96%.
4.5. Stacked Regression Model Regression Results on Different Person Data
We used the stacked regression model to carry out the regression test on the joint angle and EMG signal of shoulder and elbow activity data of different people. The results are shown in Table 5. It can be seen from the average absolute error and root mean square error that the average difference between the predicted value and the real value is within 370; while MAPE showed that the absolute errors of the five subjects were about 1.14%, 0.08%, 0.08%, 2.61%, and 5.18% of the original EMG signal, respectively. The experimental results show that the stacking model can well regress the joint angle and EMG data. Due to the large individual differences of each person, there will be great differences in the relationship between each person’s joint angle data and EMG data. Therefore, before using the smart coat for daily activity tracking and EMG signal prediction, users are required to collect standard angle data and corresponding EMG signal data in the hospital according to professional guidance [26] to build a stacked regression model. Therefore, everyone needs to train a unique stacked regression model to facilitate the subsequent regression and analysis of EMG signals.
4.6. Comparison between the Stacking Regression Model and LSTM Regression Effect
We compared the stacking model with the regression model with LSTM module. Table 6 shows the statistics of regression results of the LSTM regression model.
Compared to the regression prediction results of the stacking model in Table 4, better regression results cannot be obtained on this data set by the LSTM regression model than by the stacked regression model. The MAE value of the LSTM regression model reached 547.39, which was 235.45 higher than 311.94 of the stacked regression model. Moreover, the average absolute percentage error of the LSTM regression model is 13.23%, while that of the stacked regression model is only 0.96%. Combining the results of this section with the analysis in Section 4.2, the LSTM regression model is dominant in solving the problems of nonlinearity and hysteresis of flexible sensors, but not in the data set of this section, which requires us to select an appropriate regression model for the data from different sources. Our method also has some limitations. Because the training data is limited to the data collected by professional equipment in a short time, it is impossible to predict the irregular changes of myoelectric signals caused by muscle fatigue due to long-term exercise.
5. Conclusions
This paper presents a method for continuously estimating upper limb EMG signals by using an intelligent jacket with integrated, flexible sensors. Firstly, we use the long short-term memory network regression model to continuously estimate the sensor signals collected by the smart coat and the joint angle information collected by Kinect. Then, the shoulder elbow angle information and the corresponding EMG signals of five subjects were aligned and preprocessed, and the stacked regression model based on the extreme random tree was used for regression. The experimental results show that the long short-term memory network can effectively solve the nonlinearity and hysteresis of flexible sensors, and the stacking model can well regress the joint angle data and EMG signal data. This method provides more possibilities for home health monitoring and exercise guidance in the future; for example, users can achieve online medical treatment and online diagnosis based on this system. However, there are still some problems to be solved and studied, such as the sensor position offset and the deformation caused by clothing wrinkles, which will affect the accuracy of the prediction results. Therefore, we will continue to solve the above problems in the follow-up work and the proposed machine learning based pipeline can be applied to other fields, such as bioinformatics and computational biology [67, 71–74].
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Disclosure
The noted sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results that appear herein.
Conflicts of Interest
The authors declare no conflict of interest.
Authors’ Contributions
Zhiyong Chen and Qingsuo Wang contributed equally to this work as first authors.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (62072383, 61702433, and 62077039), the Fundamental Research Funds for the Central Universities (20720190006), and the Open Project Program of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (VRLAB2020B17).