Abstract
As exoskeleton robots are more frequently applied to impaired people to regain mobility, detection and recognition of human gait motions is important to prepare suitable control modes for exoskeletons. This paper proposes to explore the potential of the ensemble empirical mode decomposition (EEMD) method to help analyze and recognize gait motions for human subjects who wear the exoskeleton to walk. The intrinsic mode functions (IMFs) extracted from the original gait signals by EEMD are utilized to act as inputs for classification algorithms. Evident correlations are found between some IMFs and original gait kinematic sequences. Experimental results on gait phase recognition performance on 14 able-bodied subjects are shown. The performance of the composing signals extracted from the original signals as is investigated, which indicates that might be helpful when wearing exoskeleton and might be helpful when walking without exoskeleton on gait recognition. And the similarity of joint synergy between wearing and without wearing exoskeleton is analyzed, and the result shows that the joint synergy might change between with and without wearing exoskeleton. The quantitative results show that based on some IMFs of the same orders, these machine learning algorithms can achieve promising performances.
1. Introduction
Exoskeletons are encountered increasingly and frequently in assisting the movement of people in various aspects, helping them regain locomotion ability and reconstructing their lost gait. In order to achieve better gait transition for exoskeletons with different gait modulation modes switched, it is important to make accurate recognition of different gait phases for the human-exoskeleton system. In the past decades, researchers and engineers have developed a variety of methods for detecting and recognizing human gait with and without wearable robots. Aertbelien and Schutter developed a statistical model method to learn the joint gait trajectory and the variations for the control of the lower limb exoskeleton [1]. Brinker et al. proposed a gait recognition model that can fast adapt to a novel user’s movements in exoskeleton control by using previously collected other users’ data and generate one side of leg exoskeleton movement based on the other side of the leg movement [2]. Kim et al. presented a novel shape-based and model-based combined gait recognition algorithm by extracting mode-based gait cycle based on the prediction-based hierarchical active shape mode [3]. Recently, Torricelli et al. presented a methodology to predict the human joint motion based on exoskeleton motion by combining personalized skeletal models of human subject with a kinematic model of the exoskeleton [4].
As some previous works mentioned, the human-exoskeleton system is a hybrid and complicated cognitive dynamics system. In particular, different subjects with exoskeletons may present different physical and dynamic parameters to be identified, which makes the whole dynamic motion model hard to establish and personalize [5]. When the control system of exoskeleton wants to perform precise control tasks, it needs to know the accurate gait status of the human-exoskeleton system [6]. However, classification and recognition of gait phase through primal modeling and identification of the dynamic human-exoskeleton may be cumbersome. In this instance, utilization of some suitable machine learning algorithms driven by immediately acquired gait data might be feasible for the exoskeleton system to recognize different gait motions.
Since it is quite difficult to measure multiple joint torques and ground reaction forces simultaneously for the human-exoskeleton system, employing gait data acquired by motion capture devices is more convenient for the control system to process and configure [7]. Due to different influence factors such as subjects’ biomechanics and physiological movement diversity, gait kinematic signals can be still characterized as nonlinear and nonstationary signals [8]. Thus, it would be suitable to use the empirical mode decomposition (EMD) method to analyze and extract the potential features concealed in gait data, as an alternative [9]. Some researchers have utilized such pathway to analyze gait signals [10]. For example, Cui et al. proposed a novel concept as step stability index (SSI) which evaluates subjects gait stability and fall risk by utilizing the decomposed the 1st4th intrinsic mode functions (IMFs) from its original signals [11]. Wen et al. found a new way based on improved EMD using Gaussian process to denoise gait accelerator data, and it gets better performance on denoising gait data [9]. Ren et al. discovered a good performance of neurodegenerative disease characterization by utilizing EMD to extract gait rhythm features from gait rhythm fluctuations which is based on signals of vertical ground reaction force [12]. Wang et al. achieved to characterize and differentiate the single waist accelerator signals through EMD between 5 gait patterns with high accuracy [13].
In this paper, we propose a framework of gait phase recognition for a human-exoskeleton system based on EMD. In such EMD-based recognition framework, eight IMF components are extracted from the original gait signals of 14 able-bodied subjects whose kinematics data are acquired by the optical-marked motion capture device. Six kinds of machine learning algorithms are applied for gait phase classification and recognition as follows: support vector machine (SVM), Kmeans, decision tree, logistic regression, Naive Bayes, and random forest methods. The extracted IMFs are used as the input for the training models with these six algorithms. The gait recognition results based on the 5th8th IMFs with some of these algorithms generally show promising performance. In the meanwhile, nonlinear fitting is used to examine the correlation between the decomposed IMFs and the original gait signals, and such fitting results could help to improve quantitatively evident observations on the correlations between IMFs and original gait data addressed in [11]. The difference and similarity of joint synergy between wearing with or without exoskeleton is analyzed. To our best knowledge, there is little work specifically focusing on human-exoskeleton gait phase recognition based on the EMD method in such manner.
2. Materials and Methods
In this section, experiments are divided into two parts. In the 1st part, kinematics data with and without exoskeletons during locomotion for the 14 subjects are measured by the optical motion capture system. In the 2nd part, another 3 subjects’ joint angles are captured by the inertial measurement unit (IMU) system. For the 1st part and 2nd part of experiments, IMFs are extracted from the measured original kinematics data by ensemble empirical mode decomposition (EEMD) algorithm, and the method of evaluation of the correlation between them is proposed. For the 1st part experiments, joint synergy is extracted from joint angle data.
2.1. Experiment Setup
In the 1st part of experiments, 14 able-bodied subjects (all males, 22.88 1.32 years old, 173.65 5.22 cm height, and 54.59 5.21 kg weight) took part in optical-object motion capture experiments to capture their kinematics data with and without exoskeletons during locomotion. In the 2nd part of experiments, another 2 able-bodied subjects and 1 paraplegia subject (all males, 25.66 2.44 years old, 177.10 19.33 cm height, and 62.13 1.91 kg weight) also participated in the IMU-based motion acquisition experiments to measure their joint angles. All motion capture studies were performed with their consent. The experiments followed the institutional guidelines of the University of Electronic Science and Technology of China, and all the experiment operations were in accordance with the Declaration of Helsinki. They were all instructed to utilize the lower limb exoskeletons to perform the normal walking tasks during part of the optical-object motion capture experiments. The lower limb exoskeleton system used in the experiments was developed by the University of Electronic Science and Technology of China. Such lower limb exoskeleton system is with four active degrees of freedom (flexion/extension) of motion in hip and knee joints, and its ankle joints are with two passive degrees of freedom of motion (dorsi/plantar flexion). The subjects are required to use crutches to maintain the balance during their locomotion for safety reasons.
In the 1st part of experiments, during optical-object motion capture experiments, 39 infrared markers were attached to each subject’s body for the VICON (Oxford Metrics Limited, UK) motion capture system according to the Plug-in Gait full body model [14]. The VICON system captured the locomotion of these 39 markers by 8 cameras with a frame rate of 100 Hz. The locomotion on elbow, shoulder, hip, knee, and ankle joints was acquired and converted to the angles of these joints through the algorithm provided by VICON. Each subject was required to perform 12 experiment sessions with different conditions, and a group of 30 joint angle signals (i.e., hip joint, knee joint, ankle joint, shoulder joint, and elbow joint at X, Y, and Z axes on both left side and right side, respectively) were captured correspondingly for each experiment session.
In the 2nd part of experiments, 7 IMU sensors (Perception Neuron, Noitom Limited) were bonded to the subjects lower body. The Perception Neuron system captured the locomotion of these 7 sensors with a frame rate of 120 Hz. Raw data on the angles of hip, knee, and ankle joints were captured.
2.2. Experimental Protocol
In the 1st part of experiments, all the 14 subjects were required to perform walking tasks with wearing or without wearing exoskeleton on 6 different floors with different frictions and materials, which include normal floor, wood floor, ceramic floor, carpet, plastic lawn, and floor with cobble. The first session of the test for each subject was to let him walk on 6 different floor conditions wearing an exoskeleton, and the second session of the test was to let him walk on 6 different floor conditions without wearing an exoskeleton. These two sessions are independent and separate. Each subject was told to walk about 5 m for several times for both test sessions. For each experiment in different floor conditions with or without wearing exoskeleton, we clipped some of experiment sessions when some of makers are found to be lost on camera during the motion capture. Figure 1 shows the subjects walking on the floor with different materials.
2.3. EEMD Computation and IMF Extraction
In each experiment session, we need a group of 18 joint angle signals (hip joint, knee joint, and ankle joint at X, Y, and Z axes on both left side and right side, respectively) from 30 joint angle signals mentioned above to perform gait recognition. According to the EEMD design principle [10], the 18 original gait time signals for each experiment session are to be decomposed into the following intrinsic mode function (IMF) components, respectively:where denotes the th IMF component (i.e., termed as ) decomposed from the gait sequence and is the residue of . The EEMD algorithm for gait signals can be described by the following algorithm (Algorithm 1).
| ||||||||||||||||||||||||||||||||||||||||
Figure 2(a) shows one of the 18 joint angle signals generated by one experiment session and its decomposed 8 IMF signals. Figure 2(b) shows the FFT of Figure 2(a); this shows that with the increase of IMF order number, the frequency of components the IMF contains decreases.
(a)
(b)
2.4. Gait Recognition
For the purpose of investigating how the various floor materials would affect the gait on subjects and how the different machine learning classifications and the traits of IMF would affect gait recognition when it comes to processing with VICON data in the part one experiments, we try to make gait recognition based on the 14 subjects’ motion capture data, by applying classification/recognition approaches with the extracted IMF components. Before making recognition, first an algorithm is developed to tag gait phase status for every frame of dataset according to the signals of toe and heel marker on Z axis captured by VICON.
The tagged phase status contains four tags as left and right foot both on land, left foot on land and right foot off land, left foot off land and right foot on land, and the uncertain mode, where gait phase status information was used to generate dataset as label set. Then, we remove those frames labeled as uncertain mode in set and their corresponding frame in the VICON data. We make gait recognition on 3 perspectives: gait recognition modes A, B, and C.
2.4.1. Gait Recognition Mode A
In this mode, we randomly divided the dataset which we acquired from each experiment session into two parts as training set and testing set with 2 : 1 ratio for the whole gait dataset of each experiment session.
The original training dataset and labeled dataset were composed of 8 IMFs as dataset. And we let the dataset signal train different machine learning models (SVM, Kmeans, decision tree, logistic regression, Naive Bayes, and random forest) and use the corresponding testing set to count the recognition accuracy.
2.4.2. Gait Recognition Mode B
In this mode, for each subject, we set the dataset on the Normal Floor as the training set and let the others of the same subject as the testing set. Because one subject’s experiment with the floor with cobble was abandoned, we got 69 accuracies for the original dataset gait recognition. Then, the 8 IMFs as dataset were generated by decomposing the original dataset, and we let the dataset signal to evaluate different machine learning algorithms.
2.4.3. Gait Recognition Mode C
In this mode, for each floor material, we set the dataset of the same subject as the training set and let the others on the same floor material as the testing set. Because one subject’s experiment with the floor with cobble was abandoned, we got 77 accuracies for the original dataset gait recognition. Then, we decompose the original data into 8 IMFs as dataset, and we let the dataset signal go through the same procedure as its original signal dataset had been processed.
2.5. Evaluation of Correlation between IMF Components and Original Gait Trajectory
For the kinematics data of the subjects measured by VICON, 12 experiment sessions in the 1st part of experiments are performed on each subject to capture and thus produce a group of 18 original gait signals (i.e., hip, knee, and ankle angles for X, Y, and Z axes on both left side and right side, respectively) for each experiment session. For the joint data acquired by IMU in part two experiments, 3 experiment sessions are performed on all 3 subjects, and thus we generate a group of 3 original gait signals (i.e., left hip, left knee, and left ankle). For all those original signals mentioned above, we divide them with dividing parameter into and . It means that is the former part of and is the latter part of . And after the IMFs (, ) are extracted from these original gait signals , and are also generated, respectively, for two data sequences and . In order to investigate the correlation between the original gait signals and IMFs by cross validation in a nonlinear fitting manner, first, we construct the matrix for each as follows:
Next, we produce a number of new signal matrices (, ) as follows:
Thus, we describe the variance between and by index (variance accounted for) counted through the following equation:
As is closer to 1, it indicates that they may possess more correlation evidence.
2.6. Joint Synergy Extraction
In addition, it is also important to further compare the joint synergy for subjects between them with wearing exoskeleton and without wearing exoskeleton to walk. We adopt principal component analysis (PCA) to extract the principal components of the five joints’ angle data (i.e., ankle, knee, hip, shoulder, and elbow joints’ angle) in the part one experiments. Some researchers have found that utilizing PCA to analyze the movement coordination patterns for human-exoskeleton might be useful [15, 16]. Firstly, we generate the following joint motion matrix acquired for each individual.where () denotes the data about the degree of ankle, knee, hip, shoulder, and elbow joints which contain the information about joint synergy. The method of PCA creates a new set of variables called principal components. Each principal component is a linear projection of the original variables. We can generate PCA through the following equations:where denotes the covariance of the matrix we constructed, denotes the direction that the original variables project on and also is the th eigenvector for matrix , and is the th principal component corresponding to .
3. Results and Discussion
In this section, the results of gait recognition modes A, B, and C for the human-exoskeleton system based on EEMD with the six machine learning methods are presented. The influence of floor materials on gait recognition has also been analyzed. Additionally, joint synergy for the subjects who walk with and without exoskeletons is also drawn.
3.1. Gait Recognition Mode A Results
Figure 3 shows the results of gait recognition mode A for the decomposing signals of the original signals based on all the six machine learning methods (SVM, Kmeans, decision tree, logistic regression, Naive Bayes, and random forest) for the 14 subjects who walked on flat floors with different friction situations with and without exoskeletons. Tables 1 and 2 show the detailed information of Figure 3. In this mode, we could figure out the performance for the six learning algorithms recognizing the gait status on each experiment session.
(a)
(b)
According to these two subfigures in Figure 3, recognition performance with can be generally better than that with . From Tables 1 and 2, conclusion can be made that when are used for the six algorithms, the recognition performance is not satisfactory since the accuracy generally cannot reach . The performance of is generally good for recognition. These comparative results might indicate that may be considered to be involved in the recognition algorithms. Besides, in these results, especially when SVM, Kmeans, logistic regression, decision tree, and random forest approaches are applied for , promising recognition accuracy can be generated. Otherwise, in Tables 1 and 2, we can see that when we know the gait mode for someone in certain floor material, the gait recognition can reach through IMF6, which means that IMF contains information for gait recognition.
Furthermore, Figure 4 shows the frequency that the maximum accuracies may occur in each order of IMF in all experiments, which stands for the probability that each IMF contributes to the recognition accuracy for the six algorithms. We generally can see that and have a higher probability to enhance the recognition accuracy for subjects who walk with exoskeletons, and can have a higher probability for subjects who walk without exoskeletons. As we can see, the signal in that can get the highest probability changes between wearing exoskeleton and without wearing exoskeleton. The reason behind this phenomenon may be that the gait locomotion mode changes between wearing exoskeleton and without wearing exoskeleton.
(a)
(b)
In conclusion, we can see that when it comes to recognizing gait with VICON data for each experiment session, all six algorithms are useful for recognition, and might be useful for recognition when wearing exoskeleton, while might be helpful for recognition when walking without wearing exoskeleton.
3.2. Gait Recognition Mode B Results
Figure 5 shows the results of gait recognition mode B with the original signal’s decomposing signals for the same subject and various floor materials. Tables 3 and 4 show the detailed information of Figure 5. In this mode, we could evaluate the performance of the learning algorithm when we acquire one subject’s gait data in one floor material to establish the recognition model and try to recognize the same subject’s gait on other floor materials based on the model.
(a)
(b)
Kmeans could get good performance when wearing with or without exoskeleton as their highest recognition average accuracy all exceed when with or without exoskeleton in both Figure 5(a) and 5(b).
When subjects wear exoskeleton, we can see that for different gait recognition algorithms, generally the higher the order number is and the higher the accuracy is in Figure 5(a). Thus, the accuracy of is the highest () with Kmeans according to Table 3. Kmeans, logistic regression, and random forest have a good performance with in Table 3. Figure 5(b) shows that when subjects walk without exoskeleton, generally have relatively better performance among , and is the highest () with Kmeans.
Additionally, Figure 6 shows the probability for that each IMF may be the highest accuracy among for the six algorithms. From Figure 6, we could generally conclude that has the highest probability to get the maximum accuracy among when wearing exoskeleton and is most likely to get the maximum accuracy among when walking without exoskeleton. Signal in IMF1 IMF8 that can get the highest probability changes between wearing exoskeleton and without wearing exoskeleton. As with mode A, signal in that can get the highest probability in mode B changes between wearing an exoskeleton and without wearing exoskeleton. This shows again that gait locomotion mode may change between wearing exoskeleton and without wearing exoskeleton.
(a)
(b)
In general, the result of this section tells us that when we tend to utilize someone’s gait data on the normal floor as the model to recognize the same subject’s gait in other materials, Kmeans may be a more helpful algorithm to analyze gait recognition among the six algorithms. Also, may be more helpful for analyzing gait recognition when subjects walk with exoskeleton on flat floor, and meanwhile may be more useful to recognize gait among when subjects walk without exoskeleton on flat floor.
3.3. Gait Recognition Mode C Results
The results of gait recognition mode C that the decomposing data of the original data as for the same floor material and various subjects can be analyzed from Figure 7. Tables 5 and 6 show the elaborate information for Figure 7. In this mode, we can compare the performance of different machine learning algorithms when we establish the recognition model by one subject experiment session and recognize gait with the same floor material in different subjects.
(a)
(b)
When it comes to the performance of while wearing an exoskeleton in Figure 7(a) and Table 5, Kmeans is also the best algorithm compared to others and contains the highest accuracy among with Kmeans (). In Figure 7(b) and Table 6, we can see that Kmeans performs best and gets the highest accuracy among with Kmeans ().
Furthermore, Figure 8 shows the probability for that each IMF may be the highest accuracy among for the six algorithms. In Figure 8(a), we could say that the most likely to get the maximum accuracy among when wearing exoskeleton is and especially for Kmeans, Decision tree, logistic regression, and random forest. From Figure 8(b), we can see that , especially can get the maximum accuracy among when walking without exoskeleton.
(a)
(b)
In sum, from the result of mode C, we could generalize that when we try to recognize various subjects' gait based on another subject with the same floor materials, Kmeans may be better to perform gait recognition among the six algorithms on mode C. may be more helpful for analyzing gait recognition when subjects walk with exoskeleton on flat floor, and meanwhile may be more useful to recognize gait among when subjects walk without exoskeleton on flat floor. This is the same as mode B.
3.4. Correlation Analysis between IMF Components and Original Gait
As addressed in previous gait recognition results, we see that there might be potential evident correlation between the latter IMFs and the original signals. Now the aforementioned nonlinear fitting approach is adopted to investigate the trajectory correlation between the latter IMFs and the original signals. Table 7 shows the VAF results for finding correlation between the IMF combination and the original gait signals acquired from the VICON system on the 14 subjects in part one experiments and IMU sensors on the 3 subjects in part two experiments. We can see that of some combinations can be used to identify the original gait signals. It reveals that the combination of IMFs may be utilized to identify the original gait trajectory.
3.5. Joint Synergy Results
In addition to gait motion recognition results, in order to investigate the possible similarity of joint synergy, we performed PCA on the five joints: elbow, shoulder, hip, knee, and ankle joints, to extract the principal components which can represent joint synergy indexes. For these five joints on X axis, from Figure 9, we could see the comparison on the joint synergy for the subjects who walk with and without exoskeleton.
(a)
(b)
(c)
(d)
(e)
(f)
In Figures 9(a)–9(c), the first average principal components without exoskeletons on X, Y, and Z axes of left side joints are, respectively, , , and , and the first average principal components with exoskeletons on X, Y, and Z axes of left side joints are, respectively, , , and . In Figures 9(d)–9(f), the first average principal components without exoskeletons on X, Y, and Z axes of right side joints are, respectively, , , and , and the first average principal components with exoskeletons on X, Y, and Z axes of right side joints are, respectively, , , and . The second principal components also show different values for the two groups on the left side and right side joints. It can be seen that when the subjects wear exoskeletons to walk, such PCA-based synergy extraction results show that the principal components can be altered.
4. Conclusions
In this paper, EEMD-based gait recognition modes A, B, and C are proposed for the human-exoskeleton system. SVM (support vector machine), Kmeans, decision tree, logistic regression, Naive Bayes, and random forest methods are used as training algorithms for such framework, and the performance of these six algorithms is discussed. Thus, we estimate how various algorithms, IMF order number, different floor materials, and various subjects would affect the result of recognition. And the results show that when it comes to gait recognition among various subjects and various floor materials, Kmeans might be better on performance whenever with or without wearing exoskeleton. For the contribution of gait recognition with among various subjects and various floor materials, might be helpful when wearing exoskeleton, and might be useful when walking without exoskeleton. And floor materials have little influence on gait recognition.
The correlation of original gait data and their decomposing signal IMFs through EEMD is investigated, which reveals that the combination of IMFs may be utilized to identify the original gait trajectory.
At last, joint synergy of five joints for the subjects who walked with and without exoskeletons is also drawn, which showed that the joint synergy might change between with and without wearing exoskeleton.
Data Availability
The VICON and IMU data used to support the findings of this study were supplied by Jing Qiu under license and so cannot be made freely available. Requests for access to these data should be made to Jing Qiu, [email protected].
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (no. U19A2082), the Sichuan Major Scientific and Technological Special Project (no. 2018GZDZX0037), and the Fundamental Research Funds for the Central Universities (no. ZYGX2019Z010).