Recent Developments in Artificial Intelligence for Consumer Healthcare Integrative AnalysisView this Special Issue
An LSTM-Based Prediction Method for Lower Limb Intention Perception by Integrative Analysis of Kinect Visual Signal
Recently, computer vision and deep learning technology has been applied in various gait rehabilitation researches. Considering the long short-term memory (LSTM) network has been proved an excellent performance in learn sequence feature representations, we proposed a lower limb joint trajectory prediction method based on LSTM for conducting active rehabilitation on a rehabilitation robotic system. Our approach based on synergy theory exploits that the follow-up lower limb joint trajectory, i.e. limb intention, could be generated by joint angles of the previous swing process of upper limb which were acquired from Kinect platform, an advanced computer vision platform for motion tracking. A customize Kinect-Treadmill data acquisition platform was built for this study. With this platform, data acquisition on ten healthy subjects is processed in four different walking speeds to acquire the joint angles calculated by Kinect visual signals of upper and lower limb swing. Then, the angles of hip and knee in one side which were presented as lower limb intentions are predicted by the fore angles of the elbow and shoulder on the opposite side via a trained LSTM model. The results indicate that the trained LSTM model has a better estimation of predicting the lower limb intentions, and the feasibility of Kinect visual signals has been validated as well.
Stroke is a disease caused by acute rupture of blood vessels or vascular occlusion [1, 2]. About 15 million people suffer from it every year globally . Hemiplegia is the major sequela of most stroke survivors which affects the quality of their daily life in the home, workplace, and community . It presents with the weakness of one entire side of the body. Due to limb weaknesses leading to an inability to properly performing, hemiplegia patients could lose a number of motor functions especially the walking function [5, 6]. Walking abnormality makes performing everyday activities in the home, workplace, and community more difficult [7, 8].
Recovery of the walking ability for hemiplegia patients is crucial in order to perform daily activities [9, 10]. Key components of gait recovery are high-intensity, skill-oriented, and task-specific [11, 12]. Due to physically exhaustion of therapists to repeat hundreds of complex gait cycles in a training session , an amount of rehabilitation gait training robots have been developed to provide robotic assistance . Robotic-assisted gait training refers to the rehabilitation therapists how to assist the patient in performing the gait cycle . Considering the limb weaknesses leading to difficulty in supporting the body weight in training, current rehabilitation could support body weight to allow the lower limbs to maintain a pattern during gait training such as Lokomat . These gait robot trainers passively move the patients on a treadmill. However, the control systems of most commercial robotic systems are passive in nature because the training subject is not considered in the system. By increasing active participation , the dependence of patients on robot assistance can be reduced by improving the effectiveness of rehabilitation training. Thus, we should make the robots include the ability that collects quantitative gait data to generate sensory stimulation synchronized to gait patterns.
To develop a noncontact signal prediction for an active rehabilitation robotic system, the synergy that is initiated by weight-bearing over the involved limb and supporting the human body was taken into consideration [18, 19]. Twitchell et al. proved that abnormal synergy is a motor impairment in patients with stroke . The main factor that limited the motor rehabilitation of patients with stroke is abnormal synergy . Studies have shown that interlimb and intralimb coordination of lower limbs in patients after stroke is diverse from that in normal subjects . In 2018, Zebin et al. proposed a prediction method via inertial sensors and LSTM methods to predict the angle trajectory of the impaired lower limb . Simultaneously, the security and privacy of medical data are also crucial. Sandeep et al. developed a biometric-based security framework for wearable health monitoring systems to extract ECG signal, and it proved that time-domain based biometric features plays an important role in security . Wu et al. proposed an adaptive computing-based random binary sequences generation method to provide a balance between processing time and security in wireless body sensor networks . Cai et al. quantified the concurrent accuracy and the test-retest reliability of a Kinect V2-based upper limb functional assessment system . Liao et al. proposed a motion intention recognition system based on the Kinect V2 sensor. It can successfully provide an adequate assistance with a lesser time delay compared with the system without Kalman filter .
Recently, the time series prediction model has been effectively applied to several studies . Long short-term memory (LSTM) networks widely used to have done a good job on this issue in fields including gait recognition owing to the ability of processing and predicting the time series with very long intervals [29, 30]. It works effectively to extract the gait feature .
As shown in Figure 1, in this paper, a lower limb joint trajectory generation framework was proposed to drive the lower limb robot using the trajectory of healthy upper limbs. This study aimed to utilize upper limb Kinect information during walking to estimate sagittal plane hip and knee kinematics trajectories. The trajectories will be used for driving a rehabilitation robotic system in follow-up studies.
2.1. Experimental Setup and Data Acquisition
To obtain human gait data, we have built and evaluate our model that used a “virtual skeleton” produced by the Kinect sensor and software. Kinect 2.0 provides a high-quality skeletal model to one user in front of the Kinect sensor, and Kinect SDK offers the tracking and detection of 25 different skeletal points, which could apply this skeletal data for feature extraction; the experimental setup is as shown in Figure 2.
Gait data were concurrently recorded by a Kinect sensor that provides approximately 30 skeleton frames per second . Each participant wore a fitting and light color suit on the treadmill. In a 10 participants’ database, they are generally divided into four walking velocities: 3.0, 3.5, 4.0, and 4.5 km/h.
The joint angle of the shoulder, ankle, hip, knee, arm of the right side, left knee, and hip in the sagittal plane were calculated based on the quaternion. For each joint of the Kinect virtual model, the x, y, and z coordinates are recorded. This study converts the joints into a vector for angle calculation. For each joint, the current position of the angle between a joint and a sagittal vector was recorded. Finally, we generate the following features: the angle in each of the frames, the difference in angle between consecutive frames, and these angular displacements providing basic gait characteristics.
2.2. Gait Joint Angle Design
The Kinect skeletal joints 3-D coordinated data reading is less susceptible to noise compared with their distance to the acquisition [32, 33]. Thus, for each limb, a shoulder joint angle was determined by considering the location of the shoulder and elbow in the Cartesian coordinate. The shoulder, elbow, hip, and knee position in Cartesian space are defined with four-vectors, the Kinect being at the origin of the 3-D space. The vector definition is formulated in equations (1)–(5). The angle of joints definition is formulated in equations (6)–(9):where , , , and are the 3-D vectors connecting the participants’ shoulder to the elbow, elbow to the wrist, hip to the knee, and knee to the ankle, respectively, that is also depicted in Figure 3.
2.3. Long-Short Term Memory Network for Angle Prediction
To solve the difficulties in training the RNN model caused by the “vanishing gradient” effect, the long-short term memory (LSTM) architecture has been proposed. Figure 4 illustrates a typical LSTM neuron. It contains one self-connected memory cell and three multiplicative units, i.e., the input gate , the forget gate , and the output gate .
The memory cell has a self-connected recurrent edge of weight 1, ensuring that the gradient can pass across many time steps without vanishing or exploding . The input gate and forget gate govern the information flow into and out of the cell . The output gate controls how much information from the cell is passed to the output . The activations of the memory cell and three gates are given as follows:where is the logistic sigmoid function and defined as , are the weight matrices connecting and , and denotes the corresponding bias vectors.
3.1. Experiment Implementation
Since stroke patients show a lower extremity weakness of walking [38, 39], we target in studying the spatial correlations of gait features by using neural networks. To get enough training gait data, 10 healthy participants (aged years, height cm, and weight kg) were recruited from our laboratory. They were free of any physical condition or limitation which prevented them from walking on the treadmill. They were required to walk for 150s per velocity.
For maintaining a stable recognition of the human body , the Kinect was placed at a height of 1 meter above the ground and the treadmill was set within 2.6 to 4 meters from the Kinect sensor.
During the experiment, there was a total of 10 (male/female:6/4) healthy participants enrolled. We prepared 40 gait feature data of upper and lower limbs from 10 subjects, while their skeletal data were captured by Kinect 2.0. Figure 5 illustrates a participant walking session and joints behavior during a gait cycle in different velocities.
Our experiments were implemented on the Tensorflow framework , a popular deep learning framework. The base learning rate was set to 0.0005, and the LSTM step size was set to 10 frames. The maximum number of iterations was set to 1000.
This study estimated one side’s gait data by using the other side’s data based on the synergy. Figure 6 shows the estimated result of left hip joint and knee joint trajectories through using the right shoulder and elbow by LSTM. To validate the feasibility of LSTM synergy, we used right side upper limb joints and lower limb joints to predict left side lower limb and are shown in Figure 7; it shows the estimated result of one side’s hip and knee trajectories through using the other side’s shoulder, elbow, hip, and knee by LSTM. As can be seen from the figure, the error between the estimated trajectory and the original trajectory of the left hip and knee is low.
Results show that LSTM is a good approach for person identification based on gait recognition with Kinect. We also tested the quality of the prediction of the angular velocity, and we applied the root-mean-squared error (RMSE) to evaluate the model after each run. Here, we compared RMSE between the estimated angle and original angle in four different velocities on prediction based on LSTM. The result is shown in Figures 8 and 9. Especially, RMSE was poor for hip and knee joint angles at 3.0 km/h than the other three velocities by using the joints of the upper limb (Table 1); however, it was relatively good by using the joints of the upper limb and lower limb (Table 2).
To estimate the hip and knee trajectory by using the upper limb joints trajectories, we applied the Kinect 2.0 to track the upper limb and lower limb sagittal plane movement in the walking period. The human body is in a continuous dynamic state during walking. In this study, the LSTM model was developed, and its performances were compared using RMSE. Because there is no need to go through a process of selecting features and having better stability, we chose it to estimate our trajectory. The LSTM model in this study showed improved results, and RMSE has been introduced above. It can see that LSTM has a better estimation on predicting the gait trajectory, which included human interlimb synergy. This model showed excellency in modeling that with the change over time such as walking to predict the data of current time from information in the previous step.
As the pace velocity increases, we can see that the accuracy of the prediction is getting higher. In the case of 3.0 km/h velocity, the gait prediction trajectory is relatively poor; however, in the case of 4.5 km/h velocity, LSTM presents the effect of prediction is quite amazing. This result indicates that when humans walk at 4.5 km/h velocity, the upper and lower limbs on the two sides are highly correlated.
In various velocities, the trajectory prediction effect of the knee joint is generally higher than that of the hip joint, except for the velocity in 3.5 km/h.
When using the joints of the upper limb and lower limb to estimate the hip and knee trajectory, we can get an obvious better estimation accuracy. From Table 2, we can see that the RMSE is basically maintained within 2, which is better than merely used the upper limb to predict hip and knee trajectory. Otherwise, in this case, the accuracy of the estimated hip trajectory is better than estimated knee trajectory, respectively; compared with the right hip, the left hip trajectory is great. In Figure 8, the trajectory only based on the upper limb trajectory still has a good estimation performance. It was concluded that LSTM has good exploitation in gait features.
This study has a limitation of not applying data of patients with stroke to the learning model for lower limb trajectory prediction. However, the study is to suggest the possibility of estimating the lower limb trajectory by using the upper limb trajectory and an artificial neural network model. In the next research, we can apply various data for the training model.
In this paper, an artificial neural network model was developed to estimate the lower limb joints trajectory of a complete gait cycle by using the joints of the opposite side. Accuracies of using the upper limb joints and the upper and lower limb joints to estimate another side lower limb joints were compared. As a result, the model showed RMSE values within 3.0. These trials demonstrate that this model can be used safely as a gait training intervention for those stroke patients. It suggests that the exoskeletal gait rehabilitation robot can apply this model to help patients try to walk like normal people.
The raw data required to reproduce these findings cannot be shared at this time as the data also forms part of an ongoing study.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
This work was supported in part by the National Natural Science Foundation of China (61973220), Shenzhen Science and Technology Innovation Council (JCYJ20180507182040213), and Guangdong Grant “Key Technologies for Treatment of Brain Disorders” (2018B030332001).
H. Thieme, J. Mehrholz, M. Pohl, and C. Dohle, “Mirror therapy for improving motor function after stroke,” 2018, Cochrane Database of Systematic Reviews.View at: Google Scholar
G. M. Rozanski, Lower Limb Muscle Activity Underlying Temporal Gait Asymmetry Post-Stroke, medRxiv, Cold Spring Harbor, NY, USA, 2019.
R. Chamudot, S. Parush, A. Rigbi, R. Horovitz, and V. Gross-Tsur, “Effectiveness of modified constraint-induced movement therapy compared with bimanual therapy home programs for infants with hemiplegia: a randomized controlled trial,” American Journal of Occupational Therapy, vol. 72, no. 6, pp. 7206205010p1–7206205010p9, 2018.View at: Publisher Site | Google Scholar
A. Yacoby, G. Zeilig, H. Weingarden, R. Weiss, and D. Rand, “Feasibility of, adherence to, and satisfaction with video game versus traditional self-training of the upper extremity in people with chronic stroke: a pilot randomized controlled trial,” American Journal of Occupational Therapy, vol. 73, no. 1, pp. 7301205080p1–7301205080p14, 2019.View at: Publisher Site | Google Scholar
J. Bae, “Exosuit-induced improvements in walking after stroke: comprehensive analysis on gait energetics and biomechanics,” in Proceedings of the 2017 International Symposium on Wearable Robotics and Rehabilitation (WeRob), IEEE, Houston, TX, USA, November 2017.View at: Publisher Site | Google Scholar
R. Riener, “Technology of the robotic gait orthosis Lokomat,” in Neurorehabilitation Technology, pp. 395–407, Springer, Berlin, Germany, 2016.View at: Google Scholar
T. Zebin, M. Sperrin, N. Peek, and A. J. Casson, “Human activity recognition from inertial sensor time-series using batch normalized deep LSTM recurrent networks,” in Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA, July 2018.View at: Publisher Site | Google Scholar
Y.-T. Liao, “Development and evaluation of a kinect-based motion recognition system based on kalman filter for upper-limb assistive device,” in Proceeding of the 2019 58th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE), IEEE, Hiroshima, Japan, September 2019.View at: Publisher Site | Google Scholar
G. Chalvatzaki, P. Koutras, J. Hadfield, X. S. Papageorgiou, C. S. Tzafestas, and P. Maragos, “LSTM-based network for human gait stability prediction in an intelligent robotic rollator,” in Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), IEEE, Montreal, Canada, May 2019.View at: Publisher Site | Google Scholar
L. Jiasen, C. Xiong, and R. Socher, “Sentinel gate for modulating auxiliary information in a long short-term memory (lstm) neural network,” 2020, Google Patents.View at: Google Scholar