Abstract
The main purpose of developing intelligent lower limb prostheses is to improve the quality of life of the disabled and promote the development of medical care. At the same time, the research of robotic intelligent lower limb prosthesis is a research direction that has been widely concerned in the field of robotics and biomedical engineering technology in recent years. This paper aims to research and discuss the 5G virtual reality system based on the gait evaluation of prosthetic wearers and its application in the debugging of intelligent prostheses. First, this paper analyzes the combination of the concept of lower extremity rehabilitation and virtual reality. Virtual reality technology is a means of exercise rehabilitation for patients with lower extremity hemiplegia, which plays an auxiliary role in the rehabilitation process. Then, the recognition and pre-method of human lower limb motion state are described. In the field of intelligent prosthetic state recognition, in addition to neural network recognition methods, there are hidden Markov (HMM) recognition methods, auxiliary vector machines (SVM), and so on. Finally, the lower limb state phase detection experiment and the prosthetic control experiment based on state phase detection are studied. The results of this experimental study showed that, for the same experimenter, changes in pace and gait did not affect the percentage of standing and swing phases. That is, the standing phase accounts for about 60% of the entire gait cycle, while the swing phase accounts for 40% of the entire gait cycle. This rule has been verified in other experiments.
1. Introduction
In the context of today's era, with the rapid development of artificial intelligence and robotics, the artificial limb industry will also grow rapidly with the help of technological progress and gradually apply artificial intelligence and robotics to the field of intelligent prosthetics. The needs of the disabled will continue to increase, and the pursuit of intelligence, adaptability, and esthetics of prosthetics will continue to increase, so the research on intelligent prosthetics will continue to deepen. With the emergence of various prosthetic products, the intelligent prosthetic control method is an important factor affecting the development of prosthetic technology, and is also an important factor to ensure that the wearer walks more naturally and safely.
An intelligent prosthesis is an application product that combines cognitive imaging and control technology. Design and development also includes a concentration of several related technologies, such as machine design and manufacturing technology, new component technology, and recycling technology. Compared to previous prosthetics, smart prosthetics are significantly optimized in terms of safety, functionality, and comfort, enabling prosthetic movements to reach or approach healthy limbs. Amputee patients make up for limb defects by installing prosthetic limbs, restore original limb function, relieve physical and psychological pressure, regain confidence, and return to life, study, and work.
The innovation of this paper is as follows: (1) The combination of the concept of lower extremity rehabilitation and virtual reality is analyzed. Virtual reality technology is a means of exercise rehabilitation for patients with lower extremity hemiplegia, and plays an auxiliary role in the rehabilitation process. (2) Then, the recognition and pre-methods of the motion state of the lower limbs of the human body are described. At present, neural networks are widely used in pattern recognition and fuzzy statistical mode, and neural networks have self-organization ability, good robustness, good permeability, and memory reasoning ability. (3) Finally, the lower limb state phase detection experiment and the prosthetic control experiment based on state phase detection are carried out.
2. Related Work
According to the research progress at home and abroad, different scholars have also conducted cooperative research on gait evaluation of prosthetic wearers and debugging of intelligent prostheses. Pine et al. investigated initial and current concerns of prosthetic eye wearers about mucous secretions, visual perception, and appearance, and the reasons for their concerns, and designed a retrospective cross-sectional study of patients in private practice [1]. Zhao et al. redesigned the prosthetic fall protection and management system from the perspective of man-machine integration, including fall warning and fall protection control. First, after analyzing the comprehensive information of man and machine, a fall early warning system was designed, which could reasonably pre-identify and prevent the wearer's abnormal fall gait [2]. The research of flexible intelligent hand prosthesis has important practical significance for the disabled. For the grip force control of flexible hand prosthesis, Qin et al. proposed a fingertip grip force control method based on linear tension feedback for reference to the anatomical structure design of the human hand [3]. Beibei proposed and designed a software and hardware platform for gait simulation and system evaluation of intelligent lower limb prostheses, so that the wear symmetry effect of intelligent knee joint prosthesis can be quantitatively analyzed by machine testing instead of human wearing testing [4]. Su et al. proposed a new method for training an intent recognition system that provides natural transitions between horizontal walking, stair ascent/descent, and slope ascent/descent. The method he proposed can predict the motor intentions of unilateral amputees and able-bodied people, and help to adaptively calibrate the control strategy for driving powered intelligent prostheses in advance [5]. Sk A conducted a mechanical analysis of the designed biomechanical integrated tumor prosthesis in a simulation environment, made a prototype of the prosthesis, established a wireless communication and control system, and tested the performance of the system on an experimental device. The system was able to detect a limb length difference (LLD) of 1 mm or more between the healthy limb and the prosthetic limb and successfully performed the prosthetic extension procedure to overcome the greatest possible soft tissue resistance [6]. However, these scholars did not research and discuss the 5G virtual reality system based on the prosthetic wearer's gait evaluation and its application in the debugging of intelligent prostheses, but only discussed its significance unilaterally.
3. 5G Virtual Reality System for Prosthetic Wearer Gait Evaluation and Application Method in Intelligent Prosthetic Debugging
3.1. The Combination of Lower Limb Rehabilitation Concept and Virtual Reality
Virtual reality refers to the use of computer technology to establish a real computer or virtual environment. And using visual, auditory, touch, and other functions, users can recognize and manipulate all virtual objects in the virtual environment, and then interact with the virtual environment to form an immersive experience. Virtual reality is not the real world, it is just an interactive environment. Humans can enter and interact with the environment through various media of computers. The application of virtual reality technology to reconstruct stroke hemiplegia must be considered from three aspects: first, the technical advantages of virtual reality itself. Second, the basic conditions in the process of physical rehabilitation; the third aspect is the rehabilitation of cerebral palsy. In these three aspects, combined with the existing rehabilitation methods and virtual reality technology, this paper proposes a more reasonable and effective learning method. To affect rehabilitation from exercise, some specific work related to retraining must be done. In the general nervous system, responses to learning outcomes are primarily determined by visual or physical sensations. At the same time, in order to be able to receive repetitive motor training, the patient must be sufficiently motivated to integrate the entire rehabilitation process [7]. Thus, successful patient recovery requires repetition, feedback, and motivation. Figure 1 shows the virtual reality operating system. Virtual reality essentially provides people with visual, auditory, and tactile sensations through a computer, and at the same time they can feel the virtual world in a real and natural way. It greatly facilitates people's use, thereby reducing people's work burden and improving work efficiency.
Virtual reality technology proposes three key elements. First, the virtual reality system is a counter system. In the absence of a doctor to provide medical or rehabilitation services, this ensures that there is sufficient time for the patient to retrain in medical care, regardless of the intensity of the doctor's work. Second, the virtual reality system is an interactive system. The system provides the patient with visual, auditory, and tactile feedback based on the patient's movements, establishes a rehabilitation system of the virtual reality system to obtain data on the user's rehabilitation results. In addition, the virtual reality system can also allow patients to stand up and watch a vast imaginary space, combined with virtual images, to create a happy and positive attitude for patients during the rehabilitation process, which greatly helps to improve the patient's enthusiasm for sports rehabilitation [8]. Figure 2 shows the rehabilitation evaluation system based on virtual reality.
Virtual reality technology is a means of rehabilitation for paraplegic patients. As a supplement, it plays a role in the patient's recovery. The following technical tools illustrate the important role of virtual reality in this field, combining the characteristics of virtual reality with existing rehabilitation treatments [9].(1)Neural organ development technology: during the rehabilitation process, the doctor conducts rehabilitation training on the patient through the robot active rehabilitation model. In this case, a patient treated by a rehabilitated lower extremity skeletal repair robot can communicate with objects in real time. It is designed in a virtual scene and receives sound and visual signals in the patient's current motor information to stimulate their susceptibility to sensory signals, thereby promoting coordination and control of lower extremity movements [10].(2)Biofeedback method: the motion information of the skeletal robot in the lower limb rehabilitation exercise is collected by the angle sensor and transmitted to the virtual model on the computer. The virtual reality system obtains the patient's viewpoint information in the rehabilitation exercise, controls the movement of the heroic characters in the rehabilitation scene in the square and the flying plane at the rehabilitation scene. Post-processing information is displayed on the computer through virtual images and virtual sounds in the virtual reality system, and feedback is provided to the patient. Patients can communicate directly with the virtual environment and learn about their own rehabilitation training [11].(3)Sports cognitive therapy: the lower extremity movement itself is a cyclic movement, but due to the disease, the patient does not have the opportunity to coordinate the movement of the left and right legs. Therefore, in order to restore the function of the patient's lower limbs as soon as possible, the therapist can select the appropriate virtual reality in advance according to the disease degree of the patient's lower limbs. The patient is immersed in the virtual scene selected by the doctor and receives rehabilitation treatment at the same time, so as to achieve the purpose of interaction between the patient and the objects in the virtual environment, increase his subjective initiative and relaxation, and carry out active rehabilitation [12].
3.2. Recognition and Prediction of Human Lower Limb Motion State
At present, neural networks are widely used in pattern recognition and fuzzy statistical patterns. Neural networks have self-organization ability, good robustness, good permeability, and memory reasoning ability. The neural network input is an eigenvector composed of eigenvalues, and the network output is the number of lower limb movements. After a certain sample preparation, motion patterns can be effectively identified [13]. Neural networks are a relatively mature recognition method. RBF, LVQ, and BP neural networks are widely used; fuzzy recognition circuits are developed from fuzzy theory. It divides pattern types into fuzzy sets, and uses the properties and computational methods of fuzzy sets to determine and classify lower extremity motion models [14]. After continuous research and development, fuzzy pattern recognition has achieved certain results in the fields of intelligent robots, signal classification, speech recognition, character recognition, and image processing. Statistical pattern recognition, which treats the classification of patterns as a set realized by a random vector, is also called decision-theoretic recognition. Its principle is based on the premise of the known statistical model of the research object, or the condition of the known discriminant function class, according to the known limited number of sample sets, under the constraint of certain criteria, the A-dimensional feature space is divided into B regions through the learning algorithm, and each region corresponds to each category. Commonly used statistical pattern recognition methods mainly include K-nearest neighbor method, discriminant function method, and nonlinear mapping method.
In the field of intelligent prosthetic state recognition, in addition to the above recognition methods, there are hidden Markov (HMM) recognition methods, auxiliary vector machines (SVM), and so on. Considering the possible limitations of a single recognition method, we will mainly consider two recognition methods and compare their recognition effects and related features in lower extremity motion pattern classification [15].
3.2.1. Extreme Learning Machine
Extreme learning machine is a new method emerging in recent years, which is developed from implicit single-layer neural networks. Different from the commonly used algorithms before, the ELM algorithm has obvious characteristics. Its hidden layer parameters are not artificially set definite values, but are randomly generated by themselves, so that the system has a linear nature after the model is established by the network. ELM has a simple structure and strong global search capability. Many scholars have applied it to the fields of biochemical process modeling, lithology recognition, and sound recognition, and achieved good recognition and classification results [16].
For single hidden layer feedforward neural networks (SLFNs) with hidden layer nodes, the output can be expressed aswhere is determined by the activation function and the type of hidden layer nodes. It is used to represent the functional relationship between the mth hidden layer node and the input a [17].
If the hidden layer nodes can be increased and the activation function is sigmoid, there are
If the hidden layer node is of RBF type and the activation function is Gaussian function, there are
At this time, and refer to the center and influence factor of the mth RBF node, and is a set of positive real numbers.
If the error after approximating P samples is 0, then
The formula given in equation (4) is expressed in the form of a matrix, that is,
Among them, K is the hidden layer output matrix, its mth column refers to the output vector of the mth hidden layer node related to the input , and the nth row refers to the hidden layer output vector related to the input .
During the computation, the weights of the outputs can be computed in a large cycle without adjusting the learning rate and the number of iteration steps. That is to say, the hidden layer nodes and SLFN do not need to be adjusted too much during the formation process, and the allocation has a certain randomness. So, it can be thought of as a linear system, and network training is equivalent to solving the least squares solution of a linear system of formulas.
is the generalized molar inverse of K. There are many ways to find the generalized inverse of a matrix such as orthogonalization, orthogonal projection, singular value decomposition, and iteration [18]. Orthogonal projection can be used to solve the generalized inverse matrix if and only if the generalized inverse matrix is a non-singular matrix [19].
According to the gradient descent method, when solving the minimum value of the objective function, the correction method of the vector set is as follows:
The calculation process of the elm algorithm can be roughly summarized. Supposing the given training set is , the hidden layer is , and the activation function is R(a).
3.2.2. Improved Elman Algorithm
The structure of Elman neural network is different from the input layer, hidden layer, and output layer of general neural networks. It also has a special structure called the receiving layer. The function of the receiving layer is to store and record the output value of the hidden layer unit at the previous moment, which can be regarded as a delay component. It is with the existence of the receiving layer that the network has the function of dynamic memory. Its structure is shown in Figure 3.
The feedback neural network adjusts the weights according to the error to form the feedback correction of the output. The most representative one is the BP neural network. If the inside of the network is regarded as an unknown structure, the Elman network is not much different from the ordinary feedback network structure in terms of input and output [20]. Its special feature is that an inheritance layer is added inside the network. It is a reprocessing of the information of the hidden layer. The output of the hidden layer is not sent to the output layer once, but is transitioned in the successor layer as the input of the next cycle of the hidden layer.
The biggest feature of Elman neural network is that there is an extra receiver layer in the network structure. The receiver layer is used to store the output of the hidden layer and act as a delay, and then the output of the receiver layer is used as the input part of the hidden layer. In fact, it is the autocorrelation of the hidden layers. It is precisely this functional structure of delayed self-association and internal feedback that makes the network more sensitive to historical data, and this sensitivity is more conducive to the processing of time-varying information. And, the network has outstanding dynamic modeling capabilities due to its unique structural features.
The mathematical model of Elman neural network is
Among them, is the connection weight between the hidden layer and the successor layer, and is the connection weight between the input layer and the hidden layer. is the connection weight between the hidden layer and the output layer, is the output of the successor layer, is the output of the hidden layer, is the output of the output layer, and is the self-connection feedback gain factor. is the transfer function of the output neuron, generally a linear function is used. uses the sigmoid function, as follows:
Constraining the indicator function with the error sum of squares function is shown in equation (11), where is the expected value of the output value.
The objective function of the Elman neural network is as follows:where refers to the true output value of the system at step h.
Based on the gradient descent method, the partial derivative of the objective function with respect to each weight is compared, and its minimum value is obtained. Then, the corresponding Elman network learning algorithm is given as follows:
Among them, , , and are the learning steps of , , and , respectively.
The entire operation process of the Elman neural network algorithm is shown in Figure 4.
4. 5G Virtual Reality System for Prosthetic Wearer Gait Evaluation and Application in Intelligent Prosthesis Debugging and Analysis of Experimental Results
4.1. Experimental Study on Phase Detection of Lower Limb States
During the experiment, through the angle of the foot pressure acquisition module and the spatial position sensor worn on the human lower limb, the software acquisition program will collect the lower limb state data, which will be integrated and displayed in real time through serial communication. At the same time, the data obtained from the experiments are fully preserved for subsequent analysis of the experimental data.
4.1.1. Experimental Design
Each experiment was achieved in the following experiments: (1) At normal motion speed, measured in step values, the examiner passed through larger, normal, and smaller widths, respectively. (2) The experiments were performed at the normal step size level, changing the speed at fast, normal, and slow speeds, respectively. (3) The test is carried out at normal speed and normal speed.
4.1.2. Experimental Results
Two sets of data were selected for analysis. Experimenter A, the level passes through the 1.8 s step cycle, and judges the signal and step phase of each sensor. The results show that Figure 5(a) shows the relationship between the walking percentage and the experience stage in the stepping period of 1.8 s, according to the data of the shear sensor and the position angle of the sole sensor. It can be pointed out that the upright phase accounts for about 60% of the motion cycle, and the rocking phase accounts for about 40% of the motion cycle.
(a)
(b)
Figure 5(b) shows a graph of Experimenter B's gait percentage with an average stride length of 75 cm. We have also seen a similar pattern: here, we can see that the upright phase accounts for about 60% of the motion cycle, while the swinging phase accounts for about 40% of the motion cycle, which is in line with the theoretical conclusion.
4.1.3. Adaptability and Real-Time Analysis
Adaptability and real-time performance are important features of the evaluation system discussed in this paper. In order to better evaluate the state of the phase system, it is necessary to conduct a more detailed study of the experimental data, study and discuss the changes of the sensor signals in the phase judgment system under different walking forms, and provide the response time characteristics of the system.
(1) Analysis of Results under Gait Cycle Change (Pace Change). Choosing an experimenter to analyze the data at an asynchronous pace. The experimenter walked with gait cycles of 1.25 s, 1.6 s, and 1.8 s under horizontal road conditions, and recorded the signals of the two plantar force sensors and angle sensors during one gait cycle, as shown in Figure 6.
Due to the existence of sampling frequency and sensor response time, this paper gives the phase judgment time deviation of the state identification system under different step speed changes with sampling frequency of 100 Hz. As shown in Table 1, it can be seen that for the CP and Se phases, the time error of the judgment system is relatively small, and the time error of the DC phase is relatively large, and the average time error is 15 ms.
(2) Analysis of the Results under the Change of Stride Length. An experimenter was chosen to dissect the data at asynchronous vibration amplitudes, providing signal values for each sensor at asynchronous vibration amplitudes (65 cm, 75 cm, and 85 cm), as shown in Figure 7. As with the data studies at asynchronous speeds, we can observe that the spread of the individual sensors across the time axis tends to be the same when the stride changes. This is consistent with the conclusion that the system can adapt to different phase amplitude changes.
The phase judgment time error of the state judgment system under the condition of asynchronous amplitude change under the sampling frequency of 100 Hz is given, as shown in Table 2. Similarly, the time errors of CP phase and Se phase are small, and the time error of CD phase is large, with an average error of 19 ms.
(3) Analysis of the Results of Different Testers. Similarly, the experimental data of different testers are studied, and the sensor signal values of each tester in one motion cycle are provided, as shown in Figure 8. When different testers walk, the signal output of each sensor tends to be consistent, indicating that the system can adapt to different testers.
Table 3 shows the time deviations of different testers judging each stage of the state system at a sampling frequency of 100 Hz. The same pattern was observed in different testers. The experimental reaction times of the CP and Se phases were relatively short, and the reaction times of the CD phase were relatively long, with an average time error of 21 ms.
Finally, the results of the average judgment time error of each phase after multiple experiments are given, as shown in Table 4.
After analyzing the data of all the experimenters, it can be concluded that the variation of stride length and gait speed did not affect the percentage of stance phase and swing phase for the same experimenter. That is, the upright phase is about 60% of the total motion cycle, and the rocking phase is about 40% of the total motion cycle. This rule has been verified in other experiments. Therefore, the exercise phase test scheme described in this paper can basically be applied to different experimenters, different rhythm ranges, and different gait trajectories.
4.2. Prosthetic Control Experiment Based on State Phase Detection
The essential purpose of the artificial knee joint control method is to find a safe and stable control method, reduce the discomfort of the artificial knee joint user, and produce a smooth effect on the walking process of the prosthetic limb patient. This work combines the phase detection method with the host computer to send the phase information to the lower limb health control system. After advanced computer software processing, the pulse volume of the artificial knee joint is controlled. The prosthesis control link adopts a closed-loop structure, which can reduce errors and obtain better prosthesis control effects. Figure 9 is an experimental block diagram of the prosthetic control system.
When the exercise cycle starts, the tester starts to work using the plantar pressure sensor and position sensor mode, collecting and arranging the plantar pressure signal and the knee joint angle signal. On the basis of judging the state phase, obtain the movement percentage between the current phase of the healthy side and the gait cycle. According to the mapping relationship between the healthy lateral position of the human body and the lateral position of the prosthetic knee joint, the current position of the lateral position of the prosthesis is mapped. Using the mathematical model of the knee joint motion, the current motion pulse of the prosthesis is calculated. The controller drives the knee joint prosthesis to move. When movement stops, one gait cycle ends and the next gait cycle moves.
According to the established experimental prosthesis control system platform and knee joint prosthesis control process, the lower limb state phase detection system is used on the right lower limb of the tester, and the experiment is verified according to the prosthetic limb control process. The purpose of this experiment is to verify the range of walking speeds that the lower limb phase detection system can accommodate in a prototype knee prosthetic model. That is, the tester can use the lower limb phase judgment system to walk at different walking speeds, which will cause the prosthetic limb to follow the walking speed range of the healthy limb side. Taking into account the safety and stability of the experiment, the original prosthetic model was fixed on the bracket during the entire experiment.
The experimental testers used the lower limb state phase detection system to walk horizontally multiple times at different walking speeds to check whether the prosthesis could move according to the planned curve at each speed. The walking speed is calculated by the program. The walking speed of the tester each time during the test is shown in Table 5.
When the gait period is 1 s and 2.2 s, both the planned curve and the actual curve of the prosthesis have obvious distortion, which is because the mathematical motion model adopted in this paper has a certain range of applications. When the gait cycle is between 1.2 s and 2 s, the model can calculate the knee joint curve of the prosthesis well, but the model cannot be applied to other gait cycles. Jitter and hysteresis will be investigated by improving the experimental system or adding smoothing in follow-up work.
5. Discussion
In this paper, in the control research of the prosthetic knee joint, accurately obtaining the state phase information of the healthy limb or the prosthetic side is the basis for the control. We need to extract useful information from the lower limb motion information to study the method of dividing and judging the phase. For amputees, by being equipped with intelligent prosthetic legs that can self-adaptively control, it will help them learn, live, and work like normal people. It will also help to comprehensively improve the quality of life of the disabled and the ability of community activities, and will also promote the vigorous development of rehabilitation medicine and engineering. However, it has been confirmed through research on the wearing of real prosthetic legs that the number of people who use lower limb prosthetic legs is less than 1/3. The factors that cause such a situation are mainly due to the simple function of the traditional prosthetic leg, but the low sensitivity and accuracy of operation, the high quality, the easy to cause body strain, and the high price. Therefore, the production of prosthetic limbs with excellent performance and high quality and low price for the disabled is still the main task of the development of the disability cause, and the basic research work related to this is also increasingly urgent.
6. Conclusions
Aiming at the problem that most of the existing intelligent prosthetic rehabilitation robot control methods are based on the motion state stage, a phase detection method for the state of the lower limbs and the motion law of the lower limbs of the human body is proposed - the gait phase segmentation method. Using the prototype of the existing artificial knee joint model in the laboratory, the hardware platform of the experimental artificial knee joint control system was built. Using the function library provided by the controller, the experimental control system software was written, and the effect of the lower limb state phase detection system in the artificial knee joint control was verified. The lower limb motion state judgment system is ultimately applied to the prosthetic control of amputees. There is a difference between the data of the disabled and the healthy human body. This difference is a focus of follow-up research work. Due to the limitations of the experimental conditions in this paper, reliable data of disabled persons cannot be obtained for analysis, and some research work is only done on healthy human data.
Data Availability
No data were used to support this study.
Conflicts of Interest
The authors declare that there are no conflicts of interest in this study.
Acknowledgments
This work was supported by Luzhou Science and Technology planning project (2021-SYF-44), supported by the Scientific Research and Innovation Team Construction Project of Luzhou Vocational and Technical College (2021YJTD07), and Special Project for Technological Innovation and Application Development in Yongchuan District (2021yc-cxfz30011).