Abstract

Recently, as a highly infectious disease of novel coronavirus (COVID-19) has swept the globe, more and more patients need to be isolated in the rooms of the hospitals, so how to deliver the meals or drugs to these infectious patients is the urgent work. It is a reliable and effective method to transport medical supplies or meals to patients using robots, but how to teach the robot to the destination and to enter the door like a human is an exciting task. In this paper, a novel human-like control framework for the mobile medical service robot is considered, where a Kinect sensor is used to manage human activity recognition to generate a designed teaching trajectory. Meanwhile, the learning technique of dynamic movement primitives (DMP) with the Gaussian mixture model (GMM) is applied to transfer the skill from humans to robots. A neural-based model predictive tracking controller is implemented to follow the teaching trajectory. Finally, some demonstrations are carried out in a hospital room to illustrate the superiority and effectiveness of the developed framework.

1. Introduction

In the past few months, a novel coronavirus has resulted in an ongoing outbreak of viral pneumonia in the world [1, 2]. More than 98000 people are known to be infected, and over 3400 deaths have been reported [3]. The symptoms of these patients include high temperature, cough, shortness of breath, and headache, and it is a highly infectious disease [4]. More and more patients need to be isolated in independent rooms of the hospitals. How to transport meals and medicines to patients and reduce the infection for medical staff simultaneously is a hot issue. It is a safe and useful method to transport medical supplies or meals to patients using robots. Therefore, this paper focuses on how to control the robot to the destination and to enter the door like a human at the same time.

Human-robot skill transfer is currently one of the significant topics in human-assisted systems. Under different external environmental conditions, especially in the hospital room, the main challenge for assisted medical rescue robots is how to transport the medical supplies [5] safely. The robot system acquires the learning ability through the cognitive knowledge transmitted by humans [6]. At the same time, by learning the approach of human-in-the-loop, the performance of the robot is improved, making the system more intelligent [7, 8].

Human-robot interaction (HRI) is dedicated to the development of more intelligent and anthropomorphic robots, which is a subregion of human-computer interaction that researches the interaction between man and robot [9, 10]. In some hazardous areas, to minimize staff participation, robots are required to perform operations. HRI technology can be used to perform remote operations on robots efficiently [11]. Therefore, not only is it widely used in the research of robotic systems, but it also plays an essential role in the implementation of robotic systems. Furthermore, in some particular activities, human-computer interaction plays an important role. For example, remote operation of human-machine interaction for medical robots is the safest method in medical processes, allowing the user to interact with the robot through tactile signals for coordinated feedback [12]. In order to improve surgeon performance, an experimental approach to characterize the human-robot-assisted surgery system is discussed. In vision-based robotic control, the visual impedance scheme was used to implement dynamic control at the activity level. To complete the integration of the vision servosystem and the conventional servosystem, Su et al. [13] proposed the visual impedance scheme in the control scheme; namely, the characteristics of the image were applied to the impedance equation. However, the flaw is that this research is still limited in terms of a self-adaptive decision. Besides, two themes are widely used in HRI [14], one of which was the human-computer interaction interface which was represented by the operation of the keyboard, and the other was the human-machine interaction represented by a touch screen operation. Therefore, this document focuses primarily on human-robot interaction with demonstration teaching.

At the same time, another advanced technology for human-machine operation is the skill transmission through demonstration teaching [15, 16], and during this process, the motion control strategies and the generalized output [17] were learned to transfer the motor skills to the robot by the movement of the demonstrator. Behavior perception, behavior representation, and behavior reproduction are three processes that are imitating the process of learning. Some special feature methods can be used to program the learning process, such as Dynamic Motion Primitives (DMP) and Hidden Markov Models (HMM) [18, 19]. The stochastic models, like the Gaussian mixture model (GMM), have some powerful capability to code and process noise so that they have the ability to handle high-dimensional problems more effectively. The trajectory-level representation that is on the basis of the probability model uses the characteristics of the stochastic model to model the motion trajectory, thereby solving the problem more efficiently. Motion trajectory reproduction and motion control belong to the category of behavior reproduction, where trajectory reproduction is a process of transmitting coded data. It is mainly transmitted for some techniques of regression, such as Gaussian Process Regression (GPR) and Gaussian Hybrid Regression (GMR), and the feedback variable is a playback behavior learned from the presenter. In other words, it is a generalized output that maps to robot motion control for motion reproduction [2022].

The most widely used technologies for scientific research widely used in statistical models of human behavior are machine learning (ML) and deep learning (DL) technologies. In the previous work, in order to compare the recognition rates for identifying human activities, different combined sensors were adopted, and a deep convolutional neural network (DCNN) was applied to the HAR system [23]. In addition, the ML method is used to enhance HAR’s adaptive identification and real-time monitoring system to overcome time-consuming strategies. These classifiers have been proven to recognize more human actions in dynamic situations, thereby improving accuracy, enhancing robustness, and achieving time-saving effects [13, 24]. Moreover, a hybrid hierarchical classification algorithm combining DL and the method that is based on the threshold is proposed to differentiate complex events in order to calculate quickly. Although our previous research has proposed many effective frameworks, most acceptable results have been achieved with limited assumptions and conditions that cannot meet the complex environment in the medical room.

In this article, a novel human-like control framework for mobile medical service robots is presented, where Kinect sensors are used to achieve human activity recognition to generate a designed teaching trajectory. At the same time, the learning technique of dynamic movement primitives (DMP) with the Gaussian mixture model (GMM) is applied to transfer the skill from human to machine. In addition, a neural-enhanced model predictive tracking controller is implemented to follow the teaching trajectory. Finally, some demonstrations are carried out in a medical room to account for the effectiveness and superiority of the framework. The main contribution of this paper is as described below:(1)In order to control the mobile service robot to transport medical supplies or meals to the patient who suffers from the new coronavirus, a novel human-like control framework is discussed.(2)Kinect sensors technology is applied to the skill transfer of the mobile medical service robot for collecting the movement points, and the method of DMP with GMM is used to congress the points.(3)To efficiently track the teaching trajectory under uncertain disturbances, a neural network enhanced predictive tracking control scheme is presented. Also, some demonstrations are carried out to illustrate the developed structure.

The structure of this essay is as described below: Section 2 describes the overview of human-like control. Some demonstrations are discussed in Section 3. Finally, the conclusion and future point are summarized in Section 4.

2. Methodology

2.1. Human Activity Recognition Using Kinect Sensors

Human activity recognition technology can be applied to follow the position of an operator using a Kinect device. As shown in Figure 1, the operator selects the depth message on the Kinect sensor, and the color vision is collected in the Kinect depth image [25]. To effectively construct the movement information, it is combined with the color image and the depth image in such a way that the origin is located at the center of the depth camera. Thus, we assume that the coordination system in camera space follows a right-hand convention [26].

In this case, represents the three coordinates of the sequence of frame, and we define and as the depth image points and color image, respectively. Then, we can obtain the Bayes rule to evaluate the probability of :where and denote the exercise dataset and prior probabilities of skin color, respectively.

It is on account of self-occlusion or lacking of joint information at some stages that we need to incorporate other functions that can offer data about human shape so as to upgrade the precision of the classifier [6, 27]. We adopt orthogonal Cartesian planes on the depth map to obtain the positive image and the profile obtained. Then, by converting Cartesian coordinates to polar coordinates, the silhouette of a person can be efficiently processed:where and represent the coordinates of the outline of the human body and Radius and angle in polar coordinates, respectively. Besides, is the center coordinate of the human contour. The overall order of each activity with front and side views is averaged, and the average from the initial frame to the final frame is defined as follows:

2.2. Trajectory Generation via DMP with GMM

After the Kinect device has collected the path information teaching by human demonstration, the mobile robot needs to learn the created trajectory [28]. The teaching trajectory is determined by dynamic movement primitive technology (DMP) and then rebuilt by the Gaussian mixture model (GMM) to generalize the movement trajectory:where is the prior probability, is the conditional probability distribution, which follows the Gaussian distribution, and is the number of Gaussian model distribution.

Therefore, by using a Gaussian mixture model, the entire teaching dataset can be expressed as follows:where is the dimension of the GMR and determined by .

The Gaussian distribution can be addressed aswhere and .

Therefore, the average and variance of GMR of the number of Gaussian components can be evaluated aswhere is the estimation variable and is the corresponding space parameter. is the generalized points, which produces a smooth movement trajectory under the covariance constraint .

The GMM model, including a multidimensional probability density function, is composed of multiple Gaussian probability density functions. The Gaussian model is only related to two parameters, the mean and variance. As we all know, different learning mechanisms can directly affect the accuracy, convergence, and stability of the model. Assume that an -order GMM is weighted and summed with Gaussian probability density functions:where denotes dimensional random vector and is on behalf of the order of the model, while represents the weight of each Gaussian component, satisfying . Furthermore, is each Gaussian component, which is a Gaussian probability density function of dimension and can be expressed as follows:where is on behalf of the mean vector and denotes the covariance matrix. Then, GMM can be expressed by the three parameters of mean vector, covariance matrix, and mixed weight. Therefore, the GMM can be described as

2.3. Neural Approximation

In order to efficiently transfer the trajectory teaching by human demonstration, it is necessary to control the uncertain disturbance in the trajectory tracking process for the mobile robot [2931]. To overcome the hidden safety hazards in scooter operation [3234], a control scheme based on RBFNN was implemented for the elderly walker system. This scheme has certain disturbances and unknown dynamic characteristics. Design a constant smooth function to connect the approximation capability, where the RBFNN control scheme is applied for evaluating the dynamics of uncertainty, such as load friction and mechanism structure [3537]:where represents the input of ; and are the activation function depending on Gaussian function, respectively; and represents the weight in the hidden layer:where , , and is the variance.

Then, can be defined aswhere is a positive constant.

Then, we havewhere is the desired weight subjected to and .

Hence, we havewhere denotes the activation function depending on the Gaussian function.

2.4. Neural-Based Model Predictive Tracking Control

Human motion points are collected by the Kinect sensor, and then the generated trajectory can be obtained with the method of DMP and GMM [18, 38, 39]. Finally, the next task of the mobile robot is to follow the teaching trajectory [40].

Figure 2 exhibits the kinematic model of the mobile medical service robot, where and denote the coordinate of the rear axis and front axis, respectively. is the circle center, and denotes the steering radius. and represent the wheel trajectory. and denote the rear speed and front speed, respectively. and denote the steering angle and yaw angle, respectively.

The trajectory tracking control of the mobile scroll wheel system can be represented aswhere is the system state and is the control state.

Hence, the dynamic model of the mobile robot can be addressed aswhere , , , and are the wheel force of left front, right front, left rear, and right front, respectively. denotes the center yaw velocity, and represents the rotational inertia.

Besides, we assume the following condition to evaluate the lateral force of the robot tire [14]:where and are tire cornering angle. and denote cornering stiffness and is the slip angle.

The tracking error can be addressed as

Then, we discretize the error function assubjected to and is the sampling time.

In order to reliably and smoothly grasp the desired trajectory, state errors and control parameters must be constrained:where and are weighting factors, is the prediction horizon, and is the control horizon. Then, the actual control variable can be determined as

It is on account of considering the safety and stability of the robot [41, 42] that it is of necessity to restrict the control limit and control increment. Combined with the mobile robot system, the control constraint can be presented as follows:

Based on the overall control scheme, the framework of neural approximation for tracking control using DMP with GMM is shown in Figure 3. The Kinect sensor detects the human movement points and then generates the teaching trajectory using the technology of DMP with GMM. Then, the neural-based model predictive tracking controller is carried out to realize the path following.

3. Results and Discussions

In this section, the overview scenario of the medical room to transport the meals for patients is presented in Figure 4. There are two Kinect sensors (XBOX 360) used in this demonstration. The surgical medical robot (LWR4+, KUKA, Germany) is used to feed the meals to patients, where the haptic manipulator (SIGMA 7, Force Dimension, Switzerland) is applied to control the KUKA arm remotely. The main purpose of this demonstration is that the developed mobile medical service robot can safely transport the meals or medicines to the medical bed like a human without collisions.

The Kinect sensor can detect human activity points and generate a teaching trajectory based on the method of DMP and GMM. Then, the mobile robot can follow the teaching trajectory via human demonstration. The result of the learning method, including the DMP, GMM, and the regression result of teaching trajectory, is displayed in Figures 5 and 6. It is noted that there are two demonstrations considered in this section, which aims to evaluate the proposed framework for mobile medical service robot in skill transfer via teaching by demonstration.

Figure 7 exhibits teaching results of demonstration 1 in x-position, y-position, x-error, y-error, robot tracking velocity, roll angle, pitch angle, and tracking performance. It can be concluded that the mobile medical service robot can follow the teaching trajectory collected by Kinect sensors. The y-position error and x-position error can be constrained in a reasonable range within meters, indicating that the mobile robot can avoid the medical devices and surgeons. On the other hand, because of the neural-based predictive tracking controller, the velocity response of the mobile robot under uncertain disturbance is smooth. In particular, the roll angle and pitch angle can maintain a stable range.

In addition, to further illustrate the improvement of skill transfer scheme using multisensors fusion technology, demonstration 2 to avoid obstacles such as medical devices and medical staff is carried out. Figure 8 displays the teaching performance in x-position, y-position, x-error, y-error, robot tracking velocity, roll angle, pitch angle, and tracking performance. From the tracking performance of x-position and y-position, the mobile medical service robot can efficiently follow the teaching trajectory and avoid obstacles. The x-position error and y-position error also can be maintained at a high accuracy, which is within meters in x-position and meters in y-position. At the same time, the neural-based predictive controller can constrain the mobile robot body, and the pitch angle and roll angle are within degrees and degrees, respectively.

4. Conclusion

In this paper, a novel human-like control framework is implemented to control a mobile service robot using a Kinect sensor and DMP with GMM. It aims to bridge the human activity recognition techniques and assist the mobile medical service robot and allows the robot to cooperate with the medical staff. The Kinect sensor is used to detect human activities to generate a set of movement points, and then the teaching method including dynamic movement primitives with the Gaussian mixture model can generate the desired trajectory. To achieve stable tracking, a model predictive tracking control scheme based on neural networks is implemented to follow the teaching trajectory. Finally, some demonstrations are carried out in a medical room to validate the effectiveness and superiority of the developed framework.

Human-machine collaborative control based on the Internet of Things (IoT) is the future research direction. In our lasted work [43], we have successfully used IoT technology to exploit the best action in human-robot interaction for the surgical KUKA robot. Instead of utilizing compliant swivel motion, HTC VIVE PRO controllers, used as the Internet of Things technology, are adopted to detect the collision, and a virtual force is applied on the elbow of the robot, enabling a smooth rotation for human-robot interaction. Future work, combined with the IoT technology and multisensors, the concept of the intelligent medical room, will be considered to strengthen the human-robot cooperation.

Data Availability

No data were used to support this study.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

This work was supported by the National Key Research and Development Program of China under Grant 2019YFC1511401 and the National Natural Science Foundation of China under Grant 61103157.