Sensing and Intelligent Perception in Robotic ApplicationsView this Special Issue
Stability Control of Force-Reflected Nonlinear Multilateral Teleoperation System under Time-Varying Delays
A novel control algorithm based on the modified wave-variable controllers is proposed to achieve accurate position synchronization and reasonable force tracking of the nonlinear single-master-multiple-slave teleoperation system and simultaneously guarantee overall system’s stability in the presence of large time-varying delays. The system stability in different scenarios of human and environment situations has been analyzed. The proposed method is validated through experimental work based on the 3-DOF trilateral teleoperation system consisting of three different manipulators. The experimental results clearly demonstrate the feasibility of the proposed algorithm to achieve high transparency and robust stability in nonlinear single-master-multiple-slave teleoperation system in the presence of time-varying delays.
Teleoperation through which a human operator can manipulate a remote environment expands human’s sensing and decision making with potential applications in various fields such as space exploration, undersea discoveries, and minimally invasive surgery [1–3]. From the teleoperation’s point of view, a teleoperation system can be of two categories, bilateral or multilateral.
A conventional bilateral teleoperation system which consists of a pair of robots allows sensed and command signals flow in two directions between the operator and the environment: the command signals are transmitted from the master to control the slave and the contact force information is simultaneously fed back in the opposite direction in order to provide human operator the realistic experience. System stability is quite sensitive to time delays and even a small time delay may destabilize the overall system. Many researchers have been focusing on guaranteeing robust stability of a teleoperation system in the presence of time delays. Based on the passivity theory and the scattering approach, the stability analysis and controller design for the bilateral teleoperation system have been widely studied [4, 5]. The most remarkable passivity-based approach is the wave-variable method introduced by Niemeyer and Slotine . Numerous studies have explored the application of wave-variable theory to enhance the task performance of the wave-variable-based system as reported in . Yokokohji et al. design a compensator to minimize the performance degradation of the wave-based system [8, 9]. Munir and Book apply the wave prediction method which employs the Smith predictor and Kalman filter to deal with the Internet-based time-varying delay problem . Hu et al. compensate for the bias term to improve the trajectory tracking of the wave-variable-based system . Through adding correction term, Ye and Liu enhance the accuracy of the system’s force tracking . Aziminejad et al. further extend the wave-based system to the four-channel system by introducing measured force reflection . Alise et al. analyze the application of the wave variables in multi-DOF teleoperation .
A conventional bilateral teleoperation system usually involves a single slave robot which is controlled by a single operator. However, it is more effective in many applications to have multiple manipulators in a teleoperation system. Therefore, the multilateral teleoperation has been gradually becoming a popular topic and many approaches have been proposed such as control [15, 16], disturbance-observer-based control , and adaptive control . Although the wave-variable transformation can guarantee the communication channels’ passivity, most of the wave-based systems are not suitable to be extended to the multilateral teleoperation since they cannot guarantee the system stability under time-varying delays. Moreover, the wave-based systems also suffer transparency degradation and signals variation and distortion due to the existence of wave reflections. Without reducing the wave reflections, one robot with large variations can seriously influence other robots’ task performance and the users’ perception of the remote environment in the presence of large time-varying delays. Therefore, guaranteeing system stability under time-varying delays and enhancing the system transparency via wave reflections reduction are the two key criteria for the successful application of the wave-variable approach in the multilateral teleoperation.
As a part of multilateral teleoperation control, multiple-masters-single-slave (MMSS) system includes more than one single operator to collaboratively carry out the task [15, 20–23]. Unlike the MMSS system, the single-master-multi-slave (SMMS) system allows one operator to simultaneously control multiple slave robots. The SMMS teleoperation is firstly introduced in . Later, the single-master-dual-slave scenario is investigated under constant time delays for a linear one-DOF teleoperation system in [17, 25–28]. In a SMMS system, the multiple slave robots should not only coordinate their motions (e.g., robotic network as a surveillance sensor network) but also perform cooperative manipulation and grasping of a common object , as shown in Figure 1. A SMMS system is suitable for many applications where (1) a single slave robot cannot perform the required level of manipulation dexterity, mechanical strength, robustness to single point failure, and safety (e.g., distributed kinetic energy) and (2) the remote task necessarily requires the human operator’s experience, intelligence, and sensory input, but it is not desired or even impossible to send humans on site. One example of such applications is the cooperative construction/maintenance of space structures (e.g., international space station, Hubble telescope) . It requires high demand for these slave robots to have precise actions following the human operator to perform different remote environmental tasks in the presence of time-varying delays.
In this paper, a novel modified wave-variable-based control algorithm is designed to guarantee accurate position synchronization and force reflection of all the robots in the nonlinear SMMS teleoperation system in the presence of large time-varying delays. The stability of the multirobots system in different environmental scenarios is also analyzed. The theoretical work presented here is supported by experimental results based on a 3-DOF trilateral teleoperation system consisting of three different haptic devices.
2. Modeling the -DOF Multilateral Teleoperation System
In this paper, the master robot and the -slave robots are modeled as a pair of multi-DOF serial links with revolute joints. The nonlinear dynamics of such a system can be modeled aswhere , is master, and is slave. are the joint acceleration, velocity, and position, respectively, denotes master, and denotes the th slave. denotes the number of the slave robots. are the inertia matrices; are Coriolis/centrifugal effects. are the vectors of gravitational forces and are the control signals. The forces applied on the end-effector of the master and slave robots are related to equivalent torques in their joints bywhere , are the Jacobean of the master robot and the th slave robot, respectively. and stand for the human and environment forces, respectively.
Important properties of the above nonlinear dynamic model, which will be used in this paper, are as follows [25, 30].(P1)The inertia matrix for a manipulator is symmetric positive-definite which verifies , where is the identity matrix. and denote the strictly positive minimum (maximum) eigenvalue of for all configurations .(P2)Under an appropriate definition of the Coriolis/centrifugal matrix, the matrix is skew symmetric, which can also be expressed as(P3)The Lagrangian dynamics are linearly parameterizable: where is a constant -dimensional vector of inertia parameters and is the matrix of known functions of the generalized coordinates and their higher derivatives.(P4)For a manipulator with revolute joints, there exists a positive bounding the Coriolis/centrifugal matrix as(P5)The time derivative of is bounded if and are bounded.
3. Wave Variable and the Proposed Method
Figure 2 shows the standard wave-variable transformation where the wave variables ( and ) are defined aswhere denotes the wave characteristic impedance and and are the wave variables being transmitted in the communication channels. The power flow can be expressed asA system is passive if the output energy is no more than the sum of the initial stored energy and the energy injected into the system . The wave-based teleoperation system is passive when it satisfies (8), where is the initial energy stored in the system. ConsiderWhen applied to the multilateral teleoperation, the wave-variable transformation must meet two requirements, maintaining channels passivity in the presence of random time delays and transmitting signals without large variation and distortion. Considering the time delays, the power flow can be further written aswhere is the power dissipation of the communication channels. indicates passiveness of the channels. In this paper, the time-varying delays are assumed not to increase or decrease faster than time itself; that is, . is the differential of the time delays. In the presence of constant time delays (), the power dissipation is equal to zero based on (10). It means the wave-based controller assures passivity regardless of the value of constant time delay. However, when the time delay is varying, the positive results in to be negative and the system passivity will be degraded. Therefore, the conventional wave-variable transformation cannot guarantee system passivity under time-varying delays.
Wave reflection is another main drawback of the standard wave transformation, which is caused by the imperfectly matched junction impedance in the wave-based system as shown in Figure 3. There are three independent channels in the wave-variable transformation in Figure 3, the master’s direct feedback (dotted line 1), the wave reflection (dotted line 2), and the force feedback from the slave (dotted line 3). In channel 1, the master velocity signals directly return in the form of the damping . Channel 1 generates a certain amount of damping and this enhances the system stability by sacrificing transparency. Channel 3 feeds signals back from the remote slave side in order to provide useful information to the operator. Wave reflections occur in channel 2.
The phenomenon of wave reflection occurs in channel 2. The relationship between the outgoing wave variables and and the incoming wave variables and can be expressed asEach of the incoming wave variables and is reflected and returned as the outgoing wave variables and . Wave reflections can last several cycles in the communication channels and then gradually vanish. This phenomenon can easily generate unpredictable interference and disturbances that significantly influence transparency . Large signals variation and distortion can be caused by the wave reflections in the presence of large time delays. Therefore, the standard wave-variable transformation is not suitable for multilateral teleoperation when large time-varying delays exist.
In order to guarantee the passivity of the time delayed communication channels between the master robot and each slave robot, the modified wave-variable controllers proposed in  are applied in this paper as shown in Figure 4. The main advantage of the modified wave controllers is the efficient reduction in the wave-based reflections while simultaneously guaranteeing channels’ passivity as analyzed in .
The two wave-variable controllers are applied to encode the feed-forward signals and with the feedback signals and . The wave variables in the two controllers are defined as follows:where and are the characteristic impedances. and do not contain any unnecessary information from the incoming wave variables and as shown in (13) and (14). Therefore, wave reflections can be efficiently eliminated.
In the proposed SMMS teleoperation system (Figure 5) in which one master robot is used to control multiple slave robots, the main objective is to have the positions of all the slave robots accurately synchronized to the position of the master robot. A secondary objective is that all the robots should have accurate force tracking with each other, which means when one slave robot comes in contact with the remote environmental object during free motion, it will immediately feed back the force information to all of the other robots to signal them to stop. Via reaching the two targets, all the slave robots will precisely follow the human operator in different environmental scenarios. By applying the two wave controllers, the energy information such as torque, position, and velocity signals can be transmitted through the communication channels without influencing the system passivity. By setting , , , and , a new state variable for the master robot is introduced as follows:where , , and are diagonal positive-definite matrices. In the slave sides, each slave robot receives control signals from the master robot and the other slave robots. The new master-control state variable for the th slave robot is written as follows:In order to prevent the position drift between the slave robots, each slave robot should also transmit its position information to the other slave robots. Furthermore, In order to achieve the secondary objective which is the accurate force tracking, each slave robot’s environmental force information is also transmitted via slave-slave communication channels to the other slave robots. The channels’ passivity is guaranteed when the wave-variable controller proposed in  is applied to encode the th slave robot’s position signals with the transmitted th slave robot’s control environmental force ( and denote the arbitrary two slave robots in the slave robots). Therefore, the final control variable of the th slave robot is expressed aswhere denote the time-varying delays in the forward slave-slave communication channels and are diagonal positive-definite matrices. The second last term provides the position control between every two slave robots and the last terms provide force control between every two slave robots. By defining new variables,(16) and (18) can be simplified as follows:The main aim of the controller design is to provide a stable multilateral system with accurate position tracking and to enhance the force tracking during manipulations. The position synchronization is derived ifwhere is the Euclidean norm of the enclosed signal. We define the position errors , and velocity errors , between the master and the th slave manipulators as follows:The new control laws for the single master robot and the th slave robot are designed as follows:where , , and are the estimates of , , and . Substituting (24) and (25) into (1) and considering Property 3 which states that the dynamics are linearly parameterizable, the new system dynamics can be expressed aswhere are the time-varying estimates of the master’s and the th slave’s actual constant -dimensional inertial parameters given by . are the estimation errors. The time-varying estimates of the uncertain parameters satisfy the following conditions :
4. Stability Analysis
4.1. Free Motion Strategy
Theorem 1. Consider the proposed nonlinear multilateral teleoperation system described by (16)–(34) in free motion where the human-operator force and the environmental force can be assumed to be zero . For all initial conditions, all signals in this system are bounded and the master and all of the slave manipulators state are synchronized in the sense of (22) and (24).
Proof. Based on (13) and (14), and have the terms and , respectively. These two terms can be expressed as and in frequency domain. According to the well-known characteristic of the time delay element ,it is true that in the presence of large time-varying delays. It means and which are varying according to the time delays. Therefore, and can be expressed as the varying dampings and where varies between 0 and 2. The values of and are scaled by the characteristic impedances and of the applied modified wave controllers. Therefore, (20) and (21) can be expressed asDefine a storage functional , whereIn order to make positive semidefinite, and should be satisfied, which can be simplified asDue to the assumption that , by setting a small value of , (38) can be easily satisfied. By using the dynamic equations and Property 3, the derivative of can be written asBased on (39), the differential of the functional is negative semidefinite. Integrating both sides of (39), we getSince is positive semidefinite and is negative semidefinite, exists and is finite. Also, based on (37)–(40), , , , , , , , , , , . Since a square integrable signal with a bounded derivative converges to the origin [31, 33, 35], . Therefore, the master and slave manipulators state synchronize in the sense of (22)–(24).
In free motion, the system’s dynamic model (26) can also be written as Differentiating both sides of (41),For the first terms of the right sides of (42), we have According to Properties 1 and 4, are bounded. Based on Property 5, the terms in bracket of (29) are also bounded. Therefore, and are uniformly continuous (). Since , it can be concluded that based on Barbǎlat’s Lemma.
4.2. Environmental Contact with Passive Human Force
Assume the human and environmental forces are passive and can be modeled aswhere and are positive constant matrices and are the properties of the human and the environment, respectively.
Theorem 2. The multilateral nonlinear teleoperation system described by (16)–(34) is stable and all signals in this system are ultimately bounded, when the human and environmental forces satisfy (44).
Proof. Consider a positive semidefinite function for the system asThe derivative of can be written asThe Lyapunov approach requires to be negative semidefinite. Based on the first three terms of the right side of (46), the sufficient conditions to satisfy this requirement are thatBy enlarging the values of and decreasing the values of , (47) can be satisfied. Hence, will be negative semidefinite and exists and is finite.
4.3. Environmental Contact with Nonpassive Human Force
The human operator can not only dampen energy but also generate energy in order to manipulate the robots to move through the desired path. Therefore, in the common case, the human forces are not passive. In this situation, the human and environment can be modeled aswhere is a bounded positive constant vector, which generates energy as an active term. We define and . There is a linear map between and :where are nonsingular constant matrices.
Theorem 3. The proposed system is stable and all signals in this system are ultimately bounded, when the human and environmental forces satisfy (48).
Proof. By choosing the previous Lyapunov function , the new derivative can be written asNote that where vector has the same ranks as , . Therefore, it is true thatwhere . When the system satisfies (47),where is the smallest eigenvalue of , , , and . Substituting (53) into (52) and setting ,(54) can be simplified asBased on (55), for large values of , the Lyapunov function is decreasing. Therefore, and are bounded, which means , , , , , and are also bounded.
5. Experimental Validation
In this section, the performance of the proposed nonlinear multilateral teleoperation system is validated by a series of experiments. The algorithm is applied to three Phantom manipulators. The 6-DOF Phantom (TM)* model 1.5 manipulator (Sensable Technologies, Inc., Wilmington, MA) is chosen to be the master robot which remotely controls a 3-DOF Phantom Omni (Slave 1) and a 3-DOF Phantom Desktop (Slave 2) via the Internet as shown in Figure 3. The three haptic devices have different dynamics and initial parameters. PhanTorque toolkit  is applied by two computers to control the two robots. PhanTorque toolkit enables the users to work with the Sensable Phantom haptic devices in the Matlab/Simulink environment in a fast and easy way. Figure 4 shows the trilateral experiment platform.
The control loop is configured as a 1 kHZ sampling rate. Based on the controllers analysis in Section 4, the controller parameters are given as , , , , , , , , , .
5.1. Bilateral Teleoperation (1-DOF)
In this subsection, the proposed wave-based architecture is compared with the standard wave-based system in bilateral teleoperation using 1-DOF. The time delay (one way) is 400 ms constant delay.
Figures 7 and 8 show the velocity and position tracking of the two systems in free motion. Based on (10)-(11), due to the wave reflections, the useless signals remain in the communication channels for several circles to the extent that the normal signals transmissions are influenced and the transmitted velocity control signals contain large signals variations. Moreover, considering the conventional wave variables in (6), the signal transmission in the standard wave-based system can be expressed asThe biased terms and also seriously affect the accuracy of the position tracking. Since the standard wave-based system is an overdamped system, by applying the same operation force, the velocity and position of the standard wave-based system are lower than those of the proposed system and the operator feels damped when operating the system. Unlike the standard system, the proposed wave-based system has little signals variations since the wave reflections are almost eliminated. According to (20) and (21), the biased terms affecting position tracking are and . Under small time delays, the biased terms are about zero. When the time delays are nonignorable, setting large value of can also effectively reduce the biased terms. Therefore, both of the velocity and the position have accurate tracking performances.
Figures 9 and 10 show the torque tracking and position tracking of the two systems in hard contact. As shown in Figure 9, the standard wave-based system can only achieve accurate force tracking in steady state. In the transient state, when the environment undergoes unpredictable changes, wave reflections occur so that the force reflection has large perturbations and the operator can hardly feel the accurate environmental force. Moreover, according to (56), since the standard wave-based system has no direct position transmission, position drift occurs during hard contract. It means that when directly applying the conventional wave-variable transformation in the SMMS system, when one slave robot contacts with the remote environment and is forced to stop, the master robot still keeps moving which can drive other slave robots to move. Therefore, the robots’ motion synchronization will be jeopardized. As shown in Figure 10, the environmental torque quickly tracks the operator’s torque without variation and no position drift occurs during hard contact, which means when applying to the SMMS system, the proposed architecture can not only provide accurate force tracking, but also achieve motion synchronization.
5.2. Multilateral Teleoperation (3-DOF)
In this subsection, the proposed SMMS system is validated. The communication channel of the experimental platform is the Internet. In order to test the performance of the proposed system in the presence of large time-varying delays, the time delay blocks in the Simulink library are applied to introduce the overall system time delays (Figure 6). The one-way delay between the master and the slave sides is from 650 ms to 750 ms. Theoretically, in the real applications, the slave robots are close to each other, so the time delays between two slave robots are not large and not significantly different. The one-way delay between the two slave robots is set as around 100 ms in this experiment. In the first experiment, the system performance in free motion is demonstrated. During free motion, the master manipulator is guided by the human operator in the task space and the two slave robots are coupled to the master robot using the proposed system. Figure 11 demonstrates the position synchronization performances of the proposed teleoperation system. Since the wave reflections are eliminated, the slave robots can closely track the master robot without large vibration and signals distortion. The remaining slight signal perturbations in Figure 7 are caused by the time-varying delays. The two slave robots can perform exactly the same actions during free motion. In the presence of large time-varying delays, although the dynamic models of the master and slaves are quite different and affected by uncertain parameters, both of the slave robots can reasonably track the master robot’s trajectory with little errors. The root mean square errors (RMSEs) for position tracking between every two robots in Figure 7 are shown in Table 1. Therefore, it can be concluded that the main objective is that accurate position tracking of the proposed teleoperation system is achieved.
In the next experiment, the two slave robots are driven by the master robot to draw a letter “O” and a triangle “Δ” on a table as shown in Figure 8. Friction exists between the manipulators and the table. The RMSEs for position tracking between every two robots in Figure 12 are shown in Table 2. Due to the effect of the friction, the RMSEs are larger than that of free motion. The proposed algorithm still makes all of the robots have reasonable trajectory tracking without large signals distortion.
In the next experiment, slave manipulators 1 and 2 are guided by the master manipulator to come in contact with different remote environment as shown in Figure 13. The master robot firstly drives the two slave robots to perform the free motion in the first 2 seconds. Then, from the 2nd to the 5th second, Slave 1 starts to contact with a solid wall while Slave 2 is still in free motion. Slave 1 immediately feeds the contact force back to the master robots and Slave 2. The master robot keeps applying force to the two slave robots, but Slave 2 also stops moving to make the motion synchronization with Slave 1 even when no environmental force is applied to its manipulator. In the 5th second, the solid wall is suddenly removed. It can be observed that both of the two slave robots quickly track the master robot’s position with little variation, which proves that the proposed algorithm can deal with the sudden changing environment and the wave reflections will not reinstate. The RMSEs for position tracking between every two robots and the RMSEs for force tracking between the master robot and Slave 1 in Figure 13 are shown in Tables 3 and 4.
In the final experiment, the two slave robots are driven by the master robot to simultaneously contact with a solid wall. The position and force tracking are shown in Figure 14. Under the condition of hard contact, both of the two slave robots feed the environmental forces back to the master robots and the human operator can feel the mixed forces from the two slave robots. Figure 14 demonstrates that accurate force tracking between all of the three robots is achieved. The RMSEs of position and force tracking between every two robots are shown in Table 5.
In this paper, a novel wave-based control approach has been proposed for hybrid motion and force control of a multilateral teleoperation system with one-master-multiple-slave configuration in the presence of large time-varying delays in communication channels. The stability of the proposed multilateral teleoperation system in different environment scenarios is also analyzed in this paper. The feasibility of the proposed algorithm in the presence of large time-varying delays is validated using a 3-DOF nonlinear trilateral teleoperation system.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
L. F. Penin, K. Matsumoto, and S. Wakabayashi, “Force reflection for time-delayed teleoperation of space robots,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA '00), vol. 4, pp. 3120–3125, IEEE, San Francisco, Calif, USA, April 2000.View at: Publisher Site | Google Scholar
K. A. Manocha, N. Pernalete, and R. V. Dubey, “Variable position mapping based assistance in teleoperation for nuclear cleanup,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA '01), pp. 374–379, Seoul, Korea, May 2001.View at: Google Scholar
Y. Yokokohji, T. Imaida, and T. Yoshikawa, “Bilateral teleoperation under time-varying communication delay,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '99), vol. 3, pp. 1854–1859, Kyongju, Republic of Korea, October 1999.View at: Publisher Site | Google Scholar
Y. Yokokohji, T. Imaida, and T. Yoshikawa, “Bilateral control with energy balance monitoring under time-varying communication delay,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA '00), pp. 2684–2689, April 2000.View at: Google Scholar
S. Sirouspour, “A control architecture for multi-master/multi-slave teleoperation,” in Proceedings of the Tenth IASTED International Conference on Robotics and Applications, pp. 221–226, August 2004.View at: Google Scholar
N. D. Do and T. Namerikawa, “Cooperative control based on Force-Reflection with four-channel teleoperation system,” in Proceedings of the 50th IEEE Conference on Decision and Control and European Control Conference (CDC-ECC '11), pp. 4879–4884, December 2011.View at: Google Scholar
D. Lee, O. Martinez-Palafox, and M. W. Spong, “Bilateral teleoperation of multiple cooperative robots over delayed communication networks: application,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA '05), pp. 366–371, IEEE, April 2005.View at: Publisher Site | Google Scholar
K. Kosuge, J. Ishikawa, K. Furuta, and M. Sakai, “Control of single-master multi-slave manipulator system using VIM,” in Proceedings of the IEEE International Conference on Robotics and Automation, pp. 1172–1177, May 1990.View at: Google Scholar
Y. Cheung, J. H. Chung, and N. P. Coleman, “Semi-autonomous formation control of a single-master multi-slave teleoperation system,” in Proceedings of the IEEE Symposium on Computational Intelligence in Control and Automation (CICA '09), pp. 117–124, April 2009.View at: Publisher Site | Google Scholar
M. W. Spong, S. Hutchinson, and M. Vidyasagar, Robot Modeling and Control, John Wiley & Sons, New York, NY, USA, 2005.
H. K. Khalil, Nonlinear Systems, Prentice Hall, Upper Saddle River, NJ, USA, 2002.