Abstract
In Robot-Assisted Rehabilitation (RAR) the accurate estimation of the patient limb joint angles is critical for assessing therapy efficacy. In RAR, the use of classic motion capture systems (MOCAPs) (e.g., optical and electromagnetic) to estimate the Glenohumeral (GH) joint angles is hindered by the exoskeleton body, which causes occlusions and magnetic disturbances. Moreover, the exoskeleton posture does not accurately reflect limb posture, as their kinematic models differ. To address the said limitations in posture estimation, we propose installing the cameras of an optical marker-based MOCAP in the rehabilitation exoskeleton. Then, the GH joint angles are estimated by combining the estimated marker poses and exoskeleton Forward Kinematics. Such hybrid system prevents problems related to marker occlusions, reduced camera detection volume, and imprecise joint angle estimation due to the kinematic mismatch of the patient and exoskeleton models. This paper presents the formulation, simulation, and accuracy quantification of the proposed method with simulated human movements. In addition, a sensitivity analysis of the method accuracy to marker position estimation errors, due to system calibration errors and marker drifts, has been carried out. The results show that, even with significant errors in the marker position estimation, method accuracy is adequate for RAR.
1. Introduction
The application of robotics and Virtual Reality (VR) to motor neurorehabilitation (Figure 1) has been beneficial for patients, as they receive intensive, repetitive, task-specific, and interactive treatment [1–4].
The assessment of (a) patient movement compliance with the prescribed exercises and (b) patient long-term improvement is critical when planning and evaluating the efficacy of RAR therapies. In order to obtain the patient motion data to conduct the said assessments, one has to estimate patient posture (i.e., the joint angles of the limbs). Patient posture estimation methods need to be practical and easy to set up for the physician, so that the said assessments can indeed be an integral part of the therapy.
Current methods for estimating patient posture are either cumbersome or not accurate enough in exoskeleton-based therapies. In order to overcome such limitations, we propose a method where low-cost RGB-D cameras (which render color and depth images) are directly installed in the exoskeleton and colored planar markers are attached to the patient’s limb to estimate the angles of the GH joint, thereby overcoming the individual limitations of each of these systems.
2. Literature Review
Optical, electromagnetic, and inertial MOCAPs have been used in many rehabilitation scenarios for accurate posture estimation [5]. However, the use of the said MOCAPs in exoskeleton-based rehabilitation is limited by the factors discussed below:(1)Optical marker-based systems (e.g., Optotrak, CODA, Vicon) are considered the most accurate for human motion capture [5]. Reference [6] reports Optotrak errors of 0.1–0.15 mm. However, in the specific case of exoskeleton-based therapy, these systems require redundant sensors and markers to cope with occlusions caused by the exoskeletal body. Therefore, their specific usage for therapy is limited. Besides, the cost of these systems is high (50 K–300 K USD [7]) compared to nonoptical MOCAPs.(2)Electromagnetic systems do not suffer from optical occlusions. However, they are easily perturbed by surrounding metallic objects (e.g., exoskeletal body) and electric/magnetic fields [5]. An additional drawback of these systems is their limited detection volume when compared to optical systems.(3)Inertial and Magnetic Measurement Systems are robust, handy, and economical for full-body human motion detection (upper limb tracking in [8, 9]). With the use of advanced filtering techniques, inertial sensor drift errors are reduced and a dynamic accuracy of 3 deg. RMS [5] is achieved. However, these systems require patients to perform calibration motions/postures, which may not be suitable for those with neuromotor impairments.
In exoskeleton-based rehabilitation, the prevailing approach to estimate human limb joint angles (e.g., [10–13]) is to approximate them with the angles of the exoskeleton joints. However, misalignment between the axes of the exoskeleton and human joints may produce large estimation errors [14, 15]. Accurate estimation of GH joint angles is hard to achieve using this approach, since it requires an exoskeleton with a complex kinematic structure that considers the concurrent motion of the sternoclavicular and acromioclavicular joints.
Recognizing the differences in the kinematic structures of the limb and exoskeleton, [16] presents a computational method which considers the limb and exoskeleton parallel kinematic chains related by the cuff constraints joining them together. Then, the IK problem of the parallel kinematic chain can be solved to find the limb joint angles. A limitation of this method is that its performance has been demonstrated solely for analytic (1-DOF) movements of the elbow and wrist joints. The estimation accuracy of the GH joint angles has yet to be determined.
Reference [17] presents a computational method based on the estimation of the arm swivel angle (which parametrizes arm posture) for exoskeleton-based therapy. The arm IK is solved with a redundancy resolution criterion that chooses a swivel angle that allows the subject to retract the palm to the head efficiently. The approach in [17] extends their previous work in [18, 19] by considering the influence of the wrist orientation on the swivel angle estimation. Although the error of the swivel angle estimation (mean error 4 deg.) has been reported for compound movements [17], individual errors in the wrist, elbow, and GH joint angles are not indicated.
Reference [20] extends the method in [17] to estimate the wrist angles and assesses its performance for compound movements (mean RMSE 10 deg. in the swivel angle estimation). Reference [20] reports the individual errors of the arm joint angles solely for the movement task where the swivel angle was best estimated (mean RMSE 5 deg. in the swivel angle estimation). No errors of the arm joint angles were discussed for the other cases. A limitation of the work in [20] is that the MOCAP used to obtain the reference angles to assess their method performance is a custom-made inertial system with no reported measurement accuracy.
2.1. Conclusions of the Literature Review
We remind the reader that the general context of this paper is the estimation of the GH joint angles.(1)As per our literature review, no MOCAPs have been developed for the specific scenario of exoskeleton-based rehabilitation. Even if current MOCAPs and the exoskeleton could be set up for simultaneous use (e.g., [15, 16]), the setup protocol and operation are intricate and conflicting with the usual time and resources available for patient treatment.(2)Exoskeleton-based posture estimations present limitations in their accuracy due to kinematic mismatch of the limb and exoskeleton [15, 16].(3)The accuracy of the GH joint angle estimations provided by computational methods in [16, 17] is unknown. Reference [20] extends the work in [17] by estimating the wrist angles. Reference [20] solely reports the estimation accuracy of the GH angles for the best-case scenario and the precision of its ground-truth is not indicated.
2.2. Contributions of This Paper
In response to the limitations discussed in the estimation of patient joint angles in exoskeleton-based therapy (Sections 2 and 2.1), this paper introduces a hybrid approach to estimate, in real-time, the GH joint angles. This hybrid system is composed of a low-cost marker-based vision system and the rehabilitation robot, overcoming the individual limitations of its constitutive subsystems:(a)Occlusions are minimized, which are a major limitation of optical systems.(b)Accuracy of joint angle estimation is improved, which is a major limitation of exoskeleton-based systems.
This paper presents the implementation and assessment of our method using simulated human motion data. In addition, a sensitivity analysis of our method accuracy to marker position estimation errors is carried out.
We have considered the following scenarios of application for the proposed method in the RAR domain:(A)Precise estimation of GH joint angles during rehabilitation or evaluation sessions of GH joint analytic movements.(B)Acquisition of GH joint movement data enabling validation and improvement of other posture estimation methods without using expensive redundant optical MOCAPs.
3. Methods
3.1. Problem Definition
This section presents the problem of estimating the patient limb GH joint angles during the GH joint RAR using the proposed hybrid motion capture system (a detailed version of the problem definition is presented in the Appendix). This problem can be stated as follows.
Given. Consider the following:(1)Patient: (a) the kinematic model (e.g., the Denavit-Hartenberg parameters [21]) of the human upper limb () (Figure 2(a)).(2)Exoskeleton: (a) the kinematic model of the exoskeleton () and (b) the exoskeleton joint angles at any instant of the therapy () (Figure 2(b)).(3)Marker-based optical motion capture system (): (a) color and depth information captured by the RGB-D cameras installed in the exoskeleton links and (b) geometry and color of the markers attached to the patient upper limb (Figure 2(c)).
Goal. The goal is to estimate the patient GH joint angles () with minimum error during the GH joint rehabilitation exercises.
3.2. Kinematic Models
This section discusses the main features of the kinematic models of the human limb and exoskeleton used for the posture estimation method.
3.2.1. Kinematic Model of the Human Upper Body
The human kinematic model is denoted by , where and are the sets of links and joints, respectively. We use the human upper body model presented in [16] (Figure 2(a)), which includes joints of the spine, scapuloclavicular system, and arm. The upper limb is modeled with 9 DOFs: 2 DOFs of the scapuloclavicular system, 3 DOFs of the GH joint (spherical joint), 2 DOFs of the elbow, and 2 DOFs of the wrist (see further details in the Appendix). This model presents the following advantages:(a)It can be easily implemented in robotic simulators and similar tools.(b)It is suitable for simulating human-robot interaction in real-time [16].(c)The spherical model of the GH joint avoids limitations of other representations of such joint, like the Gimbal lock that occurs when using the three concurrent and orthogonal 1-DOF revolute joints’ model [22].
3.2.2. Kinematic Model of the Exoskeleton
The exoskeleton kinematic model is denoted by , where and are the sets of links and joints, respectively. In this research, the rehabilitation exoskeleton used is the Armeo Spring (Figure 2(b)), which is a passive system that supports the weight of the patient’s arm [23] with springs. The Armeo kinematic structure includes rotational joints (equipped with encoders [24, 25]) and prismatic joints (enabling exoskeleton adjustment to the size of each patient). We use the Armeo Spring kinematic model presented in [16], which includes both types of joints (see further details in the Appendix).
3.3. GH Joint Angles Estimation Method
The aim of the method is to estimate the GH joint angles with respect to (w.r.t.) a coordinate system (CS) attached to the scapuloclavicular system. Figure 2(d) shows the proposed system for the GH joint angle estimation. Our approach is based on the estimation of the upper arm orientation w.r.t. the acromion (Figure 3(a)). According to such requirements, the rationale to install the markers of the optical MOCAP is as follows:(a)Marker is rigidly installed in the acromion, so the estimated upper arm orientation can be expressed w.r.t. the CS (and therefore w.r.t. the scapuloclavicular system).(b)Marker is rigidly installed in the upper arm, so that all the rotations of the upper arm are captured by . The region that was chosen to attach to the upper arm by using a custom-made fixation (Figure 2(d)) is the distal part of the humerus (near the elbow). Elbow rotations do not affect the orientation of .
(a)
(b)
Reference [26] reports a five-marker installation procedure. This reference explicitly mentions five markers as an acceptable number for clinical upper limb tracking. In this paper, we report the usage of two markers for upper arm tracking. It is not possible to compare the performance of the marker placement protocol proposed here with the one in [26] because the work in [26] addresses (a) non-RAR scenarios, (b) tracking of the entire upper limb, and (c) protocol sensitivity w.r.t. its application on the dominant/nondominant arm and w.r.t. the age of test subjects. However, the work in [26] helps to establish the number of markers compatible with the clinical application of upper limb tracking.
The cameras of the optical motion capture system are rigidly attached (using custom-made supports) to exoskeleton links so that camera detects marker and camera detects marker during the GH joint training. Camera is mounted on link and camera is mounted on link (Figure 3(a)).
The cameras used in our system are of low cost. Commercial cameras that present similar specifications to the ones simulated here (Table 1) are Intel® SR300 (99 USD) [27, 28], DepthSense® 525 (164 USD) [29, 30], and CamBoard (690 USD) [28, 31].
Figure 3(b) shows an overview of the operation of the estimation method. In order to estimate the upper arm pose, the poses of the markers need to be expressed w.r.t. a common CS. A suitable CS to conduct such estimation is the exoskeleton base.
A summary of the steps to estimate the GH joint angles is as follows:(1)Estimate the pose of the markers w.r.t. the cameras.(2)Estimate the pose of the cameras w.r.t. the exoskeleton.(3)Estimate the pose of the markers w.r.t. the exoskeleton.(4)Estimate the upper arm pose w.r.t. the exoskeleton.(5)Refer the GH joint angles w.r.t. the acromion (marker CS).
The details of the mentioned steps are presented in the following sections.
3.3.1. Estimation of the Pose of the Markers w.r.t. the Cameras
The purpose of this step is to estimate the position and orientation of the markers (Figure 4) w.r.t. the CSs of the cameras using the color and depth images provided by each camera :(A)The RGB image is ( pixels). The pixel coordinates take values and . () contains the RGB color associated with each pixel .(B)The depth image associated with the scene in is ( pixels); and . The pixel coordinates in take values and . The CS of images and is coincident. () contains the () coordinates of the object in each pixel w.r.t. the CS.
The pose estimation of the markers w.r.t. the cameras is based on the reconstruction of the 3D position of the colored disks on the markers. The following steps are taken to estimate the marker pose:
(1)Estimation of disk coordinates in color image (Figure 5): the purpose of this step is to find the approximate coordinates of the centers of the marker disks in image . The following steps are carried out:(a)Color segmentation in image : image regions containing the colors of the marker disks are preserved and the other regions are colored in white. The resulting image is defined as .(b)Blob extraction on image : blob extraction consists of finding the connected regions in the image sharing the same color and labeling them according to their color.(c)Disk center coordinates estimation: for each () blob extracted from , the position of the center of a bounding box for the blob is obtained. This point approximates the actual center of disk (Figure 5). The resulting set of the approximate coordinates of disk centers in is . The center coordinates are referenced w.r.t. the internal image CS. Blobs are extracted with standard connected-component labeling algorithms.(2)Estimation of disk coordinates in the camera CS: this step converts disk coordinates in the internal image CS into the ones w.r.t. the sensor CS, as follows:(a)Convert the positions of the disk centers in set into the image CS. The CSs of images and match. Hence,(b)Compute the indices of the () coordinates of point in array as follows: The point contains the () coordinates of point w.r.t. the CS. The coordinates of point are obtained as follows: The approximate marker disk centers detected by camera form the set .(3)Computation of the marker CS in the camera CS: an coordinate frame is attached to each marker:(a)Make(b)Use the four disk centers in the marker (Figure 5) as follows: The submatrix is normalized to guarantee its nature. The frame describes the estimated pose of marker w.r.t. the CS of the camera .
(a)
(b)
(c)
3.3.2. Estimation of the Pose of the Cameras w.r.t. the Exoskeleton
The goal of this step is to find the transformation , which expresses the pose of the camera w.r.t. the base of the exoskeleton (Figure 6).
The rigid transformation matrices and , which describe the pose of the cameras w.r.t. the CS of the link where they are installed, are estimated during system calibration (the calibration matrix can be obtained by camera detection of a 2D/3D calibration object mounted on a known location of the exoskeleton). The poses and of the exoskeleton links and w.r.t. to the exoskeleton base CS are computed using the Forward Kinematics of exoskeleton . Then, and are estimated as follows:
3.3.3. Estimation of the Pose of the Markers w.r.t. the Exoskeleton
The objective of this step is to estimate the transformation () that describes the pose of marker w.r.t. the exoskeleton base CS (Figure 7). Transformations are estimated as follows:
3.3.4. Estimation of the Upper Arm Pose w.r.t. the Exoskeleton
The purpose of this step is to estimate the upper arm pose () w.r.t. the exoskeleton base CS using the marker poses (Figure 8). The upper arm direction vector is computed from the estimated position of the end-points of the upper arm (GH and elbow joint centers) as follows (CSs in Figure 9):(1)Estimate the position of the GH joint center: the rigid transformation matrix , which expresses the pose of the GH joint CS w.r.t. the CS, is estimated during the calibration process of the system. Hence, the GH joint center is estimated as follows:(a)Estimate , which is the pose of the GH joint CS w.r.t. the exoskeleton base CS (see (8)).(b)Extract from . The point is the position of the center of the GH joint seen from the CS:(2)Estimate the position of the elbow joint center: the rigid transformation matrix (elbow joint CS w.r.t. the CS) is estimated during the calibration process of the system. Hence, the elbow joint center is computed as follows:(a)Estimate , which is the pose of the elbow joint CS w.r.t. the exoskeleton base CS (see (9)).(b)Extract from . The point is the position of the center of the elbow joint seen from the CS:(3)Estimate the upper arm position:(a)Estimate the arm direction vector as .(b)Estimate the origin of the upper arm CS as .(4)Estimate the upper arm orientation: the estimated orientation of the upper arm is computed using Euler angle decomposition w.r.t. the base CS of exoskeleton :(a)Estimate the rotation of the arm around the -axis of the CS using the projection of on the plane of the fixed CS.(b)Compute the rotation of the arm around the mobile -axis of CS from the inner product of with the mobile -axis of CS.(c)Estimate the rotation of the upper arm around its longitudinal axis as the rotation of the marker around vector . This angle is the one between (i) the mobile -axis of CS and (ii) the projection of -axis of marker CS onto the plane of CS.(5)Express the pose of the upper arm w.r.t. the base CS as the rigid transformation .
3.3.5. Refer the Angles of the GH Joint w.r.t. the Acromion
Since is rigidly attached to the acromion, the upper arm orientation can be expressed w.r.t. the acromion by using the inverse of :
3.4. Implementation and Simulation
The arm posture estimation method was implemented by using the V-REP robotics simulator [32]. In the simulator, the scene in Figure 2(d) is created, which includes the models of (a) a human patient, (b) an Armeo Spring, (c) the RGB-D vision sensors with the couplings to attach them to the exoskeleton, and (d) the planar markers with the couplings to attach them to the human arm. The configuration of the simulated vision sensors is summarized in Table 1.
For the estimation of the coordinates of disk centers in the image , color segmentation and blob detection algorithms available in the simulator were used. Additional code was written to sort blob centers by color. All additional code was written in LUA (lightweight embeddable scripting language) scripts.
3.4.1. Generation of the Ground-Truth Poses of the Patient Upper Limb during RAR
The accuracy of the proposed method is determined by comparing its estimations of the upper arm poses with the ones of the simulated human patient (ground-truth values of ). To generate movements of the simulated patient that resemble the ones of therapy, we performed the next steps:(1)Armeo movement generation: we recorded 4 time sequence datasets of the actual Armeo joint measurements (sampled at 66.6 Hz) while performing the following shoulder movements (Figure 10): (a) shoulder horizontal abduction-adduction (SAbAd), (b) shoulder flexion-extension (SFE), (c) shoulder internal rotation (SIR), and (d) a combination of all the mentioned movements (COMB). These movement history datasets are used to guide a simulation of the Armeo model.(2)Patient movement generation: the movements of the patient upper limb that correspond to the recorded movements of the Armeo are computer-generated with the method in [16]. The said method provides an estimation of the patient posture given the joint angles of the exoskeleton by using an inverse kinematics approach.
(a)
(b)
(c)
In this way, four sets (one per movement dataset) of known poses of the upper arm are obtained by simulating patient movement and compared here against those estimated with our method. Our method accuracy is assessed without compensating any time offsets between the reference and estimated angles. In this way, real-time accuracy of the method is assessed. Table 2 presents the approximate amplitudes of the Euler angle decomposition of the GH joint movements of the simulated patient w.r.t. its local CS.
3.4.2. Measurement of the Estimation Performance
(1)Error in the estimation of the markers position: the error in the position estimation of markers is computed as the RMS of expression , where .(2)Error in the estimation of the arm pose: the error in the arm position estimation for a GH joint movement dataset () is computed as the RMS of for all samples in the movement dataset. To quantify the error in the arm orientation estimation (), the next steps are carried out:(a)Compute the matrix of rotation error , where and are the rotation submatrices of transformation matrices and , respectively.(b)Express in exponential map notation [22] as .(c)Compute as the RMS of for all samples in the movement dataset.
3.5. Sensitivity Analysis
A sensitivity analysis is carried out to study the influence of relevant parameters on the method accuracy. Formally, the sensitivity analysis determines the effect of the perturbation of the parameter on the objective function . The relative sensitivity of w.r.t. , , is given by (11) [33]. The value of is the ratio (dimensionless) between the percentual changes in and :
The upper arm pose accuracy (and, therefore, that of the GH joint angles) relies on the precise estimation of the position of the centers of the elbow and GH joints ( and ) (Section 3.3.4), which ultimately depend on the following transformations involving the markers:(a) and (markers w.r.t. exoskeleton).(b) and (GH and elbow joints w.r.t. markers).
The conducted sensitivity analysis focuses on errors in and , given that errors in the estimation of and (Section 4.2) are small. Possible causes of errors in and are as follows:(1)Inaccurate computation of and during the system calibration.(2)Relative displacement of the markers w.r.t. the GH and elbow joints due to skin movement.
In the sensitivity analysis, translations errors in matrices and are induced by disturbing the location of the markers () w.r.t. the CSs of the GH and elbow joints. Since orientation information in and is not used to estimate the upper arm pose, it is excluded from the sensitivity analysis.
For the sensitivity analysis (see (11)), the vector-valued function quantifies the estimation error of the arm position and orientation (see (12)) and the parameter set represents the marker translation errors. The parameter set is defined as , where each is a scalar representing the magnitude of a translation of a specific marker along a prescribed direction. Table 3 describes the meaning of each parameter in set :
The sensitivity analysis procedure (Figure 11) entails the following steps:(1)Load the movement dataset of the GH joint to test (SFE, SAbAd, SIR, and COMB).(2)Select the parameter to perturb (selection of a marker and a direction of translation). Marker translates along axes of the GH joint CS. Marker translates along axes of the elbow joint CS (Figure 12).(3)Apply the translation indicated by to the corresponding marker. The marker perturbation is applied for the complete movement dataset.(4)Compute the estimation errors of the upper arm position and orientation as the simulated patient moves according to the chosen GH joint movement dataset. The current iteration of the process is indicated by index .(5)Compute the position and orientation components of as per (11). The derivative of w.r.t. is given by (13). The required derivatives are computed numerically [34, 35]:(6)Increment by and go to step . Repeat the process until the desired number of iterations of the procedure is reached.
(a)
(b)
The complete sensitivity analysis was performed for each movement dataset (SFE, SAbAd, SIR, and COMB). The directions in which marker translations occur (Table 3) are chosen so that the markers do not leave the detection volume of the cameras. Table 4 summarizes the parameters of the sensitivity analysis. Translation units are in meters (mts).
4. Results and Discussion
This section presents and discusses the results of (a) estimation accuracy of the marker 3D position, (b) estimation accuracy of the upper arm pose, and (c) sensitivity analysis of the estimation accuracy of the upper arm pose w.r.t. translation errors in and .
4.1. Results of Marker Position Estimation
Table 5 presents the RMS of the estimation errors of the position of the markers per movement dataset. The mean RMS errors of the position estimation of and for all movement datasets are 0.00083 and 0.00208 mts, respectively.
Figure 13 shows the box plots of the estimation errors in the marker positions for all movement datasets. A greater variation in the position estimation accuracy of marker , in comparison to that of , is observed. We have attributed this to (a) the higher linear and rotational velocities and likewise (b) the larger translations and rotations that undergoes compared to .
4.2. Results of Upper Arm Pose Estimation
The RMS of errors in the upper arm pose estimation are presented in Table 6. By averaging the results of all movement datasets, errors of 0.00110 mts and 0.88921 deg. in the upper arm position and orientation estimation are obtained. Figure 13 shows the box plots of the estimation errors in the upper arm position and orientation for all movement datasets.
In motor rehabilitation, angular errors in the range of 3–5 degrees are considered acceptable for mobility evaluation of patients [6, 36, 37]. Figure 13 shows that our arm orientation estimation accuracy is adequate for exoskeleton-assisted rehabilitation.
4.3. Results of the Sensitivity Analysis
The results of the sensitivity analysis per movement dataset of the shoulder are presented in Figures 14, 15, 16, and 17. In each figure, the following subfigures are presented:(a)Error in upper arm position estimation () versus total marker translation (): this figure shows the evolution of the absolute error in the upper arm position estimation as the error in the translation components of matrices and increases.(b)Error in upper arm orientation estimation () versus total marker translation (): this figure shows the evolution of the absolute error in the upper arm orientation estimation as the error in the translation components of matrices and increases.(c)Position component of versus total marker translation (): this figure shows the evolution of the relative sensitivity metric corresponding to the error in the upper arm position estimation as the error in the translation components of matrices and increases.(d)Orientation components of versus total marker translation (): this figure shows the evolution of the relative sensitivity metric corresponding to the error in the upper arm orientation estimation as the error in the translation components of matrices and increases.
(a) versus
(b) versus
(c) Position component of versus
(d) Orientation component of versus
(a) versus
(b) versus
(c) Position component of versus
(d) Orientation component of versus
(a) versus
(b) versus
(c) Position component of versus
(d) Orientation component of versus
(a) versus
(b) versus
(c) Position component of versus
(d) Orientation component of versus
4.3.1. Sensitivity in Arm Position Estimation
Regarding the arm position estimation, one can observe that translations of marker produce larger absolute errors than translations of marker . This difference is due to the fact that the translations of produce a larger change in when compared to the one produced by translations of . Note that since is computed by using , any modification in directly affects the accuracy of .
Observing the behavior of the position component of , one can conclude that all translations of the markers and contribute similarly to the error in the arm position estimation. The curves obtained for the position component of resemble a logarithmic function with an asymptote along the value 1 of the ordinate axis. A value of 1 in the magnitude of the position component of means that a percentage change in the magnitude of the marker translation produced the same percentage change (also matching the sign) in the magnitude of the error in the arm position estimation.
4.3.2. Sensitivity in Arm Orientation Estimation
In Figures 14, 15, 16, and 17, one can observe that the translations of marker produce larger absolute errors in the upper arm orientation estimation when compared to those produced by translations of marker . Notice that the -axis and -axis of the elbow joint CS are always perpendicular to the upper arm vector () (Figure 12(b)). When the position of is perturbed along the said axes, the angle between (i) the actual upper arm vector () and (ii) the estimated upper arm vector () (which is inaccurate due to the perturbation of the marker position) is maximal.
A side effect of the marker position perturbation is that the marker suffers modifications of scale and changes in the level of perspective distortion in the images of camera , affecting the accuracy of the system. This situation can be observed in Figures 14(b), 15(b), 16(b), and 17(b), where translations of along the -axis of the elbow joint CS should not produce variations in the orientation estimation error. However, on the contrary, slight variations in the accuracy of the orientation estimation are indeed present in the mentioned figures.
4.3.3. Robustness of the Upper Arm Pose Estimation Method
In Figures 14(c), 14(d), 15(c), 15(d), 16(c), 16(d), 17(c), and 17(d) one can observe that the position component of increases faster than the orientation component of . The behavior of observed remains across the datasets used. Hence, the orientation estimation of the upper arm is more robust than the position estimation w.r.t. errors in the translational components of matrices and .
The results of the sensitivity analysis show that the assumption that transformations and are rigid is reasonable. Even with marker drifts of 0.02 mts, the GH joint angles can be estimated with an accuracy (RMSE 3.6 deg.) appropriate for the mobility evaluation of patients (in the range of 3–5 deg.).
Marker drifts must be mitigated by the marker attachments to the human body. Furthermore, marker attachments should be designed to minimize the effect of errors in and on the method accuracy. For example, notice how the attachment of marker (Figure 12(b)) locates marker with an offset w.r.t. the elbow joint center along the direction which least affects the upper arm orientation estimation.
The results presented suggest that the method we implemented is a feasible alternative for estimating the GH joint angles in a RAR scenario.
4.4. Comparison to Related Works
The literature review provided no references other than [16–20] for upper limb posture estimation (including the GH joint) in exoskeleton-based rehabilitation using computational methods. Among the mentioned works, only [20] reports the errors (mean RMSE 4.8 deg.) in the GH joint angles estimation. Reference [20] reports RMSE values of the GH joint angles only for the best-case scenario (swivel angle mean RMSE 5 deg.). For all the movement tasks tested, the method in [20] presents a mean RMSE of 10 deg. for the swivel angle estimations. Given that global errors of the swivel angle double those of the best-case scenario, a report of global errors of GH joint angle estimations of the method in [20] is required to reach a conclusion regarding its suitability for clinical use.
Table 7 summarizes the comparison of our contributions w.r.t. comparable works (i.e., [20]).
5. Conclusions and Future Work
In the context of RAR, this paper presents the formulation, implementation, and assessment, in silico, of a novel accurate method to estimate the patient GH joint angles during therapy. Our method does not require redundant markers or cameras and relies on simple geometric relationships and tools of standard robotics and computer vision libraries. These characteristics make it economical and readily applicable in RAR.
The accuracy and the robustness of our method are evaluated using computer-generated human movement data corresponding to actual movement datasets of the Armeo Spring. We present a formal sensitivity analysis of the pose estimation accuracy w.r.t. marker position estimation errors produced by (a) system calibration errors and (b) marker drifts (due to skin artifacts). This analysis indicates that even in the presence of large marker position errors our method presents an accuracy that is acceptable for patient mobility appraisal.
Future work includes (a) implementation of the method using commercially available RGB-D vision sensors, (b) evaluation of the method accuracy with actual human movement data, (c) adaptation of the method using solely RGB cameras, and (d) extension of our method to address other limbs.
Appendix
Problem Statement
Given. Consider the following:(1)A human patient upper body with a kinematic model (Figure 2(a)). Consider the following remarks:(a)The model is a simplified version of the spine, arm, and scapuloclavicular systems. However, since we focus on the study of the upper limb, we only describe in detail the kinematic model of the said limb.(b)The set of links is , containing the sternum, clavicle, upper arm, forearm, and hand ().(c)The set of joints is , containing the sternoclavicular, GH, elbow, and wrist joints.(i) denotes the number of DOFs of . , 2, or 3 .(ii) is an -tuple whose th component is the angle of the th DOF of joint th, .(iii) is the index of the GH joint . since the GH joint has 3 DOFs. is the 3-tuple containing the values of the DOF of the (GH) joint.(iv) registers the status, at time , of the DOF of the GH joint.(d) is an open kinematic chain, and, therefore, and are connected by joint .(2)An exoskeleton with a kinematic model , which is attached to the patient’s limb and assists the patient when performing rehabilitation exercises (Figure 2(b)). Consider the following remarks:(a)The set of links is .(b)The set of joints is .(i) denotes the number of DOFs of .(ii) is a -tuple whose th component is the angle of the th DOF of joint th, .(c) is modeled as an open kinematic chain, and, therefore, and are connected by joint .(d)The -tuple () contains the set of independent coordinates which uniquely defines a configuration of .(i)Also.(ii) registers the state, at time , of the DOF of , which is known .(e)The exoskeleton may be configured to impose specific motion constraints on the patient by blocking specific joints of the set.(3)A marker-based optical tracking system composed of two RGB-D cameras and two planar markers (Figure 2(c)). Consider the following remarks:(a)A set of planar markers that are detected by the cameras of and are installed on the patient upper limb.(i)All present the same 2D square geometry, with a disk in each corner. The position of each disk w.r.t. the marker CS is known. The set of disks is .(A) presents a color that can be detected by (Figure 2(c)).(B)The set of colors of the disks mounted on each is . Each is represented with a RGB color code.(C).(ii) is mounted on the acromion with a 0-DOF coupling (Figure 2(d)). A rigid transformation matrix defines the relative position and orientation of the GH joint CS w.r.t. the CS of .(iii) is mounted on the upper arm with a 0-DOF coupling (Figure 2(d)). A rigid transformation matrix defines the relative position and orientation of the elbow joint CS w.r.t. the CS of . Note that, to compute the GH joint angles, the calculation of the elbow joint angles is not necessary with this setup.(iv)The rigid transformation matrices and are estimated during the calibration of the system.(b)A set of low-cost cameras is installed in the exoskeleton.(i) is mounted on exoskeleton link with a 0-DOF coupling, such that the disks on are inside its detection volume during the rehabilitation exercises. The rigid transformation matrix defines the relative position and orientation of the CS of w.r.t. the CS.(ii) is mounted on the exoskeleton link with a 0-DOF coupling, such that it can detect the disks on (see Figure 2(c)). The rigid transformation matrix defines the relative position and orientation of the CS of w.r.t. the CS.(iii)The rigid transformation matrices and are estimated during system calibration.(iv)Remarks on each camera are as follows:(A) renders RGB image of pixels. The pixel coordinates take values and .(B) renders a depth image associated with the scene in , defined as , of pixels; and . The pixel coordinates in take values and . The CS of images and is coincident.(C) presents a truncated square pyramid detection volume parametrized by the minimum and maximum detection distances and the horizontal and vertical field of view of . Table 1 presents the model features of the vision sensors that have been used for the simulations.(v)The system of cameras produces the following array sequence of each :(A) () contains the RGB color associated with each pixel .(B) () contains the () coordinates of the object in each pixel w.r.t. the CS.
Goal(1)Find the values of , which approximates such that is minimum .(a) is the Euclidean norm of vector .
Glossary
| Acromion: | Region of the scapula bone above the GH joint |
| Clavicle: | Bone of the shoulder girdle located at the root of the neck |
| CS(s): | Coordinate system(s) |
| COMB: | Combination of movements of the GH joint (SAbAd, SFE, and SIR) |
| DOF(s): | Degree(s) of freedom |
| GH: | Glenohumeral |
| Humerus: | Upper arm bone |
| MOCAP(s): | Motion capture system(s) |
| mts: | Meters |
| RAR: | Robot-Assisted Rehabilitation |
| RMS: | Root mean square |
| Scapula: | Bone that connects the humerus to the clavicle |
| SAbAd: | Shoulder horizontal abduction-adduction |
| SFE: | Shoulder flexion-extension |
| SIR: | Shoulder internal rotation |
| VR: | Virtual Reality |
| V-REP: | Virtual Robot Experimentation Platform |
| w.r.t.: | With respect to |
| : | Exoskeleton kinematic model |
| : | Human upper body kinematic model |
| : | Set of planar markers mounted on the patient |
| : | Position of the GH joint w.r.t. the CS |
| : | Position of the elbow joint w.r.t. the CS |
| : | Set of vision sensors that compose the optical MOCAP |
| : | 3-tuple of joint angles of the GH joint at instant |
| : | Tuple of joint angles of the exoskeleton kinematic model at instant |
| : | Transformation matrix of marker w.r.t. the base CS |
| : | Transformation matrix of marker w.r.t. the CS |
| : | Transformation matrix of the GH joint w.r.t. the marker |
| : | Transformation matrix of the elbow joint w.r.t. the marker |
| Notation : | can be a position, transformation, and so forth, of object w.r.t. object CS. |
Competing Interests
The authors declare that there are no competing interests regarding the publication of this paper.
Acknowledgments
This research is a part of the HYPER Project funded by CONSOLIDER-INGENIO 2010, Spanish Ministry for Science and Innovation.