Abstract
A low-cost optical fiber force myography sensor for noninvasive hand posture identification is proposed. The transducers are comprised of 10 mm periodicity silica multimode fiber microbending devices mounted in PVC plates, providing 0.05 N−1 sensitivity over ~20 N range. Next, the transducers were attached to the user forearm by means of straps in order to monitor the posterior proximal radial, the anterior medial ulnar, and the posterior distal radial muscles, and the acquired FMG optical signals were correlated to the performed gestures using a 5 hidden layers, 20-neuron artificial neural network classifier with backpropagation architecture, followed by a competitive layer. The overall results for 9 postures and 6 subjects indicated a 98.4% sensitivity and 99.7% average accuracy, being comparable to the electromyographic approaches. Moreover, in contrast to the current setups, the proposed methodology allows the identification of poses characterized by different configurations of fingers and wrist joint displacements with the utilization of only 3 transducers and a simple interrogation scheme, being suitable to further applications in human-computer interfaces.
1. Introduction
The design of intuitive and feasible sensors and techniques for hand posture detection plays an important role on the establishment of advanced human-computer interfaces, with further applications in teleoperation [1], medical [2] and rehabilitation robots [3], bionic prosthesis control [4], and gesture recognition for consumer electronics [5].
Nowadays, different approaches have been developed for the direct assessment of the hand movements, mainly comprising optical tracking [6] and glove-based sensors [7]. Even though these technologies can provide reasonable results for practical applications, limitations are still observed concerning occlusion, complexity, and restrictions to the user movements [3, 6, 7].
Alternatively, myographic techniques can be utilized for monitoring the hand gestures and forces in a noninvasive way by detecting the muscular activities on the user forearm, so it is also possible to retrieve the motion intentions in case of disabled subjects. The surface electromyography (sEMG) is the most widespread approach for pose estimation [8], in which the electrical signals produced during muscle contractions are recorded by electrodes placed over the user skin and then processed to determine the related hand movements. Although several works have reported the application of sEMG on prosthesis control [9, 10], such technique still presents limitations in terms of cost and reliability. The electromyography signals are severely affected by electromagnetic noise, sweat, and variations of transducer placement. Moreover, the application of feature extraction procedure over a large number of channels is required in order to decode the subject motions with a suitable dexterity level [11, 12]. In this context, different methods such as the mechanomyography [13], sonomyography [14], optical myography [15, 16], and the force myography (FMG) [17] have been proposed in order to overcome the drawbacks presented by the myoelectric case.
The FMG can be considered as the mechanical counterpart of the sEMG, in which the hand movements are retrieved from the radial forces exerted by the forearm muscles, producing results comparable to the electrical approach [18]. In contrast to other techniques, the FMG is less affected by variations in sensor placement, exhibiting higher robustness and stability, and does not demand expensive hardware for implementation. The mechanical stimuli are typically measured using arrays of force sensing resistors (FSR) or capacitive sensors attached to the user forearm by means of orthoses or straps [18, 19], providing the pressure distribution maps for particular hand configurations. However, such transducers exhibit drawbacks in terms of hysteresis, drift, and vulnerability to electromagnetic interference [20], being also affected by skin impedance variations.
In this context, optical fiber sensors (OFS) are considered promising alternatives for biomechanical measurements, presenting intrinsic characteristics like flexibility, lightweight, high sensitivity, distributed sensing capability, and immunity to electromagnetic interference. Furthermore, silica fibers are chemically and biologically inert [21]. Currently, several examples of OFS applications in human-system interfaces have been reported, including vital sign monitoring for healthcare [22], tactile matrixes [23], and glove-based sensors [24]. Moreover, the utilization of an optical fiber sensor for FMG measurements was presented in recent studies, providing 89.9% accuracy on the prediction of individual finger movements [25].
In this paper, an intensity-based optical fiber FMG sensor for posture identification is proposed. The optomechanical transducers are attached to the forearm by means of straps to measure the forces produced by forearm muscles, so the output signals are correlated to the hand poses by artificial neural networks (ANN). The utilization of optical fiber sensors provides several advantages over the commonly adopted FSR arrays, yielding higher sensitivity and immunity to sweat and electrical noise. Additionally, in contrast to the previously reported fiber specklegram FMG sensors [25], the present setup shows improved robustness (as the guided light is less susceptible to vibration and temperature fluctuations [26]) and a simpler interrogation setup (since it is not necessary to process the spatial information of speckle fields [25]) being suitable for further applications in practical bionic prosthesis control.
2. Optical Fiber FMG Sensor
2.1. Sensor Design
The optomechanical transducer (Figure 1) is comprised of a pair of 60 × 10 × 1 mm3 polyvinyl chloride (PVC) plates with 10 mm length, 0.5 mm diameter graphite rods fixed to the internal faces in equally spaced distribution (10 mm periodicity) for assembling the deformer structure. The transducer is attached to 2 m length, 62.5 μm diameter silica multimode fiber, causing periodical mechanical disturbances along the waveguide length. According to the microbending effect, the consecutive variations on the bending radius produce the coupling between the core-propagating modes and the cladding radiation modes [27], causing light attenuation that can be modulated by the magnitude of the applied forces and displacements. Therefore, the device is attached to specific locations of the user forearm by means of Velcro straps in order to assess the muscular activities.
(a)
(b)
The measurement setup is shown in Figure 2. The continuous light emitted by a white LED source is coupled into the multimode fibers using a micrometric stage. The transmitted light intensity is modulated by the microbending transducers, being further detected by a CCD camera (u-EYE IDS UI-2230SE-C-HQ, 1024 × 768 pixels resolution, 15 fps rate). The plane-polished fiber end faces are aligned, fixated in a stable support, and placed 5 cm from the camera lens. Then, the setup is enclosed in a dark box for eliminating the external illumination effect and kept in an environment free from mechanical vibrations. An additional reference fiber is also utilized for compensating the source fluctuations. Finally, the acquired optical signals are processed by routines programmed in MATLAB, MathWorks, and the average intensity I for each frame is evaluated by computing the mean of a 40 × 40 pixels region-of-interest within the fiber output light spots.
2.2. Characterization
In order to determine the sensor response, the device was subjected to controlled forces applied by a mechanical stage provided with a load cell. The experimental setup for transducer characterization is shown in Figure 3. The actuator probe is comprised of a 2.5 mm diameter spherical-shaped structure driven by a screw mechanism, which is aligned to the deformer plate center to produce a concentrated load. As the micrometric stage is activated, the actuator position is displaced in the y-direction, causing the transducer to be pressed and producing optical attenuation. Then, the mechanical stimulus is measured by the load cell, yielding the applied force value, whereas the output light is continuously monitored by the CCD, making it possible to correlate the average intensity I with the input forces.
The static characterization was carried out by subjecting the microbending transducer to 3 upscale and downscale cycles, with the application of loads ranging from 0 to ~22 N. Once the actuator position is changed, the force level is kept static for ~100 acquisition frames. Finally, the calibration curve can be obtained by averaging the intensity values for each constant force interval regarding all upscale/downscale cycles, and the sensitivity is determined by adjusting the experimental data by a linear function.
The sensor static response is shown in Figure 4, whereas the variation of I for repeated loading cycles is illustrated in Figure 5. The device presented ~0.05 N−1 sensitivity over the investigated range. In comparison to FSR [20], the fiber sensor exhibited negligible linearity error with low hysteresis, being suitable for assessing the force magnitudes typically exerted by forearm muscles and repetitive cycles of muscular stretching and contraction [28]. In spite of the reliable results, it is worth noticing that the transducer response may be degenerated by changes on the deformer structure in case of severe loading conditions, whereas the intensity output values can be affected by variations on fiber mechanical characteristics and macrobending [27].
3. Posture Identification by FMG
3.1. Measurement Protocol
The sensor performance was evaluated on the identification of 9 postures, as shown in Table 1 and Figure 6. For retrieving the information relative to each pose, the system was implemented using 3 transducers located to monitor the posterior proximal radial (T1), the anterior medial ulnar (T2), and the posterior distal radial (T3) sections of the forearm. In practice, as the muscles are presented in layered structure [29], the measured mechanical stimuli will be equivalent to the overall radial forces produced by the muscles enclosed inside each strap rather than the activation of a single tissue.
The experiments were conducted for 6 healthy subjects (5 male and 1 female, 29 ± 4 years old, namely, S1 to S6) following the Ethical Committee recommendations, by tightly adjusting the straps to create a moderate preload, but preserving the comfort for the user. The transducer locations were determined by muscle palpation, so eventual variations in the placement of the sensors for the different subjects are expected. During the tests, the volunteers were requested to keep the wrist with neutral supination/pronation and the elbow flexed to maintain the forearm horizontally suspended in air. After receiving a visual command, the users performed a sequence of hand postures () by sustaining the static hand condition during ~100 acquisition frames. Finally, the acquired data for each pose were windowed to remove the transient components of intensity signals (due to hand configuration changes) and then reduced by computing the average value for each time interval. The experiments were repeated 10 times for each subject, with simultaneous acquisition of optical signals and further correlation to the input postures through time synchronization, that is, since each pose was performed during particular frame intervals, the corresponding intensity values can be properly retrieved by indexing the initial and final frame numbers regarding each posture in the acquired optical data array.
3.2. Data Analysis
In order to retrieve the performed poses from the FMG data, an ANN classifier was designed by addressing the acquired average intensity values to the network input, and the degree of target posture matching (ranging from 0 to 1) to the output, as illustrated in Figure 7. For the sake of simplicity, the classifier was developed using individual networks for processing each posture, yielding 9 subsystems integrated by a competitive layer. The ANN were implemented according to the feed-forward backpropagation architecture [30], utilizing 5 hidden layers of 20 neurons and tangent-sigmoidal transfer functions, whereas the training function was set to the scaled conjugate gradient limited to 3000 epochs and 0.05 target error. Subsequently, the output matching rates are submitted to a winner-take-all competitive layer, returning the final classification estimated from the FMG input signals.
For evaluating the network response, the classifications were carried out for each subject following a 10-fold cross validation analysis [31]. Given the number of true positive (TP), true negative (TN), false positive (FP), and false negative (FN) predictions, the performance metrics for a single fold can be expressed in terms of [32] where tp is the true positive rate (sensitivity), fp is the false positive rate, and ac is the accuracy. Therefore, the scores for a particular posture can be obtained by computing the average of the 10 folds, whereas the overall performance can be estimated from the average for all volunteers.
3.3. Results
The effect of hand pose on the transducer response for a single subject (S3) is shown in Figure 8, where I1, I2, and I3 correspond to the normalized intensity values obtained by transducers T1, T2, and T3, respectively. The transient parts of the curves are due to the changes in hand configuration during posture transitions. Additionally, the static response was found to be practically constant, with subtle variations possibly caused by involuntary contractions, dither, and fatigue [18].
Regarding the average results for all volunteers (Figure 9), it is observed that the optical attenuation for transducer T1 increases with the wrist and finger flexion, which can be associated to the activation of the flexor digitorum superficialis (FDS) and flexor digitorum profundus (FDP) muscles, whereas I2 and I3 present improved response to sense the joint extension due to the contribution of the extensor digitorum communis (EDC). For example, the clench poses B, F, and H, as well as the wrist flexion D, cause I1 to decrease, and I2 and I3 to increase, indicating the activation of flexors and relaxation of the extensor muscles, respectively. Conversely, performing open hand pose with finger abduction/adduction conditions or wrist extension produces attenuation of I2 and I3 and recovering of the I1 level. Concerning the abduction/adduction movements, the thumb action can be recognized from I3 value, as the transducer T3 was located close to the abductor/extensor pollicis (AEP) muscles, whereas the effect of other digits is detectable through T2 response, as noticed in postures E and G, given that the finger abduction/adduction are assisted by the EDC.
In addition to the physiological differences among subjects, the results may vary due to the sensor fixation and placement, affecting the preload and the pressure distribution over the microbending device. Moreover, even though the experiments were conducted under a strict protocol, the measurements are also expected to change with the magnitude of grip forces, as well as the degree of forearm flexion and wrist rotation. Particularly, it is relatively difficult to stipulate an exact condition for posture N (relaxed hand), so the deviations on intensity values for this case were observed. Additionally, one may notice that the system presented reproducible results considering all subjects except for the I2 value, pose C, yielding large deviation. This effect can be attributed to uncontrolled variations in the degree of wrist extension, as well as on the magnitude of forces involved in such posture. Another possible aspect is the involuntary finger abduction during the wrist movements since the EDC is also related to the abduction/adduction of the metacarpophalangeal joints.
The muscular structure of the forearm is very complex, being comprised of extensors and flexor entities arranged in a layered fashion. Although the FMG sensor was designed to monitor the muscles responsible for finger and wrist movements, structures related to the elbow flexion and forearm pronation, such as the brachioradialis and the pronator teres, respectively, also produce mechanical stimuli due their activation [29], affecting the magnitude and range of the FMG signals related to specific hand poses. This fact can be confirmed by sEMG-based studies, in which the experiments were conducted for more complicated scenarios, including sequential arm movements and application of mechanical loads, so the overall performance was deteriorated due to changes on upper limb configuration [33]. Possible solutions for such problem comprise extracting additional information from the output waveforms or eventually attaching complementary transducers in order to provide a complete characterization of the assessed poses.
Regarding the effect of applied forces, subtle variations were expected because of fatigue and subject-related uncertainties. With respect to the correlation between muscular forces and the upper limb configuration, it was reported that the electrical response in sEMG schemes can be affected by the muscle length and elastic properties, fatigue, contraction velocity and type, and the contribution of tendons and ligaments [34]. Moreover, the magnitude and range of grip forces are also influenced by the forearm and wrist configurations [35].
The ANN classification results for all postures and subjects are summarized in Table 2, whereas the confusion matrixes are shown in Figure 10. The system provided a reliable overall performance, yielding 98.4 ± 1.7% sensitivity and 99.7 ± 0.4% average accuracy. In particular, subjects S3 and S4 presented a perfect score, whereas the poorest results were observed for subject S1, resulting 98.9 ± 1.2% accuracy, which was probably caused by errors on the transducer placement or application of excessive forces during finger flexion on postures F, G, and H. Most of misclassifying occurred for gestures A and E, comprising the open hand case with the presence or absence of finger abduction/adduction, respectively. As aforementioned, the digit separation produced by metacarpophalangeal (MCP) joints can be retrieved from the EDC muscle, but such movement is bounded to the extension of proximal interphalangeal (PIP) joints, so the differentiation between postures A and E becomes very subtle. Fortunately, the thumb abduction activates the AEP muscle; therefore, most of the ambiguities can be solved based on I3 value, as also observed for poses B and H. Moreover, it is worth noticing that the wrist extension (C) and flexion (D) were successfully discriminated from the finger movement (A and B, resp.), although such displacements are partially driven by the same group of muscles (EDC and FDS, resp.). A possible explanation is given by the differences in force magnitude and the number of activated muscles involved in each situation since the optical fiber sensors actually measure the overall forces exerted by all enclosed muscles. The classifications were also succeeded concerning postures F and G, characterized by the same hand configuration except by the ring finger flexion or extension, respectively, demonstrating the proposed technique capability to track the movement of individual digits. Finally, one may observe that the overall sensitivity for pose N was reduced due to the variations on hand neutral configuration, as previously explained.
A compilation of different approaches for assessing the hand gestures by myographic techniques is summarized in Table 3. It must be stressed that the purpose of Table 3 is merely informative, as the direct comparison between techniques should consider experiments conducted under the same conditions and analyzed poses.
The utilization of sEMG on the identification of hand postures had been proposed in several works, as observed in [36], which is based on 2 electrodes applied in conjunction with a linear Bayesian classifier to evaluate 16 hand configurations, providing ~85% average accuracy. Another research demonstrated the combination of 5 sEMG electrodes and a 3D accelerometer to identify 18 postures, including dynamic gestures [37]. By applying a hidden Markov model-based data fusion technique, it was possible to obtain a 97.6% average accuracy. In contrast to the mechanical analysis, as the electromyographic signal processing requires feature extraction from noisy data, it is necessary to improve the detection scheme and increase the number of measurement channels in order to achieve higher hit rates.
A detailed comparison between sEMG and FMG was also presented in [18], in which the electrical measurements were carried out with 10 electrodes arranged along a strip, whereas the muscle mechanical forces were obtained using 6 FSR attached to an orthosis. The system was tested for recognizing the flexion of individual fingers with ridge regression method, yielding normalized root-mean-square errors (nRMSE) ranging from 10 to 20%. In spite of the comparable performances, the sEMG electrodes are typically more expensive and less robust than the FSR, making the mechanical approach more suitable for consumer applications. Moreover, the FMG do not demand the waveform feature extraction for pose classification, as shown in the present research.
Concerning the FMG implementations, in [19], an 8 FSR strap was utilized for discriminating among 6 gestures, including the elbow flexion and wrist rotation. Based on an electronic learning machine classifier, the methodology provided 92.33% accuracy with real-time performance. Alternatively, the identification of 17 hand postures comprising combinations of flexed fingers was proposed in [12]. The system was designed using a 32 FSR socket and signal processing by support vector machines, yielding >99% accuracy. However, such methodologies require an excessive number of piezoresistive transducers and signal channels, making their implementation more complicated in comparison to the proposed work, in which the measurements can be accomplished using only 3 transducers.
Finally, with respect to the previous version of the optical fiber FMG sensor [25], the measurements were carried out using 4 microbending transducers based on specklegram interrogation and ANN processing. An 89.9% accuracy was achieved regarding the identification of individual finger flexion among 11 gestures, but the acquired speckle fields are strongly affected by mechanical and thermal disturbances along the fiber length, so the system must be periodically recalibrated. Moreover, the presented intensity-based approach is more robust and can be implemented with an even simpler setup.
Although the optical fiber FMG sensor presented promising results, some aspects must be considered regarding the analysis of a more comprehensive collection of postures. For example, the identification of the angular displacements performed by a specific finger joint would demand the utilization of additional sensors and a more intensive calibration, since the individual movements of a digit are usually bounded to different muscles. Additionally, due to the layered structure of the forearm flexors and extensors, the activation of each muscle cannot be retrieved in separated form by using FMG transducers, so it is necessary to combine the information from several sensors in order to correctly estimate the hand movements. Fortunately, applications related to prosthesis control and human-computer interaction can be implemented in an efficient way according to event-driven finite state machines (EDFS) [38]; therefore, the multiple manipulator actions can be triggered by a limited set of input hand postures.
Finally, it is worth noticing that calibration must be repeated whenever the transducers are attached to the user forearm since the sensor characteristics can be affected by their location and the applied preload level. In this context, an interactive interface for sensor calibration could be implemented, in which the user is requested to perform the predefined gestures for generating the database applied on ANN classifier training. Once this preliminary step is accomplished, the FMG sensor can be utilized for retrieving the hand gestures until the transducers are unclothed or the system is turned off. In order to make this procedure less tedious, one may abbreviate the processing time by early stopping the training routine or reducing the number of classes (postures), which is applicable to EDFS control approaches.
4. Conclusions
The optical fiber FMG sensor was successfully applied on the identification of hand postures based on a simple and low-cost apparatus. The analysis of 9 gestures was carried out with 98.4% sensitivity and 99.7% accuracy regarding the average of 6 subjects, being comparable to sEMG setups. In contrast to other FMG approaches, which are based on FSR arrays, the proposed methodology was implemented using only 3 transducers, and without requiring the application of signal feature extraction procedures. Moreover, in comparison to the previous fiber specklegram sensor setup, the current system is more robust and simpler to implement, as the optical signals are less affected by mechanical vibrations and temperature fluctuations, and the data can be processed based on the average intensity values; thus, it is not necessary to compute the spatial information from the output speckle fields.
In spite of the promising results, additional developments may be necessary in order to detect the individual hand joint movements, as well as to compensate the effect of elbow flexion, forearm supination, and the magnitude of grip forces during the accomplishment of movements, causing deviation of the sensor calibration curve.
Currently, the application of the presented technology on the control of multifingered bionic prosthesis is under development [39]. According to the preliminary studies, the optical fiber FMG sensor system is capable to provide a real-time response on the identification of 4 key postures, being utilized for switching the manipulator behavior according to an EDFS machine. In this context, further researches will be focused on the improvement of the FMG sensor reliability and the design of practical implementations for this technology in the field of human-robot interaction.
Conflicts of Interest
The authors declare that there is no conflict of interest regarding the publication of this article.
Acknowledgments
This work is supported in part by Sao Paulo Research Foundation (FAPESP) under Grant no. 2014/25080-0, in part by Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), and in part by Coordenacao para a Melhoria Pessoal do Ensino Superior (CAPES).