Abstract
This paper proposes a portable system for hand motion identification (HMI) using the features from data glove with bend sensors and multichannel surface electromyography (SEMG). SEMG could provide the information of muscle activities indirectly for HMI. However it is difficult to discriminate the finger motion like extension of thumb and little finger just using SEMG; the data glove with five bend sensors is designed to detect finger motions in the proposed system. Independent component analysis (ICA) and grey relational analysis (GRA) are used to data reduction and the core of identification, respectively. Six features are extracted from each SEMG channel, and three features are computed from five bend sensors in the data glove. To test the feasibility of the system, this study quantitatively compares the classification accuracies of twenty hand motions collected from 10 subjects. Compared to the performance with a back-propagation neural network and only using GRA method, the proposed method provides equivalent accuracy (>85%) with three training sets and faster processing time (20 ms). The results also demonstrate that ICA can effectively reduce the size of input features with GRA methods and, in turn, reduce the processing time with the low price of reduced identification rates.
1. Introduction
Electromyography is a technique for evaluating and recording the activation signal of muscles including human hand and arm motions. The applications of hand motion identification (HMI) have been widely used for prosthetic control, remote control of home appliances, human computer interaction (HCI), sign language recognition (SLR), and systems for constructing virtual reality [1–5]. In engineering applications, users can control robotic manipulators using an HMI system; thus, manipulators with little strength could accomplish heavy tasks and could be protected from direct contact with dangerous objects or surroundings [6]. In medical applications, HMI system could be employed to help patients with amputated hands due to accidents and diseases. Therefore, applications of HMI contribute both to the engineering and to clinical rehabilitation fields.
Many automatic schemes have been proposed for HMI [7–11]. One method is image processing, in which motion images are captured with a camera, and features extracted during the process are identified. However, recognition accuracy can be adversely affected easily by a shaking head, insufficient light, the extraction angle, and the distance between camera and hand [7]. Another method is surface electromyography (SEMG). Neural impulses from the spinal cord arrive at the motor unit, causing muscle contractions and generating an EMG signal [8]. Notably, SEMG is a noninvasive method for detecting directly or indirectly muscle or nerve information such as neural activities and muscle force or fatigue; SEMG has a wide range of applications such as in HMI systems. However, SEMG is susceptible to noise effects and detecting deep muscle activity is difficult, thereby limiting the performance of an HMI system. Additionally, differentiating SEMG signals from many similar or weak signals from deep or far muscles groups is difficult, such as identifying finger movements by detecting arm signals. Therefore, there are some studies tried to improve the accuracy by combining other techniques like three-axis accelerometer (ACC) [9], bend sensors [10], or multiple sensors [11].
Using a data glove which is a device with embedded sensors for identifying finger movements is a reliable method. When a finger or hand moves, the data glove detects the relative signal change for HMI. Tidwell et al. (2013) developed an HMI system with SVM model using data glove and SEMG sensors to complete a verbal communication system [10]. The accuracy of their system in 10 hand motions is achieved as 93.33%. However, the potential drawback of using multiple sensors is that it dramatically increases computation load, especially when trying to identify more hand motions. Therefore, in this study, we used five bend sensors mounted on a data glove with bend sensors as signal sources for finger motions identification. Also, the SEMGs were measured by a multi-channel electrode scheme (MES) that has three active electrodes and is placed around the forearm to acquire the SEMG signals for identification of forearm motions. Because SEMG could detect the information of muscle activities and force directly [12, 13], force feedback was also considered in proposed system.
Independent component analysis (ICA) has been demonstrated that could successfully isolates power-line components from SEMG signals and decomposing SEMG signals for obtaining more information of the active motor units from specific muscle groups. It could use to reduction of required computation data for identification hand [12, 14, 15]. It could not only reduce the amount of required computation data efficiently, but also lower effect of reduced identification rates. Additionally, many core methods are proposed for signal identification in previous studies such as back-propagation neural network (BPNN), grey relational analysis (GRA), support vector machine (SVM), and log-linearized Gaussian mixture network (LLGMN) [8, 12, 15–17]. The time consumption of signal identification in real-time systems is also a concern. However, neural networks based methods often require considerable training time, especially when the program that reaches convergence is difficult. Therefore, as a classifier in this study, GRA has often been used to investigate problems with limited amounts of training data [18]. It provides the applications of relational analysis and prediction for a real-time decision system. For an adaptation application, it has a flexible pattern mechanism and can reduce training data without adjusting any parameters. In order to achieve the larger amount of hand motions and speed up the processing time, the goal of this study is to establish a new method combining ICA and GRA in portable system for HMI to recognize twenty various hand motions and, at the same time, without sacrificing the accuracy.
2. Proposed HMI System
Five channels of bend sensors in the data glove and three channels of SEMG sensors in the MES are used to detect finger and forearm motions. The acquisition component of the proposed HMI system is described below.
2.1. SEMG Signal Acquisition
The SEMG signals were measured with a MES composed of three active electrodes (BioEngineering SenseTech Corp., Taipei, Taiwan). Each electrode has two parallel conductive bars to detect the SEMG along the muscle fibers. The specifications for the active electrodes are as follows: gain, 520; common-mode rejection ratio (CMRR), 96 dB; bandwidth, 4.7 Hz to 1 kHz; quiescent current, 332 A; and DC input impedance, 6 Giga-ohms. The electrodes were placed on the skin surface of the brachioradialis, flexor carpi radialis, and pronator teres. The electrodes are held in place with a Velcro strap.
2.2. Data Glove Signal Acquisition with Bend Sensors
The data glove was developed to measure finger motion. The sensors selected for attachment to the glove are bend sensors, which are of low-cost and are highly sensitive (Images SI, Inc., NY, USA). These sensors are made with the same principle as strain gauges; the bend sensor changes resistance when bent. The resistance varies from about 10 k ohms to 40 k ohms over 0-degree to 90-degree bend angles. The sensors are inserted into five appropriately sized pockets sewn onto each finger of the glove. This design allows the sensors to be firmly and easily attached to the glove and allows for subsequent removal without damage. The analog signals from the MES and data glove were input to a 16-bit A/D converter (NI DAQ Card-6036E, Austin, TX, USA). The SEMG and data glove signals were sampled at 2 kHz. A black box mounting on the A/D card was utilized for transmitting signals and power (DC: 5 V) to the active electrodes and flex sensors.
3. Data Analysis
After signal acquisition, motion intervals were detected from the SEMG signals. Motion intervals define the time intervals for subsequent analysis of both SEMG and data glove signals. Then, preprocessing and feature extraction were applied within the identified time intervals. The features from the SEMG and data glove signals were combined in a single feature pattern. Finally, the feature patterns were input to a trained ICA and GRA-based classifier. Data analysis is described in the following sections.
3.1. Motion Interval Detection
The SEMG signals are responsive to hand and finger motions, while the bend signals respond only to finger motion. Therefore, SEMG signals are used to define the motion intervals by the mechanism for real-time detection of hand motion intervals. When the subject moved, the operator used different muscle groups for various hand motions. Thus, SEMG signals detected by individual active electrodes varied in intensity. The energy distribution of each channel was then calculated, minus the mean value of channels, and summed to get as (1). When the value exceeded a prespecified threshold , signal capture was initiated. Conversely, signal capture was stopped when the value fell below the threshold [12]. Considerwhere is SEMG channel ; is the number of SEMG channels; is the number of samples in the input record; and is the mean value of the channel , measured over samples while the arm was relaxed. Figure 1 shows a set of signal data and the detection of motion intervals from the energy distribution.
3.2. Feature Extraction
Feature processing is necessary during signal identification. Effective feature processing not only reduces the amount of data, but also increases recognition accuracy. In [8, 12, 16, 19], researchers often used time-domain and spectral estimation to extract the features from SEMG signals. This study uses only time-domain methods to extract all features. Features are computed from the signals only over the times corresponding to the detected motion intervals. Signal data obtained from the SEMG and flex sensors can be represented as and , , respectively. For each channel of SEMG, six parameters were extracted by the integral of EMG (IEMG), waveform length (WL), average total power (ATP), zero crossing (ZC), slope sign change (SSC), and Willison amplitude (WAMP) [8, 12]. Three parameters defined in this study were extracted from bend sensors in the glove, including area, length change (LC), and minimum value (MV). All the methods are defined as follows.
(i) Integral of EMG (IEMG). The IEMG can be treated as a signal power estimator and can be defined as
(ii) Waveform length (WL). The WL can estimate the complexity of the SEMG waveform corresponding to the duration of the detected motion interval and can be defined as
(iii) Average total power (ATP). The ATP is utilized to estimate the power density of the SEMG signal and can be defined as
(iv) Zero crossing (ZC). The ZC is the number of times that a signal passes the zero amplitude axes and can be defined aswhere , , and threshold = 0.02.
(v) Slope sign change (SSC). The SSC represents rate information; a suitable threshold is used to reject disturbance effects as follows:where , threshold .
(vi) Willison amplitude (WAMP). The WAMP is used to count the number of times that signal amplitude exceeds a predefined threshold and is an indicator of muscle contraction:where , , and threshold . In (2)–(7), predefined threshold values , , and were used to reduce noise and artifacts in SEMG signals.
The other three features, area, length change (LC), and minimum value (MV), were extracted from the flex sensors.
(vii) Area. It is a parameter used to calculate the duration of potential changes of a flexure sensor and can be defined as
(viii) Length change (LC). The LC is a parameter representing the number of flexure sensor windings and can be defined as
(ix) Minimum value (MV). The MV is defined as
In (8)–(10), baseline is the starting potential of a flex sensor and is the potential signal from a flex sensor. For each motion detection interval, the features IEMG, WL, ATP, SSC, ZC, WAMP, area, LC, and MV were extracted and gathered into a feature pattern represented bywherewhere is an original feature pattern, is the number of SEMG channels, and is the number of flex sensors. The original feature pattern contains 33 parameters. The number of parametric features is computed with 3 channels () multiplied by 6 parameters (IEMG, WL, VAR, SSC, ZC, and WAMP) for SEMG signals and 5 channels () multiplied by 3 parameters (area, LC, and MV) for data glove signals. In this study, twenty hand motions, that is, the general use in applications of HCI and SLR, are assigned in number order as shown in Figure 2 [1, 12].
3.3. Independent Component Analysis
ICA is a statistical technique for solving the blind source separation (BSS) problem and finding the linear transformation that can be formulated as , where is a mixing matrix. By inverting the system to , we can identify as the demixing matrix that is calculated by some independence measurement functions. For each set of experiments, the EMG data was analyzed using fast ICA MATLAB package which has been developed and proposed by the team at the Helsinki University of Technology (FastICA 2005). FastICA is a highly efficient method for performing ICA estimation. The algorithm is based on a fixed-point iteration scheme maximizing non-Gaussianity as a measure of statistical independence in SEMG analysis [8, 20, 21]. Because the features are from two different types of signal, we used two ICA processors for data glove and SEMG signal, respectively, and the ICA feature pattern represented bywherewhere is a ICA feature pattern, and are the number of independent components from SEMG and data glove features, respectively. We calculated how many independent components in the features from original SEMG and data glove by principal component analysis (PCA) [22]. After feature extraction, the feature pattern was input to the GRA-based classifier.
3.4. Pattern Discrimination Using GRA Classifier
This study uses GRA as core method to identify different hand motions. GRA is a method that determines the relationships between reference sequence data and other comparative sequence data. We assume that a reference sequence as (14), , and comparative sequences , , and it can be represented asThe comparative sequences are used as training data and reference sequence is testing data. The absolute error of reference sequence and each comparative sequence is then calculated. The deviation matrix can be represented aswhere . The grey relational grades can be calculated as [18]The grey relational grades , , where ED() is the Euclidean distance (ED) between the reference sequence and each comparative sequence . In (17), and are the minimum and maximum values of matrix , respectively. The recognition coefficient is chosen to be between 5 and 10. The physical meaning of (17) is that if the ED between the reference sequence and comparative sequence is the minimum, the grey relational grade between them is maximal. This guarantees that as the ED between two patterns decreases, the grey rational pattern grade increases. For example, if ED is approximately equal to , then is close to 1. Furthermore, similarity decreases as distance increases; if ED is approximately , then is close to 0. One can increase contrast by adjusting the recognition coefficient .
The final grey relational pattern grades as equation (19) The is defined the decision for classifying an pattern . If pattern belongs to Class , the weighting factor is equal to one and the rest of the factors are zero. The dimension of grey relational vector could be reduced from -dimension to -dimension (). The output vector could represent as (There are twenty hand motions in this study). The maximum grade indicates one gesture.
In real-time application, subjects are asked to carry out a set of 20 different gestures. After feature extraction, we can obtain comparative sequences for first 10 trials ( in this study). These feature patterns are normalized and then are input to GRA classifier for teaching. Those trials use as reference patterns for testing. In this study, signal data are obtained and transferred to laptop with a NI DAQ card (National Instruments DAQ card, analog-to-digital (AD) converter, 16 channels). Data acquisition function is designed using LabVIEW graphical programming software (National Instruments, Austin, TX, USA). Through the experimental tests, the results will show the effectiveness of the proposed method. Figure 3 shows all configurations of the proposed HMI system.
4. Experimental Setup
Ten subjects (average age years) were recruited to evaluate the performance of the proposed method. Each subject performed a set of 20 different gestures while standing. Figure 2 shows the 20 hand motions classified by the proposed method and these motions included wrist extension, wrist flexion, ulnar flexion, radial flexion, supination, pronation, opening the hand, hand grasping, thumb extension, thumb flexion with fingers extended, thumb interphalangeal (IP) flexion with fingers proximal interphalangeal (PIP) flexion, index finger extension, index and middle fingers extended, index finger extension, middle finger extension, and ring finger extension, thumb and index finger extended, little finger extension, middle finger extension, thumb and little finger extension, thumb, index and little fingers extended, and thumb and index finger in opposition. These motions selected are strongly related to the manipulation of a robotic arm and sign language.
To estimate the threshold values of , , , and , each subject was asked to open and close their hand before the real-time learning stage. In this stage, we also used PCA method to calculate the quantity of independent components. Each subject was also asked to relax before the next hand motion. For feature extraction, each subject repeats 20 different gestures randomly for 11 trials. These feature patterns are normalized in the same scale and are divided into two groups; 200 patterns are used as the comparative patterns (10 times by 20 gestures, ) for training patterns and the other 20 patterns are used as the reference patterns for testing patterns. GRA classifier has a function of feedforward mathematical operation, flexible mechanism with add-in and delete-off training patterns (dataset update) without adjusting parameters. For an adaptive application, it was potential to be a real-time mechanism for pattern recognition [23].
5. Experimental Results and Discussion
To assess the capability and accuracy of the proposed method in discriminating between different hand motions, this study used two indices, training times and accuracy, for different training datasets to evaluate the performance of this HMI system using BPNN, BPNN with ICA, GRA, and proposed methods, respectively. The details of these two indices are expressed below.
5.1. Evaluation of Training Times in Different Training Sets
We have also proposed the BPNN-based classifier with input nodes that is depending on the dimension after ICA reduction (original dimension is 33) and 20 output nodes . The number of hidden nodes in hidden layer is determined by formula in back-propagation (BP) algorithm and used sigmoid function. The convergence criterion <10−2 is set to terminate the BPNN. Figure 4 shows the training times of the twenty hand motions using BPNN, BPNN with ICA, GRA, and proposed methods, respectively. Training times increased as the training set (from 20 to 200 training patterns) increased with four methods. However, average and standard deviation (SD) of training times by GRA-based classifier were clearly lower and more stable than those with all the BPNN, indicating that training performance of GRA-based classifier was slightly affected by the increase in the amount of training data. The average training time of the proposed system combing ICA/GRA methods for identifying each hand motion was roughly 20 ms and its average SD was less than 5 ms, that was about 30% less than only using GRA method. This performance reached the requirement for real-time processing.
5.2. Evaluation of Identification Rate in Different Training Sets
Figure 5 shows the average accuracy of four methods with different training sets generated by 10 subjects. For each subject, the training data gradually increases from 1 training set (20 feature pattern) to 10 training sets (200 feature patterns). Twenty testing patterns are used to examine the proposed classifier using their own hand gestures. When training data was above three sets, the accuracy of GRA method was >85%, that was better than BPNN method. The accuracy did not clearly differ. When fewer than three training sets were used, the accuracy of GRA was more stable than that of the BPNN; however, the difference was not statistically different (). Additionally, the BPNN had difficulty reaching convergence when fewer than six training sets were used. The average accuracy of the proposed system using GRA classifier was roughly 90% with more than six training sets. The average accuracies of 10 subjects are shown in number order of hand gesture, as shown in Table 1. In earlier study [8], the average accuracy was 93.1% roughly only using SEMG signals for identifying 11 hand motions. However, it was also true that the accuracy of one of the subjects was only 63.8%. Additionally, for finger motions such as extension of the index and middle fingers and extension of all five fingers, the accuracy was low in most of the subjects. The results indicated that all the accuracy of identification rate was more than 85% that is much better than earlier study.
6. Conclusions
Compared to the BPNN methods, the identification rate varied markedly, especially when the training set was small. That is an advantage of real-time system. Because the GRA does not require strict statistical procedures or inference rules and ICA could enhance the reproducibility of features by noise isolation while on-line processing, the identification rate of proposed method is more stable. The results of this study indicate that the ICA is a reliable preprocess to reduce the amount of data and, in turn, reduce the processing time about 30% in the GRA method, but the performance of the system decreased significantly following ICA in BPNN method. That may be caused by difficult convergence in BPNN. The performance of the proposed HMI system using bend sensors and multichannel SEMG was based on accuracy, number of hand motions, and training times needed to meet the requirement for real-time processing. The architecture of the proposed system meets the requirement of portability, but with the flexible mechanism, the proposed method can be further developed for implementation in a field programmable gate array (FPGA) or digital signal processor (DSP). It has high potential to integrate as an assistive tool for many HMI applications.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.