Brain Computer Interface Systems for Neurorobotics: Methods and ApplicationsView this Special Issue
Research Article | Open Access
Noninvasive Electroencephalogram Based Control of a Robotic Arm for Writing Task Using Hybrid BCI System
A novel hybrid brain-computer interface (BCI) based on the electroencephalogram (EEG) signal which consists of a motor imagery- (MI-) based online interactive brain-controlled switch, “teeth clenching” state detector, and a steady-state visual evoked potential- (SSVEP-) based BCI was proposed to provide multidimensional BCI control. MI-based BCI was used as single-pole double throw brain switch (SPDTBS). By combining the SPDTBS with 4-class SSEVP-based BCI, movement of robotic arm was controlled in three-dimensional (3D) space. In addition, muscle artifact (EMG) of “teeth clenching” condition recorded from EEG signal was detected and employed as interrupter, which can initialize the statement of SPDTBS. Real-time writing task was implemented to verify the reliability of the proposed noninvasive hybrid EEG-EMG-BCI. Eight subjects participated in this study and succeeded to manipulate a robotic arm in 3D space to write some English letters. The mean decoding accuracy of writing task was . Four subjects achieved the optimal criteria of writing the word “HI” which is the minimum movement of robotic arm directions (15 steps). Other subjects had needed to take from 2 to 4 additional steps to finish the whole process. These results suggested that our proposed hybrid noninvasive EEG-EMG-BCI was robust and efficient for real-time multidimensional robotic arm control.
Brain-computer interface (BCI) offers a direct communication channel between the brain and external devices without relying on the brain normal output pathways of peripheral nerves and muscles . According to different signal acquisition methods, BCI systems can be divided into two main categories, invasive and noninvasive BCI systems [2–5]. Although invasive BCI systems seem suitable for some clinical applications because of their high signal-to-noise ratio (SNR) and the information transfer rates (ITR), they still suffer from potential surgical risks and postoperative immune response. Noninvasive BCIs might be more suitable for daily life applications for many socioeconomic reasons such as the user’s safety and a relatively low cost. Electroencephalography (EEG) is a popular electrophysiological monitoring method to record brain activity and is widely used in noninvasive BCI researches and applications since Berger’s discovery  and Vidal’s first BCI prototype .
Nowadays, there are huge opportunity and necessity for helping handicapped people to enhance or increase their abilities to interact with complex environment, such as rehabilitation training sessions, mind-controlled prosthetic arm applications [8, 9], and augmentative and alternative communication systems . However BCIs can also benefit healthy people for entertainment and increasing their independency, especially for elderly persons [11–13]. For decades, several EEG-based typical BCI systems have been proposed based on slow cortical potential (SCP) , motor imagery (MI) , steady-state visual evoked potential (SSVEP) , and the P300 wave of the human event-related potential [10, 17]. Each type of these BCI systems has its unique advantages and some disadvantages. For example, the SSVEP-based BCI system has many advantages such as less training and higher SNR and ITR. MI-based BCI has the advantage of fast response but is limited by the number of tasks. Therefore, there has been increasing interest in solving dimensionality issue by using hybrid BCI which has to be composed of two BCI modalities’ combination (e.g., motor imagery with P300, motor imagery with SSVEP, and P300 with SSVEP) or it can be also a combination of brain and nonbrain activity such as eye movements (EOG), muscles activity (EMG), and heart electrical activity (ECG) to improve the overall performances of BCI systems [18–22]. The electrical activity of muscles can easily interfere with EEG signal considering the anatomical locations of facial or masticatory muscles surrounding the skull. This myogenic contamination of the EEG can constitute a serious problem in BCI applications and it can be useful information in the same time for developing hybrid BCIs. Pfurtscheller et al. proposed an online system using SSVEP-based BCI and a type of ERD BCI, called a “brain switch” . Lately, Punsawad et al. controlled practical machine through hybrid EEG-EOG brain-computer interface system . Then Wang et al. controlled wheelchair directions through unilateral hand imagination and a wheelchair speed through P300 and EOG .
Controlling a robotic arm with noninvasive hybrid BCIs surely provides a desirable alternative, but prior to this study it has not been shown that such hybrid systems could achieve multidimensional control of robotic arm in three-dimensional (3D) space. These systems offer a potentially effective control for complex and naturalistic environment through the combination of brain- and nonbrain-based multifunctional BCI. They can reduce user fatigue by switching from a modality to another and increase the degree of freedom for augmentative and alternative BCI systems. The aim of this study is to improve the performance of BCI system by design a new hybrid EEG-EMG-BCI system (i.e., combination of brain activity (MI and SSVEP) with muscles activity such as teeth clenching). In this paper, a hybrid BCI system was described, including motor imagery-based brain switch, “teeth clenching” state detector, and a steady-state visual evoked potential- (SSVEP-) based BCI. In our proposed hybrid BCI, motor imagery decoding was used as a single-pole double throw brain switch (SPDTBS) which can complete multitasks, combined with 4-class SSVEP-based BCI system. In addition, “stop” command was executed by recognizing facial action by recording EMG artifact from EEG signals. For real-time application, a writing task was implemented to verify the performances of our proposed hybrid system. Healthy subjects succeed to write an English word though our proposed hybrid noninvasive BCI system. In the following sections, we describe our proposed system in detail and its real-time writing application to enhance the user’s abilities to interact with a complex environment.
2.1. Experimental Paradigm
Our proposed hybrid noninvasive EEG-EMG-BCI system mainly consists of three hardware components which are a portable EEG acquisition device (Emotiv EPOC), a host computer, and a robotic arm (see Figure 1). The EEG signals were recorded and transmitted to the host computer with an USB transceiver dongle. Then, EEG signals were processed and decoded in the host computer. Based on this proposed BCI architecture, the intension of subjects was transformed to multidimensional control commands and sent to operate the Dobot (robotic arm) via the wireless module in real time.
The hybrid BCI consists of MI-, EMG-, and SSVEP-based BCI systems. As shown in Figure 2, the subject imagines the left hand or right hand movement for 4 s as the first step. Once the imagined movements’ type (e.g., left hand and right hand movements) was confirmed, the system will enter the second hybrid BCI phase. In the second phase, SSVEP and “teeth clenching” were decoded. The presence of “teeth clenching” was detected during the second phase. Once “teeth clenching” state is confirmed, the program will execute stop command of SSVEP modality and go back to the first motor imagery modality. For SSVEP paradigm, four white blocks (with different frequencies: 6 Hz, 7.5 Hz, 8.57 Hz, and 10 Hz) of stimuli flicker were presented at the top, bottom, left, and right positions in black board.
According to each unilateral movement (right or left hand imagination), the SPDTBS was designed. The SPDTBS was combined with four tasks of SSVEP-based BCI modality to provide more commands (i.e., to achieve multidimensional BCI control) for the robotic arm movements such as the forward, backward, left, right, upward, and downward movements (see Figure 3). All these commands were shown in Table 1.
Eight healthy subjects participated in the experiment (age years (mean ± standard deviation “SD”); one female and seven males). All of them were undergraduate students, without any experience with BCI system. The subjects were seated in a comfortable chair, 50 cm away from the computer screen. The robot arm was placed on the table, about 45° in the left front of the subject. Each subject was able to look at both the monitor and the movement of the robot arm. The subjects were requested to write the word “HI” and the essential steps were shown in Figure 4. It takes at least 15 steps to complete the writing of BCI; each step represents a horizontal or vertical line. The subjects can choose the order of writing, and each step can be written repeatedly.
The following points are used to evaluate the performance of the hybrid BCI system:(1)Time: time required to complete a task.(2)Step count: the number of steps to complete the task in paper.(3)Obvious errors: number of obvious errors. For example, if the writing task is O letter but the result is Q, this result is defined as obvious error.(4)Information transfer rate, which is defined as where is the number of targets, is the accuracy rate, and is the time window length.
The EEG data were sampled at a frequency of 2048 Hz and then downsampled to 128 Hz for signal processing. The electrodes were placed at 10-20 system locations, AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4, as well as two reference electrodes located above the ears of the subject (i.e., either CMS and DRL or left/right mastoids). Electrodes P7, P8, O1, and O2 were selected to collect SSVEP-based EEG signal. Electrode FC5 and FC6 were used to capture EEG signals in MI. The signals during “teeth clenching” state were collected mainly by electrodes F7 and F8 (see Figure 5).
2.2. Processing Methods for SSVEP-Based BCI
To reduce the effect of signal-to-noise ratio (SNR), discrete wavelet transform (DWT) was employed for preprocessing of EEG signals. Assuming that is the EEG signal, the DWT of is defined as where is the wavelet basis function, is the resolution of the frequency, and is the amount of time translation.
EEG signals were decomposed in different layers (5 layers) by using Daubechies wavelet (db4) function and reconstructed by removing frequency components (0–2 Hz).
The Canonical Correlation Analysis (CCA) is a multivariable statistical method, which was used to analyze the potential correlation between two sets of data . CCA method has been widely used in SSVEP-based BCI system [16, 26].
Suppose two multidimensional random variables , ; that is, , . CCA finds a pair of weight vectors and , respectively, which maximize the correlation between linear combinations and . It is defined aswhere the maximum of is the maximum canonical correlation. and are projected onto and .
The reference signals are set aswhere is the number of sampling points, is the sampling frequency, and is the number of harmonics.
The control command is recognized aswhere are the CCA coefficients obtained from the four reference signals.
Each subject has different thresholds , so threshold was defined in training phase. To calculate the threshold in CCA, 20 offline experiments were held for each subject. The results were shown in Table 2. The average correlation coefficient of idle state is 0.1542 ± 0.0397 (mean ± SD). So, the threshold was defined to be 0.22.
2.3. Processing Methods for MI-Based BCI and “Teeth Clenching” State Detector
The motor imagery classification based on mu frequency power has been widely used for processing event-related synchronization (ERS) and event-related desynchronization (ERD) [15, 27–29]. The second-order moment energy algorithm was employed to classify the left hand with low computational complexity and simple principle . So, these algorithms could be suitable for achieving online BCI systems.
Assuming a signal of length , the second-order moment is estimated by
In MI-based BCI experiment, while imagining the left hand or right hand movement, the EEG signals are collected with band-pass filtering (0–32 Hz), mu rhythm energy change of FC5 and FC6 was computed, and the energy difference between FC5 and FC6 channels is used to calculate the threshold for classification. Mu rhythm energy change of FC5 and FC6 is denoted as .
When , which indicates the subject is imagining left hand motor imagery, the first path is closed in SPDTBS. When , which indicates the subject is imagining right hand motor imagery, then the second path is closed in SPDTBS. When , which indicates the subject is in an idle state, no command will be given to the system, and the SPDTBS is opened.
Accuracy of detecting “teeth clenching” state is higher than other facial states in EEG-based BCI system . Thus, “teeth clenching” state was detected to work as interrupt system, which can confirm motor imagery result and improve the performance of whole system.
According to the characteristics of different states (“natural” versus “teeth clenching”), the threshold β of standard deviation and threshold η of the peak distance (the absolute value of the difference between the maximum and the minimum) of the EEG signals were calculated, respectively. In online experiment, to detect the “teeth clenching” state, standard deviation and peak distance were computed and compared with thresholds β and η. where indicates that the subject is in “teeth clenching” state and, otherwise, means that the subject is in “natural” state.
3. Results of Brain-Control Tasks
In the first stage, an offline experiment was held for MI-based BCI and SSVEP-based BCI and “teeth clenching” state detector, respectively. As shown in Figure 6, the average accuracy of eight subjects in MI-based BCI and SSVEP-based BCI is and , respectively. The ITR of SSVEP-based BCI is (Figure 7). For “teeth clenching” state detector, all subjects achieved accuracy near 1.
Eight subjects joined the writing task using a robotic arm. The results were shown in Figures 4, 6, and 7. All of subjects were successful in writing the word “HI.” Eight subjects completed the writing task in seconds on average. As shown in Figure 7, four subjects took 15 steps (optimal number of steps) to finish writing, three subjects took 17 steps, and one subject took 19 steps. Only one subject has 1 significant error. The average accuracy was obtained as .
In this paper, a novel multichannel hybrid BCI system was proposed for multidimensional control purpose, which was composed of a motor-imagery-based brain switch, “teeth clenching” state detector, and SSVEP-based BCI. For achieving a multidimensional control of robotic arm, seven commands which can be up to nine were designed for real-time BCI application. Writing task was held to evaluate the performances of the proposed hybrid system. Eight subjects completed the movement tasks of the robotic arm to write the word “HI.”
Pfurtscheller et al.  used MI-based brain switch to turn on/off an SSVEP BCI for reducing the errors in resting periods, while, in this present work, MI-based brain switch was used as single-pole double throw brain switch (SPDTBS) to extend the number of commands which robotic arm needs to be fully controlled and manipulated to achieve a naturalistic hand pathway of writing a simple word. In addition, a “teeth clenching” state detector was design to initialize the statement of SPDTBS. Because of the relative lower accuracy of motor imagery BCI, the “teeth clenching” state detector can confirm the result of MI-based BCI, which can improve the accuracy of hybrid BCI system significantly. Thus, the accuracy of proposed system is almost the same as previous work , which can lead us to conclude that the proposed system is efficient and robust for real-time multifunctional BCI systems. Compared with the current BCI systems [32–34], the proposed hybrid BCI system shows higher accuracy with high degree of freedom. Moreover, wireless manner was used to build a stable and suitable connection for online experiment, which is meaningful for portable noninvasive BCI products for real-life use.
In this study, we found that a group of healthy subjects could willingly use brain and nonbrain activity to control a robotic arm with high accuracy for performing writing tasks requiring human intention, error feedback, and multiple degrees of freedom by combination of MI, SSVEP, and EMG activity (see Supplementary video in Supplementary Material available online at https://doi.org/10.1155/2017/8316485). The robotic arm could only move and write horizontal and vertical lines using our proposed BCI paradigm. There are still two commands, “no function,” in the proposed system, which can be used for writing slanted lines. This option will be added in the near future to achieve naturalistic hand writing. Thus, the users will be able to write any intended complex word in real time, which not only can be used as communication tool for the disable people, but also can be applied for education purposes for children and students using e-learning aspect which could be an innovative way to practice using teleoperation to remotely access a robotic arm using their brain activity.
This paper presented a combination of synchronous and asynchronous control using a novel hybrid EEG-EMG-based BCI which consists of motor imagery, muscle artifacts, and SSVEP to provide a multidimensional control. The synchronous control is based on SSVEP paradigm which requires the user to focus on the screen and the asynchronous control is based on the motor imagery which does not need any synchronization between the user and the screen because it is based on the imagination of the unilateral movements. Users were able to write an English word using our robust real-time control of a robotic arm through the proposed hybrid BCI. This proposed BCI was designed for multiclass control in a complex environment. Results of the study indicated that successful multidimensional control is possible using suitable combination of BCI modalities to detect and classify brain activity in different situations.
In the near future, for rehabilitation, e-learning, and entertainment, we would like to design low cost, portable, noninvasive, and hybrid EEG-EMG-based robotic arm using minimum number of wearable wireless sensors.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
This work was financially supported by Natural Science Foundation of Tianjin City (15JCYBJC51800), Tianjin Key Laboratory Foundation of Complex System Control Theory and Application (TJKL-CTACS-201702), and Young and Middle-Aged Innovation Talents Cultivation Plan of Higher Institutions in Tianjin (Grant no. 20130830).
The video demonstrates a real-time application of our proposed hybrid BCI. This video shows a healthy subject that could willingly use his brain and nonbrain activities to control a robotic arm. The subject was able in the end to write an English work “HI” using our proposed hybrid noninvasive BCI.
- J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, “Brain-computer interfaces for communication and control,” Clinical Neurophysiology, vol. 113, no. 6, pp. 767–791, 2002.
- N. Birbaumer, C. Weber, C. Neuper, E. Buch, K. Haapen, and L. Cohen, “Physiological regulation of thinking: brain-computer interface (BCI) research,” Progress in Brain Research, vol. 159, pp. 369–391, 2006.
- L. R. Hochberg and J. P. Donoghue, “Sensors for brain-computer interfaces: Options for turning thought into action,” IEEE Engineering in Medicine and Biology Magazine, vol. 25, no. 5, pp. 32–38, 2006.
- G. Schalk, P. Brunner, L. A. Gerhardt, H. Bischof, and J. R. Wolpaw, “Brain-computer interfaces (BCIs): detection instead of classification,” Journal of Neuroscience Methods, vol. 167, no. 1, pp. 51–62, 2008.
- S. G. Mason, A. Bashashati, M. Fatourechi, K. F. Navarro, and G. E. Birch, “A comprehensive survey of brain interface technology designs,” Annals of Biomedical Engineering, vol. 35, no. 2, pp. 137–169, 2007.
- H. Berger, “Uber das elektrenkephalogramm des menschen,” European Archives of Psychiatry and Clinical Neuroscience, vol. 87, no. 1, pp. 527–570, 1929.
- J. J. Vidal, “Toward direct brain-computer communication,” Annual Review of Biophysics and Bioengineering, vol. 2, pp. 157–180, 1973.
- G. Pfurtscheller and C. Neuper, “Motor imagery and direct brain-computer communication,” Proceedings of the IEEE, vol. 89, no. 7, pp. 1123–1134, 2001.
- C. Wang, S. P. Kok, K. A. Kai et al., “A feasibility study of non-invasive motor-imagery BCI-based robotic rehabilitation for stroke patients,” in Proceedings of the 4th International IEEE/EMBS Conference on Neural Engineering, pp. 271–274, May 2009.
- L. A. Farwell and E. Donchin, “Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials,” Electroencephalography and Clinical Neurophysiology, vol. 70, no. 6, pp. 510–523, 1988.
- A. Finke, A. Lenhardt, and H. Ritter, “The Mindgame: a P300-based brain-computer interface game,” Neural Networks, vol. 22, no. 9, pp. 1329–1333, 2009.
- J. A. Pineda, D. S. Silverman, A. Vankov, and J. Hestenes, “Learning to control brain rhythms: making a brain-computer interface possible,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 11, no. 2, pp. 181–184, 2003.
- T. M. Nunes, A. L. V. Coelho, C. A. M. Lima, J. P. Papa, and V. H. C. de Albuquerque, “EEG signal classification for epilepsy diagnosis via optimum path forest—a systematic assessment,” Neurocomputing, vol. 136, pp. 103–123, 2014.
- N. Birbaumer, N. Ghanayim, T. Hinterberger et al., “A spelling device for the paralysed,” Nature, vol. 398, no. 6725, pp. 297-298, 1999.
- G. Pfurtscheller and F. H. L. da Silva, “Event-related EEG/MEG synchronization and desynchronization: basic principles,” Clinical Neurophysiology, vol. 110, no. 11, pp. 1842–1857, 1999.
- Z. Lin, C. Zhang, W. Wu, and X. Gao, “Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs,” IEEE Transactions on Biomedical Engineering, vol. 53, no. 12, pp. 2610–2614, 2006.
- S. Sutton, M. Braren, and J. Zubin, “Evoked-potential correlates of stimulus uncertainty,” Science, vol. 150, no. 3700, pp. 1187-1188, 1965.
- G. Pfurtscheller, B. Z. Allison, C. Brunner et al., “The hybrid BCI,” Frontiers in Neuroscience, vol. 4, article 30, 2010.
- G. Pfurtscheller, T. Solis-Escalante, R. Ortner, P. Linortner, and G. R. Muller-Putz, “Self-paced operation of an SSVEP-based orthosis with and without an imagery-based “brain switch”: a feasibility study towards a hybrid BCI,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 18, no. 4, pp. 409–414, 2010.
- Y. Li, J. Pan, F. Wang, and Z. Yu, “A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control,” IEEE Transactions on Biomedical Engineering, vol. 60, no. 11, pp. 3156–3166, 2013.
- B. Rebsamen, E. Burdet, Q. Zeng et al., “Hybrid P300 and Mu-Beta brain computer interfaceto operate a brain controlled wheelchair,” Assistive Technol, pp. 51–55, 2008.
- A. N. Belkacem, S. Saetia, K. Zintus-Art et al., “Real-time control of a video game using eye movements and two temporal EEG sensors,” Computational Intelligence and Neuroscience, vol. 2015, Article ID 653639, 10 pages, 2015.
- Y. Punsawad, Y. Wongsawat, and M. Parnichkun, “Hybrid EEG-EOG brain-computer interface system for practical machine control,” in Proceedings of the 2010 32nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2010), pp. 1360–1363, Buenos Aires, August 2010.
- H. Wang, Y. Li, J. Long, T. Yu, and Z. Gu, “An asynchronous wheelchair control by hybrid EEG-EOG brain-computer interface,” Cognitive Neurodynamics, vol. 8, no. 5, pp. 399–409, 2014.
- H. Hotelling, “Relations Between Two Sets of Variates,” Biometrika, vol. 28, no. 3/4, p. 321, 1936.
- G. Bin, X. Gao, Z. Yan, B. Hong, and S. Gao, “An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method,” Journal of Neural Engineering, vol. 6, no. 4, Article ID 046002, 2009.
- G. Pfurtscheller, G. R. Müller-Putz, A. Schlögl et al., “15 years of BCI research at graz university of technology: current projects,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 14, no. 2, pp. 205–210, 2006.
- G. Pfurtscheller, G. R. Müller-Putz, A. Schlögl et al., “15 years of BCI research at graz university of technology: current projects,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 14, no. 2, pp. 205–210, 2006.
- E. Dong, C. Li, L. Li et al., “Classification of multi-class motor imagery with a novel hierarchical svm algorithm for brain computer interfaces,” Medical Biological Engineering & Computing, pp. 10–1007, 2017.
- G. Pfurtscheller, C. Neuper, C. Andrew, and G. Edlinger, “Foot and hand area mu rhythms,” International Journal of Psychophysiology, vol. 26, no. 1-3, pp. 121–135, 1997.
- J. Lin and Z. Y. Jiang, “An EEG-based BCI system to facial action recognition,” Wireless Personal Communications, pp. 1–16, 2016.
- I. Martišius and R. Damaševičius, “A prototype SSVEP based real time bci gaming system,” Computational Intelligence & Neuroscience, vol. 2016, no. 3, Article ID 3861425, 15 pages, 2016.
- K. Holewa and A. Nawrocka, “Emotiv EPOC neuroheadset in brain—computer interface,” in Proceedings of the 15th International Carpathian Control Conference (ICCC '14), pp. 149–152, IEEE, Velke Karlovice, Czech Republic, May 2014.
- Y. Liu, X. Jiang, T. Cao et al., “Implementation of SSVEP based BCI with Emotiv EPOC,” in Proceedings of the 10th IEEE International Conference on Virtual Environments, Human-Computer Interfaces, and Measurement Systems (VECIMS '12), pp. 34–37, Tianjin, China, July 2012.
Copyright © 2017 Qiang Gao et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.