Abstract
People suffering from paralysis caused by serious neural disorder or spinal cord injury also need to be given a means of recreation other than general living aids. Although there have been a proliferation of brain computer interface (BCI) applications, developments for recreational activities are scarcely seen. The objective of this study is to develop a BCI-based remote control integrated with commercial devices such as the remote controlled Air Swimmer. The brain is visually stimulated using boxes flickering at preprogrammed frequencies to activate a brain response. After acquiring and processing these brain signals, the frequency of the resulting peak, which corresponds to the user’s selection, is determined by a decision model. Consequently, a command signal is sent from the computer to the wireless remote controller via a data acquisition (DAQ) module. A command selection training (CST) and simulated path test (SPT) were conducted by 12 subjects using the BCI control system and the experimental results showed a recognition accuracy rate of 89.51% and 92.31% for the CST and SPT, respectively. The fastest information transfer rate demonstrated a response of 105 bits/min and 41.79 bits/min for the CST and SPT, respectively. The BCI system was proven to be able to provide a fast and accurate response for a remote controller application.
1. Introduction
The effect of a neurodegenerative disease or a spinal cord injury suffered from an accident has caused the loss of quality of life for millions of people around the world. One important reason is that they have reduced capabilities in communicating normally with people and their external environments. In order to better provide for the needs of these individuals, developments of high technology assistive devices have seen an increase in proliferation in recent years. These devices are used to help the patients cope with daily tasks such as eating, dressing, living, and mobility, among others. A field about psychologically healing that has only recently been considered is entertainment for the patients. New forms of entertainment were developed through the use of a brain computer interface (BCI). BCI is essentially a means of extracting several features from the scalp potential, or electroencephalography (EEG), with the aid of external devices. These assistive devices provide patients a distinct method to communicate with the outside world [1]. The game of chess was reinvented to make use of BCI control wherein EEG is used to determine which piece to move [2]. Previous developments on BCI made use of event-related potential (ERP) associated in time with either physical or mental occurrence.
ERP often includes the and rhythms, slow cortical potential, P300-evoked potential, and the steady-state visual-evoked potential (SSVEP) [1]. A form of slow cortical potential is the visual evoked potential (VEP), in which a visual stimulus, such as an alternating checkerboard pattern, is displayed on the screen to elicit evoked potential [3]. In a previous article the research group developed a BCI-controlled LEGO page-turner that uses VEP technology. It was able to produce a high recognition rate in turning the page of a book to the left or right. However, the duration for each trial was about 7 to 12 seconds [4]. In order to reduce the response time and increase the signal-to-noise ratio (SNR) in future projects, BCI researchers are shifting to the use of SSVEP. SSVEP is an oscillatory wave that appears in the occipital leads of the EEG in response to visual stimuli modulated at a certain frequency. It can involve the use of pattern-reversed checkerboard or flickering LEDs [3].
The frequency of the SSVEP matches that of the stimulus and its harmonics [1, 3]. The uses of LED and SSVEP were implemented in several ways, such as moving a cursor location or selecting a certain target. In larger cases such as a QWERTY keyboard spelling system, SSVEP requires the number of stimulating frequencies to be larger than that of the displayed characters [5]. In congruence with a previous study on an alternative sleep EEG electrode placement [6], the coherence of different frequency stimuli and its effect on the EEG signal at different electrode placements is also a topic of interest in this field of study.
In this paper, we propose the integration of a remote control entertainment device to a noninvasive BCI system. Throughout the years, remote control devices are one kind of the most popular toys by both children and adults alike. And in recent years a commercial remote control toy called the Air Swimmer gained popularity due to its unique concept of having a flying fish controlled with the standard remote control. Controlling the toy can be done with ease by normal people, but for those with certain disorders even this form of entertainment can be cumbersome. Therefore, the current research aims to develop a BCI remote control for the Air Swimmer and evaluate its performance. Subsequently, the proposed BCI system is based on the theory of steady state evoked potentials (SSVEP). It states that when a person stares at an oscillating visual stimulus the brain will produce a corresponding response and this response can then be used to create a BCI command. It utilizes lights or flickering boxes on a computer interface to induce SSVEP. As an object flickers at a certain frequency, it instigates a response in the visual cortex at the frequency stimulus or its harmonics. Hence, a set of frequencies can be used to perform different commands [7, 8].
The rest of this paper is organized as follows. Section 2 describes the BCI remote control and the application protocol to test the system. It also includes the methodological approach to identify spectral response including feature extraction, decision modeling, and translation command. Section 3 then describes the results of a series of experiments to examine the performance of our approach. Finally, conclusions are drawn in Section 4, and possible improvements for future development of this work are discussed.
2. Methods
The system architecture of the proposed BCI-based remote control system for recreational purposes is shown in Figure 1. The BCI system presented thereafter includes the remote control system with the following processes: visual stimulation, signal acquisition, signal prepreprocessing, feature extraction, decision model, and translational commands. First, the subjects need to focus on the flickering box that is marked with the desired icon displayed on an LCD screen. Second, electroencephalographic (EEG) signals are extracted using a single electrode from the occipital area of the subject’s head. Third, the EEG signals acquired are preprocessed to remove unnecessary drifts, trends, and noise. The signal undergoes baseline correction and a series of filtering methods prior to analysis. Fourth, the computations of the power spectra of the EEG signal’s different harmonic bands are performed. Fifth, a decision model is used to determine the required response that will then be translated to a command signal. Sixth, the hardware module control is discussed and lastly experimental procedures are stated.
2.1. Visual Stimulation
Visual stimulation was performed to elicit SSVEP. A 24′′ LCD monitor displaying the flickering boxes simulated by a program was used. The boxes were arranged and placed at different locations across the screen with the concern of reducing interference between frequencies. A program is designed to make the boxes flickering at frequencies between 6 Hz–10 Hz with 1 Hz increments. The corresponding commands are as follows: (a) 6 Hz (dive), (b) 7 Hz (left turn), (c) 8 Hz (swim forward), (d) 9 Hz (right turn), and (e) 10 Hz (elevate).
2.2. Signal Acquisition
The visual stimulation elicits signals on the visual cortex of the brain. Concurrently, EEG signals are simultaneously read from the occipital region of the subject’s head by a data acquisition device. Signal acquisition was conducted with the use of Ag-Cl electrodes connected to an EEG amplifier using the NuAmps EEG amplifier by Neuroscan Company. The EEG cap has a total of 40 electrodes and sampling frequency can be set at 125/250/1000 Hz. The device also makes use of a 22-bit analog to digital converter. EEG recordings were acquired from the Oz channel with reference placed at A2. The subjects were required to sit at a distance of 55 cm from the monitor, measured from the subject’s nasion to the screen of the BCI interface. The raw data of the EEG signals were then transmitted to a personal computer to perform further analysis and eventually control the Air Swimmer. Next, the EEG signal is then temporarily saved in a buffer array. The purpose of this buffer is to accumulate the real-time EEG signal to form an array size of 1000 samples before being processed. After the sampled data is processed, the buffer array is cleared.
2.3. Signal Preprocessing
In order to accurately/perceptively represent the EEG signal, , the drift phenomenon of EEG signal should be removed. An exponentially modified Gaussian function based on nonlinear curve fit model is used and defined as where is the offset from the -axis and is the amplitude of the data set. is the center of the data while is the width of the function. is the modification factor and is defined as The erf function in (1) represents an error function and can be defined as
After removing the drift phenomenon of the EEG signal , a 2nd order band-pass filter with cut-off frequencies between 5 and 30 Hz is used to limit the frequency range of the EEG signals to the subject's responses on visual stimulation. In order to smooth the signal and reduce noise, the Savitzky-Golay filter,, is adopted to preserve its fundamental features and is defined as where and are the number of points the signal is smoothed before and after each considered time point , respectively. are the weighting coefficients of filter, which can keep higher moments [9, 10].
2.4. Feature Extraction
To determine the subject’s selection, the power spectrum of the preprocessed EEG signal is calculated. The power spectrum is calculated by using fast Fourier transform (FFT) [11] which is defined as where is the discrete form of the continuous signal and is the length of the data. Then the power spectrum is calculated as In order to determine the target frequency, the energy at frequency bins 6 Hz, 7 Hz, 8 Hz, 9 Hz, and 10 Hz is selected as the feature which will be used in the decision model to arbitrate the desired selection.
2.5. Decision Model
A threshold is set to exclude the possibility of unnecessary peak occurrence during the initial EEG acquisition. The subjects are first exposed to the different flickering boxes and their initial responses to the stimuli are recorded. The threshold is set by obtaining the mean of the peaks from each frequency band after undergoing the preliminary EEG measurement. The selected features are required to meet the predefined threshold before any command will be sent to the remote controller.
The decision model is used to determine the command selection from feature and is defined as where is the model and is the set of data. According to Bayesian theory, (7) can be derived as
Since is constant to all other models and is assumed to be a uniform distribution, and would not affect the result of (8). Therefore, (8) can be rewritten as In this study, the is assumed to have the form of a Gaussian probability density function in the -dimension space and is defined as where is the mean value and is the variance defined as It is readily seen that for the multivariate Gaussian coincides with the univariate one. Afterwards, the location of the maximum peak is determined as the selection of the subject and a command is sent through the DAQ and transmitted to the reconfigured remote control.
2.6. Hardware Module
The remote control module was reconfigured to be able to accept input from a computer. The hardware module includes a USB data acquisition (DAQ) device and an optocoupler for data transmission. The DAQ is required to convert the digital command output from the computer to an analog input for the remote control. However, in order to reduce the transmission interference, an optocoupler was included as a bridge between the DAQ and remote control. Furthermore, modifications were also made on the input connection of the original remote control circuit board to control the Air Swimmer as shown in Figure 2. Command to the Air Swimmer is then sent via radio frequency transmission.
2.7. Experimental Design for the BCI Remote Control
To validate the proposed BCI remote control system, the subjects performed 2 different experiments, namely, command selection training (CST) and simulated path testing (SPT). The graphic user interface for the visual stimulation and a simulated path are shown in Figure 3.
(a)
(b)
2.7.1. Command Selection Training (CST)
The CST aims to determine the subject’s initial response to SSVEP stimulation. First, the CST requires the subjects to continuously look at a designated flickering box. The trial ends when the subject triggers 30 responses. After each trial the user is required to rest for 2 minutes with eyes closed. These steps are repeated until all 5 different command frequencies have been tested. The program counts the number of correct and wrong selections and calculates the accuracy of the subject. The time for each trial is also recorded for use in further analysis.
2.7.2. Simulated Path Testing (SPT)
After undergoing CST, the performances of the subjects in using the system were tested. A simulated path was developed for the subjects to follow. Arrows chain on a separate graphic user interface (GUI) serves as a guideline on which command should be selected. Subsequently, an audio command is provided to assist the subjects on the designated command. The arrow will be turned on with a green color when a correct input is made. However, if the wrong input is sent, the arrow will remain off and dark and will require the subject to make another input until the correct selection has been made. The program counts the number of correct and wrong selections and calculates the accuracy of the subject. The time for the whole trial was also recorded for use in further analysis. Moreover, after a correct selection is made, a delay was introduced by stopping the flickering boxes and letting the subject rest for 3 seconds to reduce the probability of having residual frequency from the previous selection.
3. Experimental Results
To evaluate the proposed BCI remote control system, 12 subjects (10 males and 2 females) were asked to participate in the experiments. The average age of the group is 20.9 and 21.5 years olds for males and females, respectively. In addition, they have normal or corrected-to-normal vision and had no history of any neurological or psychological disorders.
3.1. System Limitation
The first step for users to adapt to an SSVEP system is by training the brain with the set of frequencies that will be used in the experiment. During the course of the experiment it was noted that different users exhibited different responses to the same set of frequencies. Some of the subjects were removed from the study due to several reasons: strong electrocardiograph (ECG) signals were detected at the Oz region and poor response to SSVEP. Since the study only made use of a single channel EEG, independent component analysis cannot be used to remove the noise caused by the ECG signal. Subsequently, the frequencies which are adaptive to SSVEP system were divided into three main frequency bands composed of low (1–12 Hz), medium (12–30 Hz), and high (30–60 Hz) [12]. The limitation of the study includes the use of low frequency range due to the limitation of the refresh rate of the LCD screen and the number of flickers the computer can handle in real time. The subjects exhibited response to low frequencies as that of [5, 13]. Nevertheless, some of the subjects were not accustomed to low frequency and their EEG signals reflected minimal response. It was challenging to recognize the EEG feature of the subject and thus some of the initial subjects discontinued the experiments and were not included in the results. In order to correct the trigger level, the threshold was adjusted for each subject in accordance to the mean of the peak responses at different frequencies. Subsequently, the Savitzky-Golay method was also useful in the smoothing signal as it preserves the fundamental features including the maxima, minima, width of the peaks, and others [10].
3.2. Peak Detection and Control Command
The control command is sent when the specific frequency response exceeds the given threshold. However, during the experiment body movements and unconscious jitters produced unwanted noise on the signal. It would result in irrelevant peaks that interfere with the peak threshold detection as shown in Figure 4(a).
(a)
(b)
The solution chosen for the peak interference was the use of moving average. The purpose of moving average is to reduce the noise in the signal, increasing peak identification and reducing the random noise. It is optimal for this process due to its simple theory and fast calculation time [14]. In the current research the Henderson 7-term method was used. In the event that more than one peak exceeds the threshold, a possible misjudgment will cause error in the command selection. Through the use of the decision model mentioned above, the peak frequency is ascertained. Consequently, the position of the resulting peak is recorded for the command process. Once the correct peak is determined, the computer would then send the control signal to the remote control via the self-developed control module. It should also be noted that a control command is only sent if only a single peak is determined as shown in Figure 4(b).
3.3. BCI Performance Evaluation
The performance of the remote control system will be assessed by calculating the information transfer rate (ITR). In this study we used the parameters of bit rate, selections per minute (SPM), and time to determine the ITR. This method had been widely used to quantify the speeds of developed BCI systems [13, 15, 16]. The ITR calculation used in this study is the Wolpaw method based on the formula presented in [16] and is defined as where is the accuracy of the classification and is the number of selections. is presented as bits per trial. In this study, the number of selections is equivalent to the number of flickering boxes .
The classification accuracy is computed by taking the correct selection commands divided by the total number of selections. Like in most BCI studies, it can be considered that each box, flickering at different frequencies, is assumed to have an equal probability of being positively classified.
ITR is usually calculated in terms of bits/min (bpm) and is calculated using where is the total selection time in seconds, is the number of selections, and is the bit rate.
3.4. Experimental Results of the BCI Remote Control System
The CST experiment was conducted to train and understand the subject’s brain response to the different set of flickering frequencies. It was also a baseline to determine how accurate and fast the response of each subject is to visual stimulation. The results of the CST experiment are summarized in Table 1. After performing a total of 30 correct selections for each of the 5 commands, it is evident that a large disparity was manifested between the subjects. It was observed that the fastest response time shown during the training was that of subject 1 with an average time of 1.32 seconds per command while still maintaining the accuracy of 99.3% and an ITR of 105 bits/min. In contrast, subject 4 may have the same accuracy but the response time was 3.4 times slower than that of subject 1 causing significantly lower ITR of 30.98 bits/min. It was observed that although some of the subjects have an accuracy of more than 90% the response time it took to accomplish the task was 8 times slower than the best time as manifested by subject 3.
Overall, the subjects were able to achieve an average classification accuracy of 89.51% for the whole training experiments with a mean of 4.16 seconds per command. It can also be noted that the average ITR of the group is 42.92 bits/min.
Based on the results it was discovered that although the subject is only required to focus on a single flickering box there were still some instances wherein the wrong responses were produced. The erroneous results could be the effect of the other boxes flickering simultaneously near the desired frequency. It was then observed that the subjects are highly sensitive to the first 4 frequencies from 6 Hz to 9 Hz but have a slightly lower sensitivity at the 10 Hz (elevate) selection as shown in Figure 5.
The summarized results of the SPT experiment are shown in Table 2. The subjects were required to activate all the commands correctly, thus the similarity in the number of correct selections. It is also evident that a larger total number of selections can be deduced as the occurrence of multiple errors during the selection progress. The average accuracy is calculated to be 92.31% with an average time of 7.45 s. The average ITR was 20.88 bits/min. It was observed that the bit rate for each subject is relatively the same during the CST and SPT experiments as shown in Figure 6. However, more experiments will be conducted to determine the learning process and how to improve the bit rate of the subjects.
The experiments were then compared with the ITR results of the system from previous SSVEP studies that made use of LCD screen with frequencies in the low and medium frequency bands [10]. Results were satisfactory with some subject’s exhibiting results close to those of the best performing ITRs in previous studies. The average bit rate for both the CST and SPT was 31.9 bits/min and an average accuracy of 90.91% with both tests performed by the 12 subjects. Subsequently, the current application is also more convenient due to the use of only a single channel EEG as compared to multiple channels used in previous studies such as [5, 10, 15]. In comparison to the research of [13], the bit rate of their BCI spelling system was 37.62 bits per minute and an accuracy of 92.25%. Their system focuses mainly on 3 out of the 5 commands available. Studies investigated by Zhu et al. showed ITR ranging from 7.5 bits/min to 68 bits/min [12]. The peaks amplitudes of the SSVEP were observed to be large near the 10 Hz range that concurs with the study of [17]. One inherent disadvantage of the frequencies used in this study is that it falls between 5 and 25 Hz which is considered as uncomfortable compared to its higher frequency counterparts. Concurrently, the use of flash and pattern reversal stimuli within a 20 Hz interval from 5 and 25 Hz can elicit epileptic seizures and the subjects got tired more easily [18]. The task of integrating the BCI system with the remote control device offers a wide range of opportunities to entertainment devices that can help improve the quality of life of the disabled patients.
4. Conclusions
In this paper, we have successfully integrated a BCI remote-control system with a remote controlled device in the form of Air Swimmer that exhibited high accuracy and with a fast response time. It should be noted that in order to achieve high accuracy and fast response time, the subjects were trained prior to the simulated test. It can be deduced that evaluating the performance of the subject while using the BCI system requires time and training as current results of the CST and SPT showed minute change. The resulting ITR was within the range of BCI systems using LCD screens. The performance of the system using only a single channel EEG showed an average ITR of 32 bits/min while performing at an average above 90% accuracy as compared to other multielectrode BCI. In the future a more convenient EEG acquisition device could be developed and integrated with an external device that contains the visual stimulators and also a standalone signal processing module would provide a practical and convenient method for future BCI remote controls. Improvements on the feature extraction method through the use of statistical analysis will also be done to reduce misclassification.
Acknowledgment
The authors would like to thank the National Science Council for funding this project under Grant no. NSC-100-2632-E-218-001-MY3.