Abstract
Brain-computer interfaces represent a range of acknowledged technologies that translate brain activity into computer commands. The aim of our research is to develop and evaluate a BCI control application for certain assistive technologies that can be used for remote telepresence or remote driving. The communication channel to the target device is based on the steady-state visual evoked potentials. In order to test the control application, a mobile robotic car (MRC) was introduced and a four-class BCI graphical user interface (with live video feedback and stimulation boxes on the same screen) for piloting the MRC was designed. For the purpose of evaluating a potential real-life scenario for such assistive technology, we present a study where 61 subjects steered the MRC through a predetermined route. All 61 subjects were able to control the MRC and finish the experiment (mean time 207.08 s, SD 50.25) with a mean (SD) accuracy and ITR of 93.03% (5.73) and 14.07 bits/min (4.44), respectively. The results show that our proposed SSVEP-based BCI control application is suitable for mobile robots with a shared-control approach. We also did not observe any negative influence of the simultaneous live video feedback and SSVEP stimulation on the performance of the BCI system.
1. Introduction
The idea of remotely operated vehicles without the necessity of manual supervision concerns humankind since the end of 19th century. The first public demonstration of a wireless submarine was performed by Nicola Tesla in 1898 [1]. The first automatically operated vehicles utilizing autonomous behavior were Elmer and Elsie, built by neurobiologist William Grey Walter in Bristol, England, in the late 1940s [2]. Since then, numerous areas of application for robot vehicles (military, environment, and healthcare) have emerged. In healthcare applications, robotic vehicles are primarily developed as assistive devices for severely disabled or paralyzed patients or patients suffering from a stroke [3]. For some people with severe motor impairments like amyotrophic lateral sclerosis (ALS) or spinal cord injury (SCI), traditional control interfaces, such as a 2-axis joystick, a keyboard, or a mouse, are not suitable, as they require precise movement control over the limbs, which in their case is not possible due to nerve cells degradation or spinal nerve damage. For these people, accurate control interfaces can be realized through modern technologies such as brain-computer interfaces (BCIs). The term brain-computer interface (BCI) or brain-machine interface (BMI) represents a range of technologies developed intensively over the past 20 years. BCIs introduce additional communication pathways that translate brain signals directly into computer commands (without the use of muscles or nerves) and therefore provide people with minor to severe disabilities with a new opportunity to enhance their quality of life; for some people, BCIs offer a new way of communicating and, for others, a new way to control their environment (e.g., wheelchair, prosthesis) [4–6].
Service robots designed to assist paralyzed people are a popular research topic in the literature on BCIs [7]. Technical solutions for such robots are developed with different BCI paradigms. In the last decade, numerous robot control BCI applications were developed.
Ron-Angevin et al. used a SMR-based two-class BCI application to choose between four direction commands in order to control a robot [8]. Three out of four subjects were able to control their BCI robot. Tonin et al. developed a shared control of two mental states’ brain-machine interface for steering a telepresence robot located at a distance of 100 km [9]. All four subjects (two healthy and two with motor disabilities, myopathy) successfully controlled the robot with a similar performance. Huang et al. used an event-related desynchronization/synchronization of electroencephalogram signals to control a 2D, constantly moving, virtual wheelchair [10]. Four out of five healthy subjects reached mean target hit rates of 87.5% and 100% with motor imaginary in two virtual games.
Several studies used the P300 paradigm (another commonly used BCI approach using the 300 ms component of an evoked potential). Chella et al. presented a Museum Robotic Guide GUI for P300-based BCI developed at the San Camillo Institute [11]. The user can choose between two robots (Pioneer3 or PeopleBot), each located in a different place, to visit the museum. Escolano et al. reported a P300-based telepresence system that enables the user to be present in remote environments through a mobile robot using a shared-control strategy [12].
Reliable BCIs can also be realized with steady-state visual evoked potentials (SSVEPs). SSVEPs are the continuous brain responses elicited at the visual and parietal cortical areas under visual stimulation with a specific constant frequency [13]. Shortly, when looking at a light source that flickers with a constant frequency, the corresponding brain signals are measured with the EEG from the occipital cortex. As the specific stimulation frequencies are known a priori, the stimuli the user focuses its gaze on can be detected through classification algorithms that detect the stimulation frequency with the strongest amplitude, that is, through analyzing the signal to noise ratios.
Visual evoked potentials (VEP) recorded in the 1960s by Regan [14] and early tested in the 1980s and 1990s extended our knowledge about the human brain. For example, as early as 1979, Frank and Torres evaluated the use of visual evoked potentials in diagnosis of “cortical blindness” in children [15]. VEP can also support the diagnosis of multiple sclerosis [16].
At the beginning of the 21st century, the progress of computing power and technology allowed the extensive research and development of SSVEP and its use in BCI. A recent review of visual and auditory BCIs can be found in [17]. Various applications like spelling interfaces [18] and control applications for a prosthesis [19] or for navigation [20] can be implemented with the SSVEP paradigm. In a large demographic study with 86 subjects, Volosyak et al. presented a four-class, SSVEP-based BCI with LED stimulation [21]. 84 subjects reached a mean (SD) accuracy and ITR of 92.26 (7.82)% and 17.24 (6.99) bits/min, respectively, with stimulation at 13, 14, 15, and 16 Hz (medium frequency set). Only 56 subjects were able to operate the BCI at high frequencies (34, 36, 38, and 40 Hz), reaching a mean accuracy and ITR of 89.16 (9.29)% and 12.10 (7.31) bits/min. Zhao et al. introduced a three-layer decision tree for humanoid robot BCI interaction [22]. The SSVEP stimulation was presented on a 17′′ LCD monitor. Four boxes were flickering at 5, 6, 8, and 10 Hz with an additional 17 Hz red LED on top of the monitor. Five young subjects tested the developed software platform. The mean performance accuracy achieved by four of them was approximately 88%. Guneysu and Levent Akin presented a four-class SSVEP-based BCI with LED stimulation at 7, 9, 11, and 15 Hz; an Emotiv EPOC headset was used for EEG data acquisition and MATLAB software (FFT with Gaussian model) was used for dominant frequency extraction [23]. Three subjects achieved an average performance of approximately 68% at a distance of 22 cm from the LEDs. Holewa and Nawrocka presented an SSVEP-based BCI for Lego Mindstorm robot control with an Emotiv EPOC headset [24]. For visual simulation, four LED-based panels with stimulation frequencies of 28, 30, 32, and 34 Hz were used. The panels were additionally covered with different color filters. Three subjects operated this setup with a mean accuracy and ITR of 73.75% and 11.36 bits/min. Kwak et al. presented an SSVEP-control system for the exoskeleton REX [25]. Five LEDs blinking at 9, 11, 13, 15, and 17 Hz were used to select between the exoskeleton commands forward, left, standing, right, and sitting, respectively. Three subjects achieved a mean accuracy of 99.16%. In an f-VEP based BCI, each target is flashing at a different constant frequency, causing a periodic sequence of evoked responses with the same fundamental frequency as that of the flickered stimulus as well as its harmonics [26], and in a c-VEP BCI usually a pseudorandom m-sequence phase-shifted between targets is flashing (m-sequence with an autocorrelation function which is a very close approximation to a unit impulse function, and it is nearly orthogonal to its time lag sequence) [26]. Kapeller et al. evaluated a frequency-coded f-VEP (8.57, 10, 12, and 15 Hz) system and code modulated c-VEP (63-bit pseudorandom m-sequence) as a continuous robot steering input with video feedback of the robot movement [27]. Eleven healthy subjects achieved a maximum accuracy of 91.36% and 98.18% for f-VEP and c-VEP, respectively, in an online accuracy test run without zero class. Wang et al. presented a handmade remote control electrical car, which was driven using an SSVEP stimulation with blue LEDs (controlled via ATmega8L microcontroller) [28]. Combining Average and Fast Fourier Transform calculations, they achieved an average accuracy of 85, 91, 94, 98, and 100% for the different trial lengths of 0.5, 1, 2, 4, and 8 seconds, respectively. The EEG signal was recorded from electrodes and of the international 10–20 system. Gergondet et al. used SSVEP stimulation with approximated frequencies (frequency coding, method originally proposed by Wang et al. [29]) for steering a humanoid robot (HRP-2) [30]. Four stimulation arrows (6, 8, 9, and 10 Hz) were presented on a 17′′ LCD screen with 60 Hz vertical refresh rate. Also presented in a background area of the screen was a live video stream from the robot’s point of view. Their research focused on the impact of the background, dynamic compared to static. Diez et al. tested an SSVEP-driven Pioneer 3-DX mobile robot equipped with a video camera for live feedback and an obstacle avoidance ultrasonic sensor system [31]. The high frequency (37, 38, 39, and 40 Hz) visual stimulation was generated with an FPGA card and stimulation modules (LED-based) were placed on the outer edge of the monitor in a top, left, right, and bottom arrangement, and the camera image was displayed live on the monitor. BCI commands were classified using a sliding window FFT method. Seven participants navigated the robot to two different office locations. Some recent studies combine at least two of the BCI paradigms to realize a so-called hybrid BCI. Choi and Jo combined the SSVEP with event-related desynchronization (ERD, typical SMR-based BCI) for the navigation/exploration of a humanoid robot, with additional P300 BCI for interaction (object recognition, decision) [32]. They also tested and evaluated the use of a low-cost BCI system. The ERD and SSVEP BCIs were two-class systems; the number of classes for the P300 BCI varied (2 to 4, depending on the detected objects). Five subjects used a low-cost Emotiv EPOC headset to perform the experiment. The overall accuracies achieved by the ERD and SSVEP protocols were 84.6% and 84.4%, respectively; the P300 protocol reached an accuracy of 91% for two objects and 89.5% for four objects through the robot’s camera. The scored ITR value for the navigation/exploration task was 18.1 bits/min and for the objects recognition task 21 bits/min. Authors reported that switching between SSVEP and ERD caused a drop in accuracy down to 73%. The robot’s point of view was shown in the middle of the monitor (live feedback), the SSVEP stimulation was shown at the sides, and the P300 was shown directly in the live feedback of the robot’s view. Tidoni et al. presented a study where nine subjects (located in Italy) remotely controlled a humanoid robot (HRP-2, located in Japan) [33]. Subjects tested a synchronous, asynchronous audio-visual feedback and evaluated whether a mirror reflection of the robot (in its field of view) improved the performance of an SSVEP-based BCI with video feedback. The main task was to navigate the robot and to move a bottle from one table to another. A six-class SSVEP-based BCI was used, the stimulation and the video feedback were presented on the same screen. The flashing frequencies were 6, 8, 9, 10, and 14 Hz, with an additional zero class when the subject did not gaze at any target. Due to the remote distance, the feedback action was visible to the subject after approximately 800 ms. Zhao et al. compared a four-class SSVEP-based BCI to a six-class P300-based BCI by evaluating achieved control accuracies of robot navigation tasks [34]. For both systems (SSVEP and P300), the modular platform Cerebot was used and stimuli were presented on a LCD monitor. Seven subjects took part in the experiment and achieved a mean accuracy and ITR of 91.3% and 18.8 bits/min for the SSVEP approach; for the P300 approach, 90.3% and 24.7 bits/min were achieved.
Rutkowski et al. implemented a robot control application using tactile BCIs [35]. They tested two tactile stimulation configurations: a pin-pressure BCI (arranged in a matrix) and a full body vibrotactile BCI (vibrotactile generators placed on subjects backs, shoulders, arms, and legs). A humanoid NAO robot was navigated using six predefined commands. Five subjects could control the robot using the pin-pressure BCI and three subjects successfully tested the full body vibrotactile BCI. The authors demonstrate the reliability of the tactile BCI paradigm for robot navigation.
All listed studies have to deal with the main engineering problems regarding BCI technologies. EEG-based BCIs are relatively slow and can sometimes be unreliable. Maximizing the speed of a BCI while still maintaining high accuracy leads to certain limitations in the number of targets and in system speed (target selection). Moreover, it is often reported that a certain number of subjects are not able to control BCIs (so-called BCI illiterates, e.g., [21, 36]). We chose the SSVEP paradigm, as it is reported to be among the fastest and most reliable BCI strategies, with short or no training time needed [13]. In our recent study, after five years of research, we developed an SSVEP calibration software (SSVEP Wizard) for LCD monitors that determines SSVEP key parameters for each subject [37]. The selected stimulation frequencies are those with the highest SSVEP response in the visual cortex. We have shown that, with optimized parameters, the SSVEP BCI literacy rate can reach 100%. This calibration software is also used here to determine and optimize the user-specific SSVEP parameters. Some SSVEP-based applications described problems regarding video feedback. The research of Gergondet et al. was focused on the impact of the background (dynamic compared to static) on the BCI performance. They reported that a dynamic background negatively affected the SSVEP stimulation [30]. Kapeller et al. noted a slight decrease in performance for an SSVEP-based BCI with background video and small targets [27]. In the approach presented by Diez et al., the stimulation was separated from the video feedback stream (outer LEDs), but no negative influence of the live feedback was reported [31].
SSVEP-based BCI systems for robot control usually use four or five control commands. A higher degree-of-freedom control is difficult to realize, because multitarget BCIs usually have a lower accuracy. A countermeasure to the limitations in degrees of freedom can be a shared-control approach; that is, the robot can share some specific control commands with the human user which allows for the faster execution of complex tasks. The so-called shared-control principle was initially explored for BCI-controlled wheelchairs and recently applied to BCI telepresence [12]. BCIs with only a few targets are more robust and therefore more suitable for control applications. There are already numerous existing BCI technologies that patients use at their homes [38, 39]. However, for the last decade, publications about SSVEP-based BCIs have been sparse. The PubMed database shows 4200 results for the search term “Brain-computer interface”/“Brain-machine interface” but only 185 results for the term “SSVEP”. With this paper, we would like to contribute to the research of SSVEP-based BCIs.
In this paper, we present a four-class, SSVEP-based, BCI control application with live video feedback. We also present a self-made, low-cost, semiautonomous mobile robotic car (MRC) with live video feedback, controlled by a BCI application. Unlike the robotic cars developed in the abovementioned research studies, not only the vehicle itself but also the camera’s pan-tilt-position was controlled via SSVEP. The main task of this BCI study was to maneuver the MRC through a predetermined route and to examine the surroundings (live scenario task). The BCI performance was tested with 61 subjects. BCI key parameters, for example, stimulation frequencies, were determined individually by Wizard software. In this study, we tested the following hypotheses:(i)Is it possible to use an SSVEP-based BCI stimulation and a background video on the same screen, without the negative influence of the presented video, as reported in, for example, [27, 30]?(ii)How does the video feedback affect BCI performance when simultaneously presented with the SSVEP-based BCI stimulation?(iii)Is our self-made, semiautonomous mobile robotic car suitable for future use as an assistive technology?(iv)How did the subjects perform using the presented SSVEP-based BCI control application, compared to results presented in, for example, [31]?
The paper is organized as follows: Section 2 describes the experimental setup and presents details about the experimental procedure and the hardware and software solutions. The results are presented in Section 3, followed by a discussion and conclusion in Sections 4 and 5, respectively.
2. Material and Methods
2.1. Subjects
Everyone who volunteered to participate in the study became a research subject after reading the subject information sheet. This study was carried out with written informed consent from all subjects in accordance with the Declaration of Helsinki. subjects (30% female) with a mean (SD) age of 22.62 (5.00) years participated in the study, all healthy and all students or employees of the Rhine-Waal University of Applied Sciences in Kleve. The EEG recording took place in a normal PC-laboratory room (≈36 m2). The LCD screen was located in front of windows and the luminance was kept at an acceptable level, in order not to disturb the subjects. If necessary, outer blinds were pulled down during the day, or light was turned on during the evening. None of the subjects had neurological or visual disorders. Spectacles were worn when appropriate. Subjects did not receive any financial reward for their participation in this study.
2.2. Hardware
2.2.1. Mobile Robotic Car (MRC)
Four identically constructed mobile robotic cars (MRCs) were used in the experiment, all set up with the same hardware and software configurations. Four wheels were attached to a preassembled case (14 cm × 19 cm) with built-in DC motors. A rechargeable battery (6x AA NiMH, 1900 mAh) was placed inside. Four light sensors (each with a corresponding red LED, L-53SRC-F, for luminescence positioned side by side) were mounted on a prototype board at the front of the MRC. Two of those sensors were connected to the analog input pins of a BeagleBone Board Rev A6 at the center and two at the sides. The DC motors were controlled by an Atmega168-based unit. The camera, Logitech C210 (V-U0019), was mounted in front of the MRC on two servos (RB-421, RobotBase, China), simulating a pan-tilt head and connected to the BeagleBone USB port. The camera was controlled in five predefined orientations: looking down, forward, up, to the left, and to the right. The connection with the MRC was established through UDP via a HAMA N150 2in1-WLAN-ADAPTER, connected with an Ethernet cable to the BeagleBone of the MRC, and a USB WiFi adapter (D-Link, DWL-G122) connected to the desktop computer. All the electronic components (boards and HAMA N150 Access-Point) were mounted on top of the MRC. The BeagleBone was equipped with an 8 GB SD-card, running Ubuntu ARM (Linux distribution, version 12.04) with MJPG-Streamer installed (version 0.1, http://mjpg-streamer.sf.net) for video streaming. One of the used MRCs is shown in Figure 1.
2.2.2. Data Acquisition and Feature Extraction
The subjects were seated in front of a 24′′ LCD screen (BenQ XL2420T or BenQ XL2411T, resolution: pixels, vertical refresh rate: 120 Hz) at a distance of approximately cm. The computer system based on an Intel Core i7, 3.40 GHz, was running Microsoft Windows Enterprise. Standard passive Ag/AgCl electrodes were used to acquire the signals from the surface of the scalp. The ground electrode was placed over , the reference electrode was placed over , and eight signal electrodes were placed at predefined locations on the EEG-cap marked with , , , , , , , and according to the international system of EEG electrode placement. Standard abrasive electrolytic gel was applied between the electrodes and the scalp to bring impedances below 5 kΩ. An EEG amplifier g.USBamp (Guger Technologies, Graz, Austria) was utilized. The sampling frequency was set to 128 Hz. During the EEG signal acquisition, an analog bandpass filter between 2 and 30 Hz and a notch filter around 50 Hz were applied directly in the amplifier.
The minimum energy combination (MEC) method was originally introduced by Friman et al. in [40]. In this study, a refined version of the MEC, as presented by Volosyak in [18], was used. In the following, we present this method in a very short form.
2.2.3. SSVEP Signal Model
We can model the signal recorded at a given electrode (against reference electrode at time ) as follows:The first part is the SSVEP evoked response consisting of sine and cosine functions of the frequency and its harmonics , with corresponding amplitudes and . The last part represents the noise component of the electrode , other activities not related to the SSVEP evoked response, and artifacts. The th EEG signals samples stored in are in vector form , where is vector. is the SSVEP model matrix of size and contains the and pair in its columns. The vector contains the corresponding amplitudes. This is based on the assumption that the nuisance is stationary within the short time segment length; the signals from the electrodes can be stored in a vector form .
2.2.4. Minimum Energy Combination
In order to cancel out the nuisance and noise and to boost the SSVEP response, channels are created in a weighted sum of electrodes . Several sets of channels can be created with a different combination of the original electrode signals , where is matrix that contains the different weight combinations. The structure of is based on the minimum energy combination technique. In order to use this combination method for cancelling the nuisance signal, in a first step, any potential SSVEP component needs to be removed from the signal: . The next step is to find a weight vector that minimizes the resulting energy of the combination of the electrodes’ noise signals : . The last step is the optimization problem for minimalizing the nuisance and noise component:The solution to the minimize problem is the smallest eigenvector (of the symmetric matrix ) and the energy of the resulting combination equals the smallest eigenvalue from this matrix. In order to discard up to 90% of the nuisance signal, the total number of channels is selected by finding the smallest value for that satisfies the following equation:
To detect the SSVEP response for a frequency, the power of that frequency and its harmonics is predicted:To prevent overlapping between frequencies, we use in the actual system implementation. The estimation of the power of the SSVEP stimulation frequencies, and additional frequencies (), is normalized into probabilities:where is the power estimation of the th signal, . To increase the difference between the results, we apply a Softmax function:where is set to . The output values of the Softmax function are between and and their sum is equal to . The classification output of the signal of th frequency is a result of three conditions: the probability of the th frequency is the highest, it surpasses a predefined border , and the detected frequency is one of the stimulation frequencies. After each classification, to prevent wrong classifications, the classifier output was rejected for the next 9 blocks (approximately 914 ms of EEG data with the sampling rate of 128 Hz), and the visual stimulation was paused for that time period. This classification method is the next iteration of the Bremen-BCI as presented, for example, in [18]. An example of the recorded signal and the SNR can be seen in Figure 9.
The SSVEP classification was performed online every 13 samples (ca. 100 ms) on the basis of the adaptive time segment length of the acquired EEG data introduced in [18]. If no classification was possible, the time segment length window was gradually extended to the next predefined window lengths (see Table 1 and Figure 2).
2.3. Software
2.3.1. Graphical User Interface
A custom-made graphical user interface (robot-GUI) was designed to control the MRC. DirectX 9 was used to draw the images (buttons and a real-time picture from the actual camera view in background) on the monitor, and OpenCV library (version 2.4.10, http://www.opencv.org/) was used for image acquisition. Each received frame was rendered into background texture as soon as it was received (max. 30 Hz; the frame rate was limited by the used camera). Each received frame ( pixels) was rescaled to . Due to the ratio of the streamed video frames and the resolution of the monitors used, a gray background in the form of a frame was implemented to prevent unnatural video scaling. The starting dimensions of the buttons were pixels for left and right, pixels for the top, and pixels for bottom buttons (see Figures 3 and 4). During the runtime, the size of the buttons changed continuously, depending on the calculated probabilities of the th stimuli in (6). The robot-GUI had two modes: a drive mode and a camera mode. In drive mode, the subject was able to send the following commands to the MRC: “drive forward” (a discrete command, using the line-sensors to follow the taped line on autopilot to the next direction change), “turn left,” “turn right,” and “switch to camera mode.” In camera mode, the subject was able to send the following commands: “look left,” “look right,” “look up,” and “switch to drive mode.” While looking up, the button “look up” changed the content and functionality to “look forward.” The control buttons were red in drive mode and yellow in camera mode to increase user friendliness. The aspect ratio of the video stream was 4 : 3, so the computer screen was not entirely filled out. If the command “drive forward” was selected by the user, an autonomous program was executed on the MRC. The MRC used the four line-sensors to follow the white line taped to the floor until it arrived at a turning point, where the robot stopped automatically. While driving, the MRC automatically made small corrections when the line was detected by one of the inner sensors; the rotation of the wheels on the same side was decreased by 50% until another sensor detected the line. Similarly, when the line was detected by one of the outer sensors, the wheels on the same side were stopped so that the MRC could drive in a curve, in order to go back on track. If one of the turning commands was selected, the MRC made an approximately 90-degree turn in the corresponding direction. In drive mode, the camera was facing slightly the ground (approximately 20 degrees down from the frontal view for the “look down” position). When switching to camera mode (represented by the word “CAMERA” at the bottom of the robot-GUI), the camera changed to the “look forward” position (switching back to drive mode would change the camera position to looking down again). All the EEG data and the received video stream from the MRC were anonymously stored.
2.3.2. Wizard
The Wizard software is, generally spoken, a calibration program that autonomously determines BCI key parameters [37]. It is designed to select user-specific optimal stimulation frequencies for SSVEP-based BCIs on LCD screens. It is a development of the method presented in [41]. The Wizard chooses four frequencies with the best SSVEP response out of a set of 14 possible stimulation frequencies (ranged from 6 Hz to 20 Hz) that can be realized on a monitor screen with a vertical refresh rate of 120 Hz [42]. In the following, we provide a brief description.
Step 1 (alpha test). The subject was instructed by an audio feedback to close his or her eyes. 15 frequencies in the alpha wave band (8.27, 8.57, 8.87, 8.93, 9.23, 9.53, 9.70, 10.0, 10.30, 10.61, 10.91, 11.21, 11.70, 12.0, and 12.30 Hz) were measured for 10 seconds during the closed eyes period and if any of those 15 frequencies would interfere with a possible stimulation frequency ( Hz), this frequency would be filtered out.
Step 2 (stimulation frequency selection). In order to avoid the influence of harmonic frequencies, the user had to look at two circles in sequence. Each circle contained seven stimulation frequencies which were presented in pseudorandom order as small, green circle segments (see Figure 5). This step took 20 seconds.
(a)
(b)
Step 3 (optimal classification thresholds and time segment length). In this step, the Wizard software chose the best starting time-length segment as well as the optimal classification thresholds for the frequencies determined in the previous step. First, a white circle, flickering at the frequency which had the highest SSVEP response in phase 2, was displayed. It was necessary to simulate noise caused by peripheral vision when concentrating on the target object. For this purpose, the white circle was surrounded by a green ring, containing segments which presented the remaining optimal frequencies (see Figure 6). The user was instructed by an audio feedback to gaze at the white circle. The circle and the ring flickered for 10 seconds while EEG data were recorded. After a two-second break, the flickering continued. The white circle now flickered with the second-highest frequency from phase two, while the ring flickered with the remaining three frequencies. This procedure was repeated until the data for all four optimal frequencies were collected. Total recording time for phase three was 40 seconds, and based on this data, optimal thresholds and time segment length were determined. If one of the frequencies did not reach a defined, minimal threshold , this frequency was replaced with the next best frequency from Step 2, and the procedure of this step was repeated.
(a)
(b)
2.4. Procedure
In order to analyze the BCI performance variety, a pre- and postquestionnaire system (similar to [21]) was employed. After signing the consent form, each subject completed a brief prequestionnaire and was prepared for the EEG recording. Subjects were seated in a comfortable chair in front of the monitor, the electrode cap was placed, and electrode gel was applied. First, the subjects went through the steps of the Wizard and BCI key parameters (stimulation frequencies, classification thresholds, and optimal time segment length) were determined. Then, the experimenter started the robot-GUI (Figure 3). The control task was to drive the MRC through a predetermined route (Figure 8). If the robot-GUI software recognized a wrong command, for example, “turn right” instead of “turn left,” this command was not sent, so that the MRC would not drive off the track (Figure 7). If the subject made a few mistakes at the beginning and was willing to start over again (this occurred with a small number of subjects), the experimenter moved the MRC back to the starting point and restarted the robot-GUI. In the very few cases where the wireless connection to the MRC was interrupted (e.g., the MRC was mechanically unable to execute the command that was correctly selected in the robot-GUI by the user), the experimenter manually repositioned the MRC to the next turning point.
(a)
(b)
At a certain turning point (represented by a blue square shown in Figure 8, when approximately of the driving task was completed), the camera was facing a text message, which instructed the subjects to look up (user action represented with a circled number as, e.g., ⑭ in this case). At this point, the user had to switch to camera mode and select the command “look up.” Now the subject was able to see a hidden secret number which was attached to the bottom of a drawer (located at a height of 50 cm, facing the floor, not normally visible to the people in the room). The numbers varied randomly between the subjects and were only accessible via the MRC camera. After seeing the number on the screen, the subject had to change back to drive mode and continued the driving task. When arriving at the finish line, the camera faced another text message which again instructed the subjects to look up. After changing to camera mode and selecting “look up,” the camera faced a sign displaying the word “finish” and the robot-GUI stopped automatically. All of the EEG data and performed commands were stored on the hard drive of the desktop PC for further evaluation. The values of ITR and accuracies were calculated automatically by the software. The ITR was calculated using the formula presented by Wolpaw et al. in [43]; the number of possible choices for classification was equal to four; it was the same for each mode (drive mode or camera mode). After finishing the experiment, the subject filled out the postquestionnaire.
3. Results
The results of the driving performance are shown in Table 3. All 61 subjects were able to control the BCI application and completed the entire driving task. The overall mean (SD) ITR of the driving performance was 14.07 (4.44) bits/min. The mean (SD) accuracy was 97.14 (3.73)%. Except for one, all subjects reached accuracies above 80%. 40 subjects reached accuracies above 90%, and nineteen of them even completed the steering task without any errors. The mean (SD) accuracy of each action (commands required to complete the task) was 96.14 (0.02)%. In the prequestionnaire, subjects answered the questions regarding gender, the need for vision correction, tiredness, and BCI experience, as displayed in Table 2. Eighteen female subjects (30%) with an average (SD) age of 22.67 (4.30) years and 43 male subjects with an average (SD) age of 22.60 (5.32) years participated in the experiment. The influence of gender was also tested, 18 female subjects achieved a mean (SD) accuracy and ITR of 93.24 (7.98)% and 14.05 (5.73) bits/min, respectively, and 43 male achieved a mean (SD) accuracy and ITR of 92.95 (6.59)% and 14.07 (5.08) bits/min, respectively. A one-way ANOVA showed no statistically significant difference in accuracy , , and ITR , , between those two groups. The influence of vision correction on accuracy and ITR was also tested. A total of 22 subjects required vision correction in the form of glasses or contact lenses (as they stated in the questionnaire). They achieved a mean (SD) accuracy and ITR of 91.56 (6.36)% and 12.78 (4.02) bits/min, respectively. These results were compared to subjects who did not require vision correction and achieved a mean (SD) accuracy and ITR of 93.92 (5.35)% and 14.84 (4.66) bits/min, respectively. A one-way ANOVA showed no statistically significant influence of vision correction on accuracy , . There was also no significant effect of vision correction on ITR , . An evaluation of tiredness was also calculated on the basis of the answers given in a prequestionnaire (scale from 1 to 5). Results of the prequestionnaire are shown in Table 2. A one-way ANOVA revealed no significant effect of tiredness on accuracy , , and ITR , .
4. Discussion
In the results presented, the overall accuracy was calculated for all commands in drive mode and in camera mode without distinguishing between them, since four commands for SSVEP classification were always present. Even though different commands were sent, the command classification remained the same. A comparison of this study to other publications on steering a robot with SSVEP-based BCI is presented in Table 4.
In order to test the influence of video overlay, we compared the accuracy results with a demographic study [21] in which 86 subjects achieved a mean (SD) accuracy of 92.26 (7.82)% while driving a small robot through a labyrinth with a four-class SSVEP-based BCI. An unpaired -test showed no statistically significant difference , , compared to the present values. This shows that the video overlay did not negatively influence the BCI performance. Gergondet et al. listed three factors with respect to how the live video feedback in the background can influence the SSVEP stimulation: distraction due to what is happening on the video, distraction due to what is happening around the user, and variation in luminosity below the stimuli shown to the user [30]. Kapeller et al. showed that there is a slight decrease in performance for the underlying video sequence due to a distractive environment and small targets used [27]. Contrary to the findings reported by Gergondet et al. and Kapeller et al., the dynamic background did not noticeably affect the BCI performance (unpaired -test result). We assume that this was due to the fact that stimulation frequencies and minimal time segment classification windows were adjusted individually and that relatively large stimulation boxes were used (visual angle).
The high accuracies of the SSVEP-based BCI system support the hypothesis that BCI can be used to control a complex driving application. In comparison to the study by Diez et al. [31], the MRC self-driving behavior was limited to a preset number of turning points and a predefined path. Although the robot presented by Diez et al. could avoid obstacles by itself (limited algorithm reaction), it decreased its speed for trajectory adjustment and, in some cases, stopped to avoid collision, and further user commands were necessary. While Diez et al. reported a mean time of 233.20 seconds for the second destination (13.9-meter distance), in this study the achieved mean time was 207.08 seconds (15-meter distance). As pointed out by Diez et al., the overall time for completing the task depends not only on the robot but also on the subject’s performance controlling the BCI system [31]. In our study, the high accuracies of the BCI system were achieved due to the key parameter adjustment carried out by the Wizard software.
The main difference between the robot used by Diez et al. and the MRC presented in this study is the extended control possibility of the interface that also allows for controlling the camera. Both robots were designed for different purposes: Diez et al. mainly designed their robot for driving from point A to point B without any additional interaction with the environment, as they planned to use the system for wheelchair control in the future. By contrast, in our design, the MRC itself is the interaction device. Equipped with a display and a two-way audio system, it might serve as a remote telepresence in the future.
The SSVEP paradigm allows for a robust implementation of four driving commands. Compared to robots constructed for the P300 BCI approach, where the user faces large stimulation matrices spread across the entire screen, the number of simultaneously displayed targets here is only four, and hence the load on the visual channel is considerably lower in comparison. Therefore, the user might find it easier to concentrate on the driving task. On the other hand, the lesser number of command options reduces freedom and speed. However, a shared-control paradigm allows for complex tasks to be executed with minimal effort. The command “drive forward” starts an autopilot program that guides the robot through a straight distance until it reaches a predefined stopping point.
While the MRC executed the autopilot program and followed the line for some time, subjects purposely ignored the flickering targets and concentrated on the video feedback instead. As these waiting periods were added up to the actual classification times to each command that followed an autopilot command (“forward”), the ITR values are considerably low. From the researcher’s point of view, a number of errors could have been avoided if some subjects would have paid more attention to the live feedback. Especially in the task of reading the hidden number, subjects were trying to drive along the path without looking at the text message that appeared on the live feedback. After realizing that the robot was not turning, they looked at the video and saw the text message instructing them to look up. This is visible in the analysis of the accuracy of the robots commands (see Figure 10). The lowest accuracy of 87.49% was for action “change view to the camera mode” (robot command 14); this may be caused by the fact that the subjects got used to the actions “turn left”/“turn right” and “drive forward” which they repeated six times from the starting point (see Figures 10 and 9). This could easily be avoided if the driving paths were designed with evenly distributed tasks (e.g., live scenario tasks, such as reading a hidden number, should occur more often).
No difference between the time needed for the selection of the command “drive forward” in the beginning of the experiment in comparison to the end of the experiment can be observed (see Table 5); hence, neither fatigue nor practice seemed to influence performance.
The general literacy rate among test subjects of 100% indicates that SSVEP-based BCIs can be used by a larger population than previously assumed. This high literacy is due to the careful selection of individual SSVEP key parameters, which were determined by the presented calibration software.
4.1. Limitations
The SSVEP-control application has some restrictions:(i)The wireless connection was implemented with the UDP protocol and some information might therefore not be received properly.(ii)In order to maximize performance accuracy and user friendliness, the number of simultaneously displayed targets was restricted to four, which resulted in a considerably lower ITR.(iii)It should also be noted that, for long term use, recalibration might be inevitable. Although the number of targets is limited to four, the SSVEP-based BCI presented here depends on the user’s vision and control over his or her eye movements. It therefore has to compete with other approaches such as P300 and other eye tracking systems. Visual stimulation with low frequencies in particular is known to cause fatigue.(iv)Moreover, subjects in this study may not be representative of the general population; they tended to be young healthy men. Further tests with older and physically impaired persons are required.
5. Conclusions
In the present study, the SSVEP-based driving application (robot-GUI) was proven to be successful and robust. The driving application with live video feedback was tested with 61 healthy subjects. All participants successfully navigated a mobile semiautonomous robot through a predetermined 15-meter-long route. BCI key parameters were determined by the Wizard software. No negative influence of the live video feedback on the performance was observed. Such applications could be used to control assistive robots but might also be of interest for healthy subjects. In contrast to P300, the SSVEP stimulation has to be sufficiently strong (in terms of size or visual angle) to significantly influence the user. A common problem of all BCI studies that also occurred here is to keep the user motivated and focused on the task. The evaluation of the questionnaire shows that for the majority of subjects the driving task with the SSVEP-based BCI was not fatiguing, even though lower stimulation frequencies, which usually cause more fatigue than higher frequencies, were used in the experiment. More specifically, frequencies between 6 and 12 Hz were determined in the calibration procedure, as they elicited the strongest SSVEP responses. Some subjects stated that as they concentrated more on the camera video, they did not find the flickering annoying at all. Our results prove that SSVEP-based, personalized, adaptive BCI systems with a low target number can be used as an everyday assistive technology for remote robots. We have also shown that it is possible to design a four-target interface with extended capabilities such as camera and driving control. The control application presented here can be used for mobile robots, as an assistive remote technology, as a game interface, for wheelchair control, or even for steering remotely controlled drones. This study should be considered as a step towards resolving the problems of semiautonomous mobile robots controlled with SSVEP-based BCIs. In order to further improve this approach, the next steps should be an online parameter adaptation mechanism for improving the accuracy through practicing, a possibility of turning the stimulation on/off, and a two-way video and audio communication for a truly remote telepresence.
Competing Interests
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interests.
Acknowledgments
The authors would like to thank Thomas Grunenberg from the Faculty of Technology and Bionics at Rhine-Waal University of Applied Sciences for building, programming, and maintaining MRCs. The authors would also like to thank the student assistants, Leonie Hillen, Francisco Nunez, Saad Platini, Anna Catharina Thoma, and Paul Fanen Wundu, for their help with the study. This research was supported by the German Federal Ministry of Education and Research (BMBF) under Grants nos. 16SV6364 and 01DR14014 and the European Fund for Regional Development (EFRD or EFRE in Germany) under Grant no. GE-1-1-047.