In recent years, the assistive technologies and stroke rehabilitation methods have been empowered by the use of virtual reality environments and the facilities offered by brain computer interface systems and functional electrical stimulators. In this paper, a therapy system for stroke rehabilitation based on these revolutionary techniques is presented. Using a virtual reality Oculus Rift device, the proposed system ushers the patient in a virtual scenario where a virtual therapist coordinates the exercises aimed at restoring brain function. The electrical stimulator helps the patient to perform rehabilitation exercises and the brain computer interface system and an electrooculography device are used to determine if the exercises are executed properly. Laboratory tests on healthy people led to system validation from technical point of view. The clinical tests are in progress, but the preliminary results of the clinical tests have highlighted the good satisfaction degree of patients, the quick accommodation with the proposed therapy, and rapid progress for each user rehabilitation.

1. Introduction

The worldwide statistics reported by World Health Organization highlight that stroke is the third leading cause of death and about 15 million people suffer stroke worldwide each year ‎[1]. Of these, 5 million are permanently disabled needing long time assistance and only 5 million are considered socially integrated after recovering. Recovering from a stroke is a difficult and long process that requires patience, commitment, and access to various assistive technologies and special devices. Rehabilitation is an important part of recovering and helps the patient to keep abilities or gain back lost abilities in order to become more independent. Taking into account the depression installed after stroke, it is very important for a patient to benefit from an efficient and fast rehabilitation program followed by a quick return to community living ‎[2]. In the last decade, many research groups are focused on motor, cognitive, or speech recovery after stroke like Stroke Centers from Johns Hopkins Institute ‎[3], ENIGMA-Stroke Recovery ‎[4], or StrokeBack Consortium funded by European Union’s Seventh Framework Programme ‎[5]. Important ICT companies bring a major contribution to the development of technologies and equipment that can be integrated into rehabilitation systems. For example, Stroke Recovery with Kinect is a research project to build an interactive and home-rehabilitation system for motor recovery after a stroke based on Microsoft Kinect technology ‎[6].

In the last years, the virtual reality (VR) applications received a boost in development due to VR headset prices that dropped below $1000, allowing them to become a mass-market product ‎[7]. The VR was and still is especially used for military training or video games to provide some sense of realism and interaction with the virtual environment to its users ‎[8]. Now it attracts more and more the interest of physicians and therapist which are exploring the potential of VR headset and augmented reality (AR) to improve the neuroplasticity of the brain, to be used in neurorehabilitation and treatment of motor/mental disorders ‎[9]. However, considering the diversity of interventions and methods used, there is no evidence that VR therapy alone can be efficacious compared with other traditional therapies for a particular type of impairment ‎[10]. This does not mean that the potential of VR was overestimated and the results are not the ones that were expected. The VR therapy must be complemented with other forms of rehabilitation technologies like robotic therapy, brain computer interface (BCI) and functional electrical stimulation (FES) therapy, and nevertheless traditional therapy to provide a more targeted approach ‎[11].

SaeboVR is a virtual rehabilitation system exclusively focusing on activities of daily living and uses a virtual assistant that appears on the screen to educate and facilitate performance by providing real-time feedback ‎[12]. The neurotechnology company MindMaze has introduced MindMotion PRO, a 3D virtual environment therapy for upper limb neurorehabilitation incorporating virtual reality-based physical and cognitive exercise games into stroke rehabilitation programs ‎[13]. At New York Dynamic Neuromuscular Rehabilitation, the CAREN (Computer Assisted Rehabilitation Environment) based on VR is currently used to treat patients poststroke and postbrains injuries ‎[14]. EVREST Multicentre has achieved remarkable results regarding the use of VR exercises in stroke rehabilitation ‎[15].

Motor imagery (MI) is a technique used in poststroke rehabilitation for a long time ago. One of its major problems was that there was not an objective method to determine whether the user is performing the expected movement imagination. MI-based BCIs can quantify the motor imagery and output signals that can be used for controlling an external device such as a wheelchair, neuroprosthesis, or computer. The FES therapy combined with MI-based BCI became a promising technique for stroke rehabilitation. Instead of providing communication, in this case, MI is used to induce closed-loop feedback within conventional poststroke rehabilitation therapy. This approach is called paired stimulation (PS) due to the fact that it pairs each user’s motor imagery with stimulation and feedback, such as activation of a functional electrical stimulator (FES), avatar movement, and/or auditory feedback ‎[16]. Recent research from many groups showed that MI can be recorded in the clinical environment from patients and used to control real-time feedback and at the same time, they support the hypothesis that PS could improve the rehabilitation therapy outcome ‎[1721].

In a recent study, Irimia et al. ‎[22] have proved the efficacy of combining motor imagery, bar feedback, and real hand movements by testing a system combining a MI-based BCI and a neurostimulator on three stroke patients. In every session, the patients had to imagine 120 left-hand and 120 right-hand movements. The visual feedback was provided in form of an extending bar on the screen. During the trials where the correct imagination was classified, the FES was activated in order to induce the opening of the corresponding hand. All patients achieved high control accuracies and exhibited improvements in motor function. In a later study, Cho et al. ‎[23] present the results of two patients who performed the BCI training with first-person avatar feedback. After the study, both patients reported improvements in motor functions and both have improved their scores on Upper Extremity Fugl-Meyer Assessment scale. Even if the number of patients presented in these two studies is low, they support the idea that this kind of systems may bring additional benefits to the rehabilitation process outcome in stroke patients.

2. General System Architecture

The BCI-FES technique presented in this paper is part of a much more complex system designed for stroke rehabilitation called TRAVEE ‎[24], presented in Figure 1. The stimulation devices, the monitoring devices, the VR headset, and a computer running the software are the main modules of the TRAVEE system. The stimulation devices help the patient to perform the exercises and the monitoring devices are used to determine if the exercises are executed properly, according to the proposed scenarios. Actually, the TRAVEE system must be seen as a software kernel that allows defining a series of rehabilitation exercises using a series of USB connectable devices. This approach is very useful because it offers the patient the options to buy, borrow, or rent the abovementioned devices according to his needs and after connection, the therapist may choose the suitable set of exercises.

The TRAVEE system is based on a new and promising rehabilitation concept which implies the augmented/magnified feedback of the movement of the impaired limb and can be successfully applied especially in the early stages of the rehabilitation therapy in order to close the loop that may trigger the mirror neurons ‎[25]. These mirror neurons intermediate learning, indirectly controlling the brain plasticity and the technique is known as mirror therapy for stroke rehabilitation ‎[26]. Despite the advantages of mirror therapy in comparison with other standard techniques, some disadvantages are obvious: it is difficult to explain to a patient how the mirror helps him: monotony, the patient’s condition and position, the lack of challenging task, and so on. ‎[27]. By replacing the physical mirror with a VR headset the patient has the same visual feedback that is needed to close the loop that triggers the mirror neurons but without disadvantages of the mirror therapy mentioned above. Once the patient is immersed in the virtual world he is no longer a disabled person and this has a good impact on patient’s self-esteem. Within the TRAVEE project, encouraging results were obtained for the development of a virtual reality system for poststroke recovery using an inertial movement unit, a glove with sensors, a Myo Armband with electromyography sensors, and an Oculus Rift headset ‎[28]. An alternate implemented system contains a Leap Motion device for patient’s limbs movements monitoring, a VR headset, and a haptic module attached to patient’s arm also offering better results than standard therapy methods ‎[29].

3. Materials and Methods

For the current study, the BCI-FES TRAVEE subsystem is composed of FES as stimulation device, BCI and an electrooculography (EOG) system as monitoring devices, Oculus Rift as VR headset, and a laptop, Figure 2.

The rehabilitation exercise was focused on flexion and extension of hand and fingers (Figure 3). The patient is seated in a wheelchair or normal chair. The FES electrodes are mounted on extensors muscles of both hands as shown in Figure 3 and the FES software module is started in order to determine the FES parameters (intensity and timings of the current impulse: rising, front, and falling). Then, the EOG electrodes and EEG helmet are mounted and the correct acquisition of the signals is verified. Before attaching the VR headset, the therapist sits in front of the patient explaining what he will see by showing him the following: the virtual therapist will raise the hand like in Figure 3 (the left hand of the therapist is the right hand of the patient); a big arrow will appear on the upper left or right of the screen depending on virtual therapist indications and the patient will also hear sounds from the left or the right. After explanations, the VR headset is mounted on (Figure 4), EOG system is calibrated, and the recovery exercise may begin, but not before the real therapist tells the patient that he has the possibility of choosing between two views: front view (the virtual therapist is located in front of the patient) or mirror view (the virtual therapist is located on the left side and a mirror is in front of them, like in a dance room) presented in Figure 5.

For the EOG calibration, a red spot appears for 2.5 seconds on a white background displayed on the VR system in different places, in the following order: center, upper right, center, upper left, center, lower left, center, lower right, and center. The user has to gaze at the spot in each location. The calibration is very important for an accurate calculation of the gaze points (eye tracking) during the tests.

In order to provide VR and FES feedback according to the patient’s imagined movement, a set of spatial filters and classifier have to be created ‎[22]. First, we are recording 4 runs of training data. Each run consists of 20 right- and 20 left-MI trials, in a random order. We use the trial time course and signal processing algorithms presented in ‎[22]. Each trial lasts 8 seconds. At second 2 a beep informs the user about the upcoming cue. At second 3, the cue is presented and marks the moment when the user has to start imagining the movement shown by the virtual therapist until the end of the trial. While recording the test data, starting with second 4.25, the user sees the virtual hand indicated by the cue moving, and at the same time, the neurostimulator induces the patient’s corresponding hand opening. After the spatial filters and classifier are created, we are recording 2 more runs, where the VR and FES feedback are provided to the patient between seconds 4.25 and 8 of each trial only if the classification result is correct. By comparing every sample of the classification result with the presented cue for each trial during the last 2 runs, we are calculating a control error rate course for that session. Except the first session, while recording the 4 train data runs, we are using the set of spatial filters and classifier calculated in the previous session of that patient only if the control error rate for that session was smaller than 20%.

4. EEG and EOG Recording

The BCI-FES subsystem consists of a 16-channel biosignal amplifier (g.USBamp, g.tec medical engineering GmbH) and an 8-channel neurostimulator (MOTIONSTIM8, KRAUTH+TIMMERMANN GmbH). The EEG signals are collected from 12 positions over the sensorimotor areas according to the 10–20 International System, as seen in Figure 6(a). The last four channels are used in differential mode to record the vertical and horizontal EOG. Figure 6(b) presents the EOG electrodes position of the subject’s head. The EEG and EOG data are sampled at 256 Hz and notch-filtered for excluding the 50 Hz noise. The EEG data are bandpass filtered between 8 and 30 Hz and then fed to the processing algorithm that performs spatial filtering with the Common Spatial Patterns (CSP) method ‎[30, 31] and Linear Discriminant Analysis (LDA) classification ‎[22, 32]. The EOG data are filtered with a moving average filter in order to calculate the average of the last 128 samples.

To acquire EOG signals the same EEG device was used but from all the EEG electrodes of the gTec–g.USBamp, 4 of them were used for EOG signals. The eye tracking is necessary because patient needs constant motivation and attention during training/recovering session from a therapist. In fact, after a while, the patient does not pay attention any more, is falling asleep, or is looking at/thinking of something else. By using the electrooculography (EOG) based eye tracking, the system is able to determine if the patient is concentrated and warns the patient if he is not. Figure 7 presents the output of the implemented algorithm for detecting the gaze point of the subject on the image in front of him. Figure 7(a) shows the processed HEOG and VEOG while Figure 7(b) displays the movement of the gaze point based on HEOG and VEOG.

5. Technical and Clinical Testing

The online signal processing and classification of the EEG signals were done by using the Common Spatial Patterns 2 class BCI Simulink model provided by g.tec medical engineering GmbH and the offline analysis of the data was done using g.BSanalyze software provided by the same company. For the EOG processing we developed a Simulink block containing an algorithm that processes the EOG signals and outputs the - gaze normalized coordinates with respect to the center point of the image displayed on the VR system. The whole system was first tested on 3 healthy people and then some fine tunings were done based on their suggestions in order to get high accuracy and a good repeatability coefficient. All three-healthy people achieved low control error rates, comparable to the ones presented by Ortner and colleagues in ‎[33].

Before starting the tests on patients within clinical environment, this study was approved by the institutional review board of the National Institute of Rehabilitation, Physical Medicine and Balneoclimatology from Bucharest, Romania, and each patient signed informed consent and an authorization for videos and photographs release before starting the study. The general clinical profile of the patients included in the study was afebrile, aware, temporospatial oriented, and cardiorespiratory balanced, without digestive or reno-urinary complains, with poststroke central neuromotor syndrome. From the whole patients, one-third was women and two-thirds were men, with ages between 52 and 79 years old. The inclusion criteria was stable neurological status; stable consciousness state; significant and persistent neuromotor deficit; disability for at least two of the following: mobility, self-help capacity, communication, sphincter control, deglutition; sufficient cognitive functions to allow learning; communication ability; sufficient physical exercise tolerance.

The clinical tests are in progress and until this moment the proposed system was tested on 7 patients. Each of them performed three training sessions, and all of them were able to achieve a low control error rate over the whole system. Table 1 presents the mean and minimal control error rate achieved by each patient. The mean error rate is calculated as the mean of the errors for each time point between seconds 4.25 and 8 of the last 2 runs. Figure 8 presents the error rate in time for subject S6, session 8, when he achieved the lowest control error rate, indicated by the red circle at second 6.8.

Except for subjects S3 and S4, all patients exhibited control error rates lower than 20% in at least one session. At this time of the study, it is premature to make evaluations of the rehabilitation outcome of the patients, but, based on their feedback after each session, the VR system makes them remain focused on the task that they have to perform, and they see everything like an interactive game. The fact that they are cognitively involved in this task, unlike having a passive or bored attitude, obviously brings additional benefits to rehabilitation process outcome.

At the beginning, it was difficult for the patients to understand how to concentrate on imagining the movement of their impaired limb as part of the rehabilitation exercise. For those with a low-level education, it was unclear how such a concentration effort regarding their limb movement will help them. This was observed especially when the system was used only with BCI module without VR. The indications on what they had to do were very poor in information (just a simple sound and an arrow to indicate left or right). Also, the activity around the patient disturbed him very easily from imagining the movement. The patients needed around 5 training sessions in order to learn how to imagine the movements and to obtain a good neurofeedback. By adding VR, the number of training sessions was decreased to one or (very rarely) two.

Analyzing the questionnaires, it was concluded that the average user satisfaction was around 3, the answers being highly influenced by the patients’ understanding of the rehabilitation therapy because most of them expected to recover themselves based on the therapist’s activity and not to be consciously involved in the rehabilitation process. That depends also on the education degree. However, the overall patients’ impression was that they felt and saw an encouraging improvement in recovering after using the proposed system.

For the next months, we plan to organize two groups of patients: a test group and a control group. The test group will perform up to 25 sessions of training with the system, while the control group will perform only classical rehabilitation therapy. When finishing the study, the results will be compared between groups and a statistical analysis will be performed on the results to see if the test group function improvements are statistically and significantly higher than the ones of the control group.

6. Conclusions

In this paper, a BCI-FES system for stroke rehabilitation is presented. Besides stimulation device, the BCI and EOG systems supervise how exercises are performed and the patient’s commitment and Oculus Rift headset facilitates the patient’s immersion in VR. By using this system, the patient is not distracted by the real environment or by events around him. He is just immersed in VR where the virtual therapist tells and shows him how to perform every exercise and a red big arrow is shown every time. The patient is focused most of the time, but if he loses his concentration the eye tracking system detects this and gives a warning.

The technical performances were validated by testing the system on healthy persons with good knowledge in assistive technologies. The healthy people achieved low control error rates, comparable to the ones reported in the literature.

The clinical tests are in progress, but the preliminary ones are very encouraging regarding fast accommodation and satisfaction of each patient. This approach of combining VR and BCI and FES facilities can effectively speed up the rehabilitation period and increase the users’ optimism and the desire to exercise and recover lost skills. By involving the brain via BCI and VR the system proved to be more effective than the standard techniques.

The clinical tests last for several months for a significant number of subjects but once these will be completed the Likert questionnaires and technical files of all subjects will be analyzed.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.


This work was supported by the Romanian National Authority for Scientific Research (UEFISCDI), Project 1/2014 Virtual Therapist with Augmented Feedback for Neuromotor Recovery (TRAVEE).