Using Brain Waves to Control Computers and MachinesView this Special Issue
BCI Could Make Old Two-Player Games Even More Fun: A Proof of Concept with “Connect Four”
We present a brain-computer interface (BCI) version of the famous “Connect Four”. Target selection is based on brain event-related responses measured with nine EEG sensors. Two players compete against each other using their brain activity only. Importantly, we turned the general difficulty of producing a reliable BCI command into an advantage, by extending the game play and rules, in a way that adds fun to the game and might well prove to trigger up motivation in future studies. The principle of this new BCI is directly inspired from our own implementation of the classical P300 Speller (Maby et al. 2010, Perrin et al. 2011). We here establish a proof of principle that the same electrophysiological markers can be used to design an efficient two-player game. Experimental evaluation on two competing healthy subjects yielded an average accuracy of 82%, which is in line with our previous results on many participants and demonstrates that the BCI “Connect Four” can effectively be controlled. Interestingly, the duration of the game is not significantly affected by the usual slowness of BCI commands. This suggests that this kind of BCI games could be of interest to healthy players as well as to disabled people who cannot play with classical games.
Driven by the needs of people with physical disabilities, researchers have begun to work on direct brain-computer interfaces (BCIs), in the aim of enabling them to communicate and move without resorting to the usual peripheral nervous and muscular pathways. In BCIs, users have to manipulate their brain activity to produce signals that control computers or machines directly. This is challenging for both users and researchers. The users often need to learn how to control the device, which is cumbersome if not impossible, while researchers have to deal with the difficulty of processing highly variable and noisy signals online. However, this research could have a profound impact in various pathologies, including those for which patients suffer from cognitive impairments and could possibly benefit from brain or neurofeedback training. Indeed, the latter also rests upon our ability to extract online the neurophysiological markers that should be fed back to the patients, so that they could learn how to control it and yield a cognitive or behavioral improvement .
The most practical and widely applicable BCI solutions are based on noninvasive electrophysiological recordings, namely, electroencephalography (EEG). As command signals, those BCI use event-related potentials (ERPs) like the P300  or self-regulatory activities such as changes in cortical rhythms [3, 4].
Beyond medical applications [5–7], BCI has also a great potential for gaming, a domain where users are open to novelty and eager to face new challenges . Besides, developing video games based on BCI could prove useful in some patients, by yielding a better efficiency and wider acceptance of BCI-based therapies. Indeed, since the number of training sessions required by neurological rehabilitation and training protocols is usually much larger than the one in BCI control applications, a motivational (more realistic and interactive) environment as encountered in computer games could be of great interest in that context [9, 10]. In particular, it has been argued that BCI games could well boost motivation and neurofeedback training performance . Possible future investments of the gaming industry in BCI software and technologies might also stimulate the field and produce new devices, with engaging environments for future clinical applications like BCI-based stroke rehabilitation and neurofeedback therapies .
There are several examples of games that have been paired up with BCI systems yet, either by using imaginary movement-related markers , P300 responses [14, 15], or steady-state visual potentials . However, only a few have been designed for multiple players, despite the fact that competition and socializing are among the strongest motivational factors reported by users of multiplayer games online .
A crucial limitation to the use of BCI in gaming is the highly unreliable nature of brain signals. The ensuing BCI commands are difficult to interpret and provide low degrees of freedom. In most applications, a reliable command can only be achieved by accumulating data over long time windows, at the expense of the primary interest of the game. Another drawback is the obvious one of having to put an EEG cap on, although the field has made tremendous progress in that respect in the last few years . For all those reasons today, BCI hardly compete with traditional game effectors such as joysticks, mice, and keyboards, at least in healthy subjects. However, unreliable input control could be used to extend current video games and to create a motivating challenge for the users .
In this paper, we briefly report the online proof of concept of a new BCI game, based on the old and well-known “Connect Four”. It is a very popular and easy game to play, which makes it attractive for a broad audience. The proposed BCI version has several advantages and our first online trial suggests it could overcome some of the current limitations of BCI applications to gaming. Interestingly,(i)only brain signals are required to play the game;(ii)two subjects play against each other;(iii)the EEG setup has been limited to nine sensors per participant (compared to 32 in our previous experimental settings);(iv)it exploits a well-established protocol and robust electrophysiological marker: the (indeed, it has already been shown that the early visual evoked response plays a significant role in achieving higher accuracies in the classical P300 Speller paradigm. This strongly suggests that such BCIs thus highly rely on eye gaze or overt attention .) and P300 evoked response ;(v)it makes use of the unreliability of BCI commands in an elegant fashion, which extends the possibilities offered by the initial game;(vi)its duration is comparable to the one of the traditional game play;(vii)it holds promises for applications in patients, not only as an entertainment but possibly as an efficient tool for the training attention-related brain signals.
In the following, we introduce the traditional game “Connect Four” and its adaptation to a BCI version. We then describe our BCI implementation within the OpenViBE software environment. Finally, we report the results of a first online evaluation in two healthy subjects as a proof of concept to show that a BCI “Connect Four” can effectively be controlled.
1.1. Traditional “Connect Four”
“Connect Four” is a two-player strategy game in which players interact through a vertical rectangular board made of holes (6 rows and 7 columns). Each player starts with 21 coins, either all yellow or all red. Participants play in turn and place one coin at a time. They pick up a column and drop a coin which will fall down the board to the lowest available hole. The game ends when one participant first connects four coins, either horizontally, vertically, or diagonally. This participant is the winner of the game (see Figure 1).
1.2. BCI “Connect Four”
In the BCI version of “Connect Four,” we simply replace the manual drop of the coin by a P300-based selection of the target column. The column selection process follows the same principle as the item selection in the well-established P300 Speller paradigm. In the latter, a 6 × 6 matrix of symbols is usually displayed on a computer screen for spelling. The subject has to actively attend to the target symbol while rows and columns are being flashed alternatively, in a random fashion. Since the target events are rare and unpredictable, they elicit a typical EEG evoked response, the P300 whose detection allows us to identify the target symbol. In the “Connect Four” scenario, only columns matter and need to be proposed. This yields a simpler detection task and higher information transfer rate.
The P300 wave is a positive EEG deflection that occurs approximately 300 ms after stimulus onset (flash onset). It is typically recorded from central and parietal sensors. This response is evoked by paying attention to rare stimuli in a random sequence of irrelevant stimuli or distracters (the so-called oddball paradigm) . Farwell and Donchin  were the first to show that the P300 component could be used to select items displayed on a computer screen . From a cognitive perspective, the P300 can be seen as a measure of alertness and attention orientation, thus reflecting a subject’s general level of arousal .
Importantly, we build on an existing BCI system that we extensively evaluated in previous studies [26, 27]. In particular, we showed that a similar setup could be reliably controlled by naive BCI subjects (), with very high spelling accuracy (85% correct letter selection on average, for 3 flashing sequences).
2. Material and Methods
2.1. Game Rules
2.1.1. Classic Mode
In turn, players choose an empty location based on their own strategy and aim at placing a coin in the corresponding column. To drop a coin in the right column, the player focuses her attention on that particular column (Figure 2) and is advised to count the number of times it is flashed, while avoiding to be distracted by the flashes occurring next to it. After all columns have been flashed (possibly several times), the most probable target according to real-time signal processing is selected. The player receives immediate feedback on her action. The game ends whenever one player has been successful in connecting four coins. Of course, each player should prevent the other player to complete a four-coin long connection. In the classic mode, this can only be done in turn.
2.1.2. Contest Mode
The principle is the same as for the classic mode, except that players can interfere with the outcome of the action of the other player. Players still act in turn. However, when player 1 is aiming at dropping a coin, player 2 has the possibility to modify the target column for this coin, and vice versa. To do so, she has to enter a kind of mind competition and her choice will eventually prevail, provided that she could better focus her attention onto her preferred target column. The quantitative comparison between the attentional resources, respectively, allocated by the two players is based on the entropy of the posterior distribution of the target column for each player (see Section 2.3, paragraph Classification). In short, the winner is the one who produces the most accurate or less uncertain command. This mode allows for simultaneous competition between the two players and makes the interaction more challenging.
2.2. Experimental Setup
Two players, one female (P1) and one male (P2), aged 23 and 30 respectively, participated in this study. All subjects had normal or corrected to normal vision.
Players were seated 70 cm from the same 17′′ computer screen. EEG was recorded from the two players using a single 32-channel ActiCap system with Ag/AgCl electrodes (Brain Products, Germany). Only nine sensors were used for each participant. The particular (centroparietal and occipital) electrode locations were chosen in order to optimize the signal to signal-plus-noise ratio (SSNR), according to one of our previous P300 Speller experiment . We used the following sites from the extended 10-10 system: P7, P3, Pz, P4, P8, P09, O1, O2, and PO10. All electrodes were referenced to an electrode placed on the nose and impedances were kept below 5 kΩ for all sensors. Analog signals were amplified with BrainAmp amplifier (powered with a rechargeable battery, PowerPack) and digitized at a rate of 1000 Hz using the Brain Vision Recorder software (Brain Products, Germany).
To achieve good selection of one target column among seven, the two players were instructed not to move during stimulations, to stare at the targeted location, and to count how many times it was flashed.
Visual stimulations were handled by a C++/SDL software on a dedicated computer and sent to a CRT screen in random order. A trigger (labeled from 1 to 7, one per column) was also sent to the EEG amplifier via parallel port (). The flash duration was set to 90 ms and the time between two flash onsets to 200 ms. We used two repetitions per shot, meaning that each column was highlighted twice per selection process.
Since the two players had never used such a P300-based BCI before, we had to start with a short and simultaneous session to calibrate the system. Subjects were given a sequence of 63 predetermined target columns to attend to. In practice, after each sequence of flashes for one target, the new target was indicated by a green frame. As in the forthcoming game, columns were flashed twice per trial during 90 ms, with a stimulus-onset asynchrony interval (SOA) of 200 ms. Based on those data, the feature selection and classification algorithms were trained for each subject, respectively.
After calibration, the two subjects performed three games in classic mode, followed by two games in contest mode. In each game, players had 10 s to choose a target before the columns started flashing. Finally and for each shot, each player was requested to write down her/his actual target number for future (offline) evaluation.
2.3. Online Processing
We implemented this BCI game scenario in OpenViBE (OV). OV is an open source platform to design, implement, and run BCI applications . The dedicated online data processing stream consisted in a classical BCI pipeline made of successive modules in the following order: data acquisition, feature selection, feature classification, decision, and feedback (see Figure 3). Those modules operate as follows.
New chunks of raw EEG data are transmitted via TCP/IP every 20 ms. The data stream includes event markers. The Acquisition Server application converts those streams into OV format.
Data are sampled down to 100 Hz after being passed through an antialiasing filter.
For each channel independently, downsampled data are then bandpass filtered between 1 and 20 Hz using a second-order Butterworth filter to remove very low frequencies and the higher frequency content of EMG.
The channel selector limits the 32-channel data stream to two 9-channel data streams, one for each player.
To extract the most relevant signal components and to increase the signal-to-noise ratio (SNR), we used the xDAWN algorithm which has been specifically developed for the P300 Speller . Precisely, it optimizes orthogonal spatial filters (linear combinations of sensors) so as to maximize the signal to signal-plus-noise ratio. We used the first five filters (or virtual sensors) obtained from the calibration phase, for each subject.
Epochs are generated upon reception of a given event marker. Seven stimulation-based epoching are performed in parallel, per player branch, each epoch type being associated with a different column. We considered 600 ms long epochs starting at flash onset.
The temporally and spatially filtered epoched data enter as features for subsequent classification. The aim of classification is to disentangle between target and nontarget events. Therefore, we adapted the Bayesian approach we used in a previous P300 Speller experiment . The Bayesian procedure rests on computing the posterior probability that a given response pertains to the “target” or “nontarget” class type of event . This conditional density is inferred from combining two antecedents: the prior probability and the likelihood function which embeds a probabilistic model of the data to be observed. In the current scenario, we used a two-Gaussian mixture likelihood function. In a previous P300 Speller study, in twenty subjects, we had shown that this classifier was equivalent or even better than a classical LDA . Moreover, it allows for the optimal and online updating of the posterior density, after each single new observation. Hence starting with a uniform prior over columns (each column is a possible target, with equal prior probability), the posterior density is computed based on the data likelihood and the given prior. If more data pertaining to the same shot are provided, the current posterior is taken as the new prior, in a Markovian fashion. In practice, online, this is performed in two OV steps as follows. (i)Likelihood compute: this box computes the likelihood function for each class and each stimulus type. It provides the results in matrix form.(ii)Matlab Box: it communicates with the Matlab Engine to perform matrix processing. In this scenario, it is used to combine the likelihood values and current priors to compute posteriors. It also makes the final decision by indicating the most probable target a posteriori, for each player : where indicates the observed features for player .
Importantly, this Matlab Box provides a second measure which is the Shannon entropy of the posterior distribution for each player: where is a short for .
Shannon entropy is used in contest mode only, as a measure of confidence in the machine’s decision. We consider it as a proxy to the attentional resources devoted by the subject for a particular shot. Indeed, Shannon entropy can be compared between subjects. It is negative and bounded. The more informed the decision, the sharper the posterior probability distribution and the closer to zero the entropy. To summarize, the output of the Matlab Box depends on the game mode. In classic mode, the single output corresponds to the estimated target for the current player. In contest mode, the output is twofold and indicates both the winner of the contest and the target she or he most likely selected.
The final decision (the selected column) is sent via a VRPN (Virtual-Reality Peripheral Network) analog server to a VRPN client application host by the stimulator . This information is used to provide the two players with visual feedback.
In classic mode, a rectangle of the color of the current player is displayed to indicate the selected column (Figure 4(a)). In contest mode, a first feedback (red or yellow smiley) appears at the center of the screen to reveal the identity of the winner of the contest (left part of Figure 4(b)). A second feedback, similar to the one in classic mode, is finally provided and the coin is placed (right part of Figure 4(b)).
3.1. Classic Mode
Player P2 (Red) won twice and P1 (Yellow) once. The average game duration was 4 min and 6 s, corresponding to an average of 7 shots per player. Figure 5 shows an outline of the third game.
In that game, players made almost no error (Figures 5(a) and 5(b)). In turn 7, Red made an error; he wanted to select the third column in order to thwart the other’s player plan to connect four yellow coins (Figure 5(c)). Yellow won.
The average accuracy over all three games was 81.7%, which is way above chance level (14.3% for 7 classes) and corresponds to an information transfer rate of 35.3 bits/min. However, P2 (Red) reached a considerably higher accuracy than P1 (95.2% and 68.2%, resp., or 51.8 bits/min and 23.2 bits/min, resp.).
Table 1 shows the detailed results, separately for the two players and the three games in classic mode. This highlights the coherence between individual performance, in terms of accuracy, entropy, and the ensuing winner in each game.
3.2. Contest Mode
Each player won one game. The average game duration was 3 min and 37 s, corresponding to an average of 6 shots per player. Figure 6 shows an outline of the second game.
In this mode, both players compete to impose their own choice, whatever the color of the coin to be placed. An example is given in Figure 6. It was Yellow’s turn (Figures 6(a1) and 6(a2)) but the Red player won the contest (Figure 6(a1)) and enforced the yellow coin to be placed in the second column instead of the fourth one, which keeps the opportunity for the Red player to win the game in the next turn (Figure 6(a2)). However, Yellow won the next contest (Figure 6(b1)) and prevented Red’s victory by selecting the first column (Figure 6(b2)). In what follows, Yellow won again and placed the red coin in the first column, hence leaving the hole in the fourth column free (Figures 6(c1) and 6(c2)). Finally, Yellow won the last contest and hence the game by placing the coin in the fourth column (Figures 6(d1) and 6(d2)).
Over all games in contest mode, Player P1 won 48% of the contests. This indicates that both players were fully engaged in the game and close to each other in terms of devoted attentional resources.
Table 2 shows the detailed results, separately for the two players and the two games in contest mode. Interestingly, in contest mode, the player with the highest averaged accuracy is not necessarily the end winner of the game. Neither is the one who won most of the contests, since victory is the result of the complex interplay between accuracy, focus of attention, and strategy.
The average accuracy over all two games and players was 83.3%, or 37.0 bits/min. Precisely, P1 managed to increase accuracy by 18.8% compared to classic mode, reaching an average of 87.0% in contest mode, which corresponds to a bit-rate increase of 17.8 bits/min. To qualitatively evaluate this improvement, we compared the averaged features for “target” and “nontarget” stimuli, in both modes. We computed the Global Field Power (GFP)  over the five virtual sensors: where is the value at the th virtual channel, at time .
Note, in Figure 7, that GFP increased around 200 ms and 500 ms peristimulus time, for target stimuli, in the contest versus classic mode, although there is no specific difference in between the “nontarget” responses in the two modes. This might reflect the actual increase in classification accuracy and could well be a consequence of an enhanced motivation in contest mode because motivation was found to be positively correlated with the P300 amplitude in healthy subjects .
BCIs were originally developed in the context of clinical research. In this study, we developed a P300-based BCI for gaming. Moreover, beyond most existing gaming applications of BCI, we instantiated a true online interaction, involving brain commands only, from two competing subjects, in an existing but extended gameplay (“Connect Four”). We showed that it is possible to take advantage of the variable nature of brain signals and of the cognitive challenge it takes to control their stability. This could make existing games more attractive and motivating, for both healthy players and patients. Through online evaluation, we observed that a simple setup, using nine EEG sensors, was sufficient to provide high performance comparable to our previous studies [26, 27], suggesting that BCI “Connect Four” can effectively be controlled, at a pace that compares to the one of the traditional (manual) version of the game. Indeed, although motor execution remains faster than the 2.8 s that are needed in the present BCI implementation to select a command, the subjective perception of this additional cost might prove insignificant compared to the time allocated to the planning of the next move (of the order of 10 s). Moreover, following the hypothesis that the electrophysiological marker manipulated here, the P300, is known to reflect the orientation of attention, this BCI game or similar ones could prove useful in the near future, as part of a motivating system to train attention. This encouraging proof of concept calls for carefully designed experiments, involving a large number of subjects or patients, to study social interactions or effects on motivation of such an original and well-controlled BCI environment.
This work is supported by the French ANR project ANR-DEFIS 09-EMER-002 CoAdapt. Authors are grateful to Romain Bouet for technical support.
L. A. Farwell and E. Donchin, “Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials,” Electroencephalography and Clinical Neurophysiology, vol. 70, no. 6, pp. 510–523, 1988.View at: Google Scholar
G. Pfurtscheller and C. Neuper, “Motor imagery direct communication,” Proceedings of the IEEE, vol. 89, no. 7, pp. 1123–1134, 2001.View at: Google Scholar
J. R. Wolpaw, D. J. McFarland, G. W. Neat, and C. A. Forneris, “An EEG-based brain-computer interface for cursor control,” Electroencephalography and Clinical Neurophysiology, vol. 78, no. 3, pp. 252–259, 1991.View at: Google Scholar
N. Birbaumer, N. Ghanayim, T. Hinterberger et al., “A spelling device for the paralysed,” Nature, vol. 398, no. 6725, pp. 297–298, 1999.View at: Google Scholar
R. Leeb, D. Friedman, G. R. Müller-Putz, R. Scherer, M. Slater, and G. Pfurtscheller, “Self-paced (asynchronous) BCI control of a wheelchair in virtual environments: a case study with a tetraplegic,” Computational Intelligence and Neuroscience, vol. 2007, Article ID 79642, 8 pages, 2007.View at: Publisher Site | Google Scholar
E. Maby, M. Perrin, D. Morlet et al., “Evaluation in a locked-in patient of the OpenViBE P300-speller,” in Proceedings of the 5th International Brain-Computer Interface Conference, pp. 272–275, 2011.View at: Google Scholar
B. Graimann, B. Allison, and A. Gräser, “New applications for noninvasive Brain-Computer Interfaces and the need for engaging training environments,” in Brain-Computer Interfaces and Games Workshop at the International Conference on Advances in Computer Entertainment Technology, pp. 25–28, 2007.View at: Google Scholar
R. Leeb, F. Lee, C. Keinrath, R. Scherer, H. Bischof, and G. Pfurtscheller, “Brain-computer communication: motivation, aim, and impact of exploring a virtual apartment,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 15, no. 4, pp. 473–482, 2007.View at: Publisher Site | Google Scholar
J. van Aart, E. Klaver, C. Bartneck, L. Feijs, and P. Peters, “Neurofeedback gaming for wellbeing,” in Brain-Computer Interfaces and Games Workshop at the International Conference on Advances in Computer Entertainment Technology, pp. 3–5, 2007.View at: Google Scholar
A. Nijholt, D. Oude Bos, and B. Reuderink, “Turning shortcomings into challenges: brain-computer interfaces for games,” Intelligent Technologies For Interactive Entertainment, vol. 1, no. 2, pp. 153–168, 2009.View at: Google Scholar
N. Yee, “Motivations for play in online games,” Cyberpsychology and Behavior, vol. 9, no. 6, pp. 772–775, 2006.View at: Google Scholar
C. Guger, G. Krausz, B. Z. Allison, and G. Edlinger, “Comparison of dry and gel based electrodes for P300 brain-computer interfaces,” Frontiers in Neuroscience, vol. 6, article 60, 2012.View at: Google Scholar
F. Lotte, “Brain-computer interfaces for 3D games: hype or hope?” in Proceedings of the Foundations of Digital Games (FDG '11), pp. 325–327, 2011.View at: Google Scholar
M. Fabiani, D. Karis, and E. Donchin, “P300 and recall in an incidental memory paradigm,” Psychophysiology, vol. 23, no. 3, pp. 298–308, 1986.View at: Google Scholar
L. A. Farwell and E. Donchin, “Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials,” Electroencephalography and Clinical Neurophysiology, vol. 70, no. 6, pp. 510–523, 1988.View at: Google Scholar
E. Maby, G. Gibert, P. E. Aguera, M. Perrin, and O. Bertrand, “The OpenViBE P300-Speller scenario: a thorough online evaluation,” in Proceedings of the Human Brain Mapping Conference, 2010.View at: Google Scholar
M. Perrin, E. Maby, R. Bouet, O. Bertrand, and J. Mattout, “Detecting and interpreting responses to feedback in BCI,” in Proceedings of the 5th International Brain-Computer Interface Conference, pp. 116–119, 2011.View at: Google Scholar
B. Rivet, A. Souloumiac, V. Attina, and G. Gibert, “xDAWN algorithm to enhance evoked potentials: application to brain-computer interface,” IEEE Transactions on Bio-Medical Engineering, vol. 56, no. 8, pp. 2035–2043, 2009.View at: Google Scholar
R. M. Taylor, T. C. Hudson, A. Seeger, H. Weber, J. Juliano, and A. T. Helser, “VRPN: a device-independent, network-transparent VR peripheral system,” in Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '01), pp. 55–61, November 2001.View at: Google Scholar