Abstract
The Sensorium is a neurofeedback environment that allows people to experience signals from their nonperceptible body processes visually and auditorily. Various (neuro-)physiological rhythms and frequencies are projected simultaneously as soundscapes and “lightscapes” into the environment. A wireless physiological amplifier device sends signals such as EEG and ECG to a computer for real-time processing using the modified brain-computer interface software “Thought Translation Device” (TTD). The TTD performs signal filtering, parametric orchestral sonification, and light control. In a pilot study, 20 participants have been exposed to their ongoing brain and heart signals while sitting inside the Sensorium, a small room equipped with a speaker and lighting system. Almost all of them reported an increase in contentment, relaxation, happiness, and inner harmony. They also reported a widening in their body consciousness. In future, therapeutic paradigms will be developed and the treatment effects on people with psychosomatic diseases will be evaluated.
1. Introduction
Neurofeedback (NFB) is a method for training of self-regulation of physiological, especially neurophysiological body signals. It became popular in the 1970s already when the first devices for relaxation training were offered. The self-regulation training was mediated by visual or acoustical real-time display of, for example, the muscular tension measured in the electromyogram (EMG) or the amplitude of the alpha rhythm activity of the electroencephalogram (EEG). The measurement implies electrode sensors to be attached to the brain or other body parts picking up the signals ranged from V up to mV with frequencies between 0 and about 100 Hz. The EEG feedback era first started to become popular after Kamiya published his studies on operant conditioning of the EEG alpha rhythm. He found the alpha amplitude to be connected to the state of relaxation [1]. Since then, many biofeedback devices appeared on the market and often, these devices have been applied in nonscientific settings. Unfortunately, the hypothesis of a clear connection between EEG alpha power and relaxation could not maintain its position after further scientific explorations. The scientific investigations of brain physiological self regulation conducted by Sterman et al. in 1974 described the application of EEG NFB for the therapy of patients with epilepsy [2]. Birbaumer and his group demonstrated the human ability for self regulation of the slow cortical potentials (SCPs); that is, EEG potential shifts below 1 Hz [3, 4]. They also successfully applied the SCP feedback training to patients with intractable epilepsy for reduction of their seizure frequency [5, 6].
A major application of NFB currently is the treatment of attention deficit and hyperactivity disorders (ADHD) in children [7]. Kaiser and Othmer [8] did an extensive study including 1,089 patients with moderate pretraining deficits. In a NFB training of sensorimotor and beta waves a significant improvement in attentiveness and impulse control was found and positive changes as measured on the test of variables of attention (TOVA) could be demonstrated. Also Monastra [9] who compared the effect of treatment with Ritalin and NFB in a sample of 100 children found similar improvements on the TOVA and ADD evaluation scale in both groups. Leins, Strehl et al. reported behavioral and cognitive improvements in children with ADHD after NFB-treatment. They compared the feedback training of beta/theta power with feedback of slow cortical potentials (SCPs) in two groups of 19 children (aged 8–13 years) each. Both groups learned to regulate their EEG parameters and showed significant behavioural, attentional, and cognitive improvements which remained stable in a follow-up measurement six months after treatment [10, 11].
Other studies using NFB of Theta band activity successfully showed increases in intelligence scores and academic performance [12]. Egner and Gruzelier found an improvement in musical performance by alpha/theta NFB training in which the participants should try to raise the theta power over alpha [13]. Further applications of NFB can be found in the treatment of learning disabilities, conduct disorder, cognitive impairment, migraines, chronic pain, sleep dysregulation, autism-spectrum-disorders, posttraumatic stress disorder and mild traumatic brain injury, and so forth.
NFB also was used in Brain-Computer Interfaces (BCIs) for the training of direct brain-computer communication and peripheral control via self-regulation of brain signals. In contrast to a normal NFB device, a BCI is able to classify a brain response to classes with intentional meaning. This is often accomplished with certain kinds of discriminant analysis. For example, users learned to regulate their slow potentials so the computer could distinguish between a positivity or negativity response [14, 15]. Others learned to regulate their mu-rhythm activity by imagination of a hand movement which could be detected by a classification algorithm [16]. Further, a BCI is able to convert the intentional brain responses into action, for example, letter selection for verbal communication, controlling switches or prostheses. The Thought Translation Device (TTD) used for realizing the here described Sensorium is both, an NFB device for feedback training of slow cortical potentials but also oscillatory components and a BCI that allowed completely paralyzed patients to communicate verbally using the self regulation skills of their brain signals [17].
The advances in the field of BCI research demonstrate that it truly is possible to self-regulate certain brain parameters voluntarily with a high reliability. However, this research also demonstrates that not every signal component is suitable for self regulation and for some components such as the slow waves it is highly individual whether a person can learn voluntary regulation. The amount of alpha and sensorimotor rhythms also has been shown to be highly individual. And while for the NFB treatment mostly the theta, alpha or beta waves of unspecific areas were trained, for BCI control these components were not of interest at all. Here, it is consensus that motor-related rhythms such as the mu-rhythm over motor areas are the best controllable components while the alpha rhythm can hardly be used as it is closely connected to visual processing, and theta rhythms are connected to drowsiness or very special states of consciousness such as hypnagogic states or some specific styles of meditation.
Looking at the outcome of various NFB studies one can realize that there is often a rather poor learning effect in the neurophysiological self regulation training however the treatment effect often shows quite impressive results with effect sizes around 0.4–0.6 [18, 19]. This might be the reason why some researchers even do not report the neurobehavioral data but only the psychological behavioural treatment outcome. Beside the research considering NFB as an effective therapy for various different diseases, other researchers like Loo and Barkley doubt that changing the EEG is the actual mechanism for the decrease of symptoms [20]. Therefore, it could be caused by a placebo effect and rather be an unspecific treatment effect.
This would suggest that the effectiveness could be far more assigned to the feedback itself and the setting as such than to the self regulation skill of the patient. As a consequence, NFB as such or as an intervention method can be regarded to be suitable for positively changing psychological variables; however, the true mechanisms still remain unclear. This asks for a plausible explanation. Here is one attempt or hypothesis. The specificity of NFB therapy in contrast to other forms of medical or behavioural treatments is that NFB very closely connects the patients’ consciousness with their physiology. This might improve their body awareness as well as the connection between self and body. As a consequence, the patient may become more aware of the relation between body and consciousness and thus create a stronger link between both which facilitates behavioural changes as necessary for improving ADHD symptoms for example.
This consideration may serve as a basic mechanism for the success of the Sensorium as a form of a NFB intervention that optimizes the feedback experience as such without the requirement of performing self-regulation training. The Sensorium can be regarded as a whole NFB environment placing an emphasis on an enhanced sensation and extraordinary self-experience of the patient during a session.
2. Technical Development
2.1. A Historic Overview
The Sensorium as a multimodal neurobiofeedback environment resulted from the development of a multimodal brain-computer interface, the TTD [21]. The TTD initially was designed for training people to self-regulate their slow cortical potentials and thereby achieve the capability to select letters through brain responses. Using this technology, the TTD was the first brain-computer interface that enabled completely paralyzed patients suffering from the neurodegenerative disease amyotrophic lateral sclerosis (ALS) to communicate with their brain signals only [17]. In the final stage of ALS, patients can lose all means of communication while remaining conscious. This state is called “locked-in syndrome”. Most patients we trained had slight muscular reactions left such as small eye movements. However, we also encountered patients without any sign of ocular movement abilities. These patients were hardly able to stare at a computer screen and focus its content which would have been necessary for a successful EEG feedback training as all instructions and response information were given visually. Unfortunately, especially this group of patients could have profited most from the use of a BCI. Therefore, we developed an auditory driven brain-computer communication device that provided all feedback information to the patient auditorily. The computers voice asked a question or told the options for selection. The pitch of a tone sequence then directly informed about the current brain response. After this sound-mediated selection interval the computers voice informed about the selection response, that is, a “yes" or “no" or the selected letters [21].
2.2. Parametric Orchestral Sonification
Similarly to other NFB devices, this auditory BCI presented one defined brain parameter such as the slow waves, the mu rhythm amplitude, or a classifier result signal as a sound sequence to the user. However, the human ear is capable of listening to and distinguish a high number of voices, tones, or instruments simultaneously and also find connections, correlations, and higher complex order in acoustic sounds as we all know, for example, from listening to music or identifying single persons voices within a group of talking people. Similarly, the EEG consists of a mixture of various signal components, frequencies, and, depending on the number of electrodes, signal channels. Therefore, it can be hypothesized that an auditory representation of the complex EEG time series could lead to an interesting perception and possibly provide informative insights. For such investigations a module was developed that could present various EEG features and rhythms simultaneously through different (computer) musical instruments. The module and its algorithm were termed Parametric Orchestral Sonification of EEG in Real-time (POSER [22, 23]). There is an essential difference between the method of sonification typically used in NFB devices and the POSER approach. Usually, a brain rhythm or component is filtered, mostly Fourier transformed and its amplitude is being used as a measure for the pitch of a sound. The user is prompted to follow and try to control the sound sequence. In contrast, the parametric sonification approach used in the POSER module aims on presenting the nature of the signal as natural as possible. In other words, a theta rhythm consists of oscillatory patterns with frequencies between 4 and 8 Hz. Such signal changes are below the hearing threshold of 20 Hz and cannot be perceived as sound directly. However, it can be perceived as vibration and rhythm when modulated with the spectrum of musical instruments. Therefore, a theta response can be parameterized by transforming each wave cycles amplitude into loudness and the time between two cycles into pitch and playing an instrument at each wave cycle maximum with those parameters. This sonification approach maintains the rhythmic and the harmonic information of the theta rhythm which both would get lost when transforming the smoothed Fourier amplitudes into sound as typically done in auditory NFB.
In recent years, other research groups independently aimed on developing improved sonification algorithms and programs in order to provide more or more specific information from the brain, too. Baier et al. were working on EEG sonification specifically to sonify epileptic seizure events [24, 25]. The SonEnvir project of de Campo and Wallsch is another example [26]. However, they have used different approaches.
Applying the POSER sonification method to the various frequency band oscillations and playing all of them simultaneously using a specifically chosen instrument for each band we achieved an orchestral sonification of the EEG. As the TTD, including the POSER module, was a real-time processing system, users were able to listen to their actual multiple brain rhythms online like in a brain-symphonic live concert. Simultaneously recorded ECG signals from the heart could be sonified accordingly and mixed to the concert. This brain music device was used as a link between science and art, first. A portable EEG amplifier device with wireless transmission of the physiological data stream to the computer in combination with shielded electrodes allowed for movement interaction with the sonified body signals. A project called Braindance arose from this possibility in which a dancer showed an improvisation dance performance to music generated live from the own actual brain signals. This closed loop feedback experience in which cause and effect melt and sometimes become indistinguishable is both impressive for the dancer and the audience. The project including a discussion on possible movement artifacts was reported in [27].
2.3. The SymPOSER Sonification and Illumination Algorithm
For the development of the Sensorium, the POSER sonification software module was completely reprogrammed. The new system and algorithm was called SymPOSER. It includes a modified sonification algorithm plus a system for parameterized light control. Parts of the software were programmed in portable C++ classes so they can be used either directly in the TTD or in external hardware devices for sonification and light control which we also have available. Additionally, for the TTD a comfortable setup and sound configuration tool was programmed. In the following, a short description of the functional elements of the software is given.
SymPOSER consists of three basic processing classes, a filter class, a Midi control class, that is, the sonification class and a light control class. A central parameter container class holds all filter, Midi, and light control parameters. The filter class contains two selectable types of signal filters, an FIR (Finite Impulse Response) band pass filter, and an IIR (Infinite Impulse Response) filter that can either be configured as band pass, high or low pass filter. Such filters can be used for separation of the typical EEG frequency bands but also for the isolation or extraction of meaningful components within an ECG or other physiological signals. The filtered data stream can either be directly used as output signal, or be subject to further processing such as the calculation of the band power, the identification of wave extremes which later can be used for triggering the touch of a note, and the conversion of the time between two maxima into a frequency output. This provides four different output signals of a filter class. For each frequency band or signal component a separate instance of the filter class is created as shown in Figure 1. The second base class, the Midi class is the sonification unit which transforms the signals provided by the filter class into Midi commands and sends them to the Midi device. Four different parameters of a Midi note can be modulated by any of the filter output signals. These are note, touch velocity, pitch, and amplitude. With pitch and amplitude modulation, a continuously played tone can alter its frequency or loudness resulting in a vibrato or frequency modulation in accordance to a brain signal. A common sonification setting uses the wave trigger output to initiate a sound that is played with the pitch (note modulation) of the frequency output and the velocity according to the amplitude of the wave cycle. The note modulation can be tuned to 12 tone music, major, minor, or pentatonic harmony. Each Midi instance is able to play one instrument which can be selected out of 127 instruments according to the Midi wavetable to which the system is linked. Similarly to the Midi class, a number of light control instances transform the signal into light values sent to a DMX interface that controll a light system. The DMX standard is a serial interface standard used in professional studio and theatre lighting systems. Any output signal from the filter instances can be used to modulate multicoloured lighting devices.
For setting up the parameters a visual interface was programmed that allows to overview and change the parameters of all instances. Another program tool was created that provides the option for managing preset parameters and a further tool contains a sequencer for running various sonification settings in a sequence. This was used in the Sensorium study to present a fixed sequence of varying sonifications during a session in a standardized manner for each participant. Figure 1 gives an overview of the SymPOSER components as a module of the TTD.
2.4. The Feedback Environment
The complete signal diagram is illustrated in Figure 2. Basically, a PC or Notebook PC is required for running the TTD software. The TTD is capable of interfacing about ten different types of EEG and physiological amplifier systems. When connecting the Nexus 10 amplifier device or the Brainquiry PET device, one can measure and present EEG and pulse, respiration and skin conductance simultaneously. Those two devices are portable and transmit the signals wireless via Bluetooth to the computer. The TTD performs the signal processing and conversion into sound and light using the above described SymPOSER algorithms. Additional sampler software can be used to convert the MIDI commands generated by the SymPOSER into high-quality sound for being presented with a good sounding stereo amplifier and loudspeaker system. The light stimuli are sent via USB to a DMX interface that allows for driving any studio light system. We have built two types of lighting devices for the Sensorium. The first are 4 colored spot lights (red, green, blue, and yellow) with 50 W each which is suitable for indirect illumination of blank walls or curtains in a small environment. The second device is a white semitransparent ball that is enlightened by colored super bright LEDs from inside. This ball can be used as a visual focus connecting the user’s consciousness with the light ball. Through the use of a small portable amplifier the user has a high degree of freedom to either sit or lie down or even move the body in the environmental room.
According to Figure 3, we have tested the Sensorium version with indirect diffuse illumination meaning that the color and light filled the whole visual field. Signals from within the body were thus projected by sound and light into the surrounding space creating an individual and temporally changing atmosphere. Here, as an effect on consciousness we expected a more widening than introspective experience. The room itself should be as neutral as possible so that the stimulation alone is in charge of determining the conscious experience. Before exposed to the stimulation the users should be informed about the fact that every instrument or sound they hear and every change in light or color will be initiated by their own body signals. The technology was intended to be as invisible or remote as possible in order to keep the experience as pure as possible. This should create a unite consciousness between the outer experience and the own body or self.
3. Pilot Study
3.1. Study Design
20 healthy people (mean age 38 years, 11 f/9 m) have participated in a Sensorium session separately. Ten of them had meditation experience with an average total practice time of 644 hours, the other ten were nonmeditators. Eight from 10 meditators had more than 500 hours of experience. The sessions took place either in a small meditation room, or in the EEG measurement cabin of the Institute of Environmental Health Sciences at the University Clinic Freiburg, Germany in which the feedback environment was installed. After signing their informed consent, participants filled in a short introductionary questionnaire. They were seated on a meditation cushion. Five Ag/AgCl electrodes were attached to their body; one was a grounding electrode on the shoulder. ECG was measured with two electrodes between sternum and the left costal arch. One channel of EEG was measured with two electrodes between one mastoid and the position CPz of the 10/20 electrode system. After testing the signal quality the participants were instructed that the session will last for about 15 minutes in which they should sit relaxed with open eyes and just experience the lights and soundscape being aware that all changes of light and sound and all touches of an instrument will be initiated by a signal that was currently produced by their own body, that is, by their brain or heart. Then, the session was started while the experimenter remained in the background or outside the chamber. After the session, the participant could share the experience with the experimenter. Each session consisting of the introduction, setting up of electrodes, the actual session of about 15 minutes duration, a short discussion afterwards, and filling in a questionnaire lasted about 30 to 40 minutes. The final questionnaire asked for the personal experiences and allowed for performance ratings of the device.
3.2. Stimulus Design
The selection of sound and light stimuli in such kind of experiments is both an aesthetical and a scientific question. For obtaining high quality sounds the internal sound card was not appropriate. Therefore, the MIDI commands generated by the SymPOSER module were sent to an additional software sampler program offering an improved sound quality.
The beginning and final phase of the session was occupied with the basic sound of the heartbeat. The ECG was sonified with two instruments, a smooth deep bass, played at each R-peak and the ECG Padmix, that is, a drone sound with water drop like tingles played with the R, P, and T peaks of the ECG. After the initial first minute of ECG, a smooth flute sound started playing the amplitude changes of the ultraslow potential changes (USP) between 0.01 and 0.2 Hz. In addition, the delta frequencies between 1 and 4 Hz were added with strings. During the time between 5:15 minutes and 8:00 minutes the 4–8 Hz theta waves were played with a vibraphone. In the middle of the session, the slow cortical potentials between 0.01 and 1 Hz were played by harp sounds. In this setting, we preferably intended to use preferably slow EEG components which were supposed to calm down the organism and more easily allow tuning into the signal variations and rhythms. Also, it seemed to be more supportive when avoiding fast changes in the lights and flashes. Therefore, the USP were coded into a blue-red contrast. Positive slow potential shifts as generated on the cortex in relaxing moments should lead to a blue coloured environment while negative shifts normally reflect attentive moments in which the brain produces a readiness state (i.e., the “Bereitschaftspotenzial”) which turned the environment into red. This archetypical association between colour and state could result in an intensification of a certain state (which was not proven here). The smoothed band power of the alpha rhythm was used to modulate the brightness of a green spotlight that superpositioned the other lights. In the first sessions, a yellow peripheral spotlight was modulated with the heartbeat. As this appeared to be distractive for some people, this yellow spotlight was no longer used in the later sessions. Figure 4 illustrates the sound and light sequencing.
3.3. Results
3.3.1. Quantitative Results
After the session the participants had to fill in a final questionnaire assessing the subjective experiences during the session. This questionnaire was divided into two parts. The first was asking for tendencies in the emotional changes and changes in states of consciousness caused by the session. These were quantitative reports as answers were assessed in rating scales between 4 and 4. The participants had to describe their emotional state tending (or not) towards relaxation, wideness, harmony, happiness, satisfaction, luck, and love. Table 1 summarizes the quantitative results, separately for meditators and non-meditators. The major difference between both groups is a better relaxing and widening effect for non-meditators. All differences between non-meditators and meditators of the emotional reports were non-significant. As this was a pilot study with very few subjects for each group, such comparison cannot be taken too serious.
The positive tendency of the emotional states for each participant separately is visible in Figure 5. The reason for the nonrelaxing effect reported by some people in the first sessions could be associated with the distractive yellow heart beat light.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
3.3.2. Qualitative Reports
The second part of the questionnaire collected qualitative data. The participants were given the opportunity to describe in their own words their personal impressions and feelings they had during and after the session. Regarding the general spontaneous impression after the session they predominantly gave positive feedback. 12 statements were generally positive like “fascinating”, “interesting”, “novel, unique experience”, or “intensive experience”. In 7/20 participants the session initiated reactions like “experience of an enhanced consciousness” and descriptions addressing the facilitation of self-perception were given by another 6 participants. Some impressions were interindividually oppositional such as 5 persons described the session as relaxing and calming and other 5 persons described a stimulating and intensive experience. The relaxing effect pertains to a greater degree for the non-meditators (see Figure 6(a)). Only one entry mentioned some discomfort by the flood of information and another person missed the obvious connection between inner and outer perception. These were the two negative entries.
(a)
(b)
Participants also were asked whether they had the impression of a connection between the light and sound stimuli and themselves. (“Have you had the impression of a connection between the stimulation and yourself?”) 9/20 answered this question with yes, 3/20 with little, 3/20 said no, and another 3/20 reported a process of adaptation.
In a further question, the effect on the body awareness was assessed. As illustrated in Figure 7, 15/20 persons experienced an increase of their bodily awareness. In six other entries, the experience of unity concerning to the “own” music and light was described. Four persons emphasized the effect of relaxation during the session. Two of the meditator participants felt themselves irritated through the flickering light which was used in the first sessions.
Figure 8 shows the answers for the mental effect of the session. Despite the impressions were quite individual the mental effects were described positive in most entries. Important aspects were the effects of relaxation, quieting, and alteration of consciousness. The latter was reported by five meditators. Negative effects were again targeted towards the flickering light within the first sessions and the difficulty of a person to calm down the thought, which is generally a major difficulty in meditation.
4. Conclusion
The Sensorium can be regarded as a novel approach of neurofeedback to create a unifying multisensory experience with the own body, the body rhythms, and the self. In contrast to conventional neurofeedback devices the Sensorium focuses on the conscious experiencing of the complex body signals rather than on the self-regulation of a specific component. It has been demonstrated in a pilot study that this sound and light environment is a powerful tool to guide people into various experiences and alter their consciousness, body and mental awareness in a positive and conscious way. The initial results encourage us to implement a self-stabilizing and health supporting intervention for patients with psychosomatic disorders and psychological disturbances and also for people without pathological diagnosis in the near future.
Acknowledgments
The author thanks the BIAL foundation (Portugal) for the financial support that helped realizing this development within the project 148/08. He also thanks Thomas Decker for his expertise and contribution to the technical development of the devices and the software.