Abstract
A sensorimotor training system that facilitates learning to play piano was developed and tested. The system consists of three communicating units. The first unit comprises two pianos: an E-piano with a MIDI output for the teacher and an acoustic or an E-piano for the pupil. The pupil’s piano is supplied with an LED bar that illuminates the key to be struck. The second unit is a controller providing the interface between the teacher’s piano and the LED bar. The third unit consists of two pairs of gloves: one for the teacher and one for the pupil. The teacher gloves have integrated pressure sensors at every fingertip. The pupil’s gloves have vibration motors and LEDs at every finger. The pressure sensed on the teacher’s glove is transmitted to the corresponding finger on the pupil’s glove via the vibration motors and LEDs, such that the pupil knows which finger should strike which key. Additionally, two OLED displays showing the notation of the note played by the teacher can be attached to the left and right pupil’s gloves. Initially developed for people with cerebral palsy the sensorimotor system can support the learning also to all those with sensory, cognitive, and space perception impairments.
1. Introduction
The difficulties when learning to play the piano are well known and have been experienced by many people in their childhood or youth. The coordination of fingers, the observance of rhythm, the ability to strike the right keys, and especially the deciphering of an abstract text of music notes are demanding tasks.
For people with motor impairments and learning disabilities, this can be an almost hopeless task. Especially for individuals with cerebral palsy, learning to play a music instrument is a great challenge. Cerebral palsy is a multiple disability, where depending on the extent of brain damage not only are motor areas of the brain affected, but also cognitive and sensory impairments, disturbances of space perception, body image, and language and sometimes epilepsy exist [1].
Music therapy is often offered in institutions for people with mental and physical disabilities. It is frequently performed as a group exercise on Orff instruments and Veeh-harps. Other instruments, such as drums, triangle, and xylophone, are also pleasant to use and simple to operate even for people with motor impairments. Such therapies facilitate expressing feelings and improve communication skills, enhancing the self-esteem of people with disabilities [2].
Positive effects of playing piano on the recovery of hand fine motor skills and the development of neuroplastic changes have been reported in patients with acquired brain injury, especially in stroke patients [3–6]. However, the effects of playing piano on hand motor function or in initiating neuroplastic changes in patients with brain injuries occurring during development, as in cerebral palsy, have been rarely addressed in the literature. One likely contributor to this is the often coexistence of multiple disabilities that limit the ability to learn the instrument. Nevertheless, playing piano is particularly advantageous for individuals with cerebral palsy, since pianos need not be held in contrast to, for example, a violin. Moreover, immediate auditory feedback received when striking a piano key stimulates the patient to correct his own performance.
As reported in [7] regular piano lessons can contribute to improving hand motor skills in children and adolescents with cerebral palsy. Moreover, learning to play the piano can also produce changes in brain network connectivity [8]. However, during the study several difficulties were observed. Learning abstract notes was difficult because of learning disabilities. Cognitive deficits caused difficulties in reading note text so that playing by notes was hard. Тhe differentiation of fingers was furthermore insufficient. Hand sensor abilities were often disturbed, which was expressed in improper use of the fingers. In addition, participants had problems with spatial orientation at the piano. To help overcome these difficulties and facilitate independent training of piano playing a versatile piano training system was developed and is here described.
Pianos with lighted keys are already commercialized. Here music pieces are programmed and built into them so that the respective keys are illuminated while playing the piece. The user only has to strike the keys in the prescribed sequence. There are also a number of software programs that support the learning of piano playing. For instance, piano keys of a MIDI player are highlighted in color, while the piece is played from a computer. These piano learning tools provide only a visual aid and do not take into account the difficulties in recognizing and differentiating the fingers, the deficits in eye-hand coordination, impaired spatial orientation, and other sensorimotor disabilities. Moreover, only pieces can be learned that are already stored in the piano or which are available as MIDI files on the Internet.
This paper addresses the technical development of a flexible piano training system aimed to compensate the deficits described above. This system, initially developed for people with cerebral palsy, can be used also in neurorehabilitation settings. Furthermore, it can support the learning of piano playing in other medical conditions, involving also sensory and cognitive deficits.
2. Technical Description of the System
The piano system consists of three units that communicate with each other (see Figure 1). The first unit comprises the teacher’s piano, the pupil’s piano, and an LED bar. The second unit is a controller which provides an interface between the two pianos. The third unit consists of two pairs of gloves, one for the teacher and another for the pupil. The aim is to transfer the fingering executed by the teacher as well as the sequence of piano keys played to the pupil.
2.1. Unit 1: Pianos and LED Bar
The teacher’s piano is an E-piano with 88 keys and a MIDI (Musical Instrument Digital Interface) output. The pupil’s piano is an acoustic or an E-piano with also 88 keys. The LED bar is placed onto the pupil’s piano keyboard to see the use of the keys.
When a key is struck a MIDI command consisting of 3 bytes is sent to the controller. The first byte contains the status command (note on, note off) and the MIDI channel. The second byte describes the note and the third one the strength with which the key is struck. This information describes completely when and which button was pressed and released.
The keys played by the teacher are presented to the pupil via the LED bar placed over the pupil’s piano. The bar consists of 88 WS2812b which are arranged to match the size of keyboard (see Figure 2). The LED lights are RGB LEDs with a built-in drive circuit. They are connected in series, so that one data line is sufficient for all LEDs. LEDs placed over the white keys of the piano have red color while those over the black keys have green color. The red/green combination makes it easier to distinguish at first glance which type of the key is being used. By using an appropriate combination of red, green, and blue intensity values (from 0 to 255), practically any color can be chosen. Each piano key is illuminated with only one LED in a punctiform manner (see Figure 2(a)), such that it is easy to identify the keys that should be struck. The bar consists of seven segments, one for each octave, which can be connected to each other and remain aligned close together via a magnetic plug. This LED bar design is intended to facilitate the transport of the bar; it can be split into its parts and then folded together (Figure 2(b)).
(a)
(b)
2.2. Unit 2: Control Unit
Interpretation of the MIDI commands and control of the LED bar are done via the control unit. The main component is an Arduino Mega 2560 microcontroller which has 4 UART (Universal Asynchronous Receiver Transmitter) interfaces for serial communication (see Figure 1). One of the interfaces is connected through an optocoupler to the MIDI output of the teacher’s piano. The data received are interpreted and then transformed into a signal to control the LED bar placed over the pupil’s piano. NeoPixel library of Adafruit Industries is used for communication between the Arduino and the LED bar. Only one digital output is required to control an arbitrary number of LEDs. The settings for each individual LED are sent, via a resistor, through the digital input of the first LED of the series. The information is sent in reverse order (i.e., last LED first) and transferred from one component to the next. Since every single LED has its own built-in drive circuit no additional hardware is necessary to control the LEDs.
The control unit can also simultaneously drive 6 LED bars, which would allow the signal from the teacher’s piano to be transmitted to 6 different pianos simultaneously; that is, it would allow group lessons.
2.3. Unit 3: Gloves
Special gloves have been developed for the teacher and the pupil. The teacher gloves have integrated pressure sensors located at every fingertip. The pupil gloves have a vibration motor and an LED at every finger. The signal produced by a pressure sensor of the teacher gloves is transferred to the pupil gloves (to the vibration motor and to the LED light of the corresponding finger) so that the pupil is informed which finger to which key should be applied.
Each teacher glove is a tight-fitting thin leather glove (Figure 3). Sensors FSR 400 (Interlink) were incorporated in the glove’s fingertips. These sensors show a decrease in resistance, from more than 10 MΩ unloaded to less than 10 kΩ, with an increase in force applied to the surface of the sensor. They were connected in series with a 150 kΩ resistor. The pressure sensor and resistor form a voltage divider with which the change in the resistance of the sensors can be measured. The sensors were connected to the digital inputs of a Xbee Series 1 module (Digi International) that allows for wireless communication to other modules.
The module is powered by a mobile power pack and configured so that it transmits the states of the pressure sensors to a second module every 10 ms. This second module is located along with vibration motors and LEDs on the pupil’s glove. The pupil’s gloves were developed from common cyclist gloves. These have the fingertips free and therefore do not hamper piano playing. Each finger of the glove has a vibration motor and an LED that are activated by the Xbee module. Applying force on a sensor in the teacher’s glove sets the digital outputs of the pupils Xbee module to “high” state, which activates the corresponding LED and vibration motor.
The pupil’s glove is designed as follows. All discrete electronic elements such as LEDs, vibration motors, and Xbee module are embedded with silicone in a single unit (see Figure 4). Silicone provides good stability but also elasticity so that the electronic add-on can be attached to a glove of any size. In order to increase the visibility of individual LEDs they can be uncovered before embedding in silicone. In this case a plexiglass disk is used as protection against damage. Alternatively, transparent silicone can be used. The electronic add-on is fixed on the glove using either magnets or snap fasteners.
(a)
(b)
2.4. Additional Features
The following extra features and additional modules can be added/used with the sensorimotor system described above.
2.4.1. Note Display
Two displays indicating with letters the notes played by the teacher can be connected to the control unit. The 1.3″ 128 × 64 pixel OLED displays by Adafruit Industries were used here. They were controlled by an SSD1306 driver chip using SPI (Serial Peripheral Interface) for the communication with the control unit. MIDI data received from the teacher’s piano are used to identify the keys struck. For example, if the teacher plays the note C, the letter “C” appears on the display (Figure 5(a)). One display is attached to the right glove and another one to the left glove of the pupil.
(a)
(b)
2.4.2. “Single-Piano” Mode
An alternative to using two pianos during the lessons is to have both teacher and pupil seating at one piano only. In this “single-piano” mode the pupil seats right of the teacher and the LED bar lights the keys played by the teacher one octave above (Figure 5(b)). This “single-piano” mode can be turned on through a switch in the control unit. The “single-piano” mode enables simultaneous playing of the pupil and the teacher on the same piano. This proximity has the advantage that the teacher can interfere in the play of the pupil.
2.4.3. Recording a Musical Piece for Training
There is also the option of recording the teacher’s playing to reproduce it later. To this end, the control unit contains an Xbee module receiving the data with the fingering of the teacher (see Figure 6). This module is connected to the Arduino. The notes are transferred via MIDI output as described above. The Arduino contains the USB-to-TTL Serial chip Atmega 16U2 which allows communication with a computer via USB. The data transmitted from the Arduino were processed with a program written in LabVIEW. The universal I/O interface software VISA 2015 by National Instruments was used as the interface between hardware and software. The data received can be stored locally on the computer. Playback of the saved note sequences and fingering runs in reverse order by driving now the LED bar and pupil glove. The program can be run on a PC and also on a Windows tablet. This additional feature makes piano training possible in the absence of the teacher.
3. Experimental Testing of the System
The efficacy of the system in supporting the learning of playing the piano was tested experimentally on four individuals with cerebral palsy and on eight healthy persons. Two patients had bilateral leg affected cerebral palsy; one patient had unilateral cerebral palsy, and one patient had ataxic cerebral palsy. The task was to reproduce two sequences of 10 notes played by the piano teacher. The first sequence was played without the help of the system. The second sequence was presented with the help of the system. In both cases the patients had to reproduce the sequence three times on their own. The system was only used during the presentation. Sequences were always played three times by the teacher.
Individual results of persons with cerebral palsy are presented in Figures 7(a)–7(d). The number of errors versus the trial number is depicted, for sequences presentation with the system (red line and symbols) and without (blue line and symbols). For all of them, the number of errors (i.e., wrong note played) decreased from the first to the third trial, when using the system. When the system was not used, the number of errors remained approximately constant. Furthermore, the outcome in the final trial when the system was used was always better than when the system was not used.
(a)
(b)
(c)
(d)
The same task was performed by the healthy participants. Apparently, the task was too easy for them because practically no errors were made when playing with or without the system.
4. Discussion
People with cerebral palsy have multiple disabilities depending on the extent of areas affected by brain damage. In addition to motor impairments, sensorimotor, learning, and attention deficits can be present to varying degrees.
A piano training system was developed to support people with disabilities to learn to play the piano even without background knowledge of notes and despite the reduced sensory ability of the fingers. Playing the piano can be considered as a rehabilitative therapy, since it promotes hand motor function. Improvement of hand fine motor functions after learning to play the piano has been reported [7]. The system here proposed aims additionally to compensate for sensory deficits. It is a good addition to the classical therapeutic programs in cerebral palsy, which are mostly directed to improve gross motor skills like standing and walking and preventing contractures of the joints.
By means of LED bar which illuminates piano keys, the system provides the pupil with the information on the key that must be struck. Pianos with lighting keys can currently be purchased on the market. However simply having a lighted keyboard is insufficient to compensate for the sensorimotor and cognitive deficits that can be present in many patients with cerebral palsy. In the sensorimotor system described here the punctiform of lighting in the design of the LED bar allows easy differentiation of the keys to be struck, especially with adjacent keys. It is also advantageous that the black and white keys are illuminated with light of different colors. Furthermore, the LED bar is suitable for any usual acoustic piano; it can be decomposed for comfortable transportation and easily reassembled again. The commercially available pianos contain a limited number of piano pieces for training, which may not be suited for a person with multiple disabilities. The sensorimotor piano system here described allows the learning of music pieces chosen to fit the pupil’s specific limitations and needs, being played by the teacher. Many MIDI-Player computer programs offer learning aids by means of visualizing the keyboard on a computer screen in a smaller size whereas the LED bar here presented highlights the keys on the real piano.
MIDI players or illuminated piano keyboards are not yet equipped with a note translator. The system here described can contain an integrated note translator that makes it possible for the pupil to see on small displays being placed on the gloves which notes were played. This can help to learn abstract notes.
The idea to facilitate learning to play piano by transmission of electric signals to the fingers of the pupil was proposed, for example, in [9]. In the system developed, vibration signals were used because they can be easily perceived by individuals with cerebral palsy and comprise a therapeutic component. Children and adults with cerebral palsy have often an impaired sensory perception of the fingers. Treatment methods to improve body awareness through vibration have been described in the literature [10–12]. Whole body vibration plates, for example, Galileo, are increasingly being tested in cerebral palsy [13, 14]. To promote the perception of the individual fingers and reduce spasticity, sensory feedback training combined with vibration stimuli has been recently proposed [15]. The fingers of 7 patients with cerebral palsy were stimulated using small vibration motors integrated in the fingers of a hand glove. The pupil’s gloves in the presented training system work with a similar principle. Additional to vibration motors, LEDs are placed in each single finger of the glove. The gloves help to recognize the fingers by indicating through vibration and LED lights with which finger the piano key has to be struck.
Normally it requires many hours of practice and experience to acquire an appropriate predicted fingering. The latter is transmitted here by the teacher by simply playing the sequence in a piano while wearing special gloves that have pressure sensors in the tips of the fingers. These allow the recording of the fingering that is then displayed via LED and/or transferred via vibration to the pupil’s fingers, so that the pupil receives the suitable position for the hand and can play the music piece comfortably. The recording of the fingering is easy for the teacher, since he solely has to put on his gloves. With this technique the teacher can navigate the pupil during the lesson in real time or, another option, by recording his piano playing to a tablet/computer via the control unit. The student can then practice and repeat the recorded fingering at home at his own speed with the help of the vibration motors and LEDs. The self-developed software allows furthermore adjustment of the speed and timing of vibrations and LED display so that the pupil can repeat the song in different tempos according to his own individual needs.
The efficacy of the system in supporting the learning of simple sequences of notes was tested experimentally in a small sample of patients with cerebral palsy and in healthy participants. Improvements (i.e., reduction of the number of errors) could only be observed in the first group. This is a positive outcome since this is the target group for the sensorimotor system proposed. Evidence for support of learning in healthy participants, if possible, may require the use of longer sequences or for longer periods of time. The results presented provide preliminary evidence for the benefits the system can deliver. Further evidence remains nevertheless necessary to confirm the system as therapeutic option in rehabilitation programs for people with sensorimotor disabilities.
By compensating the effects of sensory deficits and learning disabilities, this piano system may support and encourage people with disabilities to learn to play the piano, this way contributing to their inclusion. It offers also a new therapeutic approach that involves motoric, sensory, and attention skills.
5. Conclusion
This work describes a novel piano training system aimed at assisting people with sensorimotor and cognitive disabilities to learn to play the piano. Compensation of deficits is achieved mainly by means of vibration and visual information to the pupil. The following options are included:(i)Training (individual or collective) with the teacher on separate pianos.(ii)Training with the teacher on a single piano.(iii)Training without the teacher using records of piano playing by the teacher.(iv)Training with the note translator.
Competing Interests
The authors declare no conflict of interests.
Acknowledgments
The work was partly funded by the German Research Society (DFG, project LA 2670/3-1), the Buhl-Strohmaier Foundation, the Friede Springer Foundation, the Gips-Schüle Foundation, the Würth Foundation, the Zeidler Research Foundation, and Casio Europe.