Abstract
Some disadvantages of optical eye tracking systems have increased the interest to EOG (Electrooculography) based Human Computer Interaction (HCI). However, text entry attempts using EOG have been slower than expected because the eyes should move several times for entering a character. In order to improve the writing speed and accuracy of EOG based text entry, a new method based on the coding of eye movements has been suggested in this study. In addition, a real time EOG based HCI system has developed to implement the method. In our method all characters have been encoded by single saccades in 8 directions and different dwell time. In order to standardize dwell times and facilitate the coding process, computer assisted voice guidance was used. A number of experiments have been conducted to examine the effectiveness of the proposed method and system. At the end of the fifth trials, an experienced user was able to write at average 13.2 wpm (5 letters = 1 word) with 100% accuracy using the developed system. The results of our experiments have shown that text entry with the eye can be done quickly and efficiently with the proposed method and system.
1. Introduction
Various human-computer interaction (HCI) methods have been proposed for the paralyzed or Amyotrophic Lateral Sclerosis (ALS) patients’ communication. Some HCI systems are designed to fulfill the most desired patient commands [1–3]. Others have focused on text entry by various virtual keyboards [4–7].
The systems referred to as the Brain Computer Interface (BCI) are, in general, based on choosing the letters of a desired word to be written with a virtual keyboard. The Electroencephalogram (EEG) signals can be used for this purpose. However, only a few words can be written per minute through the conventional EEG based systems [8].
In some studies based on the knowledge that such patients can move their eyes, saccades (30-150 ms), fixations (200-600 ms), and blinks of the eye were used to increase the text entry speed [9]. Often a virtual keyboard is used to write with eyes using optical or electrooculography (EOG) based tracking systems. The performance of optical tracking systems that use simple video cameras may fluctuate in accordance with the ambient light and head movements [9]. It is possible to determine where the eyes more precisely gaze with pupil-corneal reflection tracking devices. However, its costs are quite high. Moreover, they require precise calibration at every use [9]. In a study using such systems, typing speeds of 20 words per minute (wpm) have been achieved [6]. Kristensson et al. 2012 improved the writing speed to 46 wpm using the automatic word completion software with a similar eye tracking device [10]. These improvements have triggered the studies to develop optical eye tracking devices at low costs for the benefit of larger number of patients [11]. As an alternative to optical tracking, EOG was also used to detect eye movements.
The potential differences between the cornea and the retina during eye movements constitute the EOG signal. Clinically, EOG is used to determine the functional integrity between the retinal pigment epithelium and the photoreceptors [12, 13]. When the eyes are moved to a direction, the EOG signals taken from the electrodes placed around the eyes show changes proportional to the gaze angle. Gaze in different directions one after the other leads to drift in the base line of the EOG signals. Involuntary blinks also make it difficult to predict the gaze direction. So it is quite difficult to determine the gazing point exactly. However, it is relatively easy to determine gaze direction. In this way, control of the electric wheelchair [14, 15], the hospital alarm system [16], and the electrical hospital bed [17] has been done easily by EOG. In order to determine the gaze direction more accurately rule-based [18], neural network [19], hidden markov model [20], clustering and fuzzy logic [21], discriminant analysis [22], nearest neighborhood [7], and support vector machine [23] classification methods have been used. EOG has also been used to determine the intentions, behaviors, and cognitive processes of people and REM stage of sleep [9, 24, 25]. Further details of EOG applications can be found in the studies of Räihä et al. (2011), Istance and Hyrskykari (2012), and Majaranta et al. (2014).
The use of EOG signals to detect eye movements provides some advantages over video-based systems. It does not get affected by environment’s light and can process more easily than video images [6, 9]. However, it requires the use of electrodes attached to the patients which may cause some discomfort. Moreover, the EOG measurement system requires electrical safety precautions.
Several methods have been used in EOG based text entry systems. Some of the studies have focused on moving the cursor towards the desired letter of a virtual keyboard by moving the eye in four directions (right, left, up, and down) and the desired letter is selected by the blink of an eye [7, 26–28]. The achieved writing speeds are higher than the BCI systems but it is still insufficient when compared with optical tracking based methods. In another EOG-based approach, it has been tried to draw letters using eye movements [21, 28–31]. This method has been able to improve the writing speed by about 8 wpm, but the recognition rate of written characters is low. As another alternative, coding of letters with several consecutive eye movements has been used. Porta and Turina (2008) encoded the eye movements in different directions for transferring letters and numbers to the computer [32]. In their method, a character was encoded with at least two eye movements. Even experienced people could only write 6.8 words per minute with eyes in this method. The last two methods mentioned have more advantages when compared with others. They do not require a virtual keyboard and can be developed as an embedded system.
Although saccades of the eye are much faster, the writing speed of EOG based methods is limited to 8 wpm. In this study, firstly, various experiments have been conducted to determine moving speed and accuracy of the human eyes in different directions (right, left, up, down, and diagonals) and its combinations. In order to improve the writing speed and accuracy of EOG based text entry systems, a new EOG coding method has been suggested according to the results of these experiments. A real time computer based system was designed to implement our method. The results of the first experiments with the developed method and the system were given below.
2. Materials and Methods
2.1. Determination of the Coding Potential of the Eyes
In EOG-based HCI systems, eye movements at different directions are taken as input. Firstly, various experiments were carried out to determine the subjects’ gazing speed to different directions and the implementation accuracy.
At this stage of the study, two channels of a physiologic data acquisition system (BIOPAC MP36) were used to collect EOG data. The EOG signals on the horizontal plane were recorded through the small surface electrodes attached on the two sides of the eyes, while the vertical plane EOG was recorded by the electrodes attached over and under of an eye. Two reference electrodes were attached on the forehead. Sampling rate of the data acquisition system was set to 200 Hz and the cut-off frequency of its filters was set to 0 and 35 Hz. Motion directions were determined offline through a program we had developed in our previous study [18]. Briefly, the baseline of the EOG signals is removed first. The positive and negative thresholds of vertical and horizontal EOGs are calculated by 40% of maximum and minimum EOG amplitudes. The vertical and horizontal EOG signals that exceed these thresholds are coded by specific codes. The gaze direction is determined by a decision tree using these codes. In order to eliminate the effect of involuntary blinks, the encodings in the vertical EOG channel with a duration of less than 100 ms are neglected.
Our study protocol was approved by the ethical committee of Baskent University for clinical experiments. Four female and six male, healthy biomedical engineering senior students (22 ± 2 years old) participated in this part of the study. Their informed consent was obtained before the experiments.
In the first part of the study, the subjects were asked to look straight forward at first. Later, they moved their eyes in one direction and then looked straight again. Each subject repeated this procedure 10 times during 10 sessions (total 100 repeat). Table 1 shows participants’ coding durations and recognition rate of the vertical, horizontal, and diagonal eye movement (mean (min-max)) in the first and tenth experiments. The sequential binary permutation of these 8 eye movements can be used to encode all characters in an alphabet. Some results of this trial are also given in Table 1.
According to Table 1, the encoding speed of characters that are coded by single directional eye movements is much higher than others. Diagonal eye movements are relatively slow. As the number of trials was increased, the coding speed of a character was also increased 30-45% for one single direction and 20-30% for the others. However, the accuracy of coding decreased by about 10% with acceleration in the tenth experiment. Another striking aspect revealed in Table 1 is that sequential and/or diagonal eye movements’ recognition rate is relatively low. According to these results, we decided that using these eye movements would reduce the speed and accuracy of writing.
Another approach that could allow the encoding of eyes movement was proposed by Barea et al. 2012. They showed that EOG signals produced by eye movements with four different angles (10°, 20°, 30°, and 40°) in one single direction could be separated from each other [33]. In order to determine this methods’ efficiency at fast codding, we conducted some trials on several participants. An example EOG signal of these trials is given in Figure 1.
The obtained EOG signals can only be recognized if the subjects code the movements accurately. But our experiences have shown that recognition rate can extremely decrease when the subject tries to code these eye movements quickly.
2.2. Proposed Eyes Movement Coding Method
The experiments we have done to see the coding potential of eyes revealed that the diagonal and sequential eye movements decreased coding speed and accuracy. We found out that looking at one single direction with four different angles can shorten the coding time but it is difficult to apply this coding correctly. Consequently, we decided to use eye movements only on four main directions (up, down, right, and left).
In order to increase the number of characters encoded, eye movements in one single direction with two different angles have been used, which can be implemented much more easily than four angles. It is possible to encode only 8 characters with this method. To encode the other characters, subjects need to keep their eyes in one of the directions and angles during specified time periods. Thus, it has been possible to encode 32 and 40 characters for 4 and 5 different dwell times, respectively. An illustration of the EOG signals, which are expected to occur while all characters are encoded in one direction with two different angles for 4 different dwell times, is shown in Figure 2.
In order to standardize dwell times and facilitate the coding process, computer assisted voice guidance was used. When the user looks at one direction with an angle, the character that has the minimum dwell time in this direction and the angle is voiced by the computer. If the user turns their eyes to resting state within the first specified time, the selected character is written on the screen. Other characters in that direction are selected by the extension of dwell time. When the first dwelling period expires, the second character in that direction is voiced by computer and the user is expected to approve it, and so on. It is known that human can respond to auditory stimuli in about 150 ms [34]. According to this information the minimum waiting time has been determined to be 200 ms. Thus, this has also overcome the problems that can be caused by the involuntary eye blinking.
Finally, in order to enable coding as fast as possible, the most used letters in Turkish alphabet were assigned to the least time-consuming directions and angles. While doing this, three exceptions were applied. In the text entry studies, 5 letters are considered as one word. It can be assumed that the spacing between words is also a frequently used character. Therefore, the shortest and greatest angle eye movement to the right was set as the space between the words. Second, the shortest and greatest angle eye movement to the left was set as the deletion of the last character. Third, the longest and greatest angle eye movement to the left was set as the deletion of the last written word. The position and order of the letters in our coding system are shown in Figure 3. In this sorting, six of the most used characters (about 50% usage rate) have been encoded in the shortest possible time.
2.3. The Developed Two-Channel EOG Acquisition System
The block diagram of the developed two-channel EOG acquisition system is given in Figure 4. An instrumentation amplifier (AD624) with a 130 dB CMRR (Common Mode Rejection Ratio) was used to suppress the electrode half-cell potentials and several hundred millivolts (mV) power line noise (50/60 Hz).
In order to prevent the output offset voltage that can be caused by the difference in electrode half-cell potentials, the differential voltage gain of this amplifier was set to 20. Ground driver circuit is basically used to filter the signals induced by the 50/60 Hz noise. The noise that is received by the electrodes is applied to the patient over a very high resistor after the phase inversion. This procedure not only increases the CMRR of the amplifier but also limits the current that can pass through the patient’s body [35].
Second-order Butterwort filters were designed to filter the EOG signals. According to the ISCEV (International Society for Clinical Electrophysiology of Vision) EOG standards, a 0.5-30 Hz band pass filter is used to prevent base line shift [12]. However, the long dwell times in our coding method have caused us to pull down the low cut-off frequency (0.2 Hz) even further.
An additional amplifier was used to increase the amplitude of the EOG signals to 2-4 Volt level. Consequently, the small differences in the electrodes’ half-cell potentials caused a serious offset in the base-line. In order to eliminate this, a potentiometer was connected to the offset adjustment terminals of the amplifier to shift the EOG level to 0 volt at resting state of eyes. To prevent the macroshock hazard, the user contacted circuits were powered with 9-volt batteries. Additionally, optical isolators were used to prevent any leakage current that may be caused by the computer. A Data AcQuisition (DAQ) card (USB-NI6009, by National Instruments) was used for transferring the analog signals to computer in 12 bit resolution.
2.4. Software of the Developed System
The software of the developed system consists of 3 main sections. These are a program enabling DAQ card to receive 2-channel EOG signals, a graphical user interface to control the system, and a program for processing and converting the acquired signal to character by the help of voiced guidance of computers. MATLAB 2007 was used as the development platform for the software. General flowchart of the software is given in Figure 5.
At the beginning of the program, some initial values are assigned and the DAQ card is programed for collecting data from 2 channels. The sampling frequency was set to 200 Hz. The collected data were read at every 20 samples (every 100 ms) from the DAQ card and processed as in the flowchart.
The developed graphic user interface is illustrated in Figure 6. The interface was equipped with Start/Stop and Clear/Delete buttons to control the system by the person performing the system tests. The desired character is determined by the EOG signal that exceeded a certain threshold when looking in a certain direction. The amplitudes of the EOG signals that will emerge during the eye movement might be different for different individuals or at different disease levels. For this reason, in order to select the relevant threshold values, the edit boxes have been inserted on the GUI. Before the first use, EOG threshold values should be determined for right-left and up-down eye movement by motion monitor panels in GUI.
When the user moves the eyes in a direction, the program determines which amplitude threshold value is passed and voices the first letter in the relevant direction. As previously mentioned, most people react to acoustic stimulus in approximately 150 milliseconds [34]. Users can react slowly at the learning stage of the coding method. When they memorize the method, they may react more quickly. Therefore we have added the edit boxes to GUI, which we call pulse width threshold. If the eyes are brought to resting position within a certain time period (pulse width thresholds) after an eye movement, the corresponding character will be written on the text box. Otherwise, the next character at the same direction and angle is voiced and expected to be approved by the user. The GUI is equipped with horizontal and vertical motion monitors that are used to show EOG signals to help with the above adjustments.
3. Results and Discussion
In this section, the performance of the developed EOG based text entry system is presented. Since our ethics committee permission covers the studies with a commercial physiological measurement system, the studies in this section have only been performed on one of the authors. The subject was 23 years old and has a master’s degree in biomedical engineering. After a few attempts, the threshold values were set to the values shown in Figure 6. A timer was added to the system to determine the writing speed. The timer that is started when the first character starts to be written is stopped when the desired word is written, which ends with a space character. Elapsed time is shown in a textbox that is added to the system for experimental studies. The coding speed of a text was calculated by considering the space coding as an additional character. The writing speed is calculated in terms of words per minute (wpm) on the assumption that a word consists of 5 characters.
In the first experiments, all of the 29 letters in the Turkish alphabet were coded consecutively. 27 out of the 29 letters were written correctly in the first attempt. Two of them had to be erased and rewritten. So, the accuracy of codification in the first attempt was 93%. The writing speed of this attempt was 12.46 wpm.
In order to test our system, we have chosen words and simple phrases in Turkish (Table 2), which we think can meet the most basic needs of people who are confined to bed. The English equivalents of the words are also given in Table 2. Each word was written 5 times. Coding times, writing speed, and correct coding rate of characters are also given in Table 2.
While the writing speed was 11.84 wpm on average in the first trial, it was 13.2 words per minute in the fifth experiment. The accuracy was 86.2% on average in the first attempt. In the fifth attempt, it reached 100%.
After an adaptation period, it could be written at average 13.2 wpm with 100% accuracy by using the developed system. This performance is much better than conventional BCI systems. A comparison of the EOG-based typing and writing systems is given in Table 3. As can be seen in Table 3, only 8 wpm can be written with the fastest EOG based system developed to date. If it is compared with the other EOG based systems in the literature, our writing speed and recognition performance are the best.
On the other hand, while text entry can perform more efficiently than most of optical tracking system, it is behind the speed of the Majaranta et al. 2009 and Kristensson et.al.2012 systems. These systems are supported by a professional eye-tracking device. Kristensson et al. 2012 system also benefits from a word completion program to achieve the high speed writing.
The developed system has many advantages over the optical tracking based text entry systems. Since it is EOG based, it is not affected by ambient light. It does not need a virtual keyboard, so it can be produced as an embedded system. It can be used after a simple calibration, such as EOG threshold determination, to be performed during initial use for a user. The system can be adapted to all languages with simple assignments in the program. By adding new dwell times for character encoding, the number of total characters that can be written can be increased.
However, the system has some disadvantages when compared to optical tracking systems. Firstly, electrical safety precautions are required. Since the electrodes used are not stable for a long time, they must be replaced at regular intervals. In addition, cables of these electrodes can cause some discomfort. Furthermore, a certain amount of time is needed to learn the used coding method.
4. Conclusion
In this study, a new method enabling the patients suffering from ALS-like diseases to enter the texts easily, quickly, and accurately to the computer using EOG is proposed. For this purpose, a PC based system has been developed to enable the implementation of the proposed method and various experiments have been carried out.
In the proposed method, eye movements into four main directions with two different angles and four different dwell times were used to encode the characters. A complex pattern recognition procedure is not required to convert the encoded EOG to text. For character recognition, EOG signals exceeding certain threshold values during the respective encodings are enough. The necessary dwell times for the encoding of characters are guided by the computer through voicing the characters.
Consequently, thanks to the proposed encoding method and developed system, text entry can be done easily, quickly, and accurately with EOG. In future studies, applicability of the developed coding method together with the low-cost optical tracking systems can be investigated. Furthermore, the word completion software can be integrated to the system so as to increase the text entry speeds.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request. The dataset about determination of the coding potential of the eyes is available. However, since the system we developed work real-time, we did not record EOG data when testing our system.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.