Abstract
Imagery-based brain-computer interfaces (BCIs) aim to decode different neural activities into control signals by identifying and classifying various natural commands from electroencephalogram (EEG) patterns and then control corresponding equipment. However, several traditional BCI recognition algorithms have the “one person, one model” issue, where the convergence of the recognition model’s training process is complicated. In this study, a new BCI model with a Dense long short-term memory (Dense-LSTM) algorithm is proposed, which combines the event-related desynchronization (ERD) and the event-related synchronization (ERS) of the imagery-based BCI; model training and testing were conducted with its own data set. Furthermore, a new experimental platform was built to decode the neural activity of different subjects in a static state. Experimental evaluation of the proposed recognition algorithm presents an accuracy of 91.56%, which resolves the “one person one model” issue along with the difficulty of convergence in the training process.
1. Introduction
Brain–computer interface (BCI) [1] technology directly uses the EEG signal [2] of the cerebral cortex to avoid human nerve transmission and builds an interactive bridge between the central nervous system of the brain and the external environment. BCI technology plays a significant role in biomedicine and rehabilitation, among other fields [3–7].
The study of BCI was first conducted in the 1920s. The concept of BCI was gradually formed [8] after the German Hans Berger medical team collected electrical signals from the cerebral cortex through surface electrodes for the first time [2]. With the continuous development of science and technology, several BCI studies [9–15] have also achieved various remarkable results, among which the imagery-based BCI [16] is popular. Its research is based on the development of ERD/ERS physiology [17, 18]. By imagining limb movement consciousness tasks, users can generate distinctive brain spatial patterns in the cerebral motor cortex and then use pattern recognition algorithms to translate different activity patterns into different categories of information to realize brain–computers interaction. Compared to the visual/auditory evoked BCI, it does not require the stimulation and assistance of the external environment belonging to the active BCI [19]. Therefore, it has a prospective broader application.
The imagery-based BCI was first proposed by the team of an American scholar Wolpaw in 1993. The team designed a BCI system that used the μ-wave in the brain electrical signal to control a mouse; the recognition accuracy was 70% [16]. Subsequently, several other imagery-based BCI studies emerged. In 2015, Yao proposed a BCI based on stimulation assistance. The recognition accuracy was 80%, which improved BCI blindness to a certain extent (the recognition accuracy rate was previously lower than 70%); however, the accuracy rate had significant room for improvement [20]. In 2017, Li et al. introduced deep learning to the application of BCI [21], considering the optimal wavelet packet transform (OWPT) method and the long short-term memory (LSTM) network for feature extraction and classification, the results indicate a high recognition accuracy; however, the OWPT method is time-consuming and is not suitable for online recognition. In November 2018, Lin and Shihb embedded the following two deep learning models into the BCI system for MI-EEG signal classification to identify two imaginary movements: LSTM and generalized regression neural network (GRNN); the results indicated that the performance of GRNN is better than that of other strategies [22]. In April 2019, Professor Anumanchipalli et al., a neurosurgeon at the University of California, San Francisco (UCSF), and his colleagues developed a decoder that converts human brain nerve signals into speech, which is powerful for assisting patients who cannot speak to achieve vocal communication [15]. In 2019, Jiao et al. proposed a novel sparse group representation model (SGRM) for improving the efficiency of MI-based BCI by exploiting the intersubject information, which solves the problem that takes long time to record sufficient electroencephalogram (EEG) data for robust classifier training [23]. In a recent study, Willett et al. demonstrated a BCI in the cerebral cortex, in which a new recurrent neural network decoding method was used to decode imaginary writing actions from neural activities in the motor cortex and translate them into text in real time. The subject’s typing speed was 90 characters per minute and the accuracy rate was 99%; it also consists of a general automatic correction function.
Apparently, the feature extraction, classification, and recognition algorithms of signals in the BCI directly determine the practicability and effectiveness of the BCI. However, most existing experimental BCI paradigms use traditional classification algorithms based on feature values [1, 2, 24–29], which largely limit the development of BCI, including “BCI blind” (recognition accuracy of less than 70%), “one person, one model” (due to individual differences, recognition models cannot be shared), and model training process issues such as difficulty in convergence.
The purpose of this study is to solve the “one person, one model” issue while ensuring recognition accuracy. Thus, a composite network model is considered, for which the contribution is twofold:(1)Model: aiming to resolve the issues relative to the traditional BCI model, a new BCI model is proposed, and a new experimental platform is built to verify the feasibility of the model. Furthermore, the data collected during the experiment is used to compile a data set for model training and testing to achieve accurate decoding of signals.(2)Dense LSTM algorithm: since the EEG signals in the imagery-based BCI need to obtain deeper features when the LSTM network is used for classification, the model training process has significant jitters, considering recognition accuracy as well as difficulty in convergence. The method of machine vision is abandoned, and deep learning is introduced to effectively improve poor generalization ability. A new classification algorithm is proposed. First, the recognition accuracy of the algorithm is ensured through the LSTM network and compared for different groups of people; second, the convergence of the model training process is ensured by grafting the Dense layer, and the accuracy of gesture recognition goes up to 91.56%.
Section 2 introduces the model and experimental platform. Sections 3 and 4 present the Dense LSTM algorithm and analysis of the experimental test results, respectively. Finally, Section 5 summarizes the study.
2. Model and Experimental Platform
In this section, the new model is first proposed followed by a partial introduction of the experimental platform and process.
2.1. Model
The new imagery-based BCI model is shown in Figure 1. The model is mainly composed of the following three parts: signal collection and preprocessing, data recognition and classification, and control of application equipment. During the experiment, the subject imagined gestures in a static state while simultaneously completing the acquisition and preprocessing of the EEG signals through the device; the processed data was then transmitted to the classification module. Finally, the control of the application device was completed according to the recognition result, which realized the decoding of the subject’s “mental gesture action.”
During the experiment, each subject imagined five gestures in turn, namely, (i) thumb bending, (ii) index finger bending, (iii) middle finger bending, (iv) ring finger bending, and (v) little thumb bending. The gesture needs to be imagined eight times. During the experiment, the subjects need to be completely still, and the surrounding environment was kept silent to avoid any noise interference. The experiment recorder started to record and intercept the data for 10 seconds after the EEG was stabilized. The subjects stopped imagining the gesture with the end beep sound after 10 seconds; they can rest for half a minute until the next data collection in this cycle.
The time schedule of a single experiment for data collection is shown in Figure 2, which is divided into four steps as follows: Step 1: during the first five seconds of the acquisition, the subject was completely relaxed and still Step 2: imaginary movements of the intended gestures were performed for 10 s in the brain after hearing a prompted sound Step 3: after 10 seconds, the same sound was prompted for the subject to stop imagining the gestures Step 4: the subject rested for half a minute until the next data collection in this cycle
A total of 20 subjects participated in this experiment, including 15 boys and 5 girls. The age of the subjects ranged between 18 and 40 with an average age of 25. All subjects were in good health and met the requirements for participating in the BCI experiment. Prior to the experiment, all subjects were trained and studied imagery-based BCI; they were all informed of the experimental precautions to ensure that the experimenter understood the entire process to avoid unnecessary factors affecting the experimental results.
2.2. Experiment Platform
The EEG acquisition equipment required for the experiment included the NCERP EEG and an evoked potential instrument developed by Shanghai Nuocheng Electric Co., Ltd. The instrument consists of a computer host, a display, an audio and video stimulation box, an EEG main control box, a physiological amplification box, and electrode composition. The equipment has a high sampling rate of up to 8 kHz/CH and a 32-bit resolution; the data collected is more accurate. It can adopt multiple collected data according to actual experimental needs to obtain more EEG data characteristics. This instrument adopts a noninvasive EEG signal acquisition method and uses 24-channel silver electrodes. As shown in Figure 3, the placement position adopts the international unified standard 10/20 system method. The electrode cap and the physiological amplification box are connected to complete the physiological signal acquisition and amplification. The collected data is then transmitted to the EEG main control box through the optical fiber and finally transmitted to the computer host through the USB interface. The construction effect of the experimental platform is shown in Figure 4.
2.3. Signal Collection and Preprocessing
During the experiment, the data can be preprocessed by setting the parameters of the EEG signal collection instrument. Using the artifact correction method [30], remove the EOG artifacts from the collected EEG signal, and a low-pass filter was set to remove the EEG signal that caused interference near the 50 Hz power frequency. The valid data after preprocessing, such as interception and arrangement, is further processed which is normalized and mapped to the interval [−1, 1]. Finally, the processed data is compiled into a data set for the experiment, and the parameters of the data set are shown in Table 1.
Since each partition of the human brain corresponds to different functions, and the prefrontal cortex is responsible for processing imagination and thinking-related activities, the channels of F3 and F4 (as shown in Figure 3) data corresponding to the electrode caps of the frontal lobe are mainly used to make the data set. During the experiment, after the subjects heard the prompts, they imagined each gesture in turn. At the same time, the data recorder started the recording operation after the EEG stabilized, intercepting 10 seconds of data, and used it to make a data set. The data set used in this experiment is compiled from the experimental data of 20 subjects, from which 80% of the data is randomly selected for classification and recognition model training, and 20% is used for model testing; these 20 subjects use one model.
3. Dense LSTM Algorithm
In this section, a new recognition model is established by combining the Dense layer [31] with the LSTM network, and the model parameters are presented to achieve an optimal combination.
3.1. Dense LSTM
The classification algorithm module belongs to the data processing and recognition part of the BCI. Common classifiers such as the support vector machine (SVM), multilayer perceptron (MLP), and Bayes classifier are ubiquitous, for which recognition accuracy is not high, and the “BCI blind” issue exists, among other problems. Lin et al. used the LSTM algorithm to achieve higher efficiency, but it takes longer to extract feature values using the optimal wavelet packet transform method. Thus, the Dense LSTM model is proposed in this study. As shown in Figure 5, the first half of the model is the LSTM layer, followed by the added Dense layer.
The LSTM network consists of three designed gates. The forget gate determines whether the output information of the previous moment is retained or discarded. After the effect on the output of the previous time and the input of the current time, the output is a value in the range of 0 to 1. Utilizing the sigmoid function, the role of the forget gate is shown in the following equation:
In the formula, and U mean the weight of gate, S means the output data of the previous moment, x means the input data at the current moment, b means the bias term of gate, the subscript “f” is the name of the gate, and t and t − 1 indicate different moments.
The role of the input gate is to control the input at the current time, which directly determines how much new information will be input into the hidden layer of the LSTM. The working principle of the input gate is shown in the following equation:
The candidate gates calculate the total storage of the input at the current time and the previous input information. The working process is shown in the following equation:
The update of information while the LSTM is functioning is determined by the forget gate, the input gate, and the candidate gate. The forget gate determines if the information to be lost is equal to and the input gate and the candidate gate determine if the new information is equal to , which is added at the current moment. Combining these two, the hidden layer processes the new state as shown in the following equation:
The output gate determines how much information is input to the next moment, that is , and the calculation process is obtained by . The output gate functions as shown in the following equations, and is the weight of the output gate, which is in the range of 0 to 1. The size of will determine the information passed to the next moment:
During the data collection process, the collected EEG signal is transmitted to the LSTM as input data, and the result following the analysis and processing of the LSTM is transmitted to the connected Dense network. The Dense network part is composed of two Dense layers, and the two Dense layers have the same network structure. Each Dense network is composed of an input layer, a hidden layer, and an output layer. The structure is shown in Figure 6; all neurons between each adjacent layer are connected to each other. Through the superposition of the composite network, the feature propagation of the data can be strengthened so that the network can mine deeper features in the data and promote the convergence of the classification and recognition model in the training process.
The principle of the Dense network operation is shown in the following equations:
Since the output of each network layer is a nonlinear combination which impacts the recognition capability of the model, after the Dense network, the activation function is used to produce a nonlinear effect on the output result to solve the problem that the linear model cannot by improving the effectiveness and reliability of the classification and recognition models. The principle of the activation function of Softmax can be expressed by the following equation:
The activation function of Softmax maps the output value of the Dense network to a vector , where is a real number for which the size is in the range of (0, 1), indicating the probability of belonging to each category in the multiclassification problem; is 1. The network can finally obtain the result of classification according to the probability of each classification with the respective mapped vector.
3.2. Model Parameter Settings
The classification model of the BCI proposed in this study is formed by connecting the LSTM network and the Dense network. The connection of different network layers has varying degrees of impact on the recognition effect. Therefore, it is necessary to set appropriate network parameters to ensure optimal recognition effects.
Herein, the parameter unit size of the LSTM layer is presented. The result is shown in Figure 7. By comparing the recognition accuracy rate and the time required for each iteration when training the model, when the unit size is 128, the recognition accuracy rate is 91.56% and each iteration takes 0.52 seconds. When the unit size is 256, the recognition accuracy rate is 91.80%, but the time required for each iteration is 1.62 seconds. The recognition rate difference between them is not obvious, but the iteration time is quite different. At this time, the effect is optimal; thus, the unit of the LSTM layer is set to 128.
We compare some versions of our Dense LSTM algorithm. By testing the size of the parameter unit in different Dense networks and observing the training process of the classification recognition model as well as the accuracy of recognition, the results are shown in Table 2. When the LSTM layer is grafted to a Dense network, during the process of reducing the unit size of the Dense network from 256 to 128, the recognition accuracy rate increases rapidly from 82.32% to 87.66%; when the unit size decreases from 128 to 32, the recognition accuracy rate slowly rises to 89.52%. When the LSTM layer was grafted with two Dense networks and the unit size of the first Dense network was fixed at 256, the unit size of the second Dense network was reduced from 256 to 128, and from 64 to 32. During this process, the recognition accuracy rate increased from 85.43% to 86.42%, and from 88.59% to 89.32%. When the unit size of the first Dense network was fixed at 128, the unit size of the second Dense network was reduced from 128 to 64, and to 32, and the recognition accuracy rate sequentially increased from 89.45% to 89.45%, and to 90.22%. When the unit size of the first Dense network was fixed at 64 and the unit size of the second Dense network was fixed at 32, changing the loss function to MSE, the recognition accuracy rate was 90.68%. After the loss function is changed to Softmax, the recognition accuracy rate is 91.56%.
Based on the comprehensive recognition accuracy and the time spent in training and learning the classification recognition model, the combination of the recognition model LSTM layer and the Dense network layer and its parameter unit selects the LSTM parameter combination of (128)-DENSE (64)-DENSE (32).
4. Results and Analysis
In this section, the data set collected in the experiment is used to train and test the model to verify the feasibility of the proposed model by targeted verification of the convergence in the model training process and the recognition accuracy for different groups of people. All experiments have achieved the control of the uHand2.0 manipulator palm, that is, the subject simply imagines gestures in the brain when the subject is at rest, and the manipulator palm performs the same gesture at the same time.
4.1. Verification of Convergence
After experimental comparison, the optimal model parameters of the model are obtained, LSTM(128)-DENSE (64)-DENSE (32). The training process using the proposed Dense LSTM and LSTM algorithms is shown in Figure 8. The red broken line indicates the change of the recognition accuracy rate of the Dense LSTM algorithm during the training process, and the blue broken line indicates the accurate recognition of the LSTM model rate of change. Apparently, in the process of 0 to 100 iterations, the recognition accuracy of the two models rapidly increased to approximately 85%. After 100 iterations, the recognition accuracy increased relatively slowly. After 300 iterations, the Dense LSTM algorithm was constant; it will grow slowly, but the LSTM algorithm does not grow. Finally, the recognition accuracy of the Dense LSTM algorithm reaches 91.56%, and the recognition accuracy of the LSTM algorithm is slightly lower than 90%. During the training of the model, the recognition accuracy of the Dense LSTM algorithm changes smoothly, while the LSTM has significant jitters and several glitches, and the model is unstable. Therefore, the Dense LSTM algorithm proposed in this study presented an optimization effect.
4.2. Verification of Recognition Accuracy
As shown in Table 3, the average recognition rate and running time of the newly proposed Dense LSTM algorithm is compared with several neural networks commonly used in traditional deep learning BCI including Recurrent Neural Network (RNN), Convolutional Neural Networks (CNN) [32], and LSTM and feature extraction-classifier algorithms in machine learning including support vector machine (SVM) and Bayesian network. The results indicate that, under the same parameter settings, the Dense LSTM algorithm proposed in this study has a higher average recognition rate for all subjects, so it is more suitable for the classification of EEG signals in the imagery-based BCI. To a certain extent, it solves the LSTM network data in the imagery-based BCI. The nonconvergence problem in the classification and recognition model training process improves the practicality and generalization performance of the imagery-based BCI. All subjects used the same classification and recognition model, and the recognition accuracy rate was 91.56%. This somewhat resolves the problems relative to “BCI blindness” and “one person, one model” that are common in BCIs, making the application of imaginary BCIs more widespread.
The trained classification and recognition model is applied to the classification and recognition module of the imaginary BCI model, and 5 out of the 20 subjects are randomly selected for the gesture recognition cross-validation experiment in the imagination mode. Each gesture is done 20 times, and the experimental results are shown in Table 4, which represents the experimental results of five randomly selected subjects. Considering the experiment, 5 subjects made 100 gestures for each gesture.
The imaginary BCI data was randomly selected for the five age groups ranging 15∼20 years old, 20∼25 years old, 25∼30 years old, 30∼35 years old, and 35∼40 years old for classification. The test results are shown in Figure 9; the accuracy of gesture recognition for all ages reached more than 90.5%, of which the highest recognition accuracy for ages 25 to 30 years old reached 92.58%. The farther the remaining four age groups were from the age range of 25 to 30 years, the lower the recognition accuracy is. The experimental results show that age causes the recognition accuracy of the imaginary BCI to change, but the impact is not very significant. It is related to the attention control ability of subjects of different ages.
A total of 100 samples were randomly selected from the data of female and male subjects for classification and recognition. The experimental results are shown in Table 5. Table 5 presents the classification and recognition of the gestures of female and male subjects; among them, the overall recognition accuracy rate of female subjects is 96.8% and that of male subjects was 89.2%. The recognition accuracy of female subjects is slightly higher than that of male subjects, and the recognition accuracy rates of the five gestures are approximately the same.
In order to prove the feasibility of the proposed algorithm, we focused on one case study, taking the middle finger as an example. The number of subjects’ samples is increased on the basis of the original data set; however, the experimental process is similar to the previous one. We randomly selected 5 subjects from all collected samples, male and female sample data set, and different ages for recognition testing, compared with the previous random test results. The results are shown in Table 6. It can be seen from the results, focusing on one case study analysis and appropriate with the addition of subject samples, that the results are basically the same as the previous results.
Taking the middle finger as an example, the significance test is performed on random samples and different genders sample data. The experimental results are shown in Table 7. From the results, p(1) = 0.1381, keeping the null hypothesis, which means there is no significant difference between different subjects, and p(2) = 0.2117, the null hypothesis is also kept, and there is no significant difference between subjects of different genders.
Similarly, the significance test is performed on random samples and different age groups sample data; the results are as follows: p(1) = 0.5619, which means there is no significant difference between different subjects in different age groups; p(2) = 0.1161, which means the model is also applicable to subjects between different age groups; and p(3) = 0.5132, which means the interaction between age and different subjects is not obvious.
5. Conclusion
The model and the algorithm for the problem of the imaginary BCI were investigated in this study. The contributions of this study addressing this challenging problem are as follows: Model: a new imaginary BCI model was proposed. Model training and testing were conducted with its own data set, and a new hardware platform was built to verify the feasibility of the model. Algorithm: due to the low recognition accuracy of the traditional classification and recognition algorithms and the problem of “BCI blindness,” a Dense LSTM algorithm was proposed. This algorithm combines the LSTM network with the Dense network to improve the recognition accuracy and solve the problem of “One person, one model.”
Experimental results showed that the recognition accuracy of the proposed Dense LSTM algorithm was as high as 91.56%, which is significantly better than other algorithms and has sufficient generalization ability. Future work will focus on improving the recognition accuracy of the BCI under various interference environments to improve the practicability and effectiveness and be used in medical and rehabilitation fields in the future.
Data Availability
The data sets generated and analyzed during this current study are available from the corresponding author on reasonable request.
Conflicts of Interest
The authors declare no conflicts of interest with respect to the research, authorship, and/or publication of this article.
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant 61301250, Key Research and Development Project of Shanxi Province under Grant 201803D421035, Natural Science Foundation for Young Scientists of Shanxi Province under Grant 201901D211313, and Research Project Supported by Shanxi Scholarship Council of China under Grant HGKY2019080.