Abstract
The recognition in electroencephalogram (EEG) of action idea is to identify what action people want to do by EEG. The significance of this project is to help people who have trouble in movement. Their action ideas are identified by EEG, and then robot hands can assist them to complete the action. This paper, with comparative experiments, used OpenBCI to collect EEG action ideas during static action and dynamic action and used the EEG recognition model Conv1D-GRU to training and recognition action, respectively. The experimental result shows that the brain wave action idea is easier to recognize in static state. The accuracy of brain wave action idea recognition in dynamic state is only 72.27%, and the accuracy of brain wave action idea recognition in static state is 99.98%. The experimental result confirms that the action idea will be of great help to people with mobility difficulties.
1. Introduction
EGG action ideas are records of cerebrum activity when the human cerebrum sends out some action ideas. When the human cerebrum works, it will produce spontaneous electrophysiological activities, a big number of neurons produce postsynaptic potentials synchronously, and the amount of the potentials forms EEG. Therefore, the EEG records the changes of the waves during the brain activity, which is the general reflection of the electrophysiological activity of the brain nerve cells on the surface of the cerebral cortex or scalp [1].
At present, there are few researches on EEG action idea recognition. There is no suitable literature on the combination of EEG and action idea in CNKI. There is no result in IEEE Xplore digital library searching with EEG and action idea. There are 7 journals and 15 conferences in IEEE Xplore digital library searching with EEG and action, which mainly focus on EEG decoding [2–8]. Among them, literature 1 and literature 2 study the EEG of auditory behavior, literature 5 studies the effect of camera on motor cortex, literature 3 and literature 7 study EEG decoding of action, and literature 4 studies the significance of inhibition of EEG signals in some parts.
This paper focuses on the recognition of EEG action idea, which can fill the gap in the current research of EEG. EEG action idea recognition is a kind of noninvasive human-computer interface, which can be applied to people with limited movement. By acquiring the EEG of those people, we can identify what action they want to complete and then control the robot hand to complete the action. Therefore, EEG action idea recognition has very important application value.
The major contributions of our work are summarized as follows:(1)This paper provides a method for EEG experiment of action idea. It also gives the complete operation process and precautions about the acquisition of EEG signals by OpenBCI.(2)This paper presents a Conv1D-GRU model, which can directly process the original EEG signals and automatically extract the features of EEG signals. In addition, the model also has memory function and can process time series data.(3)The experimental results show that the Conv1D-GRU model can directly train and test the original EEG signals. It is proved that the application of action idea will be of great help to those who do not change their actions.
2. Related Work
EEG identification is feature extraction and categorization. However, the traditional classifiers, such as SVM (support vector machines), decision tree, and Bayesian network, rely on manual feature extraction, which makes it difficult to improve the accuracy of classification. Deep neural network can learn automatically the features directly from the original EEG signals [9].
The EEG signal is one-dimensional waveform data; it should use Conv1D to process the feature automatic extraction.
EEG data is a kind of time series data, which needs to use the feedback network in deep learning. The feedback network has RNN, LSTM, and GRU model, in which RNN is prone to gradient disappearance or gradient explosion. GRU is a variant of LSTM. GRU simplifies the structure of LSTM with fewer parameters. GRU is easy to train, and the overfitting problem is a little lighter than LSTM.
Based on the above EGG data and deep learning model characteristics, this paper designs an EEG recognition model Conv1D_GRU based on Conv1D and GRU, which can not only automatically extract the features of EEG signals but also memorize the previous information and apply the memory to calculate the current output.
The experiment equipment used in this paper is a 16-channel OpenBCI acquisition device, which collects the original EEG signals. At each time, the data of each channel is saved as a record, and all the data are separated by commas. According to the characteristics of the recorded data, it is easy to convert it into CSV format data.
Because CNN network has the advantage of automatic feature extraction, but CNN usually refers to 2D CNN, it is for the image classification. However, the EEG signal collected by OpenBCI is one-dimensional waveform data; therefore, this paper uses Conv1D to process the automatic feature extraction of the EEG signal [10, 11].
2.1. One-Dimensional Convolution Neural Network
Conv1D is a one-dimensional convolution operation, which mainly deals with time series data shown in following equation:where is the sequence output of the n frame of the sequence, N is the length of convolution kernel, f(m) is the m bit of convolution kernel, and g(n − m) is the n−m frame of sequence.
Conv1D requires the input data to be three-dimensional as follows:where Batch_ Size is the number of data samples in once training. Cin is the number of channels of input signal. Lin is the input width. The final output parameters are also three-dimensional as follows:where Batch_ Size is the number of data samples in once training. Cout defines the number of the convolution kernels used for scanning. Lout is the output width. The Lout calculation is shown in the following equation:where L is the representation of Lin, k is the size of each convolution kernel, p is the number of layers to supplement 0, and s is convolution step. Generally, p is 0, and then Lout is calculated as the following equation:
In EEG, Lin is time long of input; Lout is the time long of output, and the Conv1D of EEG is shown in Figure 1.
In Figure 1, the input of the convolution has many characteristics. Therefore, it is necessary to select different operators to slide through the time series in turn. After convolution of each convolution operator with the time series, the length T′ of the generated sequence is shown in the following equation:where T is the input width of time, k is the size of each convolution kernel, and s is convolution step.
2.2. Gated Recurrent Unit Neural Network
The current popular neural networks are mainly feed-forward network, memory network, and graph network.
In the feed-forward network, each neuron belongs to a different layer. Each layer of neurons can get the signal from the earlier layer of neurons and output the signal to the next layer [12]. There is no feedback in the whole network; it is mainly used in the field of image. The common models are multilayer perceptron and convolution neural network.
In the feedback network, the neurons can receive not only the information of other neurons, but also their own historical information. Compared with the feed-forward network, the neurons in the feedback network have memory function and have different states at different times. The output is not only related to the current input and the network weight but also related to the previous input. It is mainly used in the field of time series data and text data. The common time recurrent neural networks are the recurrent neural network (RNN), the long short-term memory (LSTM), and GRU [13–16].
The inputs of feed-forward network and memory network can be expressed as vectors or vector sequences, which are difficult to deal with the data of graph structure, so graph network appears. The common algorithms are GCN, GNT, and MPNN [17–19].
The EEG signal is a sequence data. Therefore, the feedback network is used in this paper. RNN is prone to the problem of the gradient disappearance or the gradient explosion; therefore, LSTM is derived, and GRU is a variant of LSTM. In many cases, GRU and LSTM have the same excellent effect [20, 21], but GRU simplifies the structure of LSTM; it has fewer parameters. That is, the GRU is relatively easy to train and has less problem of over fitting.
Figure 2 shows the network structure of the GRU. The whole GRU network is such a network that continuously circulates at time t. In order to better describe the relationship between the front and the rear moments, the two GRU time states are drawn in Figure 2. On the left is the network unit at time t − 1 and on the right is the network unit at time t. The output of the t − 1 time network unit is used as the input of the t time network unit, and the GRU network keeps its own historical information. The gating cycle unit will not clear the previous information with time; it will retain the relevant information and transfer it to the next unit; therefore, it uses all the information to avoid the gradient vanishing problem.
As can be seen from Figure 2, there are two gates in the GRU network, update gate and reset gate, which are zt and rt.
2.2.1. Update Door
At the moment of t, the calculation formula of the update gate zt is shown in the following equation:
In (7), Xt represents the input vector of time step T. W(z) and U(z) are weight matrices. ht-1 is the information of the t − 1 time step. The update gate adds Xt and ht−1 when they are linearly transformed. Then, the added result is sent into sigmoid activation function to compress to 0–1. The update gate can decide to copy all the information in the past to reduce the risk of the gradient disappearance. The value of the update gate determines how much state information is brought from the previous time to the current time.
2.2.2. Reset GATE
The reset gate rt controls how much information of the previous state is written to the current candidate set ht. The calculation formula of rt is shown in the following equation:where the calculation of rt is similar to that of zt. The reset gate adds Xt and ht-1 when they are linearly transformed. Then, the added result is sent into sigmoid activation function.
2.2.3. New Information at the Moment
The new memory content h′t will use the information about the past stored by the reset gate. The calculation expression of h′t is shown in the following equation:where ⊙ represents the product of two vectors and the operation ⊙ determines the previous information to be retained or forgotten. Because rt is a vector composed of 0 to 1, rt product U(h′) h t − 1, rt will control the size of the door opening. For example, the value of rt element corresponding to the element in h(t−1) is 0; it means that the element information of h(t − 1) would be completely forgotten.
Xt adds the result of ⊙, and the added result is put into the hyperbolic tangent activation function (tanh). The tanh compresses the value between −1 and 1, so as to adjust the output of the neural network.
2.2.4. The Final Memory of the Current Time Step
The final memory ht of the current time step will be transferred to the t+1 time step unit. The calculation formula of ht is shown in the following equation:
is the information retained in the final memory in the previous time step. is the new information at the current time. ht is the final memory information, and it is also the output of the final gating cycle unit. GRU preserves important features with various gate functions, which ensures that they will not be lost during long term propagation.
3. Design of the Classification Model of EEG Emotion
EEG signal is one-dimensional waveform data and has time dynamic. It is a time series data collected at different time points. The waveform changes with time, and the record of the former time point is related to the record of the later time point. Convolution neural network has the function of automatic feature extraction, and GRU network has the advantage of time series processing. Therefore, this paper designs the EEG emotion classification model Conv1D_GRU based on Conv1D and GRU, as shown in Figure 3.
In Conv1D_GRU, the input data are five kinds of one-dimensional 16 channel EEG signals, the input data features can be fully extracted after two Conv1D convolution operations. The parameters in two Conv1D are listed in Table 1. In order to avoid gradient disappearing, BN (batch normalization) layer is added after two Conv1D. BN suppresses the gradient vanishing. After the data passes through BN layer, sigmoid and tanh excitation functions can be used in deep network. At the same time, BN has regularization function, which makes the control overfitting not depend on the dropout.
The GRU has three layers; the parameters in the GRU are listed in Table 2. The first layer and the second layer return all the hidden states of the last layer, and the third layer returns the hidden state of the last step of the last layer. At last, the dense layer is executed to send the GRU output to the fully connected neural network, and the multiclassification is carried out with the softmax excitation function.
4. Experiment
Individuals with life activities will produce bioelectricity with different energy. EEG is a method of recording brain activity using electrophysiological indicators; it is the general reflection of the electrophysiological activities of brain nerve cells in the cerebral cortex or scalp surface. The human brain is carrying out life activities all the time. Different activities will emit different frequencies of EEG.
4.1. Introduction of EEG Acquisition Experimental Instrument
At present, the most convenient and the cost-effective equipment for the EEG acquisition is brain electrode cap. There are three kinds of electrode derivatives: 8 channels, 16 channels, and 32 channels. This experiment is based on a 16-channel brain electrode cap.
A complete set of the EEG acquisition device includes brain electrode cap, OpenBCI board and Wi-Fi shield, dry battery, and brain computer interface Open BCI_GUI (open-source brain-computer interfaces graphical user interfaces).
The electrode should be placed according to the electrode point. The placement of 16 electrodes in this experiment is shown in Figure 4.
In Figure 4, the number represents the position of the electrodes on the human brain, where positions 1 and 2 are on the forehead of the human brain, which can be placed in the bangs of hair, and positions 3 and 4 are on the top of the human brain, and the remaining electrodes are scattered in the back of the human brain. The connection of Wi-Fi shield, OpenBCI motherboard, and OpenBCI expansion board is shown in Figure 5.
In Figure 5, the red board is Wi-Fi shield, which is brain computer interface. Its function is to communicate with OpenBCI_ GUI on the computer by router. The bottom white board is the OpenBCI motherboard, and the top white board is the OpenBCI expansion board. They are used to collect channel information of EEG cap.
The two OpenBCI boards need to connect the sbr2 pin and then connect it to the ear clip. The bias pin is connected to another ear clip. When collecting signals, the ear clip must be clamped to the ear, one on each ear, regardless of the left and right. The 16 electrode channels on the electrode cap are connected according to the pins shown in Figure 5. The main OpenBCI is connected with 1–8 electrodes, and the extended OpenBCI is connected with 9–16 electrodes.
4.2. EEG Information Acquisition
4.2.1. Points for Attention before Acquisition
(i)The supply of Wi-Fi shield power should use 4v-6v dry batteries, because dry battery can reduce electromagnetic interference. The laptop should be powered by battery, and the router should only be used for acquisition.(ii)The battery should be replaced when the signal is unstable.(iii)Experimental personnel needs to keep their skin dry and clean, because skin grease will destroy the detection function of the electrode, and the electrode should directly contact the skin.(iv)In the process of acquisition, the human body needs to keep stable, reduce unnecessary movements, and especially prohibit violent shaking.
4.2.2. EEG Acquisition Experiment
Wi-Fi shield and OpenBCI_GUI are connected to the same router, then run the OpenBCI_HUB software on the computer to transmit data. OpenBCI_HUB is a bridge software, which can guarantee Wi-Fi shield and OpenBCI_ GUI to connect and communicate.
In OpenBCI_ GUI, select “search Wi-Fi shield.” After the connection is successful, an interface will pop up to display the EEG waveform collected by each electrode channel. In the experiment, OpenBCI_GUI shows the waveform variation of 16 channels with time and the real-time waveform amplitude between 0 and 60 Hz.
There are two groups of experiments. One is to make a certain posture of the hand, but the hand does not move during the collection process; the other group is the hand that has been doing a dynamic action.
Figures 6–15 show the EEG of five action idea of the same person. The intensity of each action idea is different. The EEG of static action idea and dynamic action idea were collected, respectively. When doing the static action idea, put your hand on the object to make a posture, but it does not change. It depends on your brain to imagine what you are doing. When doing dynamic action idea, the hand has been doing one kind action, such as push, pull, knock, grasp, and shear.
As can be seen in Figures 6 and 7, the signal voltage of channel 1, channel 4, channel 8, and channel 16 in Figure 7 is far more than that in Figure 6. In Figure 6, the voltage of channel 12 and channel 15 is far more than Figure 7. It indicates that the grasp action idea signal is strong in dynamic state.
As can be seen in Figures 8 and 9, the signal voltage in Figure 9 is higher than that in Figure 8. It indicates that the push action idea signal is strong in dynamic state.
In Figures 10 and 11, the signal voltage difference in the corresponding channel is very small. It indicates that the knock action idea signal is only slight difference between the two states.
As can be seen in Figures 12 and 13, the signal voltage of channel 7, channel 8, channel 10, channel 12, channel 15, and channel 16 in Figure 13 is far more than that in Figure 12. In Figure 12, only the voltage of channel 3 is far more than that in Figure 13. It indicates that the pull action idea signal is strong in dynamic state.
In Figures 14 and 15, the signal voltage of channel 3 in Figure 15 is far more than that in Figure 14. But the signal voltage of channel 4 in Figure 14 is far more than that in Figure 15. It indicates that the knock action idea signal is only slight difference between the two states.
All experiments in this paper are carried out in the same environment with the same group of equipment. Furthermore, in OpenBCI_GUI, the cutoff frequency is 50 Hz to filter noise greater than 50 Hz.
From Figures 6 to 15, we can see that the power of action idea in dynamic state is higher than that in static state. This shows that the brain power is greater when the action is taken. At dynamic state, the changes of EEG are more obvious, and the fluctuation of each channel is greater than static state.
4.2.3. EEG Data Processing
When the detection is completed, computer saves the detected data to the data of the software with a TXT file. We need to do five types of action idea EEG experiments; each time period only detects one action idea and marks start and end time. That is, there are two TXT files, one TXT file is action idea in static state, and the other TXT file is action idea in dynamic state. There are too many records in each TXT. Due to the limited performance of personal computers, we need to select part of records to train and test. The data processing method is as follows:(1)Select 1024 records of action idea randomly according to the time period, because only one movement idea is collected in a period of time.(2)There are a total of 5120 records for training, because there are five kinds of action idea in one TXT.(3)Select 256 records of action idea randomly according to the time period in the rest of the records. There are 1280 records were selected for testing.(4)The training records and the testing records are labeled with action idea types.(5)Disrupt the order of all records to prevent machine learning from over fitting the following samples.
5. Model Training and Prediction
5.1. Model Training and Prediction Results
The training process of data in the Conv1D_GRU model is shown in Figure 16.
In Figure 16, the training records are directly divided into Bach and then sent to the Conv1D network to automatically extract features. Then, the extracted features are sent to GRU network to find the time relationship between features. Finally, the results are sent to the full connection layer to get the diagnosis results. The error between the diagnosis result and the real label is calculated through the loss function, and the Adam algorithm is used to improve the loss accuracy. In the process of training, if the loss is no longer reduced, let the training stop, because it means that the model has been trained.
After 50 rounds of training, the training results are shown in Figures 17–20.
The prediction accuracy of the test samples in the model is 99.98% in static state and 72.27% in dynamic state, and the code screenshot is shown in Figures 21 and 22.
5.2. Experiment Analysis
It can be seen from the comparison between Figures 17 and 18 that the loss rate of action idea EEG in static state is lower than that in dynamic state. When the network converges, there is no fluctuation of action idea EEG in static state, which indicates that the model has good robustness.
Comparing Figures 19 and 20, it can be seen that the accuracy of action idea EEG in static state is higher than that in dynamic state.
Comparing with Figures 21 and 22, it can be seen that the loss rate of action idea EEG prediction in static state is much smaller than that in dynamic state. In other words, the prediction accuracy rate of the model in static state is much higher than that in dynamic state.
The experiment results show that Conv1D_ GRU model can recognize the action idea EEG and has higher recognition rate when people keep their hands motionless.
6. Conclusion
Five kinds of hand action EEG were collected by OpenBCI. The hand state can be divided into two situations: one is that the hand keeps still; the other is that the hand has been doing some kind of action. EEG data is one-dimensional and time-series. In order to extract the characteristics of EEG more accurately, we need to use deep learning algorithm. According to the characteristics of EEG signal, this paper proposes a Conv1D_GRU model. Conv1D is used to extract the characteristics of EEG signals. GRU analyzes the timing characteristics. The original EEG data were collected and sent to Conv1D_GRU model to train; the results showed that the hand should be kept still when doing the experiment of action idea EEG.
The advantage of this work is to find the implementation method of action idea EEG. When collecting action idea EEG experiments, people should keep all parts of the body motionless and imagine only with the brain. In addition, a deep learning model for processing EEG is proposed.
The disadvantage of this work is that the EEG cap used to collect data is very difficult to wear, and it takes a long time to collect data.
At present, there are few researches and applications on the recognition of action idea. The next step is to deeply study the characteristics and applications of action idea EEG.
Data Availability
The data and the model used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The author declares no conflicts of interest.
Acknowledgments
This work was supported in part by the Characteristic Innovation of Guangdong Provincial Department of Education under Grants 2018KTSCX343 and 2020KTSCX401.