Abstract
This work investigates the use of mixed-norm regularization for sensor selection in event-related potential (ERP) based brain-computer interfaces (BCI). The classification problem is cast as a discriminative optimization framework where sensor selection is induced through the use of mixed-norms. This framework is extended to the multitask learning situation where several similar classification tasks related to different subjects are learned simultaneously. In this case, multitask learning helps in leveraging data scarcity issue yielding to more robust classifiers. For this purpose, we have introduced a regularizer that induces both sensor selection and classifier similarities. The different regularization approaches are compared on three ERP datasets showing the interest of mixed-norm regularization in terms of sensor selection. The multitask approaches are evaluated when a small number of learning examples are available yielding to significant performance improvements especially for subjects performing poorly.
1. Introduction
Brain computer interfaces (BCI) are systems that help disabled people communicate with their environment through the use of brain signals [1]. At the present time, one of the most prominent BCI is based on electroencephalography (EEG) because of its low-cost, portability, and its noninvasiveness. Among EEG based BCI, a paradigm of interest is the one based on event-related potentials (ERP) which are responses of the brain to some external stimuli. In this context, the innermost part of a BCI is the pattern recognition stage which has to correctly recognize presence of these ERPs. However, EEG signals are blurred due to the diffusion of the skull and the skin [2]. Furthermore, EEG recordings are highly contaminated by noise of biological, instrumental, and environmental origins. For addressing these issues, advanced signal processing and machine learning techniques have been employed to learn ERP patterns from training EEG signals leading to robust systems able to recognize the presence of these events [3–8]. Note that while some ERPs are used for generating BCI commands, some others can be used for improving BCI efficiency. Indeed, recent studies have also tried to develop algorithms for automated recognition of error-related potentials [9]. These potentials are responses elicited when a subject commits an error in a BCI task or observes an error [10, 11] and thus they can help in correcting errors or in providing feedbacks to BCI users.
In this context of automated recognition of event-related potentials for BCI systems, reducing the number of EEG sensors is of primary importance since it reduces the implementation cost of the BCI by minimizing the number of EEG sensor and speeding up experimental setup and calibration time. For this purpose, some studies have proposed to choose relevant sensors according to prior knowledge of brain functions. For instance, sensors located above the motor cortex region are preferred for motor imagery tasks, while for visual event-related potential (ERP), sensors located on the visual cortex are favored [12]. Recent works have focused on automatic sensor selection adapted to the specificity of a subject [4, 13–17]. For instance, Rakotomamonjy and Guigue [18] performed a recursive backward sensor selection using cross-validation classification performances as an elimination criterion. Another approach for exploring subset sensors has been proposed by [15]; it consists in using a genetic algorithm for sensor selection coupled with artificial neural networks for prediction. Those methods have been proven efficient but computationally demanding. A quicker way is to estimate the relevance of the sensors in terms of signal to noise ratio (SNR) [4] and to keep the most relevant ones. Note that this approach does not optimize a discrimination criterion, although the final aim is a classification task. Recently, van Gerven et al. [19] proposed a graceful approach for embedding sensor selection into a discriminative framework. They performed sensor selection and learn a decision function by solving a unique optimization problem. In their framework, a logistic regression classifier is learned and the group-lasso regularization, also known as mixed-norm, is used to promote sensor selection. They have also investigated the use of this groupwise regularization for frequency band selection and their applications to transfer learning. The same idea has been explored by Tomioka and Müller [20] which also considered groupwise regularization for classifying EEG signals. In this work, we go beyond these studies by providing an in-depth study of the use of mixed-norms for sensor selection in a single subject setting and by discussing the utility of mixed-norms when learning decision functions for multiple subjects simultaneously.
Our first contribution addresses the problem of robust sensor selection embedded into a discriminative framework. We broaden the analysis of van Gerven et al. [19] by considering regularizers whose forms are mixed-norms, with , as well as adaptive mixed-norms, so as to promote sparsity among group of features or sensors. In addition to providing a sparse and accurate sensor selection, mixed-norm regularization has several advantages. First, sensor selection is cast into an elegant discriminative framework, using for instance a large margin paradigm, which does not require any additional hyperparameter to be optimized. Secondly, since sensor selection is jointly learned with the classifier by optimizing an “all-in-one” problem, selected sensors are directed to the goal of discriminating relevant EEG patterns. Hence, mixed-norm regularization helps locating sensors which are relevant for an optimal classification performance.
A common drawback of all the aforementioned sensor selection techniques is that selected set of sensors may vary, more or less substantially, from subject to subject. This variability is due partly to subject specific differences and partly to acquisition noise and limited number of training examples. In such a case, selecting a robust subset of sensors may become a complex problem. Addressing this issue is the point of our second contribution. We propose a multitask learning (MTL) framework that helps in learning robust classifiers able to cope with the scarcity of learning examples. MTL is one way of achieving inductive transfer between tasks. The goal of inductive transfer is to leverage additional sources of information to improve the performance of learning on the current task. The main hypothesis underlying MTL is that tasks are related in some ways. In most cases, this relatedness is translated into a prior knowledge, for example, a regularization term, that a machine learning algorithm can take advantage of. For instance, regularization terms may promote similarity between all the tasks [21] or enforce classifier parameters to lie in a low dimensional linear subspace [22] or to jointly select the relevant features [23]. MTL has been proven efficient for motor imagery in [24] where several classifiers were learned simultaneously from several BCI subject datasets. Our second contribution is thus focused on the problem of performing sensor selection and learning robust classifiers through the use of an MTL mixed-norm regularization framework. We propose a novel regularizer promoting sensor selection and similarity between classifiers. By doing so, our goal is then to yield sensor selection and robust classifiers that are able to overcome the data scarcity problem by sharing information between the different classifiers to be learned.
The paper is organized as follows. The first part of the paper presents the discriminative framework and the different regularization terms we have considered for channel selection and multitask learning. The second part is devoted to the description of the datasets, the preprocessing steps applied to each of them, and the results achieved in terms of performances and sensor selection. In order to promote reproducible research, the code needed for generating the results in this paper is available on the author’s website (URL: http://remi.flamary.com/soft/soft-gsvm.html.).
2. Learning Framework
In this section, we introduce our mixed-norm regularization framework that can be used to perform sensor selection in a single task or in a transfer learning setting.
2.1. Channel Selection in a Single Task Learning Setting
Typically in BCI problems, one wants to learn a classifier that is able to predict the class of some EEG trials, from a set of learning examples. We denoted as the learning set such that is a trial and is its corresponding class, usually related to the absence or presence of an event-related potential. In most cases, a trial is extracted from a multidimensional signal and thus is characterized by features for each of the sensors, leading to a dimensionality . Our aim is to learn, for a single subject, a linear classifier that will predict the class of a trial , by looking at the sign of the function defined as with the normal vector to the separating hyperplane and a bias term. Parameters of this function are learned by solving the optimization problem: where is a loss function that measures the discrepancy between actual and predicted labels, is a regularization term that expresses some prior knowledge about the learning problem, and is a parameter that balances both terms. In this work, we choose to be the squared hinge loss , thus promoting a large margin classifier.
2.1.1. Regularization Terms
We now discuss different regularization terms that may be used for single task learning along with their significances in terms of channel selection.
Norm. The first regularization term that comes to mind is the standard squared norm regularization: where is the Euclidean norm. This is the common regularization term used for SVMs and it will be considered in our experiments as the baseline approach. Intuitively, this regularizer tends to downweigh the amplitude of each component of leading to a better control of the margin width of our large-margin classifier and thus it helps in reducing overfitting.
Norm. When only few of the features are discriminative for a classification task, a common way to select the relevant ones is to use an norm of the form as a regularizer [25]. Owing to its mathematical properties (nondifferentiability at ), unlike the norm, this regularization term promotes sparsity, which means that at optimality of problem (2), some components of are exactly . In a Bayesian framework, the norm is related to the use of prior on that forces its component to vanish [19]. This is typically obtained by means of Laplacian prior over the weight. However, norm ignores the structure of the features (which may be grouped by sensors) since each component of is considered independently to the others. As such, this norm precludes grouped feature selection and allows only for feature selection.
Mixed-Norm. A way to take into account the fact that features are structured is to use a mixed-norm that will group them and regularize them together. Here, we consider mixed-norm of the form with and being a partition of the set . Intuitively, this mixed-norm can be interpreted as an norm applied to the vector containing the norm of each group of features. It promotes sparsity on each norm and consequently on the components as well. For our BCI problem, a natural choice for is to group the features by sensors yielding thus to groups (one per sensor) of features as reported in Figure 1. Note that unlike the norm as used by van Gerven et al. [19] and Tomioka and Müller [20], the use of an inner norm leads to more flexibility as it spans from the (equivalent to the -norm and leading thus to unstructured feature selection) to the which strongly ties together the components of a group. Examples of the use of norm and mixed-norm regularizations in other biomedical contexts can be found for instance in [26, 27].
(a)
(b)
Adaptive . The and norms described above are well known to lead to grouped feature selection. However, they are also known to lead to poor statistical properties (at least when used with a square loss function) [28]. For instance, they are known to have consistency issue in the sense that, even with an arbitrarily large number of training examples, these norms may be unable to select the true subset of features. In practice, this means that when used in (2), the optimal weight vector will tend to overestimate the number of relevant sensors. These issues can be addressed by considering an adaptive mixed-norm of the form [28, 29] where the weights are selected so as to enhance the sparsity pattern of . In our experiments, we obtain them by first solving the problem with , which outputs an optimal parameter , and by finally defining . Then, solving the weighted problem yields an optimal solution with increased sparsity pattern compared to since the augments the penalization of groups with norm smaller than .
2.1.2. Algorithms
Let us now discuss how problem (2) is solved when one of these regularizers is in play.
Using the norm regularization makes the problem differentiable. Hence a first- or second-order descent based algorithm can be considered [30].
Because the other regularizers are not differentiable, we have deployed an algorithm [31] tailored for minimizing objective function of the form with a smooth and differentiable convex function with Lipschitz constant and a continuous and convex nondifferentiable function having a simple proximal operator, that is, a closed-form or an easy-to-compute solution of the problem Such an algorithm, known as forward-backward splitting [31], is simply based on the following iterative approach: with being a stepsize in the gradient descent. This algorithm can be easily derived by considering, instead of directly minimizing , an iterative scheme which at each iteration replaces with a quadratic approximation of in the neighborhood of . Hence, is the minimizer of whose closed-form is given in (8). This algorithm is known to converge towards a minimizer of under some weak conditions on the stepsize [31], which is satisfied by choosing for instance . We can note that the algorithm defined in (8) has the same flavor as a projected gradient algorithm which first takes a gradient step and then “projects” back the solution owing to the proximal operator. More details can also be found in [32].
For our problem (2), we choose to be the squared hinge loss and the nonsmooth regularizer. The square hinge loss is indeed gradient Lipschitz with a constant being . Proof of this statement is available in Appendix A. Proximal operators of the and the regularization term can be easily shown to be the soft-thresholding and the block-soft thresholding operator [25]. The general norm does not admit a closed-form solution, but its proximal operator can be simply computed by means of an iterative algorithm [23]. More details on these proximal operators are also available in Appendix C.
2.2. Channel Selection and Transfer Learning in Multiple Task Setting
We now address the problem of channel selection in cases where training examples for several subjects are at our disposal. We have claimed that in such a situation, it would be beneficial to learn the decision functions related to all subjects simultaneously, while inducing selected channels to be alike for all subjects, as well as inducing decision function parameters to be related in some sense. These two hypotheses make reasonable sense since brain regions related to the appearance of a given ERP are expected to be somewhat location-invariant across subjects. For solving this problem, we apply a machine learning paradigm, known as multitask learning, where in our case, each task is related to the decision function of a given subject and where the regularizer should reflect the above-described prior knowledge on the problem. Given subjects, the resulting optimization problem boils down to be with being the training examples related to each task, being the classifier parameters for task , and being a matrix concatenating all vectors . Note that the multitask learning framework applied to single EEG trial classification has already been investigated by van Gerven et al. [19]. The main contribution we bring compared to their works is the use of regularizer that explicitly induces all subject classifiers to be similar to an average one, in addition to a regularizer that enforces selected channels to be the same for all subjects. The intuition behind this point is that we believe that since the classification tasks we are dealing with are similar for all subjects and all related to the same BCI paradigm, selected channels and classifier parameters should not differ that much from subject to subject. We also think that inducing task parameters to be similar may be more important than enforcing selected channels to be similar when the number of training examples is small since it helps in reducing overfitting. For this purpose, we have proposed a novel regularization term of the form where is the average classifier across tasks and contains nonoverlapping groups of components from matrix . The first term in (11) is a mixed-norm term that promotes group regularization. In this work, we defined groups in based on the sensors, which means that all the features across subject related to a given sensor are in the same group , leading to groups of feature, as depicted in Figure 1. The second term is a similarity promoting term as introduced in Evgeniou and Pontil [21]. It can be interpreted as a term enforcing the minimization of the classifier’s parameter variance. In other words, it promotes classifiers to be similar to the average one, and it helps improving performances when the number of learning examples for each task is limited, by reducing overfitting. Note that and , respectively, control the sparsity induced by the first term and the similarity induced by the second one. Hence, when setting , the regularizer given in (11) boils down to be similar to the one used by van Gerven et al. [19]. Note that in practice and are selected by means of a nested cross-validation which aims at classification accuracy. Thus, it may occur that classifier similarity is preferred over sensor selection leading to robust classifiers which still use most of the sensors.
Similar to the single task optimization framework given in (2), the objective function for problem (10) can be expressed as a sum of gradient Lipschitz continuous term and a nondifferentiable term having a closed-form proximal operator (see Appendix B). Hence, we have again considered a forward-backward splitting algorithm whose iterates are given in (8).
3. Numerical Experiments
We now present how these novel approaches perform on different BCI problems. Before delving into the details of the results, we introduce the simulated and real datasets.
3.1. Experimental Data
We have first evaluated the proposed approaches on a simple simulated P300 dataset generated as follows. A P300 wave is extracted using the grand average of a single subject data from the EPFL dataset described in the following. We generate simulated examples with 8 discriminative channels containing the P300 out of 16 channels for positive examples. A Gaussian noise of standard deviation is added to all signals making the dataset more realistic. of these examples have been used for training.
The first real P300 dataset we used is the EPFL dataset, based on eight subjects performing P300 related tasks [33]. The subjects were asked to focus on one of the 3 2 = 6 images on the screen while one of the images is flashed at random. The EEG signals were acquired from 32 channels, sampled at Hz, and 4 recording sessions per subject have been realized. Signals are preprocessed exactly according to the steps described in [33]: a Hz bandpass Butterworth filter of order 3 is applied to all signals followed by a downsampling. Hence, for each trial (training example), we have 8 time-sample features per channel corresponding to a 1000 ms time-window after stimulus, which leads to features for all channels (32 8 = 256 features). Overall, the training set of a given subject is composed of about 3000 trials.
Another P300 dataset, recorded by the Neuroimaging Laboratory of Universidad Autónoma Metropolitana (UAM, Mexico) [34], has also been utilized. The data have been obtained from 30 subjects performing P300 spelling tasks on a 6 6 virtual keyboard. Signals are recorded over 10 channels leading thus to a very challenging dataset for sensor selection, as there are just few sensors left to select. For this dataset, we only use the first 3 sessions in order to have the same number of trials for all subjects (4000 samples). The EEG signals have been preprocessed according to the following steps: a Hz Chebyshev bandpass filter of order 5 is first applied followed by a decimation, resulting in a poststimulus time-window of 31 samples per channels. Hence, each trial is composed of (10 31) features.
We have also studied the effectiveness of our methods on an error-related potential (ErrP) dataset that has been recorded in the GIPSA Lab. The subjects were asked to memorize the position of 2 to 9 digits and to remind the position of one of these digits; operation has been repeated times for each subject. The signal following the visualization of the result (correct/error on the memorized position) was recorded from 31 electrodes and sampled at Hz. Similar to Jrad et al. [17], a Hz Butterworth filter of order 4 and a downsampling has been applied to all channel signals. Finally, a time window of 1000 ms is considered as a trial (training example) with a dimensionality of .
3.2. Evaluation Criterion, Methods, and Experimental Protocol
We have compared several regularizers that induce feature/channel selection embedded in the learning algorithm, in a single subject learning setting as defined in (2). The performance measure commonly used in BCI competitions [3] is the area under the Roc curve (AUC). This measure is an estimate of the probability for a positive class to have a higher score than a negative class. It makes particularly sense to use AUC when evaluating a P300 speller as the letter in the keyboard is usually chosen by comparing score returned by the classifier for every column or line. In addition, AUC does not depend on the proportion of positive/negative examples in the data which makes it more robust than classification error rate. Our baseline algorithm is an SVM, which uses an regularizer and thus does not perform any selection. Using an regularizer yields a classifier which embeds feature selection, denoted as SVM-1 in the sequel. Three mixed-norm regularizers inducing sensor selection have also been considered: an denoted as GSVM-2, and referred as GSVM-q, with being selected in the set by a nested cross-validation stage, and adaptive norm, with denoted as GSVM-a.
For the multitask learning setting, two MTL methods were compared to two baseline approaches which use all features, namely, a method that treats each tasks separately by learning one SVM per task (SVM) and a method denoted as SVM-Full, which on the contrary learns a unique SVM from all subject datasets. The two MTL methods are, respectively, a MTL as described in (10), denoted as MGSVM-2s and the same MTL but without similarity promoting regularization term, which actually means that we set , indicated as MGSVM-2. For these approaches, performances are evaluated as the average AUC of the decision functions over all the subjects.
The experimental setup is described in the following. For each subject, the dataset is randomly split into a training set of trials and a test set containing the rest of the trials. The regularization parameter has been selected from a log-spaced grid () according to a nested -fold cross-validation step on the training set. When necessary, the selection of is also included in this CV procedure. Finally, the selected value of is used to learn a classifier on the training examples and performances are evaluated on the independent test set. We run this procedure 10 times for every subject and report average performances. A Wilcoxon signed-rank test, which takes ties into account, is used to evaluate the statistical difference of the mean performances of all methods compared to the baseline SVM. We believe that such a test is more appropriate for comparing methods than merely looking at the standard deviation due to the high intersubject variability in BCI problems.
3.3. Results and Discussions
We now present the results we achieved on the above-described datasets.
3.3.1. Simulated Dataset
Average (over 10 runs) performance of the different regularizers on the simulated dataset is reported in Table 1 through AUC, sensor selection rate, and -measure. This latter criterion measures the relevance of the selected channels compared to the true relevant ones. F-measure is formally defined as where and are, respectively, the set of selected channels and true relevant channels and here denotes the cardinality of a set. Note that if the selected channels are all the relevant ones, then the -measure is equal to one. Most of the approaches provide similar AUC performances. We can although highlight that group-regularization approaches (GSVM-2, GSVM-p, GSVM-a) drastically reduce the number of selected channels since only and of the sensors are selected. A clear advantage goes to the adaptive regularization that is both sparser and is more capable of retrieving the true relevant channels.
3.3.2. P300 Datasets
Results for these datasets are reported in Table 2. For the EPFL dataset, all methods achieve performances that are not statistically different. However, we note that GSVM-2 leads to sensor selection (80% of sensor selected) while GSVM-a yields to classifiers that, on average, use of the sensors at the cost of a slight loss in performances (1.5% AUC).
Results for the UAM dataset follow the same trend in terms of sensor selection but we also observe that the mixed-norm regularizers yield to increased performances. GSVM-2 performs statistically better than SVM although most of the sensors (9 out of 10) have been kept in the model. This shows that even if few channels have been removed, the group-regularization improves performances by bringing sensor prior knowledge to the problem. We also notice that GSVM-a performance is statistically equivalent to the baseline SVM one while using only half of the sensors and GSVM-p consistently gives similar results to GSVM-2.
To summarize, concerning the performances of the different mixed-norm regularization, we outline that on one hand, GSVM-2 is at worst equivalent to the baseline SVM while achieving sensor selection and on the other hand GSVM-a yields to a substantial channel selection at the expense of a slight loss of performances.
A visualization of the electrodes selected by GSVM-a can be seen in Figure 2 for the EPFL dataset and in Figure 3 for the UAM dataset. Interestingly, we observe that for the EPFL dataset, the selected channels are highly dependent on the subject. The most recurring ones are the following: FC1 C3 T7 CP5 P3 PO3 PO4 Pz and the electrodes located above visual cortex O1, Oz, and O2. We see sensors from the occipital area that are known to be relevant [12] for P300 recognition, but sensors such as T7 and C3, from other brain regions, are also frequently selected. These results are however consistent with those presented in the recent literature [4, 18].
The UAM dataset uses only 10 electrodes that are already known to perform well in P300 recognition problem, but we can see from Figure 3 that the adaptive mixed-norm regularizer further selects some sensors that are essentially located in the occipital region. Note that despite the good average performances reported in Table 2, some subjects in this dataset achieve very poor performances, of about 50% of AUC, regardless of the considered method. Selected channels for one of these subjects (Subject 25) are depicted in Figure 3 and, interestingly, they strongly differ from those of other subjects providing rationales for the poor AUC.
We have also investigated the impact of sparsity on the overall performance of the classifiers. To this aim, we have plotted the average performance of the different classifiers as a function of the number of selected sensors. These plots are depicted in Figure 4 for the EPFL dataset and on Figure 5 for the UAM dataset. For both datasets, GSVM-a frequently achieves a better AUC for a given level of sparsity. For most of the subjects, GSVM-a performs as well as SVM but using far less sensors. A rationale may be that in addition to selecting the relevant sensors, GSVM-a may provide a better estimation of the classifier parameters leading to better performances for a fixed number of sensors. As a summary, we suggest thus the use of an adaptive mixed-norm regularizer instead of an mixed-norm as in van Gerven et al. [19] when sparsity and channel selection are of primary importance.
3.3.3. ErrP Dataset
The ErrP dataset differs from the others as its number of examples is small (72 examples per subject). The same experimental protocol as above has been used for evaluating the methods but only 57 examples out of 72 have been retained for validation/training. Classification performances are reported on Table 2. For this dataset, the best performance is achieved by GSVM-2 but the Wilcoxon test shows that all methods are actually statistically equivalent. Interestingly, many channels of this dataset seem to be irrelevant for the classification task. Indeed, GSVM-2 selects only 30% of them while GSVM-a uses only 7% of the channels at the cost of 10% AUC loss. We believe that this loss is essentially caused by the aggressive regularization of GSVM-a and the difficulty to select the regularization parameter using only a subset of the 57 training examples. Channels selected by GSVM-2 can be visualized on Figure 6. Despite the high variance in terms of selected sensors, probably due to the small number of examples, sensors in the central area seem to be the most selected one, which is consistent with previous results in ErrP [35].
3.3.4. Multitask Learning
We now evaluate the impact of the approach we proposed in (10) and (11) on the P300 datasets. We expect that since multitask learning allows transferring some information between the different classification tasks, it will help in leveraging classification performances especially when the number of available training examples is small. Note that the ErrP dataset has not been tested in this MTL framework, because the above-described results suggest an important variance in the selected channels for all subjects. Hence, we believe that this learning problem does not fit into the prior knowledge considered through (11).
We have followed the same experimental protocol as for the single task learning except that training and test sets have been formed as follows. We first create training and test examples for a given subject by randomly splitting all examples of that subject and then gather all subject’s training/test sets to form the multitask learning training/test sets. Hence, all the subjects are equally represented in these sets. A -fold nested cross-validation method is performed in order to automatically select the regularization terms ( and ).
Performances of the different methods have been evaluated for increasing number of training examples per subject and are reported in Figure 7. We can first see that for the EPFL dataset, MGSVM-2 and MGSVM-2s yield a slight but consistent improvement over the single-task classifiers (SVM-Full being a single classifier trained on all subject’s examples and SVM being the average performances of subject-specific classifiers). The poor performances of the SVM-Full approach are probably due to the high intersubject variability in this dataset, which includes impaired patients.
(a)
(b)
For the UAM dataset, results are quite different since the SVM-Full and MGSVM-2s show a significant improvement over the single-task learning. We also note that when only the joint channel selection regularizer is in play (MGSVM-2), multitask learning leads to poorer performance than the SVM-Full for a number of trials lower than . We justify this by the difficulty of achieving appropriate channel selection based only on few training examples, as confirmed by the performance of GSVM-2. From Figure 8, we can see that the good performance of MGSVM-2s is the outcome of performance improvement of about 10% AUC over SVM, achieved on some subjects that perform poorly. More importantly, while performances of these subjects are significantly increased, those that perform well still achieve good AUC scores. In addition, we emphasize that these improvements are essentially due to the similarity-inducing regularizer.
(a)
(b)
For both datasets, the MTL approach MGSVM-2s is consistently better than those of other single-task approaches thanks to the regularization parameters and that can adapt to the intersubject similarity (weak similarity for EPFL and strong similarity for UAM). These are interesting results showing that multitask learning can be a way to handle the problem related to some subjects that achieve poor performances. Moreover, results also indicate that multitask learning is useful for drastically shortening the calibration time. For instance, for the UAM dataset, 80% AUC was achieved using only 100 training examples (less than 1 minute of training example recordings). Note that the validation procedure tends to maximize performances and does not lead to sparse classifiers for MTL approaches. As shown in Figures 2 and 3, the relevant sensors are quite different between subjects thus a joint sensor selection can lead to a slight loss of performances, hence the tendency of the cross-validation procedure to select nonsparse classifiers.
4. Conclusion
In this work, we have investigated the use of mixed-norm regularizers for discriminating event-related potentials in BCI. We have extended the discriminative framework of van Gerven et al. [19] by studying general mixed-norms and proposed the use of the adaptive mixed-norms as sparsity-inducing regularizers. This discriminative framework has been broadened to the multitask learning framework where classifiers related to different subjects are jointly trained. For this framework, we have introduced a novel regularizer that induces channel selection and classifier similarities. The different proposed approaches were tested on three different datasets involving a substantial number of subjects. Results from these experiments have highlighted that the regularizer has been proven interesting for improving classification performance whereas adaptive mixed-norm is the regularizer to be considered when sensor selection is the primary objective. Regarding the multitask learning framework, our most interesting finding is that this learning framework allows, by learning more robust classifiers, significant performance improvement on some subjects that perform poorly in a single-task learning context.
In future work, we plan to investigate a different grouping of the features, such as temporal groups. This kind of group regularization could be for instance used in conjunction with the sensors group in order to promote both feature selection and temporal selection in the classifier. While the resulting problem is still convex, its resolution poses some issues so that a dedicated solver would be necessary.
Another research direction would be to investigate the use of asymmetrical MTL. This could prove handy when a poorly performing subject will negatively influence the other subject performances in MTL while improving his own performances. In this case one would like subject classifier to be similar to the other's classifier without impacting their classifiers.
Appendices
A. Proof of Lipschitz Gradient of the Squared Hinge Loss
Given the training examples , the squared Hinge loss is written as and its gradient is
The squared Hinge loss is gradient Lipschitz if there exists a constant such that
The proof essentially relies on showing that is Lipschitz itself; that is, there exists such that
Now let us consider different situations. For a given and , if and , then the left-hand side is equal to and any would satisfy the inequality. If and , then the left-hand side (lhs) is
A similar reasoning yields to the same bound when and and . Thus, is Lipschitz with a constant . Now, we can conclude the proof by stating that is Lipschitz as it is a sum of Lipschitz function and the related constant is .
B. Lipschitz Gradient for the Multitask Learning Problem
For the multitask learning problem, we want to prove that the function is gradient Lipschitz, being the square Hinge loss. From the above results, it is easy to show that the first term is gradient Lipschitz as the sum of gradient Lipschitz functions.
Now, we also show that the similarity term is also gradient Lipschitz.
This term can be expressed as where is the vector of all classifier parameters and is the Hessian matrix of the similarity regularizer of the form with the identity matrix and a block matrix with a -diagonal matrix where each block is an identity matrix with appropriate circular shift. is thus a row-shifted version of .
Once we have this formulation, we can use the fact that a function is gradient Lipschitz of constant if the largest eigenvalue of its Hessian is bounded by on its domain [36]. Hence, since we have the Hessian matrix of the similarity term has consequently bounded eigenvalues. This concludes the proof that the function is gradient Lipschitz continuous.
C. Proximal Operators
C.1. Norm
The proximal operator of the norm is defined as and has the following closed-form solution for which each component is
C.2. Norm
The proximal operator of the norm is defined as The minimization problem can be decomposed into several ones since the indices are separable. Hence, we can just focus on the problem whose minimizer is
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgment
This work was partially supported by the French Agence Nationale de la Recherche (Grant ANR-11-JS02-10).