Multinomial Regression with Elastic Net Penalty and Its Grouping Effect in Gene Selection

Abstract

For the multiclass classification problem of microarray data, a new optimization model named multinomial regression with the elastic net penalty was proposed in this paper. By combining the multinomial likeliyhood loss and the multiclass elastic net penalty, the optimization model was constructed, which was proved to encourage a grouping effect in gene selection for multiclass classification.

1. Introduction

Support vector machine [1], lasso [2], and their expansions, such as the hybrid huberized support vector machine [3], the doubly regularized support vector machine [4], the 1-norm support vector machine [5], the sparse logistic regression [6], the elastic net [7], and the improved elastic net [8], have been successfully applied to the binary classification problems of microarray data. However, the aforementioned binary classification methods cannot be applied to the multiclass classification easily. Hence, the multiclass classification problems are the difficult issues in microarray classification [9–11].

Besides improving the accuracy, another challenge for the multiclass classification problem of microarray data is how to select the key genes [9–15]. By solving an optimization formula, a new multicategory support vector machine was proposed in [9]. It can be successfully used to microarray classification [9]. However, this optimization model needs to select genes using the additional methods. To automatically select genes during performing the multiclass classification, new optimization models [12–14], such as the 𝐿₁ norm multiclass support vector machine in [12], the multicategory support vector machine with sup norm regularization in [13], and the huberized multiclass support vector machine in [14], were developed.

Note that the logistic loss function not only has good statistical significance but also is second order differentiable. Hence, the regularized logistic regression optimization models have been successfully applied to binary classification problem [15–19]. Multinomial regression can be obtained when applying the logistic regression to the multiclass classification problem. The emergence of the sparse multinomial regression provides a reasonable application to the multiclass classification of microarray data that featured with identifying important genes [20–22]. By using Bayesian 𝐿₁ regularization, the sparse multinomial regression model was proposed in [20]. By adopting a data augmentation strategy with Gaussian latent variables, the variational Bayesian multinomial probit model which can reduce the prediction error was presented in [21]. By using the elastic net penalty, the regularized multinomial regression model was developed in [22]. It can be applied to the multiple sequence alignment of protein related to mutation. Although the above sparse multinomial models achieved good prediction results on the real data, all of them failed to select genes (or variables) in groups.

For the multiclass classification of the microarray data, this paper combined the multinomial likelihood loss function having explicit probability meanings [23] with multiclass elastic net penalty selecting genes in groups [14], proposed a multinomial regression with elastic net penalty, and proved that this model can encourage a grouping effect in gene selection at the same time of classification.

2. Problem Formulation and Preliminary

Given a training data set of 𝐾-class classification problem (𝑥₁,𝑦₁),(𝑥₂,𝑦₂),…,(𝑥_𝑛,𝑦_𝑛), where 𝑥_𝑖=(𝑥_𝑖1,𝑥_𝑖2,…,𝑥_𝑖𝑝) represents the input vector of the 𝑖th sample and 𝑦_𝑖∈{1,2,…,𝐾} represents the class label corresponding to 𝑥_𝑖. For the microarray data, 𝑛 and 𝑝 represent the number of experiments and the number of genes, respectively. Restricted by the high experiment cost, only a few (less than one hundred) samples can be obtained with thousands of genes in one sample. Let 𝑌=(𝑦₁,…,𝑦_𝑛)^𝑇 and 𝑋=(𝑥₍₁₎,𝑥₍₂₎,…,𝑥_(𝑝)), where 𝑥_(𝑗)=(𝑥_1𝑗,…,𝑥_𝑛𝑗)^𝑇, 𝑗=1,…,𝑝. Without loss of generality, it is assumed that ^𝑛_𝑖=1𝑦_𝑖1=0,𝑛^𝑛_𝑖=1𝑥_𝑖𝑗=0,^𝑛_𝑖=1𝑥²_𝑖𝑗=1. (1)

For the binary classification problem, the class labels are assumed to belong to 𝐿={1,−1}. The logistic regression model represents the following class-conditional probabilities; that is, 𝑦logPr_𝑖=+1∣𝑥Pr𝑦_𝑖=−1∣𝑥=𝑏+𝑤^𝑇𝑥, (2) and then Pr𝑦_𝑖=1=+1∣𝑥1+𝑒^{−(𝑏+𝑤𝑇𝑥)},Pr𝑦_𝑖=1=−1∣𝑥1+𝑒^{+(𝑏+𝑤𝑇𝑥)}=1−Pr𝑦_𝑖=+1∣𝑥. (3) According to the common linear regression model, 𝑌 can be predicted as 𝑌=𝑤𝑋=^𝑝_𝑗=1𝑤_(𝑗)𝑥_(𝑗)+𝑏̂, (4) where 𝑏̂ represents bias and 𝑤=(𝑤₍₁₎,…,𝑤_(𝑝))^𝑇 represents the parameter vector.

In this paper, we pay attention to the multiclass classification problems, which imply that 𝐾≥3. Let 𝑓=(𝑓₁,𝑓₂,…,𝑓_𝐾) be the decision function, where 𝑓_𝑘(𝑥)=𝑏_𝑘+𝑤^𝑇_𝑘𝑥,(𝑘=1,2,…,𝐾). The multiclass classifier can be represented as 𝜙(𝑥)=argmax_{𝑘=1,2,…,𝐾}𝑓_𝑘(𝑥). (5) Let 𝑏=(𝑏₁,⋯,𝑏_𝐾)^𝑇 and 𝑤𝑤=¹¹𝑤₁₂⋯𝑤_1𝑝𝑤₂₁𝑤₂₂⋯𝑤_2𝑝𝑤⋮⋮⋮⋮_𝐾1𝑤_𝐾2⋯𝑤_𝐾𝑝. (6) For convenience, we further let 𝑤_𝑘=(𝑤_𝑘1,…,𝑤_𝑘𝑝)^𝑇 and 𝑤_(𝑗)=(𝑤_1𝑗,…,𝑤_𝐾𝑗)^𝑇 represent the 𝑘th row vector and 𝑗th column vector of the parameter matrix 𝑤. Then extending the class-conditional probabilities of the logistic regression model to (𝐾−1)-logits, we have the following formula: 𝑦logPr_𝑖=𝑘∣𝑥Pr𝑦_𝑖=𝐾∣𝑥=𝑏_𝑘+𝑤^𝑇_𝑘𝑥(𝑘=1,2,…,𝐾−1), (7) where (𝑏_𝑘,𝑤_𝑘) represent a pair of parameters which corresponds to the sample (𝑌=𝑘∣𝑥), and 𝑏_𝑘∈𝑅¹, 𝑤_𝑘∈𝑅^𝑝. Similarly, we can construct the 𝐾th as 𝑦logPr_𝑖=𝐾∣𝑥Pr𝑦_𝑖=𝐾∣𝑥=log1=0=𝑏_𝐾+𝑤^𝑇_𝐾𝑥 (8) holds if and only if (𝑏_𝐾,𝑤_𝐾)=(0,^→0). It can be easily obtained that 1=Pr𝑦_𝑖=1∣𝑥+Pr𝑦_𝑖=2∣𝑥+⋯+Pr𝑦_𝑖=𝑒=𝐾∣𝑥_{𝑏1+𝑤𝑇1𝑥}+𝑒^{𝑏2+𝑤𝑇2𝑥}+⋯+𝑒^{𝑏𝐾+𝑤𝑇𝐾𝑥}⋅Pr𝑦_𝑖=𝐾∣𝑥. (9) that is, Pr𝑦_𝑖=1=𝐾∣𝑥∑^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥)}. (10) It should be noted that (𝑏_𝐾,𝑤_𝐾)=(0,^→0) if 𝑘=𝐾. Therefore, the class-conditional probabilities of multiclass classification problem can be represented as Pr𝑦_𝑖=𝑒=𝑘∣𝑥^{𝑏𝑘+𝑤𝑇𝑘𝑥}∑^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥)}. (11)

3. Main Results

3.1. Multinomial Regression with the Multiclass Elastic Net Penalty

Following the idea of sparse multinomial regression [20–22], we fit the above class-conditional probability model by the regularized multinomial likelihood. Let 𝑝_𝑘(𝑥_𝑖)=Pr(𝑦_𝑖=𝑘∣𝑥_𝑖). It is easily obtained that 𝑝_𝑦𝑖𝑥_𝑖=𝑒^{(𝑏𝑦𝑖+𝑤𝑇𝑦𝑖𝑥𝑖)}∑^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑖)}. (12) Hence, 1𝑛^𝑛_𝑖=1log𝑝_𝑦𝑖𝑥_𝑖=1𝑛log𝑝_𝑦1𝑥₁+log𝑝_𝑦2𝑥₂+⋯+log𝑝_𝑦𝑛𝑥_𝑛=1𝑛𝑒log^{(𝑏𝑦1+𝑤𝑇𝑦1𝑥1)}∑^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥1)}𝑒+log^{(𝑏𝑦2+𝑤𝑇𝑦2𝑥2)}∑^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥2)}𝑒+⋯+log^{(𝑏𝑦𝑛+𝑤𝑇𝑦𝑛𝑥𝑛)}∑^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑛)}=1𝑛𝑏_𝑦1+𝑤^𝑇_𝑦1𝑥₁+⋯+𝑏_𝑦𝑛+𝑤^𝑇_𝑦𝑛𝑥_𝑛−log^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥1)}−⋯−log^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑛)}=1𝑛𝑏_𝑦1+𝑤^𝑇_𝑦1𝑥₁+⋯+𝑏_𝑦𝑛+𝑤^𝑇_𝑦𝑛𝑥_𝑛−1𝑛^𝑛_𝑖=1log^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑖)}. (13) Let 𝑦_𝑖𝑘=𝐼𝑦_𝑖=𝑘=1,𝑦_𝑖=𝑘,0,𝑦_𝑖≠𝑘. (14) Then (13) can be rewritten as 1𝑛^𝑛𝐾_𝑖=1𝑘=1𝑦_𝑖𝑘𝑏_𝑘+𝑤^𝑇_𝑘𝑥_𝑖−1𝑛^𝑛_𝑖=1log^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑖)}=1𝑛^𝑛_𝑖=1^𝐾_𝑘=1𝑦_𝑖𝑘𝑏_𝑘+𝑤^𝑇_𝑘𝑥_𝑖−log^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑖)}. (15) Note that log𝑝_𝑦𝑖𝑥_𝑖𝑒=log^{(𝑏𝑦𝑖+𝑤𝑇𝑦𝑖𝑥𝑖)}∑^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑖)}−1<log1=0,𝑛^𝑛_𝑖=1log𝑝_𝑦𝑖𝑥_𝑖>0. (16) Hence, the multinomial likelihood loss function can be defined as 𝑙𝑏_𝑘,𝑤_𝑘^𝐾₁1=−𝑛^𝑛_𝑖=1^𝐾_𝑘=1𝑦_𝑖𝑘𝑏_𝑘+𝑤^𝑇_𝑘𝑥_𝑖−log^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑖)}. (17)

In order to improve the performance of gene selection, the following elastic net penalty for the multiclass classification problem was proposed in [14] 𝐽(𝑏,𝑤)=𝜆₂^𝐾𝑝_𝑘=1𝑗=1𝑤²_𝑘𝑗+𝜆₁^𝐾𝑝_𝑘=1𝑗=1𝑤_𝑘𝑗. (18) By combing the multiclass elastic net penalty (18) with the multinomial likelihood loss function (17), we propose the following multinomial regression model with the elastic net penalty: argmin_(𝑏,𝑤)−1𝑛^𝑛_𝑖=1^𝐾_𝑘=1𝑦_𝑖𝑘𝑏_𝑘+𝑤^𝑇_𝑘𝑥_𝑖−log^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑖)}+𝜆₂^𝐾𝑝_𝑘=1𝑗=1𝑤²_𝑘𝑗+𝜆₁^𝐾𝑝_𝑘=1𝑗=1𝑤_𝑘𝑗, (19) where 𝜆₁,𝜆₂ represent the regularization parameter. Note that (𝑏_𝐾,𝑤_𝐾)=(0,^→0). Hence, the optimization problem (19) can be simplified as argmin_(𝑏,𝑤)−1𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝑦_𝑘𝑖𝑏_𝑘+𝑤^𝑇_𝑘𝑥_𝑖−log1+^𝐾−1_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑖)}+𝜆₂^𝐾−1𝑝_𝑘=1𝑗=1𝑤²_𝑘𝑗+𝜆₁^𝐾−1𝑝_𝑘=1𝑗=1𝑤_𝑘𝑗. (20)

3.2. Grouping Effect

For the microarray classification, it is very important to identify the related gene in groups. In the section, we will prove that the multinomial regression with elastic net penalty can encourage a grouping effect in gene selection. To this end, we must first prove the inequality shown in Theorem 1.

Theorem 1. Let (𝑏̂,𝑤) be the solution of the optimization problem (19) or (20). For any new parameter pairs which are selected as (𝑏^∗,𝑤^∗)=([𝑏^∗₋;^→0],[𝑤_𝑘;^→0]), the following inequality log1+^𝐾−1_𝑘=1𝑒^{(𝑏∗𝑘+𝑤𝑘∗𝑇𝑥𝑖)}−log1+^𝐾−1_𝑘=1𝑒⁽𝑏̂_𝑘+𝑤^𝑇_{𝑘𝑥𝑖)}≤^𝐾−1_𝑘=1𝑏^∗_𝑘+𝑤_𝑘^∗𝑇𝑥_𝑖−𝑏̂_𝑘+𝑤^𝑇_𝑘𝑥_𝑖 (21) holds, where 𝑏^∗₋ and 𝑏̂₋ represent the first 𝐾−1 rows of vectors 𝑏^∗ and 𝑏̂ and 𝑤^∗₋ and 𝑤₋ represent the first 𝐾−1 rows of matrices 𝑤^∗ and 𝑤.

Proof. Note that the inequality 𝑛+|𝑚−𝑛|≥𝑚 holds for the arbitrary real numbers 𝑚 and 𝑛. Hence, the following inequality 𝑏̂_𝑡+𝑤^𝑇_𝑡𝑥_𝑖+𝑏^∗_𝑡+𝑤_𝑡^∗𝑇𝑥_𝑖−𝑏̂_𝑡+𝑤^𝑇_𝑡𝑥_𝑖≥𝑏^∗_𝑡+𝑤_𝑡^∗𝑇𝑥_𝑖 (22) holds for any pairs (𝑏^∗_𝑡,𝑤^∗_𝑡), (𝑏̂_𝑡,𝑤_𝑡). From (22), it can be easily obtained that 𝑏̂_𝑡+𝑤^𝑇_𝑡𝑥_𝑖+^𝐾−1_𝑘=1𝑏^∗_𝑘+𝑤_𝑘^∗𝑇𝑥_𝑖−𝑏̂_𝑘+𝑤^𝑇_𝑘𝑥_𝑖≥𝑏^∗_𝑡+𝑤_𝑡^∗𝑇𝑥_𝑖. (23) that is, 𝑒⁽𝑏̂_𝑡+𝑤^𝑇_{𝑡𝑥𝑖)+∑}^𝐾−1_𝑘=1|(𝑏^∗_{𝑘+𝑤𝑘}^∗𝑇_{𝑥𝑖)−(}𝑏̂_𝑘+𝑤^𝑇_{𝑘𝑥𝑖)|}≥𝑒^{(𝑏∗𝑡+𝑤𝑡∗𝑇𝑥𝑖)}. (24) Note that 𝑒^{∑𝐾−1𝑘=1|(𝑏∗𝑘+𝑤𝑘∗𝑇𝑥𝑖)−(}𝑏̂_𝑘+𝑤^𝑇_{𝑘𝑥𝑖)|}≥𝑒⁰=1. (25) Hence, from (24) and (25), we can get Δ₁Δ₂≤1, (26) where Δ₁=1+𝑒^{(𝑏∗1+𝑤1∗𝑇𝑥𝑖)}+⋯+𝑒^{(𝑏∗𝑡+𝑤𝑡∗𝑇𝑥𝑖)}+⋯+𝑒^{(𝑏∗𝐾−1+𝑤∗𝑇𝐾−1𝑥𝑖)},Δ₂=𝑒^{∑𝐾−1𝑘=1|(𝑏∗𝑘+𝑤𝑘∗𝑇𝑥𝑖)−(}𝑏̂_𝑘+𝑤^𝑇_{𝑘𝑥𝑖)|}+𝑒⁽𝑏̂₁₊𝑤^𝑇_{1𝑥𝑖)+∑}^𝐾−1_𝑘=1|(𝑏^∗_{𝑘+𝑤𝑘}^∗𝑇_{𝑥𝑖)−(}𝑏̂_𝑘+𝑤^𝑇_{𝑘𝑥𝑖)|}+⋯+𝑒⁽𝑏̂_𝑡+𝑤^𝑇_{𝑡𝑥𝑖)+∑}^𝐾−1_𝑘=1|(𝑏^∗_{𝑘+𝑤𝑘}^∗𝑇_{𝑥𝑖)−(}𝑏̂_𝑘+𝑤^𝑇_{𝑘𝑥𝑖)|}+⋯+𝑒⁽𝑏̂_𝐾−1+𝑤^∗𝑇_𝐾−1^𝑥_𝑖^{)+∑𝐾−1}_𝑘=1^|(𝑏∗_𝑘^+𝑤_𝑘^∗𝑇𝑥_𝑖⁾⁻⁽𝑏̂_𝑘+𝑤^𝑇_𝑘^𝑥_𝑖^)|. (27) Equation (26) is equivalent to the following inequality: 1+𝑒^{(𝑏∗1+𝑤1∗𝑇𝑥𝑖)}+⋯+𝑒^{(𝑏∗𝐾−1+𝑤∗𝑇𝐾−1𝑥𝑖)}×1+𝑒⁽𝑏̂₁₊𝑤^𝑇₁^𝑥_𝑖⁾+⋯+𝑒⁽𝑏̂_𝐾−1+𝑤^𝑇_𝐾−1^𝑥_𝑖⁾⋅𝑒^{∑𝐾−1𝑘=1|(𝑏∗𝑘+𝑤𝑘∗𝑇𝑥𝑖)−(}𝑏̂_𝑘+𝑤^𝑇_{𝑘𝑥𝑖)|}⁻¹≤1⟺log1+𝑒^{(𝑏∗1+𝑤1∗𝑇𝑥𝑖)}+⋯+𝑒^{(𝑏∗𝐾−1+𝑤∗𝑇𝐾−1𝑥𝑖)}1+𝑒⁽𝑏̂₁₊𝑤^𝑇_1𝑥𝑖)+⋯+𝑒⁽𝑏̂_𝐾−1+𝑤^𝑇_{𝐾−1𝑥𝑖)}≤^𝐾−1_𝑘=1𝑏^∗_𝑘+𝑤_𝑘^∗𝑇𝑥_𝑖−𝑏̂_𝑘+𝑤^𝑇_𝑘𝑥_𝑖⟺log1+^𝐾−1_𝑘=1𝑒^{(𝑏∗𝑘+𝑤𝑘∗𝑇𝑥𝑖)}−log1+^𝐾−1_𝑘=1𝑒⁽𝑏̂_𝑘+𝑤^𝑇_{𝑘𝑥𝑖)}≤^𝐾−1_𝑘=1𝑏^∗_𝑘+𝑤_𝑘^∗𝑇𝑥_𝑖−𝑏̂_𝑘+𝑤^𝑇_𝑘𝑥_𝑖. (28)
Hence, inequality (21) holds. This completes the proof.

Using the results in Theorem 1, we prove that the multinomial regression with elastic net penalty (19) can encourage a grouping effect.

Theorem 2. Give the training data set (𝑥₁,𝑦₁),(𝑥₂,𝑦₂),…,(𝑥_𝑛,𝑦_𝑛) and assume that the matrix 𝑋 and vector 𝑌 satisfy (1). If the pairs (𝑏̂,𝑤) are the optimal solution of the multinomial regression with elastic net penalty (19), then the following inequality 𝑤_(𝑚)−𝑤_(𝑙)2≤2√𝐾√𝑛𝜆₂21−𝜌. (29) holds, where 𝜌=𝑥^𝑇_(𝑚)𝑥_(𝑙)=∑^𝑛_𝑖=1𝑥_𝑖𝑚𝑥_𝑖𝑙, 𝑤_(𝑚) is the 𝑚th column of parameter matrix 𝑤, and 𝑤_(𝑚) is the 𝑙th column of parameter matrix 𝑤.

Proof. First of all, we construct the new parameter pairs (𝑏^∗,𝑤^∗)=([𝑏^∗₋;^→0],[𝑤^∗_𝑘;^→0]), where 𝑏^∗₋=𝑏̂₋,𝑤^∗_𝑘𝑗′=12𝑤_𝑘𝑚+𝑤_𝑘𝑙,𝑗^′𝑤=𝑚,𝑙,_𝑘𝑗′,𝑗^′≠𝑚,𝑙. (30) Let 1𝐿=−𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝑦_𝑘𝑖𝑏_𝑘+𝑤^𝑇_𝑘𝑥_𝑖−log1+^𝐾−1_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑖)}+𝜆₂^𝐾−1𝑝_𝑘=1𝑗=1𝑤²_𝑘𝑗+𝜆₁^𝐾−1𝑝_𝑘=1𝑗=1𝑤_𝑘𝑗. (31) Since the pairs (𝑏̂,𝑤) are the optimal solution of the multinomial regression with elastic net penalty (19), it can be easily obtained that 0≤𝐿𝜆₁,𝜆₂,𝑏^∗₋,𝑤^∗₋−𝐿𝜆₁,𝜆₂,𝑏̂₋,𝑤₋. (32) Note that the function 𝐿_𝑘𝑖(𝑏̂_𝑘,𝑤_𝑘,𝑥_𝑖)=𝑏_𝑘+𝑤^𝑇_𝑘𝑥_𝑖 is Lipschitz continuous. Hence, we have 𝐿_𝑘𝑖𝑏^∗_𝑘,𝑤^∗_𝑘,𝑥_𝑖−𝐿_𝑘𝑖𝑏̂_𝑘,𝑤_𝑘,𝑥_𝑖≤𝑤^∗_𝑘−𝑤_𝑘^𝑇𝑥_𝑖. (33) From (33) and (21) and the definition of the parameter pairs (𝑏^∗,𝑤^∗), we have −1𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝐿_𝑘𝑖𝑏^∗_𝑘,𝑤^∗_𝑘,𝑥_𝑖−𝐿_𝑘𝑖𝑏̂_𝑘,𝑤_𝑘,𝑥_𝑖−log1+^𝐾−1_𝑘=1𝑒^{(𝑏∗𝑘+𝑤𝑘∗𝑇𝑥𝑖)}+log1+^𝐾−1_𝑘=1𝑒⁽𝑏̂_𝑘+𝑤^𝑇_{𝑘𝑥𝑖)}≤−1𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝐿_𝑘𝑖𝑏^∗_𝑘,𝑤^∗_𝑘,𝑥_𝑖−𝐿_𝑘𝑖𝑏̂_𝑘,𝑤_𝑘,𝑥_𝑖+1𝑛^𝑛_𝑖=1log1+^𝐾−1_𝑘=1𝑒^{(𝑏∗𝑘+𝑤𝑘∗𝑇𝑥𝑖)}−log1+^𝐾−1_𝑘=1𝑒⁽𝑏̂_𝑘+𝑤^𝑇_{𝑘𝑥𝑖)}≤1𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝐿_𝑘𝑖𝑏^∗_𝑘,𝑤^∗_𝑘,𝑥_𝑖−𝐿_𝑘𝑖𝑏̂_𝑘,𝑤_𝑘,𝑥_𝑖+1𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝑏^∗_𝑘+𝑤_𝑘^∗𝑇𝑥_𝑖−𝑏̂_𝑘+𝑤^𝑇_𝑘𝑥_𝑖≤1𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝑤^∗_𝑘−𝑤_𝑘^𝑇𝑥_𝑖+1𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝑤^∗_𝑘−𝑤_𝑘^𝑇𝑥_𝑖=12𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝑤_𝑘𝑚−𝑤_𝑘𝑙𝑥_𝑖𝑚−𝑥_𝑖𝑙+12𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝑤_𝑘𝑚−𝑤_𝑘𝑙𝑥_𝑖𝑚−𝑥_𝑖𝑙=12𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝑤_𝑘𝑚−𝑤_𝑘𝑙𝑥_𝑖𝑚−𝑥_𝑖𝑙+12𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝑤_𝑘𝑚−𝑤_𝑘𝑙𝑥_𝑖𝑚−𝑥_𝑖𝑙=12𝑛^𝑛_𝑖=1𝑥_𝑖𝑚−𝑥_𝑖𝑙⋅^𝐾−1_𝑘=1𝑤_𝑘𝑚−𝑤_𝑘𝑙+12𝑛^𝑛_𝑖=1^𝐾−1_𝑘=1𝑤_𝑘𝑚−𝑤_𝑘𝑙𝑥_𝑖𝑚−𝑥_𝑖𝑙=1𝑛𝑥_(𝑚)−𝑥_(𝑙)1⋅𝑤_(𝑚)−𝑤_(𝑙)1. (34) Analogically, we have ^𝐾−1𝑝_𝑘=1𝑗=1𝑤^∗_𝑘𝑗−𝑤_𝑘𝑗=^𝐾−1_𝑘=12𝑤_𝑘𝑚+𝑤_𝑘𝑙2−𝑤_𝑘𝑚−𝑤_𝑘𝑙≤0,_{𝐾−1𝑝𝑘=1𝑗=1}𝑤^∗_𝑘𝑗²−𝑤²_𝑘𝑗1=−2^𝐾−1_𝑘=1𝑤_𝑘𝑚−𝑤_𝑘𝑙²1=−2𝑤_(𝑚)−𝑤_(𝑙)²₂. (35) Substituting (34) and (35) into (32) gives 10≤𝑛𝑥_(𝑚)−𝑥_(𝑙)1⋅𝑤_(𝑚)−𝑤_(𝑙)1−𝜆₂2𝑤_(𝑚)−𝑤_(𝑙)²₂. (36) that is, 20≤𝑛𝜆₂𝑥_(𝑚)−𝑥_(𝑙)1⋅𝑤_(𝑚)−𝑤_(𝑙)1−𝑤_(𝑚)−𝑤_(𝑙)²₂≤2√𝐾𝑛𝜆₂𝑥_(𝑚)−𝑥_(𝑙)1⋅𝑤_(𝑚)−𝑤_(𝑙)2−𝑤_(𝑚)−𝑤_(𝑙)²₂. (37) From (37), it can be easily obtained that 𝑤_(𝑚)−𝑤_(𝑙)²₂≤2√𝐾𝑛𝜆₂𝑥_(𝑚)−𝑥_(𝑙)1≤2√𝐾𝑛𝜆₂⋅√𝑛𝑥_(𝑚)−𝑥_(𝑙)2=2√𝐾√𝑛𝜆₂2−2𝑥^𝑇_(𝑚)𝑥_(𝑙)=2√𝐾√𝑛𝜆₂21−𝜌, (38) where 𝜌=𝑥^𝑇_(𝑚)𝑥_(𝑙)=∑^𝑛_𝑖=1𝑥_𝑖𝑚𝑥_𝑖𝑙. This completes the proof.

According to the inequality shown in Theorem 2, the multinomial regression with elastic net penalty can assign the same parameter vectors (i.e., 𝑤_(𝑚)=𝑤_(𝑙)) to the high correlated predictors 𝑥_(𝑚),𝑥_(𝑙) (i.e., 𝜌=1). This means that the multinomial regression with elastic net penalty can select genes in groups according to their correlation. According to the technical term in [14], this performance is called grouping effect in gene selection for multiclass classification. Particularly, for the binary classification, that is, 𝐾=2, inequality (29) becomes 𝑤_1𝑚−𝑤_1𝑙≤2√𝑛𝜆₂21−𝜌. (39) This corresponds with the results in [7].

3.3. Solving Algorithm

Microarray is the typical small 𝑛, large 𝑝 problem. Because the number of the genes in microarray data is very large, it will result in the curse of dimensionality to solve the proposed multinomial regression. To improve the solving speed, Friedman et al. proposed the pairwise coordinate decent algorithm which takes advantage of the sparse property of characteristic. Therefore, we choose the pairwise coordinate decent algorithm to solve the multinomial regression with elastic net penalty. To this end, we convert (19) into the following form: argmin_(𝑏,𝑤)−1𝑛^𝑛_𝑖=1^𝐾_𝑘=1𝑦_𝑖𝑘𝑏_𝑘+𝑤^𝑇_𝑘𝑥_𝑖−log^𝐾_𝑘=1𝑒^{(𝑏𝑘+𝑤𝑇𝑘𝑥𝑖)}+𝜆𝛼^𝐾𝑝_𝑘=1𝑗=1𝑤²_𝑘𝑗+(1−𝛼)^𝐾𝑝_𝑘=1𝑗=1𝑤_𝑘𝑗. (40) Equation (40) can be easily solved by using the R package “glmnet” which is publicly available.

4. Conclusion

By combining the multinomial likelihood loss function having explicit probability meanings with the multiclass elastic net penalty selecting genes in groups, the multinomial regression with elastic net penalty for the multiclass classification problem of microarray data was proposed in this paper. The proposed multinomial regression is proved to encourage a grouping effect in gene selection. In the next work, we will apply this optimization model to the real microarray data and verify the specific biological significance.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

This work is supported by Natural Science Foundation of China (61203293, 61374079), Key Scientific and Technological Project of Henan Province (122102210131, 122102210132), Program for Science and Technology Innovation Talents in Universities of Henan Province (13HASTIT040), Foundation and Advanced Technology Research Program of Henan Province (132300410389, 132300410390, 122300410414, and 132300410432), Foundation of Henan Educational Committee (13A120524), and Henan Higher School Funding Scheme for Young Teachers (2012GGJS-063).