Cost-Sensitive Support Vector Machine Using Randomized Dual Coordinate Descent Method for Big Class-Imbalanced Data Classification

Abstract

Cost-sensitive support vector machine is one of the most popular tools to deal with class-imbalanced problem such as fault diagnosis. However, such data appear with a huge number of examples as well as features. Aiming at class-imbalanced problem on big data, a cost-sensitive support vector machine using randomized dual coordinate descent method (CSVM-RDCD) is proposed in this paper. The solution of concerned subproblem at each iteration is derived in closed form and the computational cost is decreased through the accelerating strategy and cheap computation. The four constrained conditions of CSVM-RDCD are derived. Experimental results illustrate that the proposed method increases recognition rates of positive class and reduces average misclassification costs on real big class-imbalanced data.

1. Introduction

The most popular strategy for the design of classification algorithms is to minimize the probability of error, assuming that all misclassifications have the same cost and classes of dataset are balanced [1–6]. The resulting decision rules are usually denoted as cost insensitive. However, in many important applications of machine learning, such as fault diagnosis [7] and fraud detection, certain types of error are much more costly than others. Other applications involve significantly class-imbalanced datasets, where examples from different classes appear with substantially different probability. Cost-sensitive support vector machine (CSVM) [2] is one of the most popular tools to deal with class-imbalanced problem and unequal misclassification problem. However, in many applications, such data appear with a huge number of examples as well as features.

In this work we consider the cost-sensitive support vector machine architecture [1]. Although CSVMs are based on a very solid learning-theoretic foundation and have been successfully applied to many classification problems, it is not well understood how to design big data learning of the CSVM algorithm. CSVM usually maps training vectors into a high dimensional space via a nonlinear function. Due to the high dimensionality of the weight vector, one solves the dual problem of CSVM by the kernel trick. In some applications, data appear in a rich dimensional feature space; the performances are similar with/without nonlinear mapping. If data are not mapped, we can often train much data set.

Recently, many methods have been proposed for linear SVM in large-scale scenarios [4]. In all methods, dual coordinate descent methods for dual problem of CSVM are one of popular methods to deal with large-scale convex optimization problem. However, they do not focus on big data learning of CSVM. We focus on big data class-imbalanced learning by CSVM.

This paper is organized as follows. In Section 2 basic theory of cost-sensitive support vector machine is described. In Section 3 we derive our proposed algorithm. Section 4 discusses both speed-ups and four constrained conditions of cost-sensitive support vector machine. Implementation issues are investigated in Section 5. Experiments show efficiently our proposed method.

2. Basic Theory of Cost-Sensitive Support Vector Machine

Cost-sensitive support vector machine such as 2C-SVM is proposed in [1]. Consider examples set {(𝐱_𝑖,𝑦_𝑖)}^𝑙_𝑖=1, 𝐱_𝑖∈𝑅^𝑑, 𝑦_𝑖∈𝑌={+1,−1}, 𝐼₊={𝑖∶𝑦_𝑖=+1}, 𝐼₋={𝑖∶𝑦_𝑖=−1}. The 2C-SVM has primal optimization problem: min_{𝑤,𝑏,𝜉}12‖𝐰‖²+𝐶𝛾_𝑖∈𝐼+𝜉_𝑖+𝐶1−𝛾_{𝑖∈𝐼−}𝜉_𝑖s.t.𝑦_𝑖𝐰^𝑇𝐱_𝑖+𝑏≥1−𝜉_𝑖𝜉𝑖=1,2,…,𝑙_𝑖≥0𝑖=1,2,…,𝑙𝐶>0. (1) Lagrange multipliers method can be used to solve the constrained optimization problem. In this method, the Lagrange equation is defined as follows: 𝐿()=1𝐰,𝑏,𝜉2‖𝐰‖²+𝐶𝛾_𝑖∈𝐼+𝜉_𝑖+𝐶1−𝛾_{𝑖∈𝐼−}𝜉_𝑖+^𝑙_𝑖=1𝛼_𝑖1−𝜉_𝑖−𝑦_𝑖𝐰^𝑇𝐱_𝑖+𝑏+^𝑙_𝑖=1𝛽_𝑖−𝜉_𝑖, (2) where the 𝛼_𝑖, 𝛽_𝑖’s are called the Lagrange multipliers. 𝐿’s partial derivatives are set to be zero as follows: 𝜕𝐿(𝐰,𝑏,𝜉)()𝜕𝐰=0,𝜕𝐿𝐰,𝑏,𝜉()𝜕𝑏=0,𝜕𝐿𝐰,𝑏,𝜉𝜕𝜉_𝑖=0. (3)

And solve 𝐰,𝑏,𝜉. The formula (3) is extended to be 𝜕𝐿(𝐰,𝑏,𝜉)𝜕𝐰=𝐰+^𝑙_𝑖=1𝛼_𝑖−𝑦_𝑖𝐱_𝑖()=0,𝜕𝐿𝐰,𝑏,𝜉=𝜕𝑏^𝑙_𝑖=1𝛼_𝑖−𝑦_𝑖()=0,𝜕𝐿𝐰,𝑏,𝜉𝜕𝜉_𝑖=𝐶𝛾−𝛼_𝑖−𝛽_𝑖=0,𝑖∈𝐼₊,𝜕𝐿(𝐰,𝑏,𝜉)𝜕𝜉_𝑖=𝐶1−𝛾−𝛼_𝑖−𝛽_𝑖=0,𝑖∈𝐼₋. (4)

Equation (4) can be reformatted as 𝐰=^𝑙_𝑖=1𝛼_𝑖𝑦_𝑖𝐱_𝑖,_𝑙𝑖=1𝛼_𝑖𝑦_𝑖=0,𝐶𝛾−𝛼_𝑖−𝛽_𝑖=0,𝑖∈𝐼₊,𝐶1−𝛾−𝛼_𝑖−𝛽_𝑖=0,𝑖∈𝐼₋. (5)(6)(7)

By incorporating (5) to (7), Lagrange equation (2) is rewritten as 𝐿()=1𝑤,𝑏,𝜉2^𝑛_𝑖=1𝛼_𝑖𝑦_𝑖𝐱_𝑖²+𝐶𝛾_𝑖∈𝐼+𝜉_𝑖+𝐶1−𝛾_{𝑖∈𝐼−}𝜉_𝑖+^𝑙_𝑖=1𝛼_𝑖1−𝜉_𝑖−𝑦_𝑖^𝑛_𝑖=1𝛼_𝑖𝑦_𝑖𝐱_𝑖^𝑇𝐱_𝑖+𝑏+_𝑖∈𝐼+𝐶𝛾−𝛼_𝑖−𝜉_𝑖+_{𝑖∈𝐼−}𝐶1−𝛾−𝛼_𝑖−𝜉_𝑖. (8)

Lagrange equation (8) is simplified as 𝐿()1𝑤,𝑏,𝜉=−2^𝑙_𝑖=1𝛼_𝑖𝑦_𝑖^𝑙_𝑗=1𝛼_𝑗𝑦_𝑗𝐱_𝑗^𝑇𝐱_𝑖+^𝑙_𝑖=1𝛼_𝑖. (9)

Recall that the equation above is obtained by minimizing 𝐿 with respect to 𝐰 and 𝑏. Putting this together with the constraints 𝛼_𝑖≥0 and the constraints (5) to (7), the following dual optimization problem of 2C-SVM is obtained as min_𝛼𝑓(𝜶)=12^𝑙𝑙_𝑖=1𝑗=1𝛼_𝑖𝛼_𝑗𝑦_𝑖𝑦_𝑗𝐱^𝑇_𝑖𝐱_𝑗−^𝑙_𝑖=1𝛼_𝑖s.t.0≤𝛼_𝑖≤𝐶𝛾,𝑖∈𝐼₊0≤𝛼_𝑖≤𝐶1−𝛾,𝑖∈𝐼_{−𝑙𝑖=1}𝛼_𝑖𝑦_𝑖=0, (10) where 𝛼_𝑖, 𝑖=1,…,𝑙 are Lagrange multipliers and 𝐶𝛾, 𝐶(1−𝛾), 𝐶>0, 𝛾∈[0,1] are misclassification cost parameters. The matrixes 𝑄 and 𝜶 are defined, respectively, as follows: 𝑄=𝑄_{𝑖𝑗𝑙×𝑙},where𝑄_𝑖𝑗=𝛼_𝑖𝛼_𝑗𝑦_𝑖𝑦_𝑗𝐱^𝑇_𝑖𝐱_𝑗,𝜶=𝛼₁,…,𝛼_𝑖,…,𝛼_𝑗,…,𝛼_𝑙. (11)

3. Cost-Sensitive Support Vector Machine Using Randomized Dual Coordinate Descent Method

In this section, randomized dual coordinate descent method is used to solve 2C-SVM that is one version of cost-sensitive support vector machine. The optimization process starts from an initial point 𝜶⁰∈𝐑^𝑙 and generates a sequence of vectors 𝜶^𝑘, 𝑘=0,…,∞. The process from 𝜶^𝑘 to 𝜶^𝑘+1 is called an outer iteration. In each outer iteration 𝑙 is called inner iteration, so that 𝛼₁,𝛼₂,…,𝛼_𝑙 are sequentially updated. Each outer iteration thus generates vectors 𝜶^𝑘,𝑖∈𝐑^𝑙, 𝑖=1,2,…,𝑙+1, such that 𝜶^𝑘,1=𝜶^𝑘𝜶,𝑖=1,_𝑘,𝑖=𝛼₁^𝑘+1,…,𝛼^𝑘+1_𝑖−1,𝛼^𝑘_𝑖,…,𝛼^𝑘_𝑙𝜶,∀𝑖=2,…,𝑙,_𝑘,𝑙=𝜶^𝑘,𝑖=𝑙+1. (12)

For updating 𝜶^𝑘,𝑖 to 𝜶^𝑘,𝑖+1, the following one-variable subproblem is solved as min_𝑑𝑓𝜶^𝑘,𝑖+𝑑𝑒_𝑖, (13) where 𝑒_𝑖=[0,…,1,…,0]^𝑇. This is general process of one-variable coordinate descent method. However, the sum constrained condition of the optimization problem (13) is denoted as follows: ^𝑛_𝑖=1𝛼_𝑖𝑦_𝑖=0, (14) where 𝛼_𝑖 is exactly determined by the other 𝛼_𝑖,s, and if we were to hold {𝛼_𝑖}^𝑛_𝑖=1⧵𝛼_𝑖 fixed, then we cannot make any change 𝛼_𝑖 without violating the constraint condition (14) in the optimization problem.

Thus if we want to update some subject of the 𝛼_𝑖,s, we must update at least two of them simultaneously in order to keep satisfying the constraints. We currently have some setting of the 𝛼_𝑖s that satisfy the constraint conditions (14) and suppose we have decided to hold {𝛼_𝑖}^𝑛_𝑖=1⧵𝛼_𝑖,𝛼_𝑗 fixed and reoptimize the dual problem of CSVM (10) with respect to 𝛼_𝑖, 𝛼_𝑗 (subject to the constraints). Equation (6) is reformatted as 𝛼_𝑖𝑦_𝑖+𝛼_𝑗𝑦_𝑗=^𝑙_𝑚=1𝛼_𝑚𝑦_𝑚−𝛼_𝑖𝑦_𝑖−𝛼_𝑗𝑦_𝑗. (15)

Since the right hand side is fixed, we can just let it be denoted by some constant 𝜁: 𝜁=^𝑙_𝑚=1𝛼_𝑚𝑦_𝑚−𝛼_𝑖𝑦_𝑖−𝛼_𝑗𝑦_𝑗. (16)

We can form the updated coordinates before and after with respect to 𝛼_𝑖, 𝛼_𝑗: 𝛼^𝑘_𝑖𝑦_𝑖+𝛼^𝑘_𝑗𝑦_𝑗=𝛼_𝑖^𝑘+1𝑦_𝑖+𝛼_𝑗^𝑘+1𝑦_𝑗=𝜁. (17)𝛼^𝑘_𝑖𝑦_𝑖, 𝛼_𝑖^𝑘+1𝑦_𝑖 are updated coordinates before and after with respect to 𝛼^𝑘_𝑖. Consider the following dual two-variable subproblem of 2C-SVM: 𝑓𝜶^𝑘,𝑖+𝑑_𝑖𝑒_𝑖+𝑓𝜶^𝑘,𝑗+𝑑_𝑗𝑒_𝑗=𝑄_𝑖𝑖𝑑²_𝑖+2(𝛼^𝑘,𝑖𝑄)_𝑖−1𝑑_𝑖+𝑄_𝑗𝑗𝑑²_𝑗+2(𝛼^𝑘,𝑗𝑄)_𝑗−1𝑑_𝑗+𝑐, (18) where 𝑐 is constant, 𝑄_𝑖=[𝑄_𝑖,𝑠,…,𝑄_{𝑖,𝑠−1},𝑄_𝑖,𝑠,…,𝑄_𝑖,𝑙], and (𝛼^𝑘,𝑖𝑄)_𝑖=^𝑙_𝑠=1𝑄_𝑖,𝑠𝛼_𝑖^𝑘,𝑠. (19)

From (18), we obtain 𝑓^′(𝜶)=𝜶^𝑇𝑄−𝟏,𝜕𝑓𝜶^𝑘,𝑖𝜕𝛼_𝑖=2(𝜶^𝑘,𝑖𝑄)_𝑖−1, (20)(21) where 𝜕𝑓(𝜶^𝑘,𝑖)/𝜕𝛼_𝑖 is the 𝑖th component of the gradient 𝑓^′(𝜶). By incorporating (21), (18) is rewritten as 𝑓𝜶^𝑘,𝑖+𝑑_𝑖𝑒_𝑖+𝑓𝜶^𝑘,𝑗+𝑑_𝑗𝑒_𝑗=𝑄_𝑖𝑖𝑑²_𝑖+𝜕𝑓𝜶^𝑘,𝑖𝜕𝛼_𝑖𝑑_𝑖+𝑄_𝑗𝑗𝑑²_𝑗+𝜕𝑓𝜶^𝑘,𝑗𝜕𝛼_𝑗𝑑_𝑗+𝑐. (22)

From (17), we have 𝑑_𝑗=−𝑑_𝑖𝑦_𝑖𝑦_𝑗. Equation (22) is reformed as follows: 𝑓𝜶^𝑘,𝑖+𝑑_𝑖𝑒_𝑖+𝑓𝜶^𝑘,𝑗+𝑑_𝑗𝑒_𝑗=𝑄_𝑖𝑖𝑑²_𝑖+𝜕𝑓𝜶^𝑘,𝑖𝜕𝛼_𝑖𝑑_𝑖+𝑄_𝑗𝑗𝑑²_𝑖−𝜕𝑓𝜶^𝑘,𝑗𝜕𝛼_𝑗𝑑_𝑖𝑦_𝑖𝑦_𝑗=𝑄+𝑐_𝑖𝑖+𝑄_𝑗𝑗𝑑²_𝑖+𝜕𝑓𝜶^𝑘,𝑖𝜕𝛼_𝑖−𝜕𝑓𝜶^𝑘,𝑗𝜕𝛼_𝑗𝑦_𝑖𝑦_𝑗𝑑_𝑖+𝑐. (23)

Treating (𝑄_𝑖𝑖+𝑄_𝑗𝑗), ((𝜕𝑓(𝜶^𝑘,𝑖)/𝜕𝛼_𝑖)−(𝜕𝑓(𝜶^𝑘,𝑗)/𝜕𝛼_𝑗)𝑦_𝑖𝑦_𝑗), 𝑐 as constants, we should be able to verify that this is just some quadratic function in 𝑑_𝑖. We can easily maximize this quadratic function by setting its derivative to zero and solving the optimization problem. The following two-variable optimization problem is obtained as min_𝑑𝑖𝑓𝜶^𝑘,𝑖+𝑑_𝑖𝑒_𝑖+𝑓𝜶^𝑘,𝑗+𝑑_𝑗𝑒_𝑗. (24)

The closed form is derived as follows: 𝑑_𝑖=𝜕𝑓𝜶^𝑘,𝑖/𝜕𝛼_𝑖−𝜕𝑓𝜶^𝑘,𝑗/𝜕𝛼_𝑗𝑦_𝑖𝑦_𝑗2𝑄_𝑖𝑖+𝑄_𝑗𝑗. (25)

We now consider the box constraints (10) of the two-variable optimization problem. The box constraints as 0≤𝛼_𝑖≤𝐶𝛾, 𝑖∈𝐼₊, 0≤𝛼_𝑖≤𝐶(1−𝛾), 𝑖∈𝐼₋ are classified to four boxes constraints according to the labels of examples from some two coordinates.

Firstly, suppose that the labels of examples are as 𝑦_𝑖=+1,𝑦_𝑗=−1. (26)

The sum constraints of Lagrange multipliers according to these labels are as 𝛼_𝑖𝑦_𝑖+𝛼_𝑗𝑦_𝑗=𝜁. (27)

Another expressing of sum constraints of Lagrange multipliers is as 𝛼_𝑖−𝛼_𝑗=𝜁,or𝛼_𝑖=𝛼_𝑗+𝜁. (28)

The box constraints of Lagrange multipliers are defined as 0≤𝛼_𝑖≤𝛾𝐶,0≤𝛼_𝑗≤1−𝛾𝐶. (29)

We obtain the new expressing of box constraints of Lagrange multipliers as the following: 0≤𝜁≤1−𝛾𝐶,0≤𝛼_𝑖≤𝛼_𝑖−𝛼_𝑗−𝛼1−𝛾𝐶,or−1−𝛾𝐶≤𝜁≤0,_𝑖−𝛼_𝑗≤𝛼_𝑖≤𝛾𝐶. (30)

Thus we obtain stricter box constraints of Lagrange multipliers 𝛼_𝑖, 𝛼_𝑗 according to 𝑦_𝑖=+1, 𝑦_𝑗=−1 as the following: 𝑈₁=max0,𝛼_𝑖−𝛼_𝑗,𝑉₁=min𝛼_𝑖−𝛼_𝑗−1−𝛾𝐶,𝛾𝐶. (31)

Secondly, suppose that the labels of examples are as 𝑦_𝑖=+1,𝑦_𝑗=+1. (32)

Similarly, similar stricter box constraints of Lagrange multipliers 𝛼_𝑖, 𝛼_𝑗 are obtained as follows: 𝑈₂=max0,𝛼_𝑖+𝛼_𝑗,𝑉−𝛾𝐶₂=min𝛼_𝑖+𝛼_𝑗,𝛾𝐶. (33)

Thirdly, suppose that the labels of examples are as 𝑦_𝑖=−1,𝑦_𝑗=−1. (34)

Similarly, the similar stricter box constraints of Lagrange multipliers 𝛼_𝑖,𝛼_𝑗 are obtained as follows: 𝑈₃=max0,−𝛼_𝑖−𝛼_𝑗−𝐶,𝑉1−𝛾₃=min−𝛼_𝑖−𝛼_𝑗,1−𝛾𝐶. (35)

Finally, suppose that the labels of examples are as 𝑦_𝑖=−1,𝑦_𝑗=+1. (36)

Similarly, the similar stricter box constraints of Lagrange multipliers 𝛼_𝑖,𝛼_𝑗 are obtained as follows: 𝑈₄=max0,−𝛼_𝑖−𝛼_𝑗,𝑉₄=min𝛾𝐶−𝛼_𝑖−𝛼_𝑗,1−𝛾𝐶. (37)

For the simplification, set 𝛼^temp_𝑖=𝛼_𝑖^𝑘,𝑖+𝑑_𝑖, (38)𝛼_𝑖^𝑘,𝑖+1 is defined as the temp solution, which would be edited to satisfy 𝛼_𝑖^𝑘,𝑖+1=𝑉_𝑖,if𝛼^temp_𝑖>𝑉_𝑖𝛼,𝑖=1,2,3,4_temp𝑖,if𝑈_𝑖<𝛼^temp_𝑖<𝑉_𝑖𝑈,𝑖=1,2,3,4_𝑖,if𝛼^temp_𝑖<𝑈_𝑖,𝑖=1,2,3,4. (39)

From linear constraints with respect to 𝛼_𝑖^𝑘,𝑖+1 in (17), the value of 𝛼_𝑖^𝑘,𝑗+1 is obtained as 𝛼_𝑖^𝑘,𝑗+1=𝑦_𝑗𝑦_𝑖𝛼_𝑖^𝑘,𝑖+𝑦_𝑗𝛼_𝑖^𝑘,𝑗−𝑦_𝑖𝛼_𝑖^𝑘,𝑖+1. (40)

4. The Modified Proposed Method

4.1. Two Speeding-Up Strategies

To avoid duplicate and invalid iterative process calculations, the two given conditions are not updated. If one of two conditions is met, then the algorithm skips this iteration that can significantly reduce the computational amount and accelerate the convergence speed.

Condition 1. If 𝛼_𝑖^𝑘,𝑖=𝛼_𝑖^𝑘,𝑗=0 or 𝛼_𝑖^𝑘,𝑖=𝛼_𝑖^𝑘,𝑗=𝐶, then the constrained conditions are not updated.

Due to the box constraints of (10), there will appear a lot of boundary points of coordinates (𝛼_𝑖^𝑘,𝑖=𝐶 or 𝛼_𝑖^𝑘,𝑖=0) in computing process. If there are two coordinates (𝛼_𝑖^𝑘,𝑖 and 𝛼_𝑖^𝑘,𝑖) as the value of 0 or C in an iteration process, the analytical solution of the two subvariables updates coordinates without calculating. The reason is that the formula (17) guarantees 𝛼_𝑖^𝑘,𝑖+𝛼_𝑖^𝑘,𝑗=0 or 𝛼_𝑖^𝑘,𝑖+𝛼_𝑖^𝑘,𝑗=2𝐶, while double restricted box constrained optimization 0≤𝛼_𝑖^𝑘,𝑖≤𝐶 if the result is 0 or 𝐶. Constrained conditions will be edited ultimately as 0 or 𝐶. The constrained conditions are not updated.

Condition 2. If projected gradient is 0, the constrained conditions are not updated.

4.2. Dual Problem of Cost-Sensitive Support Vector Machine Using Randomized Dual Coordinate Descent Method and Its Complexity Analysis

From the above algorithm derivation of view, solving of CSVM seems to have been successful, but the computational complexity of the solving process is also larger. Assume that the average value of nonzero feature of each sample is 𝑛. Firstly, the computational complexity of the inner product matrix 𝑄_𝑖𝑖=𝑦_𝑖𝑥^𝑇_𝑖𝑥_𝑖𝑦_𝑖, 𝑖=1,2,…,𝑙 is 𝑂(𝑙𝑛), but the process can be operated in advance and stored into memory. Secondly, the calculation of (𝛼^𝑘,𝑖𝑄)_𝑖=∑^𝑙_𝑠=1𝑄_𝑖,𝑠𝛼_𝑖^𝑘,𝑠 takes the computational complexity 𝑂(𝑙𝑛). The amount of calculation is very great when the data size is large. However, there is a linear relationship model CSVM: 𝑤=^𝑙_𝑠=1𝛼_𝑖^𝑘,𝑠𝑥_𝑠𝑦_𝑠. (41)

Thus, (𝛼^𝑘,𝑖𝑄)_𝑖 is further simplified as follows: (𝛼^𝑘,𝑖𝑄)_𝑖=^𝑙_𝑠=1𝑄_𝑖,𝑠𝛼_𝑖^𝑘,𝑠=^𝑙_𝑠=1𝛼_𝑖^𝑘,𝑠𝑦_𝑠𝑥^𝑇_𝑠𝑥_𝑠𝑦_𝑠=𝑤^𝑇𝑥_𝑖𝑦_𝑖. (42)

Solving the corresponding formula (23) can be simplified as follows: 𝑑_𝑖=2(𝜶^𝑘,𝑖𝑄)_𝑖−1−2(𝜶^𝑘,𝑗𝑄)_𝑗−1𝑦_𝑖𝑦_𝑗2𝑄_𝑖𝑖+𝑄_𝑗𝑗=2𝑤^𝑇𝑥_𝑖𝑦_𝑖−1−2𝑤^𝑇𝑥_𝑗𝑦_𝑗−1𝑦_𝑖𝑦_𝑗2𝑄_𝑖𝑖+𝑄_𝑗𝑗. (43)

As can be seen, computing 𝑑_𝑖 with the complexity of 𝑂(𝑙𝑛) becomes computing 𝑤^𝑇𝑥_𝑖𝑦_𝑖 with the complexity of 𝑂(𝑛). Thus the calculation times reduce 𝑙−1. However, 𝑤=^𝑙_𝑠=1𝛼_𝑖^𝑘,𝑠𝑥_𝑠𝑦_𝑠, (44) where 𝑤 is the updated still computational complexity 𝑂(𝑙𝑛). The amount of calculation can be reduced significantly when 𝑤 is updated by changing 𝛼_𝑖^𝑘,𝑠. Let 𝛼_𝑖^𝑘,𝑖, 𝛼_𝑖^𝑘,𝑗 be the values of the current selection; 𝛼_𝑖^𝑘,𝑖+1, 𝛼_𝑖^𝑘,𝑗+1 are updated values, which can be updated via a simple way: 𝑤⟵𝑤+𝛼_𝑖^𝑘,𝑖−𝛼_𝑖^𝑘,𝑖+1𝑥_𝑖𝑦_𝑖+𝛼_𝑖^𝑘,𝑗−𝛼_𝑖^𝑘,𝑗+1𝑥_𝑗𝑦_𝑗. (45)

Its computational complexity only is 𝑂(𝑛). So, whether calculating 𝑑_𝑖 or updating 𝑤, coordinate gradient computation complexity is 𝑂(𝑛), which is one of the coordinate gradient method rapid convergence speed reasons.

When assigned an initial value and constraints based on ^𝑛_𝑖=1𝛼_𝑖𝑦_𝑖=0,0≤𝛼_𝑖≤𝛾𝐶,0≤𝛼_𝑗≤1−𝛾𝐶, (46) set the initial point 𝜶⁰=[…,𝛼_𝑖,…,𝛼_𝑗,…,]^𝑇_1×𝑙, (47) where 𝛼_𝑖=1/𝑡, 𝛼_𝑗=(1/(𝑙−𝑡))𝑙,𝑡 are the total number of samples and the number of positive samples, respectively. Thus the weight vector 𝑤 of the original problem is obtained by optimizing Lagrangian multipliers 𝜶 of the dual problem.

4.3. Description of Cost-Sensitive Support Vector Machine Using Randomized Dual Coordinate Descent Method

Accelerated conditions are judged by Section 4.1. One chooses the coordinate optimized number 𝑖,𝑗. We use formula (43) to calculate 𝑑_𝑖, formulas (39) and (40) to update 𝛼_𝑖^𝑘,𝑖, 𝛼_𝑖^𝑘,𝑗, and the formula (45) to update 𝑤, respectively. We can see that inner iteration takes 𝑂(𝑛) effort. The computer memory is mainly used to store samples information 𝑥₁,𝑥₂,…,𝑥_𝑙 and each sample point and their inner products 𝑄_𝑖𝑖=𝑦_𝑖𝑥^𝑇_𝑖𝑥_𝑖𝑦_𝑖. Cost-sensitive support vector machine using dual randomized coordinate gradient descent algorithm is described as follows.

Algorithm  3. Cost-sensitive support vector machine using randomized dual coordinate descent algorithm (CSVM-RDCD). Input: sample information {(𝑥_𝑖,𝑦_𝑖)}^𝑙_𝑖=1, 𝜶⁰. Output: 𝜶^𝑇𝑤. Initialize 𝜶⁰ and the corresponding 𝑤=∑^𝑙_𝑖=1𝛼_𝑖𝑥_𝑖𝑦_𝑖. For 𝑘=1,2,…,𝑇 do 𝜶^𝑘,1⟵𝜶^𝑘. (48)

Step 1. Randomly choose 𝑖,𝑗∈{1,2,…,𝑙}, 𝑖≠𝑗 and the corresponding 𝑦_𝑖, 𝑦_𝑗.

Step 2. If𝑦_𝑖=+1,𝑦_𝑗𝑈=−1₁=max0,𝛼_𝑖−𝛼_𝑗;𝑉₁=min𝛼_𝑖−𝛼_𝑗−;𝑦1−𝛾𝐶,𝛾𝐶if_𝑖=+1,𝑦_𝑗𝑈=+1₂=max0,𝛼_𝑖+𝛼_𝑗;𝑉−𝛾𝐶₂=min𝛼_𝑖+𝛼_𝑗;𝑦,𝛾𝐶if_𝑖=−1,𝑦_𝑗𝑈=−1₃=max0,−𝛼_𝑖−𝛼_𝑗−𝐶;𝑉1−𝛾₃=min−𝛼_𝑖−𝛼_𝑗,𝐶;𝑦1−𝛾if_𝑖=−1,𝑦_𝑗𝑈=+1₄=max0,−𝛼_𝑖−𝛼_𝑗;𝑉₄=min𝛾𝐶−𝛼_𝑖−𝛼_𝑗,1−𝛾𝐶;. (49)

Step 3. 𝛼^temp_𝑖=𝛼_𝑖^𝑘,𝑖+2𝑤^𝑇𝑥_𝑖𝑦_𝑖−1−2𝑤^𝑇𝑥_𝑗𝑦_𝑗−1𝑦_𝑖𝑦_𝑗2𝑄_𝑖𝑖+𝑄_𝑗𝑗. (50)

Step 4. if𝑦_𝑖=1,𝑦_𝑗𝑦=−1𝑚=1;if_𝑖=1,𝑦_𝑗𝑦=1𝑚=2;if_𝑖=−1,𝑦_𝑗𝑓𝑦=−1𝑚=3;_𝑖=−1,𝑦_𝑗𝛼=+1𝑚=4;_{𝑖𝑘,𝑖+1}=𝑉_𝑚,if𝛼^temp_𝑖>𝑉_𝑚𝛼^temp_𝑖,if𝑈_𝑚<𝛼^temp_𝑖<𝑉_𝑚𝑈_𝑚,if𝛼^temp_𝑖<𝑈_𝑚. (51)

Step 5. 𝛼_𝑖^𝑘,𝑗+1=𝑦_𝑗𝑦_𝑖𝛼_𝑖^𝑘,𝑖+𝑦_𝑗𝛼_𝑗^𝑘,𝑗−𝑦_𝑖𝛼_𝑖^𝑘,𝑖+1. (52)

Step 6. 𝑤⟵𝑤+𝛼_𝑖^𝑘,𝑖−𝛼_𝑖^𝑘,𝑖+1𝑥_𝑖𝑦_𝑖+𝛼_𝑖^𝑘,𝑗−𝛼_𝑖^𝑘,𝑗+1𝑥_𝑗𝑦_𝑗. (53) Until A stopping condition is satisfied, 𝜶^𝑘+1⟵𝜶^𝑘. (54) End for.

5. Experiments and Analysis

5.1. Experiment on Big Class-Imbalanced Benchmark Datasets Classification Problem

In this section, we analyze the performance of the proposed cost-sensitive support vector machine using randomized dual coordinate descent method (CSVM-RDCD). We compare our implementation with the state-of-the-art cost-sensitive support vector. Three implementations of related cost-sensitive SVMs are compared. We proposed the cost-sensitive SVM using randomized dual coordinate descent method (CSVM-RDCD) by modifying the LibSVM [8, 9] source code. Eitrich and Lang [10] proposed parallel cost-sensitive support vector machine (PCSVM). [6] proposed cost-sensitive support vector machines (CSSVM).

Table 1 lists the statistics of data sets. KDD99, Web Spam, Covertype, MNIST, SIAM1, and SIAM11 are obtained from the following data website. KDD99 is at http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html. Web Spam is at http://www.cc.gatech.edu/projects/doi/WebbSpamCorpus.html. Covertype is at http://archive.ics.uci.edu/ml/datasets/Covertype. MNIST is at http://yann.lecun.com/exdb/mnist/. SIAM is at https://c3.nasa.gov/dashlink/resources/138/.

Table 1 

Specification of the benchmark datasets. Number of ex. is the number of example data points. Number of feat. is the number of features. Ratio is the class imbalance ratio. Target specifies the target or positive class. Number is the number of the datasets.

To evaluate the performance of CSVM-RDCD method, we use a stratified selection to split each dataset to 9/10 training and 1/10 testing. We briefly describe each set below. For each dataset we choose the class with the higher cost or fewer data points as the target or positive class. All multiclass datasets were converted to binary data sets. In particular, the binary datasets SIAM1 and SIAM2 are datasets which have been constructed from the same multiclass dataset but with different target class and different imbalance ratios.

Evaluation of the performance of the three algorithms using CSVM was 10-fold cross-validation tested average misclassification cost, the training time, and the recognition rate of the positive class (i.e., the recognition rate of the minority class).

Average misclassification cost (AMC) represents 10-fold cross-validation average misclassification cost on test datasets for related CSVMs, described as follows: AMC=𝑓𝑝×𝐶𝛾+𝑓𝑛×𝐶1−𝛾𝑁, (55) where 𝑓𝑝 and 𝑓𝑛 represent the number of the positive examples misclassified as the negative examples in the test dataset and the number of the negative examples misclassified as the positive examples in the test data set, respectively. 𝐶𝛾 and 𝐶(1−𝛾) denote the cost of the positive examples misclassified as the negative examples in the test data set and the cost of the negative examples misclassified as the positive examples in the test data set, respectively. 𝑁 denotes the number of test examples.

The recognition rate of the positive class is the number of classified positive classes and the number of the positive classes on testing dataset.

Training time in seconds is used to evaluate the convergence speeding of three algorithms using CSVM on the same computer.

Cost-sensitive parameters of three algorithms using CSVM are specified as the following Table 2. Cost-sensitive parameters 𝐶𝛾 and 𝐶(1−𝛾) are valued according to the class ratio of datasets, namely, the class ratio of minority class and majority class.

Table 2 

Cost-sensitive parameters of three algorithms using CSVM on 6 datasets.

Three datasets with relative class imbalance are examined. Namely, KDD99 (intrusion detection), Web span, and MNIST datasets are considered. Three datasets with severe class imbalance are examined. Namely, Covertype, SIAM1, and SIAM11 datasets are considered. The average misclassification cost comparison of three algorithms using CSVM is shown in Figure 1 for each of the datasets. The CSVM-RDCD algorithm outperforms the PCSVM and CSVM on all datasets.

Figure 1 

Average misclassification cost comparison of three algorithms using CSVM on 6 datasets.

The recognition rate of positive class comparison of three algorithms using CSVM is shown in Figure 2 for each of the datasets. The CSVM-RDCD algorithm outperforms the PCSVM and CSSVM on all datasets, surpasses the PCSVM on four datasets, and ties with the PCSVM on two datasets.

Figure 2 

The positive class recognition rate comparison of three algorithms using CSVM.

We examine large datasets with relative imbalance ratios and severe imbalance ratios to evaluate the convergence speed of CSVM-RDCD algorithm. The training time comparison of three algorithms using CSVM is shown in Figure 3 for each of the datasets. The CSVM-RDCD algorithm outperforms the PCSVM and CSSVM on all datasets.

Figure 3 

Training time (s) comparison of three algorithms using CSSVM.

5.2. Experiment on Real-World Big Class-Imbalanced Dataset Classification Problems

In order to verify the effectiveness of the proposed algorithm CSVM-RDCD on real-world big class-imbalanced data classification problems, it was evaluated using the real vibration data measured in the wind turbine. The experimental data were from the SKF WindCon software and collected from a wind turbine gearbox type TF138-A [11]. The vibration signals were continuously acquired by an accelerometer mounted on the outer case of the gearbox.

All parameter settings for the dataset are listed in Table 3. The statistical results of the big class-imbalanced data problems that measure the quality of results (average misclassification cost, recognition rate of positive class, and training time) are listed in Table 4. From Table 4, it can be concluded that CSVM-RDCD is able to consistently achieve superior performance in the big class-imbalanced data classification problems.

Table 3 

Specification of the benchmark datasets and cost-sensitive parameters on the dataset.

Table 4 

Classification performance of three algorithms using CSVM on the dataset.

Experimental results show that it is applicable to solve cost-sensitive SVM dual problem using randomized dual coordinate descent method on the large-scale experimental data sets. The proposed method can achieve superior performance in the average misclassification cost, recognition rate of positive class, and training time. Large-scale experimental data sets show that cost-sensitive support vector machines using randomized dual coordinate descent method run more efficiently than both PCSVM and CSSVM; especially randomized dual coordinate descent algorithm has advantage of training time on large-scale data sets. CSSVM needs to build complex whole gradient and kernel matrix 𝑄 and needs to select the set of complex work in solving process of decomposition algorithm. Decomposition algorithm updates full uniform gradient information as a whole, the computational complexity 𝑂(𝑙𝑛) for the full gradient update. PCSVM also has similar computational complexity. Randomized dual coordinate gradient method updates linearly the coordinates, its computational complexity as 𝑂(𝑛), which increases considerably the convergence speed of the proposed method.

6. Conclusions

Randomized dual coordinate descentmethod (RDCD) is the optimization algorithm to update the global solution which is obtained by solving an analytical solution of the suboptimal problem. The RDCD method has the rapid convergence rate, which is mainly due to the following: (1) the subproblem has formal analytical solution, which is solved in solving process without complex numerical optimization; (2) the next component of RDCD method in solving process is updated on the basis of a previous component; compared with the full gradient information updated CSSVM method as a whole, the objective function of RDCD method can decline faster; (3) the single coordinate gradient calculation of RDCD method is simpler and easier than the full gradient calculation.

Randomized dual coordinate descent method is applied to cost-sensitive support vector machine, which expanded the scope of application of the randomized dual coordinate descent method. For large-scale class-imbalanced problem, a cost-sensitive SVM using randomized dual coordinate descent method is proposed. Experimental results and analysis show the effectiveness and feasibility of the proposed method.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

This work is partially supported by the National Science Fund for Distinguished Young Scholars of China (no. 61025015), the National Natural Science Foundation of China (nos. 51305046 and 61304019), the Research Foundation of Education Bureau of Hunan Province, China (no. 12A007), Open Fund of Hunan Province University Key Laboratory of Bridge Engineering (Changsha University of Science and Technology), the Key Laboratory of Renewable Energy Electric-Technology of Hunan Province (Changsha University of Science and Technology), the Key Laboratory of Efficient and Clean Energy Utilization, College of Hunan Province, and the Introduction of Talent Fund of Changsha University of Science and Technology.