Piecewise-Smooth Support Vector Machine for Classification

Abstract

Support vector machine (SVM) has been applied very successfully in a variety of classification systems. We attempt to solve the primal programming problems of SVM by converting them into smooth unconstrained minimization problems. In this paper, a new twice continuously differentiable piecewise-smooth function is proposed to approximate the plus function, and it issues a piecewise-smooth support vector machine (PWSSVM). The novel method can efficiently handle large-scale and high dimensional problems. The theoretical analysis demonstrates its advantages in efficiency and precision over other smooth functions. PWSSVM is solved using the fast Newton-Armijo algorithm. Experimental results are given to show the training speed and classification performance of our approach.

1. Introduction

In the last several years, support vector machine (SVM) has become one of the most promising learning machines because of its high generalization performance and wide applicability for classification, forecasting, and estimation in small-sample cases [1–6]. In addition, SVMs have surpassed the performance of artificial neural networks in many areas such as text categorization, speech recognition, and bioinformatics [7–11].

Basically, the main idea behind SVM is the construction of an optimal hyper plane, which has been used widely in classification [5, 8–16]. It can be formulated into an unconstrained optimization problem [17–21], but the objective function is nonsmooth. To overcome this disadvantage, Lee and Mangasarian used the integral of the sigmoid function to get a smooth SVM(SSVM) model in 2001 [17]. It is a very important and significant result to SVM since many famous algorithms can be used to solve it. In 2005, Yuan et al. proposed two polynomial functions, namely, the smooth quadratic polynomial function and the smooth forth polynomial function, and got QPSSVM and FPSSVM models [20, 21]. Xiong et al. derived an important recursive formula and a new class of smoothing functions using the technique of interpolation functions in [20]. In 2007, Yuan et al. used a three-order spline function to smooth the objective function of unconstrained optimization problem of SVM and obtained TSSVM model [21]. However, the efficiency or the precision of the algorithms was limited.

A natural problem is whether there is another smooth function to get a more efficient smooth SVM than existing works. In this paper, we introduce a piecewise function to smooth SVM and obtain a novel piecewise smooth support vector machine (PWSSVM). Theoretical analyses show that approximation accuracy of the piecewise smooth function to the plus function is higher than the available. Rough set theory is used to prove the global convergence of PWSSVM and the upper bound of convergence is proposed. The fast Newton-Armijo algorithm [22, 23] is employed to train the PWSSVM. Our new method is implemented in batches and can efficiently handle large-scale and high dimensional problems. Numerical experiments confirm the theoretical results and demonstrate that PWSSVM is more effective than the previous smooth support vector machine models.

The paper is organized as follows. In Section 2, we state the pattern classification and describe the PWSSVM. The approximation performance of smooth functions to the plus function is compared in Section 3. The convergence performance of PWSSVM is given in Section 4. The Newton-Armijo algorithm is applied to train PWSSVM in Section 5. Section 6 shows numerical comparisons. Finally, a brief conclusion of this paper is made.

In this paper, unless otherwise stated, all vectors are column vectors. For a vector 𝑥 in the 𝑛-dimensional real space 𝑅^𝑛, the plus function 𝑥₊ is defined as (𝑥₊)_𝑖=max(0,𝑥_𝑖), 𝑖=1,…,𝑛. The scalar (inner) product of two vectors 𝑥,𝑦 in the 𝑛-dimensional real space will be denoted by 𝑥⋅𝑦 and the 𝑝-norm will be denoted by ‖⋅‖_𝑝. For a matrix 𝐴∈𝑅^𝑚×𝑛, 𝐴_𝑖 is the 𝑖th row of 𝐴 which is a row vector in 𝑅^𝑛. A column vector of ones of 𝑛 dimension will be denoted by 𝑒. If 𝜑 is a real valued function defined in the 𝑛-dimensional real space 𝑅^𝑛, the gradient of 𝜑 is denoted by ∇𝜑(𝑥) which is a row vector in 𝑅^𝑛 and 𝑛×𝑛 the Hessian matrix of 𝜑 at 𝑥 is denoted by ∇²𝜑(𝑥).

2. Piecewise-Smooth Support Vector Machine

In this paper, let us consider a binary classification problem with 𝑚 training samples in the 𝑛-dimensional real space 𝑅^𝑛. It is represented by the 𝑚×𝑛 matrix 𝐴, according to membership of each point 𝐴_𝑖 in the class 1 or −1 as specified by a given 𝑚×𝑚 diagonal matrix 𝐷 with 1 or −1 along its diagonal. For this problem, the standard SVM with a linear kernel is given by the following quadratic program with parameter 𝜐>0min_{(𝑤,𝛾,𝑦)∈𝑅𝑛+1+𝑚}𝜐𝑒^Τ1𝑦+2𝑤^Τ𝑤s.t.𝐷𝐴𝑤−𝑒𝛾+𝑦≥𝑒𝑦≥0, (1) where 𝑒 is a vector of ones, 𝑤 is the normal to the bounding plane, and 𝑏 is the distance of the bounding plane to the origin. The linear separating plane is defined as follows: 𝑃=𝑥_𝑖∣𝑥_𝑖∈𝑅^𝑛,𝑤^Τ𝑥_𝑖=𝑏. (2) The first term in the objective function of (1) is the 1-norm of the slack variable 𝑦 with weight 𝜐. Replace the first term with the 2-norm vector 𝑦. Add (1/2)𝛾^Τ𝛾 to the objective function which induces strong convexity but has little or no effect on the problem. SVM model is replaced by the following problem: min_{(𝑤,𝛾,𝑦)∈𝑅𝑛+1+𝑚}𝜐2𝑦^Τ1𝑦+2𝑤^Τ𝑤+𝛾^Τ𝛾s.t.𝐷𝐴𝑤−𝑒𝛾+𝑦≥𝑒𝑦≥0. (3) Let 𝑦=(𝑒−𝐷(𝐴𝑤−𝑒𝛾))₊, where (⋅)₊ replaces negative components of a vector by zeros, then we can convert the SVM problem (3) into the following unconstrained optimization problem: min_{(𝑤,𝛾)∈𝑅𝑛+1}12𝜐𝑒−𝐷𝐴𝑤−𝑒𝛾₊²+12𝑤^Τ𝑤+𝛾^Τ𝛾. (4) This is a strongly convex minimization problem and it has a unique solution. Let max{0,𝑥}=𝑥₊. The function (𝑥)₊ is a continuous but nonsmooth function. Therefore, many optimization algorithms based on derivative and gradient cannot solve the problem (4) directly.

In 2001, Lee and Mangasarian [17] employed the integral of the sigmoid function 𝑝(𝑥,𝑘) to approximate the nondifferentiable function 𝑥₊ as follows: 𝑝()1𝑥,𝑘=𝑥+𝑘log1+𝜀^−𝑘𝑥,𝑘>0, (5) where 𝜀 is the base of natural logarithm and 𝑘 is a smoothing parameter. They got the SSVM model.

In 2005, Yuan et al. [18] presented two polynomial functions as follows: 𝑞()=1𝑥,𝑘𝑥,𝑥≥𝑘,𝑘4𝑥²+121𝑥+14𝑘,−𝑘1<𝑥<𝑘1,𝑘>0,0,𝑥≤−𝑘,ℎ()=1𝑥,𝑘𝑥,𝑥≥𝑘,−1()16𝑘𝑘𝑥+1³()1𝑘𝑥−3,−𝑘1<𝑥<𝑘1,𝑘>0,0,𝑥≤−𝑘. (6) Using the above smooth functions to proximate plus function 𝑥₊, they got two smooth polynomial support vector machine models (FPSSVM and QPSSVM). The authors also showed that FPSSVM and QPSSVM were more effective than SSVM in [18].

In 2007, Yuan et al. [21] presented a three-order spline function as follows: 𝑇()=1𝑥,𝑘0,𝑥<−𝑘,𝑘²6𝑥³+𝑘2𝑥²+121𝑥+16𝑘,−𝑘−𝑘≤𝑥<0,₂6𝑥³+𝑘2𝑥²+121𝑥+16𝑘,0≤𝑥≤𝑘,1𝑥,𝑥>𝑘. (7) They used the smooth function 𝑇(𝑥,𝑘) to approach the plus function and got a new smooth SVM model TSSVM. However, the efficiency or the precision of these algorithms above was limited.

In this paper, we propose a novel smooth function 𝜑(𝑥,𝑘) with smoothing parameter 𝑘>0 to approximate to the function 𝑥₊ as follows:𝜑()=1𝑥,𝑘0,𝑥<−,33𝑘2𝑘²1𝑥+3𝑘³1,−33𝑘≤𝑥<0,𝑥+2𝑘²13𝑘−𝑥³1,0≤𝑥≤,13𝑘𝑥,𝑥>.3𝑘 (8) The first- and second-order derivatives of 𝜑(𝑥,𝑘) are ()=1∇𝜑𝑥,𝑘0,𝑥<−,93𝑘2𝑘²1𝑥+3𝑘²1,−93𝑘≤𝑥<0,1−2𝑘²13𝑘−𝑥²1,0≤𝑥≤,13𝑘1,𝑥>,∇3𝑘₂𝜑()=1𝑥,𝑘0,𝑥<−,3𝑘9𝑘²1𝑥+13𝑘,−3𝑘≤𝑥<0,9𝑘²113𝑘−𝑥,0≤𝑥≤,13𝑘0,𝑥>.3𝑘 (9) The solution of the problem (3) can be obtained by solving the following smooth unconstrained optimization problem with the smoothing parameter 𝑘 approaching infinity as min_{𝑤,𝛾∈𝑅𝑛+1}Φ_𝑘=1𝑤,𝛾2𝜐𝜑𝑒−𝐷𝐴𝑤−𝑒𝛾,𝑘²+12𝑤^Τ𝑤+𝛾^Τ𝛾. (10) Thus, we develop a new smooth approximation for problem (3).

3. Approximation Performance Analysis of Smooth Functions

In this section, we will compare the approximation performance of smooth functions to plus function.

Lemma 1 (see [17]). 𝑝(𝑥,𝑘) is defined as the integral of the sigmoid function in [17], and 𝑥₊ is the plus function. The following conclusions can be obtained:(1)𝑝(𝑥,𝑘) is arbitrary rank smooth about 𝑥;(2)𝑝(𝑥,𝑘)≥𝑥₊;(3)for 𝜌>0,|𝑥|<𝜌, 𝑝(𝑥,𝑘)²−𝑥²₊≤(log2/𝑘)²+(2𝜌/𝑘)log2.

Lemma 2 (see [18]). 𝑞(𝑥,𝑘) and ℎ(𝑥,𝑘) are defined as (6), and 𝑥₊ is plus function. We can obtain the following conclusions:(1)𝑞(𝑥,𝑘) is one rank smooth about 𝑥, and ℎ(𝑥,𝑘) is twice rank smooth about 𝑥;(2)𝑞(𝑥,𝑘)≥𝑥₊ and ℎ(𝑥,𝑘)≥𝑥₊;(3)for any 𝑥∈𝑅, 𝑞(𝑥,𝑘)²−𝑥²₊≤1/11𝑘² and ℎ(𝑥,𝑘)²−𝑥²₊≤1/19𝑘².

Lemma 3 (see [19]). 𝑇(𝑥,𝑘) is defined as (7), and 𝑥₊ is plus function. The following results are easily obtained:(1)𝑇(𝑥,𝑘) is twice rank smooth about 𝑥;(2)𝑇(𝑥,𝑘)≥𝑥₊;(3)for any 𝑥∈𝑅, 𝑇(𝑥,𝑘)²−𝑥²₊≤1/24𝑘².

Theorem 4. The piecewise approximation function 𝜑(𝑥,𝑘) defined in (8) has the following properties:(1)𝜑(𝑥,𝑘) is twice rank smooth about 𝑥;(2)for any 𝑥∈𝑅, 𝜑(𝑥,𝑘)≥𝑥₊,(3)for any 𝑥∈𝑅, then 𝜑(𝑥,𝑘)²−𝑥²₊≤1/216𝑘².

Proof. (1) According to the formulas (8) and (9), one can easily obtain the results in (1).
(2) In the following, we verify the fact 𝜑(𝑥,𝑘)≥(𝑥)₊≥0. (i) The equation 𝜑(𝑥,𝑘)=(𝑥)₊ holds while 𝑥∈(−∞,−(1/3𝑘))∪(1/3𝑘,∞). (ii) Since 𝜑(𝑥,𝑘) is a monotone increasing function, we have the following result 𝜑(𝑥,𝑘)−(𝑥)₊=𝜑(𝑥,𝑘)≥𝜑(−(1/3𝑘),𝑘)=0 while 𝑥∈[−(1/3𝑘),0). (iii) For 𝑥∈[0,1/3𝑘], we have 𝜑(𝑥,𝑘)−(𝑥)₊=(3/2)𝑘²(1/3𝑘−𝑥)³≥0. Hence, we have the conclusion 𝜑(𝑥,𝑘)≥(𝑥)₊≥0.
(3) For 𝑥∈(−∞,−(1/3𝑘))∪(1/3𝑘,∞), 𝜑(𝑥,𝑘)²−𝑥²₊=0, the inequality in conclusion (3) is satisfied naturally.
For −(1/3𝑘)≤𝑥≤0, since 𝑥₊=0, 𝜑(𝑥,𝑘)²−𝑥²₊=𝜑(𝑥,𝑘)². Because 𝜑(𝑥,𝑘) is positive value, continuous, and increasing function for −(1/3𝑘)≤𝑥≤0, we have 𝜑(𝑥,𝑘)²≤𝜑(0,𝑘)²=1/324𝑘²≤1/216𝑘².
For 0<𝑥≤1/3𝑘, let 𝑠(𝑥)=𝜑(𝑥,𝑘)²−𝑥²₊=3𝑥+2𝑘²13𝑘−𝑥³²−𝑥²=94𝑘⁴1𝑥−3𝑘⁶−3𝑘²𝑥1𝑥−3𝑘³. (11) In order to obtain the result, making the variable substitution 𝑡=𝑘𝑥 (obviously 𝑡∈(0,1/3)), then we have 𝑠(𝑡)=(3/𝑘²)[(3/4)(𝑡−(1/3))⁶−𝑡(𝑡−(1/3))³]. For 𝑡∈(0,1/3), the maximum point of 𝑠(𝑡) is 𝑡=0.0605 and 𝑠(𝑡)=𝜑²(𝑥,𝑘)−(𝑥)²₊≤𝑔(0.0605)≈0.0046/𝑘²≤1/216𝑘².
In conclusion, we have 𝜑(𝑥,𝑘)²−𝑥²₊≤1/216𝑘².

Theorem 5. Let 𝜌=1/𝑘, and 𝑘>0. Consider the following.(1)If the smooth function is defined as (5), then by Lemma 1, we have 𝑝(𝑥,𝑘)²−𝑥²₊≤log2𝑘²+2𝜌𝑘=log2log²12+2log2𝑘²1≈0.6927𝑘². (12)(2)If the smooth functions are defined as (6), by Lemma 2, 𝑞(𝑥,𝑘)²−𝑥²₊≤111𝑘²1≈0.0909𝑘²,ℎ(𝑥,𝑘)²−𝑥²₊≤119𝑘²1≈0.0526𝑘². (13)(3)If the smooth function is defined as (7), by Lemma 3, 𝑇(𝑥,𝑘)²−𝑥²₊≤124𝑘²1≈0.0417𝑘². (14)(4)If the smooth function is defined as (8), by Theorem 4, 𝜑(𝑥,𝑘)²−𝑥²₊≤1216𝑘²1≈0.0046𝑘². (15)
Theorem 5 shows that the proposed piecewise smooth function 𝜑(𝑥,𝑘) achieves the best degree of approximation to the plus function 𝑥₊. When 𝑘 is fixed, it is easy to obtain the different smooth capability of the above smooth functions. The smooth performance comparison is given in Figure 1, where we set the smooth parameter 𝑘=10 and 𝜌=1/𝑘.

Figure 1 

Comparison of approximation performance of smooth functions (𝑘=10).

4. Convergence Performance of PWSSVM

In this section, the convergence of PWSSVM will be presented. By using rough set theory, we prove that the solution of PWSSVM can closely approximate the optimal solution of the original model (4) when 𝑘 goes to infinity. Furthermore, a formula for computing the upper bound of convergence is deduced.

Theorem 6. Let 𝐴∈𝑅^𝑚×𝑛 and 𝑏∈𝑅^𝑚×1. Define the real-valued functions in the 𝑛-dimensional real space 𝑅^𝑛 as follows: 𝑓(𝑥)=12(𝐴𝑥−𝑏)₊²₂+12‖𝑥‖²₂,𝑔()=1𝑥,𝑘2𝜑(𝐴𝑥−𝑏,𝑘)²₂+12‖𝑥‖²₂, (16) where 𝜑(⋅) is defined in (8), 𝑘>0. Then we have the following results:(1)𝑓(𝑥) and 𝑔(𝑥,𝑘) are strongly convex functions; (2)there exists a unique solution 𝑥^∗ to min_𝑥∈𝑅^𝑛𝑓(𝑥), and a unique solution 𝑥^∗_𝑘 to min_𝑥∈𝑅^𝑛𝑔(𝑥,𝑘);(3)for any 𝑘>0, 𝑥^∗_𝑘 and 𝑥^∗ satisfy the following condition: 𝑥^∗_𝑘−𝑥^∗2≤𝑚432𝑘²; (17)(4)lim_𝑘→∞‖𝑥^∗_𝑘−𝑥^∗‖=0.

Proof. (1) For any 𝑘>0, 𝑓(𝑥) and 𝑔(𝑥,𝑘) are strongly convex functions because ‖⋅‖² is strong convex function.
(2) Let 𝐿_𝑣(𝑓(𝑥)) be the level set of 𝑓(𝑥) and let 𝐿_𝑣(𝑔(𝑥,𝑘)) be the level set of 𝑔(𝑥,𝑘). Since 𝑥₊≤𝜑(𝑥,𝑘), it is easy to obtain 𝐿_𝑣(𝑔(𝑥,𝑘))⊂𝐿_𝑣(𝑓(𝑥))⊂{𝑥∣‖𝑥‖²₂≤2𝑣}. Therefore, 𝐿_𝑣(𝑓(𝑥)) and 𝐿_𝑣(𝑔(𝑥,𝑘)) are compact subsets in 𝑅^𝑛. Using the strong convexity property of 𝑓(𝑥) and 𝑔(𝑥,𝑘) for 𝑘>0, there is a unique solution to min_𝑥∈𝑅^𝑛𝑓(𝑥) and min_𝑥∈𝑅^𝑛𝑔(𝑥,𝑘), respectively.
(3) By using the first order optimization condition and considering convex property of 𝑓(𝑥) and 𝑔(𝑥,𝑘), we have 𝑓𝑥^∗_𝑘−𝑓𝑥^∗≥∇𝑓𝑥^∗𝑥^∗_𝑘−𝑥^∗+12𝑥^∗_𝑘−𝑥^∗2₂=12𝑥^∗_𝑘−𝑥^∗2₂,𝑔𝑥^∗,𝑘−𝑔𝑥^∗_𝑘,𝑘≥∇𝑔𝑥^∗_𝑘,𝑘𝑥^∗−𝑥^∗_𝑘+12𝑥^∗_𝑘−𝑥^∗2₂=12𝑥^∗_𝑘−𝑥^∗2₂. (18) Add the two formulas above and notice that 𝜑(𝑥,𝑘)≥𝑥₊, and then we have 𝑥^∗_𝑘−𝑥^∗2₂≤𝑓𝑥^∗_𝑘−𝑓𝑥^∗+𝑔𝑥^∗,𝑘−𝑔𝑥^∗_𝑘=𝑔𝑥,𝑘_∗,𝑘−𝑓𝑥^∗−𝑔𝑥^∗_𝑘,𝑘−𝑓𝑥^∗_𝑘≤𝑔𝑥^∗,𝑘−𝑓𝑥^∗=12𝜑𝐴𝑥^∗−𝑏,𝑘²₂−12𝐴𝑥^∗−𝑏₊²₂. (19) According to the third result of Theorem 4, we obtain the conclusion ‖𝑥^∗_𝑘−𝑥^∗‖²≤𝑚/432𝑘².
(4) According to ‖𝑥^∗_𝑘−𝑥^∗‖²≤𝑚/432𝑘², we have lim_𝑘→∞𝑥^∗_𝑘=𝑥^∗.

5. The Newton-Armijo Algorithm for PWSSVM

Following the results of the previous section, one can obtain the twice continuous differentiability of the objective function of problem (10). In order to take advantage of this feature, we use the Newton-Armijo method to train PWSSVM since it is a faster method than the BFGS algorithm [18, 19, 21]. The Newton-Armijo algorithm for problem (10) works as follows.

5.1. Newton-Armijo Algorithm

Step 1 (initialization). Start with any (𝑤⁰,𝛾⁰)∈𝑅^𝑛+1, 𝜏 and set 𝑖∶=0.

Step 2. Compute Φ^𝑖=Φ_𝑘(𝑤^𝑖,𝛾^𝑖) and 𝑔^𝑖=∇Φ_𝑘(𝑤^𝑖,𝛾^𝑖).

Step 3. If ‖𝑔^𝑖‖₂≤𝜏, then stop and accept (𝑤^𝑖,𝛾^𝑖). Otherwise, compute Newton direction 𝑑^𝑖∈𝑅^𝑛+1 from the following linear system: ∇²Φ𝑤^𝑖,𝛾^𝑖;𝑘^𝑖𝑑^𝑖=−𝑔^𝑖Τ, (20) where “Τ” denotes transpose symbol.

Step 4 (Armijo stepsize). Choose a stepsize 𝜆_𝑖=max{1,1/2,1/4,…} such that Φ_𝑘𝑤^𝑖,𝛾^𝑖−Φ_𝑘𝑤^𝑖,𝛾^𝑖+𝜆_𝑖𝑑^𝑖≥−𝜌𝜆_𝑖𝑔^𝑖𝑑^𝑖, (21) where 𝜌∈(0,1/2) and set 𝑤^𝑖+1,𝛾^𝑖+1=𝑤^𝑖,𝛾^𝑖+𝜆_𝑖𝑑^𝑖. (22)

Step 5. Replace 𝑖 by 𝑖+1 and go to Step 2.

We need to only solve a linear system of (20) instead of a quadratic program in our smooth approach. Because the objective function is strong convex, it is not difficult to obtain that our Newton-Armijo algorithm for training PWSSVM converges globally to the unique solution [17, 23]. Hence, the start point is not important. In this paper, we always set (𝑤⁰,𝛾⁰)=𝑒, where 𝑒 denotes a column vector of ones of 𝑛 dimension.

PWSSVM described above can solve the linear classification problems. In fact, we can extend some of the results in Section 2 to nonlinear PWSSVM with kernel technique as [17]. Furthermore, The Newton-Armijo algorithm can also solve nonlinear PWSSVM successfully.

6. Numerical Experiments

Newton-Armijo cannot be applied to QPSSVM model due to lack of the second-order derivative. In fact, the classification capacity of FPSSVM is slightly better than QPSSVM [18–21]. In our experiment, we do not compare QPSSVM with the other smooth SVM method. To demonstrate the effectiveness and speed of PWSSVM, we compare the performance numerically among SSVM, FPSSVM, TSSVM, and PWSSVM. The four smooth SVMs are all trained by the fast Newton-Armijo algorithm. All experiments are run on Personal Computer with 3.0 GHz and a maximum of 1.99 GB of the memory available. The programs of PWSVM, FPSSVM, and TSSVM are written in the MATLAB language. This computer runs Win7 with MATLAB 7.0.1. The source code of SSVM, ‘‘ssvm.m,’’ is obtained from the author’s website for the linear problem [24], and ‘‘lsvmk.m’’ for the nonlinear problem. In our experiments, all of the input data and the variables needed in programs are kept in the memory. For SSVM, TSSVM, FPSSVM, and PWSSVM, an optimality tolerance of 10⁻⁸ is used to determine when to terminate. Gaussian kernel is used in all our experiments.

The first experiment is used to demonstrate the capability of PWSSVM in solving larger problems. The results in Table 1 are designed to compare the training correctness, the testing correctness, and the training time among the four smooth SVMs on a massively sized dataset. The datasets are created using Musicants NDC Data Generator [25] with different sizes. The test samples are 5% of the training samples. The experiment results show that PWSSVM has the highest training accuracy and testing accuracy. Furthermore, PWSSVM can be used to solve problems more quickly than the other three smooth SVMs when the number of the sample data is relative small.

Table 1 

PWSSVM compared with SSVM, FPSSVM, and TSSVM on NDC generated dataset with difference sizes (𝐶=10).

The second experiment is designed to demonstrate the effectiveness of PWSSVM through the “tried and true” checkerboard dataset [26]. One highly nonlinearly separable but simple example is the checkerboard dataset which has often been used to show the effectiveness of nonlinear kernel methods [17]. The checkerboard dataset is generated by uniformly discretizing the regions [0,1]×[0,1] to 200²=40000 points and labeling two classes “White” and “Black” spaced by 3×3 grid as Figure 2 shows.

Figure 2 

The figure of the checkerboard dataset.

In the first trial of this experiment, the training set contains 1000 points randomly sampled from the checkerboard (for comparison, they are obtained from [17]) which contain 514 “white” samples and 486 “black” samples and the rest 39,000 points are in the testing set. Gaussian kernel function 𝐾(𝑥,𝑦)=exp(−0.5||𝑥−𝑦||²) is used and 𝐶=100. Total time for the 1000-point training set using PWSSVM with a Gaussian kernel is 4.01 s. The train accuracy of PWSSVM is 99.80%. The test accuracy of PWSSVM is 98.76% on a 39,000-point test set. TSSVM solves the same problem within 4.23 s, and the train accuracy and the test accuracy are 99.62% and 98.28%. FPSSVM and SSVM obtain the train accuracy of 99.60% within 4.35 s and 4.61 s, respectively. The test accuracy of them are 98.28%.

The rest results are presented in Table 2. The training set is randomly selected from the checkerboard with different sizes. The remaining samples are used as test samples. We compare the classification results of PWSSVM, TSSVM, FPSSVM, and SSVM with the same Gaussian kernel function. The results in Table 2 demonstrate that PWSSVM can solve massive problems quickly, followed by TSSVM, FPSVM, and SSVM in turn. The experimental results show that PWSSVM can obtain the highest train precision and test precision.

Table 2 

PWSSVM compared with SSVM, FPSSVM, and TSSVM on checkerboard dataset with difference sizes (𝐶=100).

7. Conclusions

A novel PWSSVM is proposed in this paper. It only needs to find the unique minima of the unconstrained differentiable convex quadratic function. The proposed method has many advantages over those available, such as good classification performance and less training time cost. The numerical results show that PWSSVM has excellent generalization ability.

Acknowledgments

The authors would like to thank the anonymous reviewers for their valuable comments. This work was supported in part by the National Natural Science Foundation of China under Grants 61100165, 61100231, and 51205309 and the Natural Science Foundation of Shaanxi Province (2010JQ8004, 2012JQ8044).