Smooth Diagonal Weighted Newton Support Vector Machine

Abstract

Based on diagonal weighted support vector machine, a smooth model with Newton algorithm is proposed and is called SDWNSVM for short. SDWNSVM introduces the entropy function to approximate the plus function of the slack in the diagonal weighted SVM and is thus different from traditional SSVM that treats a reformulation problem. SDWNSVM utilizes the dual technique to rewrite the objection function by the connotative relation between the primal and dual program, which induces an exact smooth program and differs from traditional SSVM that uses Lagrangian multipliers to roughly substitute for the hyperplane weight. SDWNSVM proves the equivalence between the obtained model and the original one and proposes Newton algorithm to figure out the optimal solution. Numerical experiments on UCI data demonstrate that SDWNSVM has higher accuracies and less iteration than existing methods.

1. Introduction

Based on statistical learning theory and dual programming, support vector machine (SVM) has been developed by Vapnik as an efficient method for small-sample data [1–3] and has gained wide application in classification and regression area. When the sample points are completely separable, the hard margin classifiers are introduced which can separate the positive class from the negative class with the maximal margin. When the sample points are nonseparable, the soft margin classifiers are introduced, which allow erroneousness of the misclassified points. Both 1-norm and 2-norm soft margin SVM are proposed, while the former is called box constrained SVM and the latter is called diagonal weighted SVM.

Denote the nonnegative constraints in the plus function form for SVM error Lee and Mangasarian propose smooth support vector machine (SSVM) [4, 5], which has strong mathematical properties, such as strong convexity and infinitely often differentiability. SSVM applied smoothing techniques to one reformulation of the 2-norm soft margin SVM by appending an additional bias term. The approach was originally proposed to induce strong convexity and was later used in [6–8]; this was equivalent to adding a constant feature to the training data and finding a separating hyperplane passing through the origin, which had been proved to have little or no effect on the solutions to the original program. Based on SSVM, many scholars research from various aspects and presented diverse kinds of smooth models [9–20]. Some scholars proposed polynomial function based smooth approach [12–15], quarter penalty function based smooth approach [16], or rational function based smooth approach [10] and some scholars generalize to the regression or forecasting area [11, 17]. Among these methods, the widely used models are the fourth polynomial function based smooth approach called FPSSVM; however, the accuracies are limited. Wu and wang proposed a piecewise-smooth support vector machine PWSSVM for classification [9], which uses a twice continuously differentiable piecewise polynomial function to approximate the plus function and get satisfying results. A detailed study shows that the above smooth models have two drawbacks. (1) The 2-norm of the weight of the separating hyperplane is substituted for that of the Lagrangian multipliers vector. (2) The equivalences of the obtained programs and the original program are proved in none of these methods.

A natural problem exists whether an exact model can be constructed for SVM and whether the proof can be given with regard to the equivalence between the obtained program and the original one. This paper investigates a smooth diagonal weighted newton support vector machine (SDWNSVM), which applies entropy function to the diagonal weighted support vector machine and proves the equivalence between the transformed program and the original one, by making use of the connotative relation between the primal and dual program. We begin from the linear space to obtain the unconstrained smooth model by directly adopting the entropy penalty function. As for the kernel space, we use the dual technique to rewrite the objection. Upon obtaining the smooth models, Newton algorithm is utilized to figure out the solution to SDWNSVM. Numerical experiments are carried out to demonstrate the effectiveness and superiority.

This paper is organized as follows. Section 2 derives the exact smooth model in the linear space. Section 3 proposes the two smooth models in the kernel space with the key transformations illustrated. Section 4 proposes Newton algorithm to determine the solutions to SDWNSVM. Numerical experiments are given in Section 5 to demonstrate the performances. The conclusions are made in the final section.

2. Linear Smooth Model

Given is the training set 𝑇={(𝑥₁,𝑦₁),(𝑥₂,𝑦₂),…,(𝑥_𝑙,𝑦_𝑙), in which 𝑥_𝑖∈𝑅^𝑛 and 𝑦_𝑖∈{1,−1}. The training of standard diagonal weighted SVM equals the following program: 1min2𝑤^𝑇𝐶𝑤+2^𝑙_𝑖∑₌₁𝜉_𝑖²,s.t.𝑦_𝑖𝑤^𝑇𝑥_𝑖+𝑏≥1−𝜉_𝑖,𝑖=1,…,𝑙,𝜉_𝑖≥0. (1)

Here 𝑤∈𝑅^𝑛 is the normal to the separating hyperplane, 𝑏∈𝑅 is the bias, and 𝜉∈𝑅^𝑙 is the misclassified error. The linear separating hyperplane is 𝑤^𝑇𝑥+𝑏=0, with 𝑥∈𝑅^𝑛 as the input point.

Denote the training set in the matrix form by 𝐴_𝑙×𝑛=[𝑥₁,𝑥₂,…,𝑥_𝑖,…,𝑥_𝑙]^𝑇, and denote the labels 𝑦_𝑖∈{1,−1}(𝑖=1,…,𝑙) by the diagonal matrix 𝐷_𝑙×𝑙 with ones or minus ones along its diagonal. Having these notations, we can rewrite (1) as 1min2𝑤^𝑇𝐶𝑤+2𝜉^𝑇()𝜉,s.t.𝐷𝐴𝑤+𝑏𝑒≥𝑒−𝜉,𝜉≥0. (2)

For 𝑟∈𝑅, we define the plus function as 𝑟₊=max(𝑟,0) and denote the entropy function with smooth parameter 𝛽>0𝑃_𝛽(𝑟)=𝑟+𝛽⁻¹ln1+exp−𝛽𝑟. (3)

Lemma 1. 𝑃_𝛽(𝑟) is a strict convex function.

Proof. Figuring out the first order derivative for 𝑃_𝛽(𝑟) with respect to variable 𝑟 gives the following equation, which guarantees that Lemma 1 holds: 𝑃_𝛽^′(𝑟)=1+−𝛽exp−𝛽𝑟𝛽1+exp−𝛽𝑟>0. (4)

Lemma 2. For any 𝑟∈𝑅 and |𝑟|<𝜌∶𝑃_𝛽(𝑟)²−(𝑟₊)²≤(ln2/𝛽)²+(2𝜌/𝛽)ln2 holds, where 𝑃_𝛽(𝑟) is defined as in (3) with smooth parameter 𝛽>0.

Proof. For 0<𝑟<𝜌, we have 𝑃_𝛽(𝑟)²−𝑟₊²=1𝛽²ln²1+𝜀^−𝛽𝑟+2𝑟𝛽ln1+𝜀^−𝛽𝑟≤ln2𝛽²+2𝜌𝛽ln2. (5)
For −𝜌<𝑟<0, 𝑃_𝛽(𝑟) is a monotonically increasing function, and 𝑃_𝛽(𝑟)²−𝑟₊²=𝑃_𝛽(𝑟)²≤𝑃_𝛽(0)²=ln2𝛽². (6)
Thus 𝑃_𝛽(𝑟)²−(𝑟₊)²≤(ln2/𝛽)²+(2𝜌/𝛽)ln2 holds, which completes the proof.

Corollary 3. The entropy penalty function converges to the plus function; that is, lim_𝛽→∞𝑃_𝛽(𝑟)=𝑟₊=max(𝑟,0).
By figuring out the limit of 𝑃_𝛽(𝑟) as 𝛽 approaches infinity, it is easy to show that Corollary 3 holds.

Corollary 3 guarantees that the entropy function can be used to approximate the plus function. It should be pointed out that, if we generalize 𝑟∈𝑅 to 𝑟=(𝑟₁,𝑟₂,𝑟₃,…,𝑟_𝑛)∈𝑅^𝑛, the element 𝑟_𝑖∈𝑅 will still satisfy the corresponding results of Lemmas 1 and 2 and Corollary 3, and Lemma 2 becomes Lemma 4.

Lemma 4. For any vector 𝑟=(𝑟₁,𝑟₂,𝑟₃,…,𝑟_𝑛)∈𝑅^𝑛 with |max_{1≤𝑖≤𝑛}(𝑟_𝑖)|<𝜌, the inequality |||𝑃_𝛽(𝑟)||²−||𝑟₊||²|≤(ln2/𝛽)²+(2𝜌/𝛽)ln2 holds, where 𝑃_𝛽(𝑟) is defined as in (3) with parameter 𝛽>0.
Write the slack vector in the plus function form: 𝜉=max(𝑒−𝐷(𝐴𝑤+𝑏𝑒),0)=(𝑒−𝐷(𝐴𝑤+𝑏𝑒))₊. (7)
Then one converts the diagonal weighted SVM into the equivalent unconstrained formulation: min𝐹_𝛽𝑤𝑏=12𝑤^𝑇𝐶𝑤+2𝑃_𝛽(𝑒−𝐷(𝐴𝑤+𝑏𝑒))². (8)
For any vector 𝛾=(𝛾₁,𝛾₂,𝛾₃,…,𝛾_𝑛)∈𝑅^𝑛, the diagonal matrix is defined as follows with the elements of 𝛾 along the diagonal: 𝛾=𝛾Diag¹⋱𝛾_𝑛, (9)
Now one figures out the gradient vector ∇𝐹_𝛽(^𝑤_𝑏) and the Hessian matrix ∇²𝐹_𝛽(^𝑤_𝑏). For the sake of simplicity, five new vectors are introduced; namely, 𝑡=𝑒−𝐷(𝐴𝑤−𝑒𝛾),𝑣=exp(−𝛽𝑡),𝑀=(𝐼+𝛽Diag(𝑣))⋅Diag(𝑃_𝛽(𝑡)),𝑄=𝐷𝐴, and 𝑌=𝐷𝑒: ∇𝐹_𝛽𝑤𝑏=𝑤0+𝐶⋅−𝑄^𝑇−𝑌^𝑇Diag𝑃_𝛽(𝑡)Diag(𝑒+𝑣)⁻¹∇⋅𝑒,₂𝐹_𝛽𝑤𝑏=𝐼000+𝐶⋅−𝑄^𝑇−𝑌^𝑇Diag(𝑒+𝑣)⁻²𝑀−𝑄−𝑌. (10)(11)
Following from (10) and (11), one concludes that 𝐹_𝛽(^𝑤_𝑏) is convex and strict convex. Theorem 5 guarantees the existences of the global minima for the convex functions.

Theorem 5. Suppose 𝑆⊂𝑅^𝑛 is a nonempty convex set and 𝑓∶𝑆→𝑅 is a convex function; then the local minima of 𝑓on 𝑆 are also global minima.

Proof. Respectively, denote by 𝑥∈𝑆 and 𝑦∈𝑆 the local and global minimal point of the function 𝑓; then 𝑓(𝑦)<𝑓(𝑥) and the following inequality holds for any real number 𝜆∈[0,1]: 𝑓𝜆𝑦+(1−𝜆)𝑥≤𝜆𝑓𝑦+(1−𝜆)𝑓(𝑥)<𝑓(𝑥). (12)
It is noted that lim_𝜆→0⁺𝜆𝑦+(1−𝜆)𝑥=𝑥; for arbitrary 𝜀>0 there exists 𝜆₀ sufficiently small such that 𝜆₀𝑦+1−𝜆₀𝑥∈𝑈₀(𝑥,𝜀). (13)
Here 𝑈₀(𝑥,𝜀) is a small hollow region with 𝑥 as the center and 𝜀 as the radius.
Take inequality (12) and (13) into account; we directly derive the following inequality, which implies that 𝑥 is not a local minimal and thus contradicts the hypothesis: 𝑓𝜆₀𝑦+1−𝜆₀𝑥<𝑓(𝑥). (14) Now we prove Theorem 5.

Theorem 6. Let 𝐴=𝐴_𝑙×𝑛 and 𝑏∈𝑅. Define the two programs for the real valued functions 𝐹(^𝑤_𝑏) and 𝐹_𝛽(^𝑤_𝑏) in 𝑅^𝑛+1 as follows: 𝑤𝑏=1min𝐹2𝑤^𝑇𝐶𝑤+2(𝑒−𝐷(𝐴𝑤+𝑏𝑒))₊²,min𝐹_𝛽𝑤𝑏=12𝑤^𝑇𝐶𝑤+2𝑃_𝛽(𝑒−𝐷(𝐴𝑤+𝑏𝑒))², (15)(16) where 𝛽>0 is the smooth parameter; then(1)there, respectively, exist global minima of min𝐹(^𝑤_𝑏) and min𝐹_𝛽(^𝑤_𝑏);(2)let 𝑤𝑏 and 𝑤_𝛽𝑏_𝛽 be the global minima of min𝐹_𝛽(^𝑤_𝑏) and min𝐹(^𝑤_𝑏); the following upper bound inequality holds with 𝜌=max_{1≤𝑖≤𝑙}|(𝑒−𝐷(𝐴𝑤+𝑏𝑒))_𝑖|: 𝑤_𝛽−𝑤²+𝑏_𝛽−𝑏²≤𝑙ln2𝛽²+2𝜌𝛽ln2. (17)
Thus, 𝑤_𝛽𝑏_𝛽 converges to 𝑤𝑏 as 𝛽 approaches infinity.

Proof. (1) Firstly, we prove the existence of local minima.
Define the level set of 𝑓 as 𝐿_𝜇(𝑓)={𝑥∣𝑓(𝑥)≤𝜇} for any real number 𝜇∈𝑅.
Since 𝑃_𝛽(𝑥) dominates the plus function 𝑥₊(𝑥₊≤𝑃_𝛽(𝑥)), the two level sets satisfy inequality 𝐹_𝛽(𝑥)−𝐹(𝑥)≥0, and the two level sets satisfy the following for 𝑟∈𝑅: 𝐿_𝑉𝐹_𝛽(𝑥)⊆𝐿_𝑉(𝐹(𝑥))⊆𝑥∣‖𝑥‖²₂≤2𝑟, (18) Hence, both 𝐿_𝑉(𝐹(𝑥)) and 𝐿_𝑉(𝐹_𝛽(𝑥)) are compact subsets in 𝑅^𝑛+1.
It is easy to prove the convexity of 𝐹(^𝑤_𝑏) and the strong convexity of 𝐹_𝛽(^𝑤_𝑏). Theorem 5 guarantees that there, respectively, exist global minima.
(2) Secondly, we establish the convergence.
Utilizing the first order optimality conditions of the convex function 𝐹(^𝑤_𝑏) and the strict convex function 𝐹_𝛽(^𝑤_𝑏), 𝐹𝑤_𝛽𝑏_𝛽−𝐹𝑤𝑏≥∇𝐹𝑤𝑏𝑤_𝛽−𝑤,𝑏_𝛽−𝑏𝐹=0,_𝛽𝑤𝑏−𝐹_𝛽𝑤_𝛽𝑏_𝛽≥∇𝐹_𝛽𝑤_𝛽𝑏_𝛽𝑤−𝑤_𝛽,𝑏−𝑏_𝛽+12𝑤−𝑤_𝛽,𝑏−𝑏_𝛽²₂=12𝑤−𝑤_𝛽,𝑏−𝑏_𝛽²₂. (19)
Adding up the above two inequalities, we derive the following inequality: 12𝑤_𝛽−𝑤,𝑏_𝛽−𝑏²₂≤𝐹_𝛽𝑤𝑏−𝐹𝑤𝑏−𝐹_𝛽𝑤_𝛽𝑏_𝛽−𝐹𝑤_𝛽𝑏_𝛽≤𝐹_𝛽𝑤𝑏−𝐹𝑤𝑏=12𝑃_𝛽𝑒−𝐷𝐴𝑤+𝑏𝑒²₂−12(𝑒−𝐷(𝐴𝑤+𝑏𝑒))₊²₂. (20) Define 𝛿=max_{1≤𝑖≤𝑙}|(𝑒−𝐷(𝐴𝑤+𝑏𝑒))_𝑖|; we apply Lemma 2 to the right of and obtain the following inequality, which testifies the convergence: 𝑤_𝛽−𝑤²+𝑏_𝛽−𝑏²≤𝑙ln2𝛽²+2𝜌𝛽ln2. (21)

3. Kernel Smooth Model

In the kernel space, the following program is minimized, where 𝜙(𝑥) is the nonlinear map: 1min2𝑤^𝑇𝐶𝑤+2^𝑙_𝑖∑₌₁𝜉_𝑖²,s.t.𝑦_𝑖𝑤^𝑇𝜙𝑥_𝑖+𝑏≥1−𝜉_𝑖𝜉,𝑖=1,…,𝑙,_𝑖≥0. (22)

3.1. Rough Kernel Smooth Model

Similar to SSVM, the 2-norm of the Lagrangian column vector 𝛼=(𝛼₁,𝛼₂,…,𝛼_𝑙)^𝑇∈𝑅^𝑙 is used to replace that of the weight of the hyperplane. The matrix formation for kernel diagonal weighted SVM is as follows: 1min2𝛼^𝑇𝐶𝛼+2𝜉^𝑇𝐾𝜉,s.t.𝐷𝐴,𝐴^𝑇𝐷𝛼+𝑏𝑒≥𝑒−𝜉,𝜉≥0. (23)

The unconstrained program in the plus function form and the smooth form for the kernel 2-norm soft margin SVM is, respectively, illustrated as follows: 𝛼𝑏=1min𝐹2𝛼^𝑇𝐶𝛼+2𝑒−𝐷𝐾𝐴,𝐴^𝑇𝐷𝛼+𝑏𝑒₊²,min𝐹_𝛽𝛼𝑏=12𝛼^𝑇𝐶𝛼+2𝑃_𝛽𝑒−𝐷𝐾𝐴,𝐴^𝑇𝐷𝛼+𝑏𝑒². (24)(25)

As for the gradient vector and Hessian matrix, we define 𝑡=𝑒−𝐷(𝐾(𝐴,𝐴^𝑇)𝛼+𝑏𝑒) and 𝑄=𝐷𝐾(𝐴,𝐴^𝑇)𝐷; the gradient vector ∇𝐹_𝛽(𝑥) and the Hessian matrix ∇²𝐹_𝛽(𝑥) are computed according to (26) and (27), where 𝑣,𝑀, and 𝑌 are the same as defined in Section 2: ∇𝐹_𝛽𝛼𝑏=𝛼0+𝐶−𝑄^𝑇−𝑌^𝑇Diag𝑃_𝛽(𝑡)Diag(𝑒+𝑣)⁻¹∇⋅𝑒,₂𝐹_𝛽𝛼𝑏=𝐼000+𝐶−𝑄^𝑇−𝑌^𝑇Diag(𝑒+𝑣)⁻²𝑀−𝑄−𝑌. (26)(27)

Theorem 7. Let 𝐴=𝐴_𝑙×𝑛 and 𝑏∈𝑅. For the two programs 𝐹(^𝛼_𝑏) and 𝐹_𝛽(^𝛼_𝑏) in (24) and (25),(1)there, respectively, exist global minima of min𝐹(^𝛼_𝑏) and min𝐹_𝛽(^𝛼_𝑏).(2)let 𝛼𝑏 and 𝛼_𝛽𝑏_𝛽 be the global minima of min𝐹_𝛽(^𝛼_𝑏) and min𝐹(^𝛼_𝑏) one has the following inequality with 𝜌=max_{1≤𝑖≤𝑙}|(𝑒−𝐷(𝐾(𝐴,𝐴^𝑇)𝐷𝛼+𝑏𝑒))_𝑖|: 𝛼_𝛽−𝛼²+𝑏_𝛽−𝑏²≤𝑙ln2𝛽²+2𝜌𝛽ln2. (28)
Thus, 𝛼_𝛽𝑏_𝛽 converges to 𝛼𝑏 as 𝛽 approaches infinity.

Theorem 7 can be proved similarly as Theorem 6; thus its detailed process is omitted.

3.2. Exact Kernel Smooth Model

Now we will propose the exact smooth model in the kernel space.

Since the nonlinear map 𝜙(𝑥) is unknown, the connotative conditions between the primal program and the dual one are exploited.

It is known that at the optimal solution 𝑤=^𝑙_𝑖=1𝑦_𝑖𝛼_𝑖𝜙𝑥_𝑖. (29)

Using this relation, we have the following two equations, in which 𝑄_𝑖𝑗=𝑦_𝑖𝑦_𝑗𝐾(𝑥_𝑖,𝑥_𝑗): 𝑦_𝑖𝑤^𝑇𝜙𝑥_𝑖=^𝑙_𝑗=1𝑦_𝑖𝑦_𝑗𝛼_𝑗𝜙𝑥_𝑗^𝑇𝜙𝑥_𝑖=^𝑙_𝑗=1𝑄_𝑖𝑗𝛼_𝑗=(𝑄𝛼)_𝑖,𝑤^𝑇𝑤=^𝑙_𝑖=1𝑦_𝑖𝛼_𝑖𝜙𝑥_𝑖^𝑇𝑦_𝑗𝛼_𝑗𝜙𝑥_𝑗=^𝑙_𝑖=1𝛼_𝑖(𝑄𝛼)_𝑖=𝛼^𝑇𝑄𝛼. (30)

Put the above two equations back into (22); we get the following program: 1min2𝛼^𝑇𝐶𝑄𝛼+2𝜉^𝑇𝜉,s.t.𝑄𝛼+𝑏𝑦≥𝑒−𝜉,𝜉≥0. (31)

Though (31) is different from (22), the below theorem will prove that, for any optimal solution 𝛼^∗ to (31), 𝑤^∗=∑^𝑙_𝑖=1𝑦_𝑖𝛼^∗_𝑖𝜙(𝑥_𝑖) is also an optimal solution to (22).

Theorem 8. Let (𝛼^∗,𝑏^∗,𝜉^∗) be the optimal solution to program (31); then (𝑤^∗,𝑏^∗,𝜉^∗) is the optimal solution to program (22), where 𝑤^∗ is defined as 𝑤^∗=∑^𝑙_𝑖=1𝑦_𝑖𝛼^∗_𝑖𝜙(𝑥_𝑖).

Proof. Evidently, if (𝑤,𝑏,𝜉) and (𝛼,𝑏,𝜉) are the primal and dual optimal solutions, then (𝛼,𝑏,𝜉) is feasible for (31).
For feasible (𝑤,𝑏,𝜉) and (𝛼,𝑏,𝜉), inequality (31) can be deduced from (29): 12𝑤^𝑇𝐶𝑤+2𝜉^𝑇𝜉=12𝛼^𝑇𝐶𝑄𝛼+2𝜉^𝑇𝜉≥12𝛼^∗𝑇𝑄𝛼^∗+𝐶2𝜉^∗𝑇𝜉^∗=12𝑤^∗𝑇𝑤^∗+𝐶2𝜉^∗𝑇𝜉^∗. (32)
Thus, (𝑤^∗,𝑏^∗,𝜉^∗) is the optimal solution to (31).
From the constraint in (31), we write the slack 𝜉 in the form of plus function as 𝜉=𝑒−𝑄𝛼−𝑏𝑦₊=𝑒−𝐷𝐾𝐴,𝐴^𝑇𝐷𝛼+𝑏𝑒₊. (33)
Then we obtain the model in the plus function form and the model in the entropy function form in the kernel space: 𝛼𝑏=1min𝐹2𝛼^𝑇𝐷𝐾𝐴,𝐴^𝑇+𝐶𝐷𝛼2𝑒−𝐷𝐾𝐴,𝐴^𝑇𝐷𝛼+𝑏𝑒₊²min𝐹_𝛽𝛼𝑏=12𝛼^𝑇𝐷𝐾𝐴,𝐴^𝑇+𝐶𝐷𝛼2𝑃_𝛽𝑒−𝐷𝐾𝐴,𝐴^𝑇𝐷𝛼+𝑏𝑒². (34)(35)

The gradient vector and the Hessian matrix of (35) can be computed as follows, where 𝑡 and 𝑄 are the same as defined in Section 3.1, while 𝑣, 𝑀, and 𝑌 are the same as defined in Section 2: ∇𝐹_𝛽𝛼𝑏=0𝑄𝛼+𝐶⋅−𝑄^𝑇−𝑌^𝑇⋅Diag𝑃_𝛽(𝑡)⋅Diag(𝑒+𝑣)⁻¹∇⋅𝑒,₂𝐹_𝛽𝛼𝑏=𝑄000+𝐶⋅−𝑄^𝑇−𝑌^𝑇⋅Diag(𝑒+𝑣)⁻²⋅𝐼+𝛽Diag(𝑣)⋅Diag𝑃_𝛽(𝑡)⋅−𝑄−𝑌. (36)(37)

Theorem 9. Let 𝑄=𝑄_𝑙×𝑙(𝑄=𝐷𝐾(𝐴,𝐴^𝑇)𝐷), 𝛼∈𝑅^𝑙, and 𝑏∈𝑅. For the two programs 𝐹(^𝛼_𝑏) and 𝐹_𝛽(^𝛼_𝑏) in (36) and (37),(1)there, respectively, exist global minima of min𝐹(^𝛼_𝑏) and min𝐹_𝛽(^𝛼_𝑏).(2)let 𝛼𝑏 and 𝛼_𝛽𝑏_𝛽 be the global minima of min𝐹_𝛽(^𝛼_𝑏) and min𝐹(^𝛼_𝑏), the following inequality holds with 𝜌=max_{1≤𝑖≤𝑙}|(𝑒−𝐷(𝐾(𝐴,𝐴^𝑇)𝐷𝛼+𝑏𝑒))_𝑖|; 𝑄^1/2𝛼−𝛼_𝛽²₂+𝑏−𝑏_𝛽²≤𝑙ln2𝛽²+2𝜌𝛽ln2. (38)

Corollary 10. As 𝛽 approaches infinity, the unique solution 𝛼_𝛽𝑏_𝛽 to min𝐹_𝛽(^𝛼_𝑏) converges to the unique solution 𝛼𝑏 to min𝐹(^𝛼_𝑏); that is, lim_𝛽→∞^𝛼𝛽_𝑏𝛽=^𝛼_𝑏.

4. Implementation

In this section, we will first prove the existences of the unique solutions to the smooth models (16), (25), and (35) and next propose Newton algorithm to figure out solutions to the smooth models of diagonal weighted SVM.

Theorem 11. Programs (16), (25), and (35) have unique solutions.

Proof. We define 𝑡=𝑒−(𝐷𝐴𝑤−𝐷𝑏𝑒) in the linear space and 𝑡=𝑒−𝐷(𝐾(𝐴,𝐴^𝑇)𝛼−𝑏𝑒) in the kernel space.
Let 𝑣=exp(−𝛽𝑡),𝑀=(𝐼+𝛽Diag(𝑣)⋅Diag(𝑃_𝛽(𝑡)); then for any column vector 𝛾∈𝑅^𝑛, the inequalities Diag(𝑒+𝑣)⁻²𝑀>0 and 𝛾^𝑇⋅Diag(𝑒+𝑣)⁻²𝑀⋅𝛾>0 hold.
Using the results for ∇²𝐹_𝛽(𝑥) in (11), (27), and (37), we know that ∇²𝐹_𝛽(𝑥)>0 holds, which guarantee that programs (16), (25) and (35) are convex and they all have unique solutions. Now, we complete the proof.

SDWNSVM Algorithm

Step 0 (initialization). Input training set 𝑇={(𝑥₁,𝑦₁),(𝑥₂,𝑦₂),…,(𝑥_𝑙,𝑦_𝑙)}, the initial point ^𝛼0_𝑏0∈𝑅^𝑙+1, and 𝜀>0. Calculate the gradient vector ∇𝐹_𝛽^𝛼0_𝑏0 and the Hessian matrix ∇²𝐹_𝛽^𝛼0_𝑏0. Set 𝑖=0.

Step 1 (termination). Having 𝛼_𝑖𝑏_𝑖, stop if the gradient vector is zero, that is, ||∇𝐹_𝛽^𝛼𝑖_𝑏𝑖||<𝜀; else, go to step 2.

Step 2 (newton direction). Determine the direction 𝑑^𝑖∈𝑅^𝑙+1 by setting the linearization of 𝐹_𝛽(^𝛼_𝑏) around 𝛼_𝑖𝑏_𝑖 equal to zero, which gives 𝑙+1 linear equations in 𝑙+1 variables: ∇²𝐹_𝛽𝛼_𝑖𝑏_𝑖𝑑^𝑖=−∇𝐹_𝛽𝛼_𝑖𝑏_𝑖. (39)

Step 3 (Armijo stepsize). Find the search direction 𝜆_𝑖 using Armijo stepsize, that is, choose 𝜆_𝑖=max{1,1/2,1/4,…}, such that for 𝛿∈(0,1/2), 𝐹_𝛽𝛼_𝑖𝑏_𝑖𝛼−𝐹^𝑖𝑏_𝑖+𝜆_𝑖𝑑^𝑖≥−𝛿𝜆_𝑖∇𝐹_𝛽𝛼_𝑖𝑏_𝑖𝑑^𝑖. (40)

Step 4 (update). Update the point according to 𝛼_𝑖+1𝑏_𝑖+1=𝛼_𝑖𝑏_𝑖+𝜆_𝑖𝑑^𝑖. (41)

Compute the gradient vector ∇𝐹_𝛽^𝛼𝑖+1_𝑏𝑖+1. Let 𝑘∶=𝑘+1. Go to step 1.

Here we are solving a linear system of 𝑙+1 equations in SDWNSVM instead of solving a quadratic program, which is the case for classical SVM.

Comments. Although the presented SDWNSVM looks similar to SSVM, they are different in nature. Their main differences can be summarized into two points.(1)The unconstrained programs have different convexities in SDWNSVM and SSVM. The program 𝐹(^𝛼_𝑏) is convex in SDWNSVM, while it is strict convex in SSVM.(2)The distances between 𝛼_𝛽𝑏_𝛽 and 𝛼𝑏 have different upper bounds equality in SDWNSVM and SSVM. By the first order optimality condition of the strict convex function, SSVM has the upper bound inequality as 𝛼_𝛽−𝛼²+𝑏_𝛽−𝑏²≤𝑙2ln2𝛽²+2𝜌𝛽ln2. (42)

Obviously, with regard to the distance between 𝛼_𝛽𝑏_𝛽 and 𝛼𝑏, the upper bound of the rough kernel smooth model in SDWNSVM is twice that in SSVM, while the upper bound of the exact kernel smooth model in SDWNSVM is absolutely different form that in SSVM, as illustrated in  (38)

5. Experiments and Comparisons

We demonstrate now the effectiveness and speed of SDWNSVM on several UCI datasets. All the experiments are carried out on a PC with P4 CPU, 3.06 GHz, and 1 GB memory. The programs are written in pure MATALAB 7.01 language. The parameters are chosen for the optimal performances on a tuning set by the ten-fold cross validation.

5.1. Parameter Setting

In the linear space, only the penalty 𝐶 needs to be specified. In the kernel space, the penalty 𝐶 and the kernel width 𝜎 need to be specified with respect to the radial basis kernel function 𝐾(𝑥,𝑦)=exp(−||𝑥−𝑦||²/𝜎²).

The detailed procedure of ten-fold cross validation consists of the following three steps.(1)Setting the Parameter-Ranges. Specify penalty 𝐶∈{0.01,0.1,0.5,0.8,1,5,10,50,100}, and specify 𝜎∈{0.01,0.02,0.05,0.1,0.5,1,10,50,100} as the kernel width.(2)Estimating the Performance. Select 𝐶 or (𝜎,𝐶) from the parameter-ranges, run ten-fold cross validation to train a classifier, and test its performances. Firstly, divide the training set into ten subsets; secondly, select nine subsets as the training set to construct a classifier; finally, take the one subset left as the tuning set to test the generalization performances of the obtained classifier.(3)Selecting the Optimal Parameters. Select the parameters as the optimal parameters, which are corresponding to the highest ten-fold testing accuracies.

Obviously, the training and testing procedures need to be repeated ten times. The ten-fold training accuracy is defined as the averaged accuracy obtained on the nine training subsets, while the ten-fold testing accuracy is defined as the averaged accuracy obtained on the tuning set.

We carry out the experiments on the ionosphere data with 𝐶=5 in the linear space and illustrate performances of the ten-fold cross validation for SSVM and SDWNSVM.

Data in Table 1 lead to the following conclusions.(1)SSVM and SDWNSVM both have satisfying training and testing accuracies. The former has the training and testing accuracy of about 93.86% and 89.48%, while the latter has the training and testing accuracy of about 94.21% and 89.19%.(2)SSVM and SDWNSVM both have low iteration number and training time. The former has an iteration number of about 7.6 and a training time of about 0.14 while the latter has an iteration number of about 5 and a training time of about 0.12.(3)SSVM and SDWNSVM both have small variances. SSVM has the training and testing accuracy variance of about 0.0062 and 0.2844, while SDWNSVM has the training and testing accuracy variance of about 0.0044 and 0.2803. SSVM has an iteration number variance of about 0.2667 and a training time variance of about 0.0032, while SDWNSVM has an iteration number variance of 0 and a training time variance of about 0.0031.(4)The differences are not statistically significant in the ten-fold cross validation.

Table 1 

Performances of the ten-fold cross validation.

The differences among the ten folds are in Table 2

Table 2 

Differences among the ten different folds.

Data in Table 2 bring about the following observations, which demonstrate that the differences among the ten different folds are trivial.(1)The differences of the training and testing accuracies are mostly not larger than 0.03.(2)The differences of the iterations are mostly about 1 in SSVM and 0 in SDWNSVM.(3)The differences of the training time are mostly not larger than 0.03.

Tables 1 and 2 demonstrate that the ten-fold cross validation has satisfying training and testing accuracies; also the differences among the ten different folds are trivial and they are not statistically significant.

5.2. Performances in the Linear and Kernel Space

The experiment is carried out in the liner space on several datasets. SDWNSVM is compared with RLP, SVM_||1||, SVM_||2||, and FSV in terms of ten-fold training and testing accuracies. The results are illustrated in Table 3 with the penalty parameter specified by 𝐶∈{0.01,0.1,0.5,0.8,1,3,5,8,10}.

Table 3 

Comparisons with various algorithms.

Obviously, SDWNSVM has the highest ten-fold training and testing correctness.

SDWNSVM is also compared with SSVM, SOR (successive over relaxation), and SVM^light on the large-scale adult data in Figure 1, whose training sizes vary from 1605 to 32562. The linear kernel is used and the penalty parameter is selected from 𝐶∈{0.1,0.3,0.5,0.8,1,5,10,50,100} by the ten-fold cross validation.

Figure 1 

Training accuracies of SDWNSVM with various algorithms.

Evidently, SDWNSVM has the highest ten-fold training accuracies.

SDWNSVM is compared with SSVM on the Bupa Liver and Pima Indians data. The former data set is composed of 241 “train” examples and 104 “test” examples, containing five attributes for each sample. The latter Pima Indians diabetes data is composed of 500 plus examples and 268 minus examples, eight attributes for each example. The results are illustrated in Table 4 with parameters selected by ten fold cross validation with 𝐶∈{0.1,0.5,1,5,10,15,30,50,80,100} and 𝜎∈{0.01,0.05,0.3,0.5,0.8,1,1.2,3,5,8}.

Table 4 

Comparisons with various algorithms.

The following conclusions are directly drawn from Table 4.(1)The iterations of SDWNSVM are identical to those of SSVM.(2)The training time of SDWNSVM remains the same as that of SSVM in linear space, but a little higher than that of SSVM in the kernel space.(3)The training and testing accuracies of SDWNSVM are higher than those of SSVM. In the linear space, SDWNSVM improves the training and testing accuracies by about 1.42% and 2.11%. In the kernel space, SDWNSVM improves the testing accuracies by about 0.96%.

To further demonstrate the superiority of SDWNSVM, we will compare SDWNSVM with earlier SSVM algorithm, as well as the currently frequently used algorithms with good performance, namely, PWSSVM and FPSSVM. The training time, the training and testing accuracies are selected as the assessment indexes.

The “tried and true” checkerboard data set is used, which is generated by uniformly discrediting the regions [0,1]×[0,1] to 100²=10000 points and labeling two classes “White” and “Black.” For the sake of comparisons, the same parameters are erected for various algorithms. The Gaussian kernel is used; the kernel width is set to be 𝜎=0.5 and the penalty is set to be 𝐶=1. The training set is randomly sampling with increasing sizes while the others are selected as the testing set. The averaged results are presented in Table 5 of ten random samplings.

Table 5 

Comparisons with various algorithms.

It is obvious that SDWNSVM has the highest training and testing accuracies with the lowest training time among the four different algorithms.

The final experiment is designed to show the accuracies and computational advantages over datasets with larger feature.

The NDC data generator is used to produce data of various dimensional data with different scales. Randomly choose 20% as the training set and choose the others as the testing set. Select the Gaussian kernel; the averaged performances of ten random samplings are illustrated in Table 6 for the four algorithms under the same parameters with the kernel width 𝜎=0.1 and the penalty 𝐶=1.

Table 6 

Comparisons with various algorithms.

From data in Table 6, we draw the following conclusions.(1)The training and testing accuracies increase with the training scales but decrease with the dimension by a certain degree for the four algorithms.(2)For various scale and various dimensional data, SDWNSVM has the highest training and testing accuracies but has the lowest training time among the four different algorithms. Take 200 dimensional data of 10000, for example; SDWNSVM has a training accuracy, respectively, about 1.14%, 1.18%, and 0.48% higher than FPSSVM, FPSSVM, and PWSSVM, while it has a testing accuracy respectively about 0.4%, 0.54%, and 0.17% higher than SSVM, FPSSVM, and PWSSVM. SDWNSVM has a training time respectively about 20.44, 58.24, and 8.05 seconds lower than SSVM, FPSSVM, and PWSSVM.

6. Conclusion

This paper deals with diagonal weighted SVM and introduces smoothing technique to obtain an efficient training algorithm SDWNSVM. By using the connotative relation found by dual technique, SDWNSVM can be easily extended to the kernel space and has higher training accuracies; what is more, SDWNSVM proves the equivalence between the new smooth model and the original one. By introducing Newton algorithm to figure out the optimal solution, SDWNSVM has lower training time than existing methods. Numerical experiments on several UCI data demonstrate the effectiveness and superiority. Future work includes finding other smooth penalty functions or searching new efficient algorithms to figure out the optimal solution for the unconstrained smooth model.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

The work is supported by the Natural Science Foundation of Shaanxi Educational Commission (2010JK773), the Research Fund of the Doctoral Program of Xi’an Shiyou University (YS29030903), and the Natural Science Foundation of China (no. 60974082).