A Novel Sparse Least Squares Support Vector Machines

Abstract

The solution of a Least Squares Support Vector Machine (LS-SVM) suffers from the problem of nonsparseness. The Forward Least Squares Approximation (FLSA) is a greedy approximation algorithm with a least-squares loss function. This paper proposes a new Support Vector Machine for which the FLSA is the training algorithm—the Forward Least Squares Approximation SVM (FLSA-SVM). A major novelty of this new FLSA-SVM is that the number of support vectors is the regularization parameter for tuning the tradeoff between the generalization ability and the training cost. The FLSA-SVMs can also detect the linear dependencies in vectors of the input Gramian matrix. These attributes together contribute to its extreme sparseness. Experiments on benchmark datasets are presented which show that, compared to various SVM algorithms, the FLSA-SVM is extremely compact, while maintaining a competitive generalization ability.

1. Introduction

The last decade has seen widespread applications of Least Squares Support Vector Machines (LS-SVM) [1, 2] to a variety of classification problems. The LS-SVM involves finding a separating hyperplane of maximal margin and minimizing the empirical risk via a Least Squares loss function. Here the optimization is subject to equality constraints, as opposed to the inequality ones used with a standard SVM [3, 4]. Thus the LS-SVM successfully sidesteps the quadratic programming (QP) required for the training of the standard SVM. As a result, an LS-SVM classifier is equivalent to a set of linear algebraic equations, which are usually solved by the conjugate gradient (CG) method [5]. Empirical studies suggest that an LS-SVM possesses very competitive generalization abilities compared to a regular SVM [6]; improvement has been made to Suyken’s algorithm by Chu et al. [7] whose proposal is also based on CG method but with a reduced time complexity. Meanwhile Keerthi et al. advocated training an LS-SVM using the sequential minimal optimization (SMO) algorithm [8].

While these algorithms have indeed made an LS-SVM more computationally attractive, the nonsparseness of its solution still remains a major, and as yet unsolved, bottleneck. Sparseness in solutions is essential for reducing the time required in predicting the class membership of unlabelled data. The general approach to addressing this issue for an LS-SVM is iterative shrinking of the training set. Here the importance of training samples is evaluated from the weights assigned to them after training. Samples of less importance are removed, and then the remainder forms a reduced training set to be learnt again. In view of the linear relationship between the approximation error and the support value for a training sample, a straightforward pruning strategy is to remove samples whose absolute support values are trivial [9]. A later paper recommended pruning a sample which introduces the smallest error if omitted [10]. However, this necessitates the inversion of the kernel matrix, which is indicative of high computation complexity. Zeng and Chen [11] suggested a pruning method based on the SMO formulation of an LS-SVM, which leads to faster retraining. Other authors [12] proposed solving the dual form of an LS-SVM by iterative addition of basis functions from the kernel dictionary either until all the training samples are traversed or until the approximation error is lower than a preset threshold. Despite the lower computation cost with the backfitting scheme adopted, Lagrangian multipliers associated with previously selected training samples were inherited into the next pass, which could compromise the generalization abilities of the resultant LS-SVM.

This paper presents a new LS-SVM formulation called the Forward Least Squares Approximation SVMs (FLSA-SVM). It is trained by Forward Least Squares Approximation(FLSA) [13], a function approximation method using a Least Squares loss function. The FLSA-SVM loss function decreases monotonically with an increasing number of support vectors, allowing for the removal of slack variables from the equality constraints. Another novelty with the FLSA-SVM lies in the fact that it cleverly transforms the number of support vectors to be the regularization parameter, which indicates the tradeoff between generalization abilities and empirical risk. The FLSA-SVM builds a classifier by iteratively selecting a single basis function for the solution, which contributes the largest reduction to the quadratic cost function.

The paper is organized as follows. Section 2 briefly reviews Least Squares SVM principles. The new sparse LS-SVM—Forward Least Squares Approximation SVM (FLSA-SVM)—is introduced in Section 3. Experimental results are given in Section 4 and concluding remarks in Section 5.

2. Least Squares Support Vector Machines

For a classification problem of ℓ pairs of training samples (𝐱₁,𝑦₁),…,(𝐱_ℓ,𝑦_ℓ), where 𝐱∈ℝ^𝑁 and 𝑦∈{−1,1}, LS-SVM algorithms seek the optimal separating hyperplane with the orientation vector of the least 𝐿₂ norm.

To ensure the presence of the optimal hyperplane, the input data are translated into a reproducing kernel hilbert space (RKHS) by a mapping function denoted by 𝜙(𝐱). To avoid the curse of dimensionality, the mapping is implemented implicitly by the introduction of “kernel trick.” Its idea is that dot products in the RKHS space can be represented by a Mercer kernel function in input space [14]: 𝐾𝐱,𝐱^′=Φ(𝐱)^⊤Φ𝐱^′. (1) Thus the linear discriminant function in the feature space can be formulated as 𝑓(𝐱)=𝐰^⊤𝜙(𝐱)+𝑏, (2) where 𝐰 is the orientation vector and 𝑏 is the bias term.

An LS-SVM finds the hyperplane parameterized by (𝐰,𝑏) by solving the following optimization problem [15]: min_{𝐰,𝑏,𝐞}12𝐰^⊤1𝐰+2𝛾^ℓ_𝑖=1𝑒²_𝑖,s.t.𝐰^⊤𝜙𝐱_𝑖+𝑏=𝑦_𝑖−𝑒_𝑖,𝑖=1,…,ℓ, (3) where the slack variable 𝑒_𝑖 denotes the deviation between the actual output 𝑓(𝐱) of the LS-SVM on sample 𝐱_𝑖 and its target value 𝑦_𝑖. 𝛾 is a parameter which imposes penalty on deviations.

Introducing the Lagrange multipliers 𝛼_𝑖  (𝑖=1,…,𝑙) for each of the equality constraints gives ℒ()=1𝐰,𝑏,𝐞,𝛼2𝐰^⊤1𝐰+2𝛾^ℓ_𝑖=1𝑒²_𝑖−^ℓ_𝑖=1𝛼_𝑖𝐰^⊤𝜙𝐱_𝑖+𝑏−𝑦_𝑖+𝑒_𝑖. (4)

Due to the equality constraints, 𝛼_𝑖 can either be positive or negative according to the Karush-Kuhn-Tucker (KKT) conditions [16]: 𝐰=^ℓ_𝑖=1𝛼_𝑖𝜙𝐱_𝑖,_ℓ𝑖=1𝛼_𝑖𝛼=0,_𝑖=𝛾𝑒_𝑖,𝐰^⊤𝜙𝐱_𝑖+𝑏=𝑦_𝑖−𝑒_𝑖. (5)(6)(7)(8)

The linear equations can be further simplified to 𝐇+𝛾⁻¹𝐈𝟏𝟏^⊤0𝛼𝑏=𝐲0, (9) where 𝐇∈ℝ^ℓ×ℓ and 𝐇_𝑖𝑗=𝐾(𝐱_𝑖,𝐱_𝑗),  𝐲=[𝑦₁,…,𝑦_ℓ]^⊤,  𝜶=[𝛼₁,…,𝛼_ℓ]^⊤, 𝐈 is unity matrix of rank 𝑛, and 𝟏 is a column vector of 1s of 𝑛 length. The solution of an LS-SVM is a linear combination of basis function 𝐾(⋅,𝐱_𝑖) whose associated 𝛼_𝑖 is nonzero: 𝑓(𝐱)=_𝑖𝛼_𝑖𝐾𝐱,𝐱_𝑖+𝑏. (10)

It has been noted that the LS-SVM is almost equivalent to ridge regression [17] since the two methods corresponds to an identical optimization problem. Meanwhile, the equivalence between the LS-SVM and the Kernel Fisher Discriminant method [18] has also been established [19].

3. Least Squares Approximation Sparse SVM

Equation (3) shows that an LS-SVM can also be viewed as a ridge regression model. And the optimality conditions (7) indicate that the introduction of slack variable 𝑒_𝑖 is the root of the nonsparseness problem, since in regression applications, most slack variables end as nonzero [20]. But the introduction of 𝑒_𝑖 seems inevitable for the representation of training cost and thus the penalty parameter to indicate the tradeoff between training cost and generalization abilities.

The section introduces a new formulation of Least Squares SVM, namely, Forward Least Squares Approximation SVM (FLSA-SVM). The proposed FLSA-SVM algorithm is motivated by the Least Squares “Forward Approximation & Backward Refinement” method for function approximation, which was first developed in the dynamic system identification community by Li et al.

FLSA-SVM is rid of slack variables by employing an approximation function of Least Squares loss function which is Forward Least Squares Approximation (FLSA) algorithm. FLSA approximation function iteratively selects a basis function which causes the steepest reduction in the loss function. The features enable the number of support vectors (SVs), which equals to the number of basis functions, to be the tradeoff between training cost and generalization abilities. In FLSA-SVMs, the sparseness of its solution is ensured and confirmed by the experiments section in which FLSA-SVMs are more sparse than standard SVMs.

In this section, a description of  “Forward Approximation” is given, from a machine learning perspective, as a preface to the introduction of LSA-SVM algorithm. The strategy to reduce the computation complexity of FLSA-SVMs is then presented.

3.1. Forward Least Squares Approximation [13]

Given ℓ values (𝑦₁,…,𝑦_ℓ) of an unknown target function 𝑓 in a Hilbert space at ℓ input data (𝐱₁,…,𝐱_𝑙). A “dictionary” {𝜑₁,…,𝜑_𝑁} of 𝑁 functions in the Hilbert space is also given in which 𝜑_𝑖  (𝑖=1,…,𝑁) is termed as a “basis function.” Forward Least Squares Approximation (FLSA) addresses the estimation of 𝑓 with a linear combinations of 𝑚 basis functions chosen from the dictionary: 𝑓_𝑚=^𝑚_𝑖=1𝜃_𝑖𝜙_𝑖, (11) where 𝑚is the number of basis functions which expand 𝑓_𝑚,  {𝜙₁,…,𝜙_𝑚} is the set of selected basis functions, and (𝜃₁,…,𝜃_𝑚)∈ℝ^𝑚 is the associated weight vector,

so that the squared norm of the residue vector, denoted by 𝐿, is minimized: 𝐿=𝑓_𝑚−𝐲^⊤𝑓_𝑚−𝐲, (12) where 𝐲=(𝑦₁,…,𝑦_ℓ) and 𝑓_𝑚=(𝑓_𝑚(𝐱₁),…,𝑓_𝑚(𝐱_ℓ))^⊤ is the output vector of 𝑓_𝑚 on the ℓ input data: 𝑓_𝑚=^𝑚_𝑖=1𝜃_𝑖𝜙_𝑖=ΩΘ, (13) where 𝜙_𝑖=(𝜙_𝑖(𝐱₁),…,𝜙_𝑖(𝐱_ℓ))^⊤ is the vector of decision values of basis function 𝜙_𝑖 on the input data and Ω=[𝜙₁,…,𝜙_𝑚]∈ℝ^ℓ×𝑚. Equation (13) suggests that FLSA can be understood as working entirely in the ℝ^ℓ space.

The weight vector Θ that minimizes the loss function (12) is given as Θ=Ω^⊤Ω⁻¹Ω^⊤𝐲 (14) for a matrix Ω which is of full column rank [21]. The resultant minimal loss function is 𝐿(Θ)=𝐲^⊤𝐲−Θ^⊤Ω^⊤𝐲=𝐲^⊤𝐈−ΩΩ^⊤Ω)⁻¹Ω^⊤𝐲. (15) Starting from (15), FLSA defines a set of residue matrices 𝐑_𝑖∈ℝ^ℓ×ℓ  (𝑖=1,…,𝑚) for the measurement of the contribution of each basis function to the reduction of loss function: 𝐑_𝑖=𝐈−Ω_𝑖Ω^⊤_𝑖Ω_𝑖⁻¹Ω^⊤_𝑖, (16) where matrix Ω_𝑖=(𝜙₁,…,𝜙_𝑖) is full column rank and is composed of output vectors of 𝑖 basis function. 𝐈 is the unity matrix and set 𝐑₀=𝐈. Then, the following equation holds: (1)𝐑_𝑖𝜙_𝑗(2)𝐑=01≤𝑗≤𝑖,_⊤𝑖=𝐑_𝑖,𝐑²_𝑖=𝐑_𝑖. (17)(18)

For a 𝜑_𝑖=(𝜑_𝑖(𝐱₁),…,𝜑_𝑖(𝐱_ℓ))^⊤  (1≤𝑘≤𝑁) which produces a full column rank (Ω_𝑖,𝜑_𝑘), it is proved that 𝐑_𝑖+1 has the following properties: (3)𝐑_𝑖+1=𝐑_𝑖−𝐑_𝑖𝜑_𝑘𝜑^⊤_𝑘𝐑^⊤_𝑖𝜑^⊤_𝑘𝐑_𝑖𝜑_𝑘,(4)𝐑_𝑖𝐑_𝑖+1=𝐑_𝑖+1𝐑_𝑖=𝐑_𝑖+1. (19)(20)

Meanwhile, the introduction of 𝐑_𝑖 simplifies the formulation of 𝐿_𝑖 which is the evaluation of loss function 𝐿 at Ω_𝑖: 𝐿_𝑖=𝐲^⊤𝐑_𝑖𝐲. (21) And thus 𝐿_𝑖+1=𝐲^⊤𝐑_𝑖+1𝐲=𝐿_𝑖−𝐲^⊤𝐑_𝑖𝜑_𝑘𝜑^⊤_𝑘𝐑^⊤_𝑖𝐲𝜑^⊤_𝑘𝐑_𝑖𝜑_𝑘. (22)

The contribution, denoted by 𝛿𝐿_𝑘, of a column vector 𝜑_𝑘 makes the loss function 𝐿_𝑖 can able to be explicitly expressed: 𝛿𝐿_𝑘=𝐲^⊤𝐑_𝑖𝜑_𝑘𝜑^⊤_𝑘𝐑^⊤_𝑖𝐲𝜑^⊤_𝑘𝐑_𝑖𝜑_𝑘=𝐑_𝑖𝜑_𝑘^⊤𝐑_𝑖𝐲²𝐑_𝑖𝜑_𝑘^⊤𝐑_𝑖𝜑_𝑘. (23)

FLSA algorithm proceeds in a greedy manner which selects one basis function per iteration. The 𝑖th iteration identifies the index of the 𝑖th basis function 𝜙_𝑖 by solving the optimization problem: argmax_{𝑘=1,…,𝑁}𝛿𝐿_𝑘=𝜑_𝑘^{(𝑖−1)⊤}𝐲^(𝑖−1)2𝜑_𝑘^{(𝑖−1)⊤}𝜑_𝑘^(𝑖−1), (24) where 𝜑_𝑘^(𝑖−1)=𝐑_𝑖−1𝜑_𝑘 and 𝐲^(𝑖−1)=𝐑_𝑖−1𝐲.

The 𝑖th iteration of FLSA algorithm, in actual fact, establishes the following linear system whose solutions are the last 𝑚−𝑖+1 elements of 𝜃 in (13): 𝜙_𝑖^(𝑖−1)𝜙^(𝑖−1)_𝑖+1⋯𝜙_𝑚^(𝑖−1)𝜃_𝑖𝜃_𝑖+1⋮𝜃_𝑚=𝐲^(𝑖−1). (25)

Thus [𝜙_𝑖^(𝑖−1)𝜑_𝑗^(𝑖−1),𝜙_𝑖^(𝑖−1)𝐲^(𝑖−1)]  (𝜑_𝑗∉{𝜙₁,…,𝜙_𝑖}) which represents a linear equation is stored for the 𝑖th iteration. Eventually, 𝑚 iterations build up an upper triangle which represents a linear system of (26). The solution Θ can be computed by performing a back substitution procedure that is used by a typical Gaussian elimination process: 𝜙^⊤₁𝜙₁𝜙^⊤₁𝜙₂…𝜙^⊤₁𝜙_𝑚0𝜙₂^(1)⊤𝜙₂⁽¹⁾…𝜙₂^(1)⊤𝜙_𝑚⁽¹⁾𝜙⋮⋮⋱⋮00…_𝑚^{(𝑚−1)⊤}𝜙_𝑚^(𝑚−1)𝜃₁𝜃₂⋮𝜃_𝑚=𝜙^⊤₁𝐲𝜙₂^(1)⊤𝐲⁽¹⁾⋮𝜙_𝑚^{(𝑚−1)⊤}𝐲^(𝑚−1). (26)

The FLSA algorithm, in fact, is closely related to the Orthogonal Least Squares (OLS) method [22–24], which also allows for the explicit formulation of the contribution of a basis function made to the reduction of the squared error loss.

3.2. Forward Least Squares Approximation SVMs

As with standard SVMs, the formulation of LS-SVMs embodies the Structural Risk Minimization (SRM) principle which can be illustrated by Figure 1, in which the dotted line represents the upper bound on the complexity term of function set from which solution is chosen and the dash line the empirical risk. SRM minimizes the upper bound on the expected risk (generalization error), for which the best tradeoff between the complexity term and the empirical risk is required to be found.

Figure 1 

Structural risk minimization (SRM) principle.

To this end, a regularization term is introduced to indicate the tradeoff between the complexity term and the empirical risk, both of which LS-SVMs explicitly formulate. Model selection is performed in the domain of ℝ>0 in search for its optimal value.

While in FLSA algorithm, it can be concluded from (22) that the training cost 𝐿 monotonously decreases to the increase in the number of selected basis functions 𝑚. The training cost 𝐿 can be then plotted against 𝑚 into a curve similar to that of the empirical risk in Figure 1.

This fact motivates the following two innovations to traditional formulation of LS-SVMs: (1) the employment of the parameter 𝑚 as the regularization term; (2) the avoidance of the term to represent empirical risk by using the FLSA as the training algorithm, which minimizes the summed squared residues for any value of 𝑚. As a result, a new formulation of LS-SVMs—namely, Forward Least Squares Approximation Sparse SVM (FLSA-SVM), which is restricted to be trained by FLSA algorithm, is proposed: min_𝐰,𝑏12𝐰^⊤𝐰+𝑏²,s.t.𝐰^⊤𝜙𝐱_𝑖+𝑏=𝑦_𝑖|Γ|,𝑖=1,…,ℓ,=𝑚, (27)(28)(29) where Γ=𝛾₁,…,𝛾_𝑚 is composed of the indices of the support vectors and |Γ| is the cardinality of the set. The addition of term 𝑏² to (27) is first introduced by [25].

In an FLSA-SVM, the parameter 𝑚 is interpreted as the number of nonzeros of Lagrangian multipliers, that is, the number of support vectors (SVs), which is seen more obviously in its Lagrangian by introducing the Lagrange multipliers 𝛼_𝑖  (𝑖=1,…,ℓ) for each of the equality constraints giving ℒ()=1𝐰,𝑏,𝛼2𝐰^⊤𝐰+𝑏²−^𝑚_𝑖=1𝛼_𝑖𝐰^⊤𝜙𝐱_𝑖+𝑏−𝑦_𝑖−𝛽|Γ|. (30) The optimal point requires that 𝐰=^𝑚_𝑖=1𝛼_𝑖𝜙𝐱_𝑖,_𝑚𝑖=1𝛼_𝑖𝐰=𝑏,_⊤𝜙𝐱_𝑖+𝑏=𝐲_𝑖,|Γ|=𝑚. (31)

The linear equations can be further simplified to 𝟏|Γ|𝐊+𝜶=𝐃𝜶=𝐲,=𝑚, (32)(33) where 𝐊∈ℝ^ℓ×ℓ and 𝐊_𝑖𝑗=𝐾(𝐱_𝑖,𝐱_𝑗),  𝐲=[𝑦₁,…,𝑦_ℓ]^⊤,  𝜶=[𝛼₁,…,𝛼_ℓ]^⊤, 𝟏 is a ℓ-by-ℓ matrix of 1s, and 𝐃∈ℝ^ℓ×ℓ, where 𝐃_𝑖𝑗=𝐊_𝑖𝑗+1.

It is worth attention that, in this paper, 𝐃 in (32) is also referred to as 𝐃_𝑚 as it has 𝑚 support vectors. Thus, if all ℓ data samples are used as support vectors, then 𝐃 becomes 𝐃_ℓ.

FLSA can be applied to train an FLSA-SVM using a kernel-based dictionary 𝐃_ℓ={𝑑_𝑖=𝐊(⋅,𝑗)+1∣𝑖=1,…,ℓ} of candidate basis functions. Since 𝑚 is defined to be the number of SVs, it thus achieved a more direct control of the sparseness of its solution.

3.3. Automatic Detection of Linear Dependencies

Since the kernel matrix 𝐊 in (32) is a semipositive definite, it is easy to prove the semipositive definiteness of the matrix 𝐃. Then it is likely the occurrence of linear dependencies among column vectors, each being evaluation of a basis function on the training data. Assume that columns (𝐝_𝛾1,𝐝_𝛾2,…,𝐝_𝛾𝑘) are selected iteratively and linearly independent, where 𝛾_1,…,𝑘 are their column indices in chronological sequence. Denote 𝐝 to be any column which remains as available candidates in 𝐃 and meanwhile can be expressed as a linear combination of (𝐝_𝛾1,𝐝_𝛾2,…,𝐝_𝛾𝑘). It is thus naturally desired to remove 𝐝 as candidates in order to (1) ensure the sparseness of the solution; (2) avoid any undue computation concerning 𝐝 since

𝐝=𝛼₁𝐝_𝛾1+𝛼₂𝐝_𝛾2+⋯+𝛼_𝑘𝐝_𝛾𝑘, (34) where 𝜶=[𝛼₁,…,𝛼_𝑘]^⊤≠𝟎.

With the introduction of the residue matrix at each iteration as 𝐑_𝛾𝑖(𝑖=1,…,𝑘), the following property holds according to (18) and (20): 𝐑_𝛾𝑘𝐝=𝛼₁𝐑_𝛾𝑘𝐝_𝛾1+𝛼₂𝐑_𝑘𝐝_𝛾2+⋯+𝛼_𝑘𝐑_𝛾𝑘−1𝐝_𝛾𝑘=𝛼₁𝐑_𝛾𝑘𝐑_𝛾1𝐝_𝛾1+𝛼₂𝐑_𝛾𝑘𝐑_𝛾2𝐝_𝛾2+⋯+𝟎=𝟎. (35) Thus updating the dictionary 𝐃 by 𝐑_𝛾𝑘, the column vector(s) 𝐝 becomes 𝟎, that is, automatically pruned. Hence, at each iteration of the FLSA-SVM, any column vector(s) which can be represented by a linear combination of previously selected columns can be automatically pruned. This merit of the FLSA-SVM is one contributor to the sparseness of the resultant solution.

Algorithm 1 gives the pseudocode of the FLSA-SVM algorithm in detail.

INPUT: (i) The data set {(𝐱1, 𝑦1),…, (𝐱𝑙, 𝑦𝑙)} (ii) 𝑚 which is the number of support vectors desired in the expansion of the solution and  1≤𝑚≤𝑙 (iii) A dictionary of 𝑙 basis functions 𝐃0={𝐝1,…,𝐝𝑙} INITIALIZATION: (i) Current residue vector y, current dictionary 𝐃 which is initially a matrix of evaluations  of 𝑙 candidate basis functions on training data:          𝑦𝐲⟵⋮1𝑦𝑙𝑑,𝐃⟵1𝐱1⋯𝑑𝑙𝐱1𝑑⋮⋱⋮1𝐱𝑙⋯𝑑𝑙𝐱𝑙 (ii) The matrix 𝐀 and the vector 𝐛 both starts as empty 𝐀 is appended a row and 𝐛 grows   by one extra element at each iteration, which in the end forms a linear system. (iii) A variable 𝑛 which is the count of candidate basis functions and a vector Γ={𝛾1,…,𝛾𝑙}   which contains the indices of basis functions. At the start, 𝛾𝑖=𝑖 for 1≤𝑖≤𝑙 and 𝑛=𝑙. FOR  𝑖=1,…,𝑚             𝑘⟵argmax𝑘=𝑖,…,𝑛𝐃⋅,𝛾𝑘⊤𝐲𝐃⋅,𝛾𝑘 (iv) 𝛾𝑖 is made a pointer to the current selected basis functions:   𝛾𝑘↔𝛾𝑖               𝑏𝑖⟵𝐃⋅,𝛾𝑖⊤𝐲𝐃⋅,𝛾𝑖2 (v) The residue vector is reduced by 𝑏𝑖𝐝𝛾𝑖 as the target values for the next linear system of size 𝑙:                𝐲⟵𝐲−𝑏𝑖𝐃(⋅,𝛾𝑖) (vi) Update the dictionary matrix and prune the candidate basis functions which can be  represented as a linear combinations of the previously selected ones:   FOR 𝑗=𝑖+1,…,𝑛              𝛽𝛾𝑗⟵𝐃⋅,𝛾𝑖⊤𝐃⋅,𝛾𝑗𝐃⋅,𝛾𝑖2      𝐃⋅,𝛾𝑗←𝐃⋅,𝛾𝑗−𝛽𝑗𝐃⋅,𝛾𝑖    IF 𝐃⋅,𝛾𝑗=𝟎         𝛾𝑙↔𝛾𝑖         𝑛←𝑛−1  𝛽𝛾𝑖←1  𝐃⋅,𝛾𝑗←𝟎  𝛽𝐀←1,…,𝛽𝑙𝐀  𝐛←𝑏𝑖𝐛⊤ (vii) If equations 𝑖=𝑛 and 𝑚>𝑛 hold where 𝑖 is the number of selected basis functions and   𝑛 the count of available candidates, it suggests that the initial value setting on 𝑚 has   exceeded the rank of 𝐃0. Terminate the loop and reset 𝑚:         𝑚←𝑛 BACK SUBSTITUTION: (i) 𝑚 basis functions are chosen whose indices {𝛾1,…,𝛾𝑚} are the first 𝑚 elements of Γ. 𝑚1   columns of matrix 𝐀 with the indices {𝛾1,…,𝛾𝑚} and 𝐛 forms a linear system, on which   the process of back substitution is performed for the solution:         𝛼𝑚←𝑏𝑖     FOR 𝑖=𝑚−1,…,1         𝛼𝑖←𝑏𝑖−∑𝑚𝑗=𝑖+1𝛼𝑗𝐀(𝑖,𝛾𝑗) OUTPUT:  (i) The solution is defined by 𝑓(𝐱)=∑𝑚𝑖=1𝛼𝑖𝐝𝛾𝑖(𝐱)

Algorithm 1 

Forward Least Squares Approximation SVMs.

3.4. Computation Complexity

As discussed in [26], for a single round of training, the computational complexity of FLSA-SVM is 𝑂(𝑚ℓ²) and space complexity for FLSA-SVM is 𝑂(ℓ²).

4. Experimental Results

A set of experiments were performed to evaluate the performance of the proposed FLSA-SVM algorithm. The FLSA-SVMs were first applied to the two-spiral benchmark [27] for an illustrative view of their generalization abilities. The following Gaussian kernel function was used for the experiments in this paper: 𝐾𝐗_𝑖,𝐗_𝑗=𝑒^{−𝜆‖𝐗𝑖−𝐗𝑗‖2}. (36) The standard SVMs were implemented by LIBSVM [28]. The LS-SVM trained by CG method was implemented by the toolbox of LS-SVMlab [29] and all experiments were run on a Pentium 4 3.2 GHz processor under Windows XP with 2 GB of RAM.

4.1. Experiments on Two-Spiral Dataset

The 2D “two-spiral” benchmark is known to be difficult for pattern recognition algorithms and poses great challenges to neural networks [30]. The training set consists of 194 points of the 𝑋-𝑌 plane, half of which has a target value of +1 output and half a target value of −1. These training points describe two intertwining spirals that go around the origin three times, as shown in Figure 2, where the two categories are marked, respectively, by “+” and “o.”

Figure 2 

Two-spiral dataset.

For the FLSA-SVM, the parameter setting of Gaussian kernel was 𝜆=1 as in (36). With a feasible range of [85,193], the optimal regularization parameter was found to be 134 which gave a leave-one-out cross-validation (LOOCV) accuracy of 97.94%. With standard SVMs, the parameter setting was 𝐶=32 and 𝜆=0.5, whose LOOCV accuracy is 98.97%. The SVM classifier was required in 164 support vectors (SVs). The graphical outputs of the FLSA-SVM and the SVM were given by Figure 3. It showed that generally both SVM and FLSA-SVM algorithms recognized the pattern successfully, outputting “two-spiral” in a very smooth and regular manner. But at the area around the coordinate of [0,0], the FLSA-SVM showed better performance than the conventional SVM whose decision hyperplane was biased towards the “o” class.

(a)

(b)

Figure 3 

Comparison between standard SVMs and FLSA-SVMs: (a) two-spiral pattern recognized by the FLSA-SVM trained on 134 data points with 𝜆=1 in (36); (b) two-spiral pattern recognized by the standard SVM trained on 164 data points. The penalty constant 𝐶=32 and 𝜎=0.5.

Despite the reduction in the number of SVs which is very notable for such a “hard” classification problem, the FLSA-SVMs are comparatively superior to the following two aspects. In SVMs, it often occurred that given a fixed kernel parameter, the optimal CV accuracy can be obtained from multiple value settings on the regularization parameter. The “two-spiral” problem is a case in point. For the best LOOCV accuracy, with a fixed 𝜆=0.5, the value of 𝐶 can also opt for (2⁶,2⁷,2⁸,2⁹,2¹⁰) beyond 2⁵. There is no specific rule as to which option is the best, and normally the smallest is chosen for a scaled-down feasible region. While in FLSA-SVMs, the setting for the value of the regularization term 𝑚 is more tractable since different options correspond to different learning errors which can be easily tracked with (23). For multiple values of 𝑚 which produce the same optimal CV accuracy, the largest 𝑚 is chosen for a smaller learning error.

In fact, for 𝑚∈[160,193], the LOOCV accuracy remained stable at 96.91%. Thus an FLSA-SVM was also trained with the parameter settings of 𝑚=193 which was depicted in Figure 4(a). For comparisons, an LS-SVM was trained whose parameter settings are 𝛾=173.7427 and 𝜆=5.4899 for Gaussian RBF. The solution was parameterized by the entire 194 points and illustrated in Figure 4(b). The decision boundaries of both of the LS-SVM and the FLSA-SVM were generally smooth and followed the pattern satisfactorily, despite the slightly biased segments around the coordinates of [1.5,1.5] and [−1.5,−1.5]. But still the FLSA-SVM performed much better at the origin area than the LS-SVM.

(a)

(b)

Figure 4 

Comparison between LS-SVMs and FLSA-SVMs: (a) two-spiral pattern recognized by the conventional LS-SVM trained on 194 data points with the penalty parameter 𝛾=173.7427 and 𝜆=5.4899 for Gaussian RBF; (b) two-spiral pattern recognized by the FLSA-SVM trained on 180 data points with 𝜆=1 in (36).

Table 1 also compares the time cost of 10-fold cross-validation (CV) on the regularization parameter by FLSA-SVMs, SVMs and LS-SVMs. For FLSA-SVMs, the regularization term 𝑚 was assigned 21 integers evenly with an interval of 10 within the range of (1,194). For SVMs and LS-SVMs, the parameter 𝐶 was sequentially increased from 2⁻¹⁰ to 2¹⁰ in multiple of 2. Each fold was split into a training set and a validation set, with a division pattern of 176 training versus 18 validation or 174 training versus 20 validation. The total computation time of the 21 classifiers on various folds for each algorithm was reported in Table 1. The time cost of the 10-fold altogether was given in the last row entry of Table 1. It was clearly shown that the 10-fold cross-validation (CV) of FLSA-SVMs is over 4 times faster than SVMs. The time complexity of FLSA-SVMs remained competitive to that of LS-SVMs using the SMO algorithm and much more reduced than LS-SVMs using the CG method.

Table 1 

Time cost of 10-fold cross-validation on two-spiral data by FLSA-SVMS, SVMs, and LS-SVMs with fixed kernel parameters: 𝜆=1 for FLSA-SVMs, 𝜆=0.5 for SVMs, and 𝜆=5.4899 for LS-SVMs.

These comparisons prove that the FLSA-SVM is very promising in easing the nonsparseness problem of an LS-SVM, in addition to its outstanding generalization performance. And it can obtain a solution whose sparseness is competitive, or even superior, to that of a standard SVM.

4.2. More Benchmark Problems

The FLSA-SVM algorithm was applied to 3 small-scale binary problems: Banana, Image, and Splice which are accessible at http://theoval.cmp.uea.ac.uk/matlab/#benchmarks/. Among all the realizations for each benchmark, the first one of them was used. Experiments were also performed on Banana dataset [31], which is a medium-scale binary learning problem. The detailed information of the datasets was given in Table 2.

Table 2 

Benchmark information.

FLSA-SVMs were compared with SVMs, LS-SVMs, the fast sparse approximation scheme for LS-SVM (FSALS-SVM), and its variant, called PFSALS-SVM, both of which were proposed by Jiao et al. [12]. The parameter 𝜖 of FSALS-SVMs and PFSALS-SVMs was uniformly set to be 0.5 which was empirically proved to work well with most datasets [12]. Comparisons were also made against D-optimality orthogonal forward regression (D-OFR) [32] which is a technique for nonlinear function estimation, promised to yield sparse solutions. The parameters, which were the penalty constant and 𝜆 in (36), were tuned by tenfold cross-validation (CV).

Table 3 presented the classification accuracy of the SVM algorithms. The best results among the four SVM algorithms were highlighted. It can be seen that the FLSA-SVM achieved comparable classification accuracy to the standard SVM, the conventional SVM, FSAL-SVM, PFSALS-SVM, and D-optimality OFR.

Table 3 

Test correctness (%).

The numbers of SVs were compared in Table 4. For all the learning problems, the FLSA-SVM required much less SVs than the SVM and the conventional SVM. In particular, the reduction in SVs reached over 98% and 80%, respectively, on Ringnorm and Banana datasets compared with the SVM. FLSA-SVM has maintained its edges over FSAL-SVM and PFSALS-SVM, particularly with the Ringnorm and Banana datasets. Although the D-OFR method falls into the category of unsupervised learning algorithms, FLSA-SVM, mathematically, has the closest link with the D-OFR method. The results in Table 3 demonstrated that the D-OFR method remained competitive compared with FLSA-SVM on Ringnorm and Banana datasets. However, on the Splice and Image datasets, the D-OFR method failed to achieve any sparse solutions.

Table 4 

Number of support vectors (best in bold).

5. Conclusions

The paper proposed a new LS-SVM algorithm—the FLSA-SVM which is trained specifically by the FLSA method of minimized squared error loss. The FLSA-SVM iteratively selects an optimal basis function which is associated with a specific training example into the solution. The algorithm cleverly adapts the number of SVs into the regularization term as the tradeoff between generalization abilities and training cost. As a result, the solution of the FLSA-SVMs is extremely sparse compared to LS-SVMs. Experiments showed that the FLSA-SVM algorithm maintained a comparable accuracy compared to the standard SVM, the LS-SVM and a number of recently developed sparse learning algorithms. Yet the FLSA-SVM showed definite advantages to its counterparts regarding the sparseness of the solution. On small datasets like the two-spiral benchmark, the FLSA-SVM training algorithm also proved to be more efficient than the CG method.