A Novel Sparse Least Squares Support Vector Machines

Algorithm 1

Forward Least Squares Approximation SVMs.

INPUT:

(i) The data set {(𝐱₁, 𝑦₁),…, (𝐱_𝑙, 𝑦_𝑙)}

(ii) 𝑚 which is the number of support vectors desired in the expansion of the solution and

 1≤𝑚≤𝑙

(iii) A dictionary of 𝑙 basis functions 𝐃₀={𝐝₁,…,𝐝_𝑙}

INITIALIZATION:

(i) Current residue vector y, current dictionary 𝐃 which is initially a matrix of evaluations

 of 𝑙 candidate basis functions on training data:

         𝑦𝐲⟵⋮¹𝑦_𝑙𝑑,𝐃⟵¹𝐱₁⋯𝑑_𝑙𝐱₁𝑑⋮⋱⋮₁𝐱_𝑙⋯𝑑_𝑙𝐱_𝑙

(ii) The matrix 𝐀 and the vector 𝐛 both starts as empty 𝐀 is appended a row and 𝐛 grows

  by one extra element at each iteration, which in the end forms a linear system.

(iii) A variable 𝑛 which is the count of candidate basis functions and a vector Γ={𝛾₁,…,𝛾_𝑙}

  which contains the indices of basis functions. At the start, 𝛾_𝑖=𝑖 for 1≤𝑖≤𝑙 and 𝑛=𝑙.

FOR  𝑖=1,…,𝑚

            𝑘⟵argmax_{𝑘=𝑖,…,𝑛}𝐃⋅,𝛾_𝑘^⊤𝐲𝐃⋅,𝛾_𝑘

(iv) 𝛾_𝑖 is made a pointer to the current selected basis functions:

  𝛾_𝑘↔𝛾_𝑖

              𝑏_𝑖⟵𝐃⋅,𝛾_𝑖^⊤𝐲𝐃⋅,𝛾_𝑖²

(v) The residue vector is reduced by 𝑏_𝑖𝐝_𝛾𝑖 as the target values for the next linear system of size 𝑙:

               𝐲⟵𝐲−𝑏_𝑖𝐃(⋅,𝛾_𝑖)

(vi) Update the dictionary matrix and prune the candidate basis functions which can be

 represented as a linear combinations of the previously selected ones:

  FOR 𝑗=𝑖+1,…,𝑛

             𝛽_𝛾𝑗⟵𝐃⋅,𝛾_𝑖^⊤𝐃⋅,𝛾_𝑗𝐃⋅,𝛾_𝑖²

     𝐃⋅,𝛾_𝑗←𝐃⋅,𝛾_𝑗−𝛽_𝑗𝐃⋅,𝛾_𝑖

   IF 𝐃⋅,𝛾_𝑗=𝟎

        𝛾_𝑙↔𝛾_𝑖

        𝑛←𝑛−1

 𝛽_𝛾𝑖←1

 𝐃⋅,𝛾_𝑗←𝟎

 𝛽𝐀←¹,…,𝛽_𝑙𝐀

 𝐛←𝑏_𝑖𝐛^⊤

(vii) If equations 𝑖=𝑛 and 𝑚>𝑛 hold where 𝑖 is the number of selected basis functions and

  𝑛 the count of available candidates, it suggests that the initial value setting on 𝑚 has

  exceeded the rank of 𝐃₀. Terminate the loop and reset 𝑚:

        𝑚←𝑛

BACK SUBSTITUTION:

(i) 𝑚 basis functions are chosen whose indices {𝛾₁,…,𝛾_𝑚} are the first 𝑚 elements of Γ. 𝑚₁

  columns of matrix 𝐀 with the indices {𝛾₁,…,𝛾_𝑚} and 𝐛 forms a linear system, on which

  the process of back substitution is performed for the solution:

        𝛼_𝑚←𝑏_𝑖

    FOR 𝑖=𝑚−1,…,1

        𝛼_𝑖←𝑏_𝑖−∑^𝑚_𝑗=𝑖+1𝛼_𝑗𝐀(𝑖,𝛾_𝑗)

OUTPUT:

 (i) The solution is defined by 𝑓(𝐱)=∑^𝑚_𝑖=1𝛼_𝑖𝐝_𝛾𝑖(𝐱)