A Simpler Approach to Coefficient Regularized Support Vector Machines Regression

Abstract

We consider a kind of support vector machines regression (SVMR) algorithms associated with 𝑙^𝑞(1≤𝑞<∞) coefficient-based regularization and data-dependent hypothesis space. Compared with former literature, we provide here a simpler convergence analysis for those algorithms. The novelty of our analysis lies in the estimation of the hypothesis error, which is implemented by setting a stepping stone between the coefficient regularized SVMR and the classical SVMR. An explicit learning rate is then derived under very mild conditions.

1. Introduction

Recall the regression setting in learning theory, and let 𝑋 be a compact subset of ℝ^𝑛, 𝑌⊂[−𝑀,𝑀], for some 𝑀>0. 𝜌 is an unknown probability distribution endowed on 𝑍∶=𝑋×𝑌, and 𝐳∶={𝑧_𝑖}^𝑚_𝑖=1={(𝑥_𝑖,𝑦_𝑖)}^𝑚_𝑖=1∈𝑍^𝑚 is a set of samples independently drawn according to 𝜌. Given samples 𝐳, the regression problem aims to find a function 𝑓_𝐳∶𝑋→ℝ, such that 𝑓_𝐳(𝑥) is a satisfactory estimate of output 𝑦 when a new input 𝑥 is given.

Support vector machines regression (SVMR) is a kind of kernel-based regression algorithms with the 𝜀-insensitive loss defined by 𝑉(𝑦,𝑡)=max{0,|𝑦−𝑡|−𝜀} for some fixed 𝜀≥0. A function 𝐾∶𝑋×𝑋→ℝ is called a Mercer kernel if it is continuous, symmetric, and positive semidefinite; that is, for any finite set of distinct points {𝑥₁,𝑥₂,…,𝑥_𝑙}⊂𝑋, the matrix (𝐾(𝑥_𝑖,𝑥_𝑗))^𝑙_𝑖,𝑗=1 is positive semidefinite. The reproducing kernel Hilbert space (RKHS) ℋ_𝐾 associated with a Mercer kernel 𝐾 is defined (see [1]) to be the completion of the linear span of the set of functions {𝐾_𝑥∶=𝐾(𝑥,⋅)∶𝑥∈𝑋} with the inner product ⟨⋅,⋅⟩_𝐾 satisfying 𝐾_𝑥,𝐾_𝑢𝐾=𝐾(𝑥,𝑢), (1) and the reproducing property is given by 𝐾_𝑥,𝑓_𝐾=𝑓(𝑥),∀𝑥∈𝑋,𝑓∈ℋ_𝐾. (2) Let 𝜅∶=sup_𝑥∈𝑋𝐾(𝑥,𝑥)<∞, (3) and then the reproducing property tells us the following: 𝑓_∞≤𝜅𝑓_𝐾,∀𝑓∈ℋ_𝐾. (4)

The classical SVMR proposed by Vapnik and his coworkers [2, 3] is given by the following regularization scheme: 𝑓_𝐳,𝜇∶=argmin_{𝑓∈ℋ𝐾}ℰ_𝐳𝑓+𝜇𝑓²_𝐾, (5) where ℰ_𝐳(𝑓)∶=(1/𝑚)∑^𝑚_𝑖=1𝑉(𝑦_𝑖,𝑓(𝑥_𝑖)) is the empirical error with respect to 𝐳 and 𝜇 is a regularization parameter. It is well known, see for example, [4, Proposition 6.21], that the solution is of the form 𝑓_𝐳,𝜇=^𝑚_𝑖=1𝛼_𝑖𝐾𝑥_𝑖,⋅, (6) where the coefficients 𝛼_𝑖 are a solution of the optimization problem maximize^𝑚_𝑖=1𝛼_𝑖𝑦_𝑖−𝜀^𝑚_𝑖=1𝛼_𝑖−12^𝑚_𝑖,𝑗=1𝛼_𝑖𝛼_𝑗𝐾𝑥_𝑖,𝑥_𝑗subjectto𝛼_𝑖≤12𝜇𝑚∀𝑖=1,2,…,𝑚. (7)

Remark 1. The equality constraint ∑^𝑚_𝑖=1𝛼_𝑖=0 needed in [4, Proposition 6.21] is superfluous since we do not include an offset term 𝑏 in the primal problem (5).

The mathematical analysis of algorithm (5) has been well understood with various techniques in extensive literature; see, for example, [5–7]. In this paper, we are interested in a different regularized SVMR algorithm. In our setting, the regularizer is not the RKHS norm but an 𝑙^𝑞-norm of the coefficients in the kernel ensembles.

Definition 2. For 1≤𝑞<∞, let ℋ_𝐾,𝐳∶=^𝑚_𝑖=1𝛼_𝑖𝐾_𝑥𝑖∶𝛼_𝑖,Ω∈ℝ,𝑖=1,2,…,𝑚_𝐳𝑓=inf^𝑚_𝑖=1𝛼_𝑖^𝑞∶𝑓=^𝑚_𝑖=1𝛼_𝑖𝐾_𝑥𝑖. (8) Then, the SVMR with 𝑙^𝑞-coefficient regularization learning algorithm that we study in this paper takes the form 𝑓_𝐳,𝜆∶=argmin_{𝑓∈ℋ𝐾,𝐳}ℰ_𝐳𝑓+𝜆Ω_𝐳𝑓. (9)

Remark 3. The regularization parameter 𝜆 in (9) may be different from 𝜇 in scheme (5), but a relationship between 𝜆 and 𝜇 will be given in Section 3 as we derive the learning rate of algorithm (9).

Learning with coefficient-based regularization has attracted a considerable amount of attention in recent years, on both theoretical analysis and applications. It was pointed out in [8] that by taking 𝑞<2, and especially for the limit value 𝑞=1, the proposed minimization procedure in (9) can promote the sparsity of the solution; that is, it tends to result in a solution with a few nonzero coefficients [9]. This phenomenon has been also observed in the LASSO algorithm [10] and the literature of compressed sensing [11].

However, it should be noticed that there are essential differences between the learning schemes (9) and (5). On one hand, the regularizer Ω_𝐳(𝑓) is not a Hilbert space norm, which causes a technical difficulty for mathematical analysis. On the other hand, both hypothesis space ℋ_𝐾,𝐳 and regularizer Ω_𝐳(𝑓) depend on samples 𝐳, and this increases the flexibility and adaptivity of algorithm (9) but causes the standard error analysis methods for scheme (5) which are not appropriate to scheme (9) any longer. To overcome these difficulties, [12] introduces a Banach space ℋ of all functions of the form 𝑓(𝑥)=^∞_𝑗=1𝛼_𝑗𝐾𝑢_𝑗,𝑥,𝑢_𝑗∈𝑋, (10) with norm 𝑓=inf^∞_𝑗=1𝛼_𝑗∶𝑓=^∞_𝑗=1𝛼_𝑗𝐾_𝑢𝑗. (11) An error analysis framework is established and then a series of papers start to investigate the performance of kernel learning scheme with coefficient regularization (see [13–16]). In those literatures, an 𝐿_𝜏 condition imposed on the marginal distribution of 𝜌 on 𝑋 plays a critical role in the error analysis. A probability measure 𝜌_𝑋 on 𝑋 is said to satisfy 𝐿_𝜏 condition if there exist some 𝜏>0 and 𝑐_𝜏>0 such that 𝜌_𝑋({𝑢∈𝑋∶|𝑢−𝑥|<𝑟})≥𝑐_𝜏𝑟^𝜏,∀𝑥∈𝑋,0<𝑟≤1. (12) In general, the index 𝜏 is hard to estimate. If 𝑋 satisfies some regularity conditions (such as an interior cone condition) and 𝜌_𝑋 is uniform distribution on 𝑋, then (12) holds with 𝜏=𝑛. It leads to a low convergence rate and depends on 𝑛, the dimension of the input space 𝑋, which is often large in learning problem.

In this paper, we succeed to remove 𝐿_𝜏 condition (12) and provide a simpler error analysis for scheme (9). The novelty of our analysis is a stepping stone technique applied to bound the hypothesis error. As a result, we derive an explicit learning rate of (9) under very mild conditions.

2. Error Decomposition and Hypothesis Error

The main purpose of this paper is to provide a convergence analysis of the learning scheme (9). With respect to the 𝜀-insensitive loss 𝑉, the prediction ability of a measurable function 𝑓 is measured by the following generalization error: ℰ𝑓∶=_𝑍𝑉(𝑥)=𝑦,𝑓𝑑𝜌_𝑋𝑌𝑉𝑦,𝑓(𝑥)𝑑𝜌𝑦∣𝑥𝑑𝜌_𝑋(𝑥), (13) where 𝜌_𝑋 is the marginal distribution on 𝑋 and 𝜌(⋅∣𝑥) is the conditional probability measure at 𝑥 induced by 𝜌. Let 𝑓^∗ be a minimizer of ℰ(𝑓) among all measurable functions on 𝑋. It was proved in [6] that |𝑓^∗(𝑥)|≤𝑀+𝜀 for almost every 𝑥∈𝑋. To make full use of the feature of the target function 𝑓^∗, one can introduce a projection operator, which was extensively used to the error analysis of learning algorithm; see, for example, [17, 18].

Definition 4. The projection operator 𝜋=𝜋_𝑀+𝜀 is defined on the space of measurable functions 𝑓∶𝑋→ℝ as 𝜋𝑓𝑥)=(𝑥)(𝑥)𝑓(𝑥)(𝑥)(𝑀+𝜀,if𝑓>𝑀+𝜀,−𝑀−𝜀,if𝑓<−𝑀−𝜀,,if−𝑀−𝜀≤𝑓≤𝑀+𝜀. (14)

It is easy to see that 𝑉(𝑦,𝜋(𝑓)(𝑥))≤𝑉(𝑦,𝑓(𝑥)), so ℰ𝜋𝑓≤ℰ𝑓,ℰ_𝐳𝜋𝑓≤ℰ_𝐳𝑓. (15) We thus take 𝜋(𝑓_𝐳,𝜆) instead of 𝑓_𝐳,𝜆 as our empirical target function and analyze the related learning rates.

2.1. Error Decomposition

The error decomposition is a useful approach to the error analysis for the regularized learning schemes. With sample-dependent hypothesis space ℋ_𝐾,𝐳, [12] proposes a modified error decomposition with an extra hypothesis error term, by introducing a regularization function as 𝑓_𝜇∶=argmin_{𝑓∈ℋ𝐾}ℰ𝑓+𝜇𝑓²_𝐾,𝜇>0. (16) We can conduct the error decomposition for scheme (9) with the same underlying idea of [12].

Proposition 5. Let 𝑓_𝐳,𝜆, 𝑓_𝜇 be defined in (9) and (16). Then, ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^∗≤𝑆𝐳,𝜆,𝜇+𝑃𝐳,𝜆,𝜇+𝐷𝜇. (17) Here, 𝑆𝐳,𝜆,𝜇∶=ℰ𝜋𝑓_𝐳,𝜆−ℰ_𝐳𝜋𝑓_𝐳,𝜆+ℰ_𝐳𝑓_𝜇−ℰ𝑓_𝜇,𝑃𝐳,𝜆,𝜇∶=ℰ_𝐳𝜋𝑓_𝐳,𝜆+𝜆Ω_𝐳𝑓_𝐳,𝜆−ℰ_𝐳𝑓_𝜇+𝜇𝑓_𝜇²_𝐾,𝐷𝜇∶=ℰ𝑓_𝜇−ℰ𝑓^∗+𝜇𝑓_𝜇²_𝐾=inf_{𝑓∈ℋ𝐾}ℰ𝑓−ℰ𝑓^∗+𝜇𝑓²_𝐾. (18)

Proof. A direct computation shows that ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^∗≤ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^∗+𝜆Ω_𝐳𝑓_𝐳,𝜆=ℰ𝜋𝑓_𝐳,𝜆−ℰ_𝐳𝜋𝑓_𝐳,𝜆+ℰ_𝐳𝜋𝑓_𝐳,𝜆+𝜆Ω_𝐳𝑓_𝐳,𝜆+ℰ_𝐳𝑓_𝜇−ℰ𝑓_𝜇−ℰ_𝐳𝑓_𝜇+𝜇𝑓_𝜇²_𝐾+ℰ𝑓_𝜇−ℰ𝑓^∗+𝜇𝑓_𝜇²_𝐾=𝑆𝐳,𝜆,𝜇+𝑃𝐳,𝜆,𝜇+𝐷𝜇. (19) This proves the proposition.

𝑆(𝐳,𝜆,𝜇) is usually called the sample error; it will be estimated by some concentration inequality in the next section. 𝐷(𝜇) is independent of the sample and is often called the approximation error, the decay of 𝐷(𝜇), as 𝜇→0 characterizes the approximation ability of ℋ_𝐾. We will assume that, for some 0<𝛽≤1 and 𝑐_𝛽>0, 𝐷𝜇≤𝑐_𝛽𝜇^𝛽,∀𝜇>0. (20)

Remark 6. Since ℰ(𝑓)−ℰ(𝑓^∗)≤‖𝑓−𝑓^∗‖_{𝐿1𝜌𝑋}, 𝐷(𝜇) concerns the approximation of 𝑓^∗ in 𝐿¹_𝜌𝑋 by functions from ℋ_𝐾. In fact, (20) can be satisfied when 𝑓^∗ is in some interpolation spaces of the pair (𝐿¹_𝜌𝑋,ℋ_𝐾) (see, e.g., [19, 20]).

𝑃(𝐳,𝜆,𝜇) is called hypothesis error since the regularization function 𝑓_𝜇 may not be in the hypothesis space ℋ_𝐾,𝐳. The major contribution we make in this paper is to give a simpler estimation of 𝑃(𝐳,𝜆,𝜇) by a stepping stone between 𝑓_𝐳,𝜆 and 𝑓_𝐳,𝜇.

2.2. Hypothesis Error Estimate

The solution 𝑓_𝐳,𝜆 of scheme (9) has a representation similar to 𝑓_𝐳,𝜇 in scheme (5); it is reasonable to expect close relations between the two schemes. So, the latter may play roles in the analysis of the former.

Theorem 7. Let 𝜆, 𝜇>0, 1≤𝑞<∞, and then 𝑃𝐳,𝜆,𝜇≤𝑚𝜆2𝑚𝜇^𝑞. (21)

Proof. Let 𝑓_𝐳,𝜇=∑^𝑚_𝑖=1𝛼_𝑖𝐾_𝑥𝑖 be the solution to (5). By (7), we have Ω_𝐳𝑓_𝐳,𝜇≤^𝑚_𝑖=1𝛼_𝑖^𝑞≤𝑚2𝑚𝜇^𝑞. (22) Noting that 𝑓_𝐳,𝜇∈ℋ_𝐾,𝐳, it can be derived from (15), (9), and (22) that ℰ_𝐳𝜋𝑓_𝐳,𝜆+𝜆Ω_𝐳𝑓_𝐳,𝜆≤ℰ_𝐳𝑓_𝐳,𝜆+𝜆Ω_𝐳𝑓_𝐳,𝜆≤ℰ_𝐳𝑓_𝐳,𝜇+𝜆Ω_𝐳𝑓_𝐳,𝜇≤ℰ_𝐳𝑓_𝐳,𝜇+𝑚𝜆2𝑚𝜇^𝑞≤ℰ_𝐳𝑓_𝐳,𝜇+𝜇𝑓_𝐳,𝜇²_𝐾+𝑚𝜆2𝑚𝜇^𝑞. (23) By taking 𝑓=𝑓_𝜇 in (5), one can get ℰ_𝐳𝑓_𝐳,𝜇+𝜇𝑓_𝐳,𝜇²_𝐾≤ℰ_𝐳𝑓_𝜇+𝜇𝑓_𝜇²_𝐾. (24) Putting (24) into (23), one then has ℰ_𝐳𝜋𝑓_𝐳,𝜆+𝜆Ω_𝐳𝑓_𝐳,𝜆≤ℰ_𝐳𝑓_𝜇+𝜇𝑓_𝜇²_𝐾+𝑚𝜆2𝑚𝜇^𝑞. (25) This proves the theorem.

Remark 8. The stepping stone method was first introduced in [21] to the error analysis of linear programming SVM classifiers. This technique was also used in [22, 23] to study 𝑙^𝑞-coefficient regularized least square regression with 𝑞∈[1,2]. While, in this paper, we extend the index 𝑞 to a large range [1,+∞), it will be helpful to improve the understanding of those coefficient-based regularized algorithms.

Remark 9. Theorem 7 presents a simpler approach for estimating the hypothesis error. Different from the former literature (see, e.g., [13–16]), we conduct the estimation without imposing any assumptions on the input space 𝑋, kernel 𝐾, and marginal distribution 𝜌_𝑋.

3. Sample Error and Learning Rate

This section is devoted to estimating the sample error 𝑆(𝐳,𝜆,𝜇) and deriving the learning rate of algorithm (9).

3.1. Sample Error Estimate

We will adopt some results from the literature to estimate the sample error. To this end, we need some definitions and assumptions. For a measurable function 𝑓∶𝑍→ℝ, denote 𝔼𝑓∶=∫_𝑍𝑓(𝑧)𝑑𝜌.

Definition 10. A variance power 𝑠 of the pair (𝑉,𝜌) is a number in [0,1] such that for any 𝑓∶𝑋→[−𝑀−𝜀,𝑀+𝜀], there exists some constant 𝑐_𝑠>0 satisfying 𝔼𝑉𝑦,𝑓(𝑥)−𝑉𝑦,𝑓^∗(𝑥)²≤𝑐_𝑠ℰ𝑓−ℰ𝑓^∗𝑠. (26)

Equation (26) is usually called a variance-expectation condition for the pair (𝑉,𝜌). It is easy to see that (26) always holds for 𝑠=0 and 𝑐_𝑠=4(𝑀+𝜀)². When 𝜀=0, the target function 𝑓^∗ becomes the median 𝑓_𝜌,1/2 of 𝜌 (see, [6]). In this case, as it points out in [24], if 𝜌 has a median 𝑓_𝜌,1/2 of 𝑎-average type 𝑏 for some 𝑎∈(0,∞] and 𝑏∈[1,∞), then (26) can be satisfied with 𝑠=min{2/𝑏,𝑎/(𝑎+1)}∈(0,1]. Here, we say 𝜌 has a median 𝑓_𝜌,1/2 of 𝑎-average type 𝑏 if, for every 𝑥∈𝑋, there exist constants 𝑐_𝑥∈(0,2] and 𝑑_𝑥>0 such that, for all 𝑣∈[0,𝑐_𝑥], 𝜌𝑦∈𝑓_𝜌,1/2(𝑥)−𝑣,𝑓_𝜌,1/2(𝑥)∣𝑥≥𝑑_𝑥𝑣^𝑏−1,𝜌𝑦∈𝑓_𝜌,1/2(𝑥),𝑓_𝜌,1/2(𝑥)+𝑣∣𝑥≥𝑑_𝑥𝑣^𝑏−1, (27) and that the function on 𝑋 taking value (𝑑_𝑥𝑐_𝑥^𝑏−1)⁻¹ at 𝑥∈𝑋 lies in 𝐿^𝑎_𝜌𝑋. But for 𝜀>0, as we know, it is still open to find a meaningful condition for 𝜌 to guarantee that (26) holds with a positive index 𝑠>0.

Definition 11. Let ℱ be a class of functions on 𝑍 and let 𝐳={𝑧_𝑖}^𝑚_𝑖=1∈𝑍^𝑚. The 𝑙²-metric 𝑑_2,𝐳 is defined on ℱ by 𝑑_2,𝐳1𝑓,𝑔∶=𝑚^𝑚_𝑖=1𝑓𝑧_𝑖−𝑔𝑧_𝑖^21/2,∀𝑓,𝑔∈ℱ. (28) For every 𝜂>0, the covering number of ℱ with respect to 𝑑_2,𝐳 is 𝒩_2,𝐳ℱ,𝜂∶=inf𝑙∈ℕ∶∃𝑓_𝑖^𝑙_𝑖=1suchthatℱ=^𝑙_𝑖=1𝑓∈ℱ∶𝑑_2,𝐳𝑓,𝑓_𝑖≤𝜂. (29)

Let ℬ_𝑅∶={𝑓∈ℋ_𝐾∶‖𝑓‖_𝐾≤𝑅}. The 𝑙²-empirical covering number of the unit ball ℬ₁ is defined as 𝒩ℬ₁,𝜂∶=sup_𝑚∈ℕsup_{𝐱∈𝑋𝑚}𝒩_2,𝐱ℬ₁,𝜂. (30) We assume that ℋ_𝐾 satisfies the following capacity assumption.

There exists an exponent 𝑝 with 0<𝑝<2 and a constant 𝑐_𝑝>0 such that log𝒩ℬ₁,𝜂≤𝑐_𝑝𝜂^−𝑝,∀𝜂>0. (31) We now set out to bound the sample error. Write 𝑆(𝐳,𝜆,𝜇) as 𝑆𝐳,𝜆,𝜇=ℰ_𝐳𝑓_𝜇−ℰ_𝐳𝑓^∗−ℰ𝑓_𝜇−ℰ𝑓^∗+ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^∗−ℰ_𝐳𝜋𝑓_𝐳,𝜆−ℰ_𝐳𝑓^∗=∶𝑆₁𝐳,𝜇+𝑆₂(𝐳,𝜆). (32) Applying [6, Proposition 4.1], we yield the following estimation for 𝑆₁(𝐳,𝜇).

Lemma 12. For any 𝑡>0, under the assumption (26), with confidence 1−2𝑒^−𝑡, one has 𝑆₁𝐳,𝜇≤7𝜅𝑡6𝑚𝐷𝜇𝜇+8(𝑀+𝜀)𝑡+3𝑚2𝑐_𝑠𝑡𝑚^1/(2−𝑠)+𝐷𝜇. (33)

The estimation for 𝑆₂(𝐳,𝜆) is based on the following concentration inequality which can be found in [25].

Lemma 13. Let ℱ be a set of measurable functions on 𝑍, and let 𝐵,𝑐>0, and 𝑠∈[0,1] be constants such that each 𝑓∈ℱ satisfies ‖𝑓‖_∞≤𝐵 and 𝔼(𝑓²)≤𝑐(𝔼𝑓)^𝑠. If, for some 𝐴>0 and 𝑝∈(0,2), sup_𝑚∈ℕsup_{𝐳∈𝑍𝑚}log𝒩_2,𝐳ℱ,𝜂≤𝐴𝜂^−𝑝,∀𝜂>0, (34) then there exists a constant 𝑐^′_𝑝 depending only on 𝑝 such that for any 𝑡>0, with probability 1−𝑒^−𝑡, there holds 1𝔼𝑓−𝑚^𝑚_𝑖=1𝑓𝑧_𝑖≤12𝜃^1−𝑠𝔼𝑓^𝑠+𝑐^′_𝑝𝜃+2𝑐𝑡𝑚^1/(2−𝑠)+18𝐵𝑡𝑚,∀𝑓∈ℱ, (35) where 𝜃=max{𝑐^{(2−𝑝)/(4−2𝑠+𝑝𝑠)}(𝐴/𝑚)^{2/(4−2𝑠+𝑝𝑠)},𝐵^{(2−𝑝)/(2+𝑝)}(𝐴/𝑚)^2/(2+𝑝)}.

We may apply Lemma 13 to a set of functions ℱ_𝑅 with 𝑅>0, where ℱ_𝑅∶=𝑉𝑦,𝜋𝑓(𝑥)−𝑉𝑦,𝑓^∗(𝑥)∶𝑓∈ℬ_𝑅. (36)

Proposition 14. If assumptions (26) and (31) are satisfied, then for any 𝑡>0, with confidence 1−𝑒^−𝑡, there holds ℰ𝜋𝑓−ℰ𝑓^∗−ℰ_𝐳𝜋𝑓−ℰ_𝐳𝑓^∗≤12ℰ𝜋𝑓−ℰ𝑓^∗+12+𝑐^′_𝑝𝜃_𝑅𝑐+2^𝑠𝑡𝑚^1/(2−𝑠)+36(𝑀+𝜀)𝑡𝑚 (37) for all 𝑓∈ℬ_𝑅, where 𝜃_𝑅=max𝑐_𝑠^{(2−𝑝)/(4−2𝑠+𝑝𝑠)}𝑐_𝑝𝑅^𝑝𝑚^{2/(4−2𝑠+𝑝𝑠)},[2(𝑀+𝜀)]^{(2−𝑝)/(2+𝑝)}𝑐_𝑝𝑅^𝑝𝑚^2/(2+𝑝). (38)

Proof. Each function 𝑔∈ℱ_𝑅 has a form 𝑔(𝑧)=𝑉𝑦,𝜋𝑓(𝑥)−𝑉𝑦,𝑓^∗(𝑥) (39) with some 𝑓∈ℬ_𝑅. We can easily see that ‖𝑔‖_∞≤2(𝑀+𝜀) and 𝔼𝑔=ℰ𝜋𝑓−ℰ𝑓^∗,1𝑚^𝑚_𝑖=1𝑔𝑧_𝑖=ℰ_𝐳𝜋𝑓−ℰ_𝐳𝑓^∗. (40) The assumption (26) tells us that 𝔼(𝑔²)≤𝑐(𝔼𝑔)^𝑠 with 𝑐=𝑐_𝑠. Moreover, for any 𝑓₁,𝑓₂∈ℬ_𝑅, and (𝑥,𝑦)∈𝑍, 𝑉𝑦,𝜋𝑓₁(𝑥)−𝑉𝑦,𝜋𝑓₂(𝑥)≤𝜋𝑓₁(𝑥)−𝜋𝑓₂(𝑥)≤𝑓₁(𝑥)−𝑓₂(𝑥), (41) we get 𝒩_2,𝐳ℱ_𝑅,𝜂≤𝒩_2,𝐱ℬ_𝑅,𝜂=𝒩_2,𝐱ℬ₁,𝜂𝑅. (42) This together with (31) implies sup_𝑚∈ℕsup_{𝐳∈𝑍𝑚}log𝒩_2,𝐳ℱ_𝑅,𝜂≤log𝒩ℬ₁,𝜂𝑅≤𝑐_𝑝𝑅^𝑝𝜂^−𝑝. (43) Hence, all the conditions in Lemma 13 hold, and we know that, for any 𝑡>0, with confidence 1−𝑒^−𝑡, there holds, for every 𝑓∈ℬ_𝑅, ℰ𝜋𝑓−ℰ𝑓^∗−ℰ_𝐳𝜋𝑓−ℰ_𝐳𝑓^∗≤12𝜃_𝑅^1−𝑠ℰ𝜋𝑓−ℰ𝑓^∗𝑠+𝑐^′_𝑝𝜃_𝑅𝑐+2^𝑠𝑡𝑚^1/(2−𝑠)+36(𝑀+𝜀)𝑡𝑚. (44) Here, 𝜃_𝑅=max𝑐_𝑠^{(2−𝑝)/(4−2𝑠+𝑝𝑠)}𝑐_𝑝𝑅^𝑝𝑚^{2/(4−2𝑠+𝑝𝑠)},[2(𝑀+𝜀)]^{(2−𝑝)/(2+𝑝)}𝑐_𝑝𝑅^𝑝𝑚^2/(2+𝑝). (45) Recall an elementary inequality 1𝜄+1𝜈1=1,with𝜄,𝜈>1⟹𝜓𝜔≤𝜄𝜓^𝜄+1𝜈𝜔^𝜈,∀𝜓,𝜔>0. (46) Applying it with 𝜓=[ℰ(𝜋(𝑓))−ℰ(𝑓^∗)]^𝑠, 𝜔=𝜃_𝑅^1−𝑠, 𝜄=1/𝑠 to the first term of (44), we can derive the conclusion.

It remains to find a ball containing 𝑓_𝐳,𝜆 for all 𝐳∈𝑍^𝑚.

Lemma 15. Let 1≤𝑞<∞, 𝑓_𝐳,𝜆 be defined by (9). Then, for any 𝐳∈𝑍^𝑚, one has 𝑓_{𝐳,𝜆𝐾}≤𝜅𝑚^1−(1/𝑞)𝑀𝜆^1/𝑞. (47)

Proof. For any 𝜁>0, there exists {𝛼_𝑖}^𝑚_𝑖=1∈ℝ^𝑚, such that 𝑓_𝐳,𝜆=∑^𝑚_𝑖=1𝛼_𝑖𝐾_𝑥𝑖 and ^𝑚_𝑖=1𝛼_𝑖^𝑞≤Ω_𝐳𝑓_𝐳,𝜆+𝜁. (48) Taking 𝑓=0 in (9), we can see that 𝜆Ω_𝐳𝑓_𝐳,𝜆≤ℰ_𝐳𝑓_𝐳,𝜆+𝜆Ω_𝐳𝑓_𝐳,𝜆≤ℰ_𝐳(0)≤𝑀. (49) It follows that ^𝑚_𝑖=1𝛼_𝑖^𝑞≤𝑀𝜆+𝜁. (50) When 𝑞>1, by the Hölder inequality, we can see that 𝑓_{𝐳,𝜆𝐾}=^𝑚_𝑖=1𝛼_𝑖𝐾_𝑥𝑖𝐾≤𝜅^𝑚_𝑖=1𝛼_𝑖≤𝜅𝑚^{1−(1/𝑞)𝑚}_𝑖=1𝛼_𝑖^𝑞1/𝑞≤𝜅𝑚^1−(1/𝑞)𝑀𝜆+𝜁^1/𝑞. (51) It is easy to check that (51) still holds for 𝑞=1. Let 𝜁→0; we then get the assertion.

From Lemma 15 and Proposition 14, we can get the following.

Corollary 16. If assumptions (26) and (31) hold, then, for any 𝑡>1, with confidence 1−𝑒^−𝑡, there holds 𝑆₂()≤1𝐳,𝜆2ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^∗+𝐶₁𝑡𝑚^{𝑝−(𝑝/𝑞)−1}𝜆^{−𝑝/𝑞2/(4−2𝑠+𝑝𝑠)}+𝑚^{𝑝−(𝑝/𝑞)−1}𝜆^{−𝑝/𝑞2/(2+𝑝)}+𝑚^1/(𝑠−2), (52) where 𝐶₁1∶=2+𝑐^′_𝑝𝑐_𝑠^{(2−𝑝)/(4−2𝑠+𝑝𝑠)}𝑐_𝑝𝜅^𝑝𝑀^{𝑝/𝑞2/(4−2𝑠+𝑝𝑠)}+[2(𝑀+𝜀)]^{(2−𝑝)/(2+𝑝)}𝑐_𝑝𝜅^𝑝𝑀^{𝑝/𝑞2/(2+𝑝)}+2𝑐_𝑠^1/(2−𝑠)+36(𝑀+𝜀). (53)

3.2. Deriving Learning Rates

Combining the estimation in Sections 2.2 and 3.1, we can derive an explicit learning rate for scheme (9) by suitably selecting the regularization parameters 𝜆 and 𝜇.

Theorem 17. Suppose that assumptions (20), (26), and (31) are satisfied, for any 0<𝛿<1, by taking 𝜆=𝑚^𝑞−1𝜇^𝛽+𝑞, 𝜇=(1/𝑚)^{min{2/(1+𝛽),2𝑞/((4−2𝑠+𝑝𝑠)𝛽𝑞+2𝑝(𝛽+𝑞))}}, and we have, with confidence 1−𝛿, ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^∗3≤𝐶log𝛿1𝑚^{min{2𝛽/(1+𝛽),2𝑞𝛽/((4−2𝑠+𝑝𝑠)𝛽𝑞+2𝑝(𝛽+𝑞))}}, (54) where 𝐶 is a constant independent of 𝑚 or 𝛿.

Proof. Putting Theorem 7, Lemma 12, Corollary 16, and assumption (20) into Proposition 5, by taking 𝜆=𝑚^𝑞−1𝜇^𝛽+𝑞, we find that, for any 𝑡>1, with confidence 1−3𝑒^−𝑡, ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^∗≤2𝐶₁𝑡𝑚⁻¹𝜇^{−(𝑝𝛽/𝑞)−𝑝2/(4−2𝑠+𝑝𝑠)}+𝑚⁻¹𝜇^{−(𝑝𝛽/𝑞)−𝑝2/(2+𝑝)}+𝑚^1/(𝑠−2)+7𝜅𝑡𝑐_𝛽𝜇6𝑚^(𝛽−1)/2+8(𝑀+𝜀)𝑡+3𝑚2𝑐_𝑠𝑡𝑚^1/(2−𝑠)+2𝑐_𝛽𝜇^𝛽+𝜇^𝛽2^𝑞≤𝐶₂𝑡1𝑚^𝑞𝜇^{𝑝(𝛽+𝑞)2/(4−2𝑠+𝑝𝑠)𝑞}+1𝑚^𝑞𝜇^{𝑝(𝛽+𝑞)2/(2+𝑝)𝑞}+1𝑚^1/(2−𝑠)+𝜇^(𝛽−1)/2𝑚+𝜇^𝛽. (55) Here, 𝐶₂∶=2[𝐶₁+(7/6)𝜅𝑐_𝛽+8(𝑀+𝜀)/3+(2𝑐_𝑠)^1/(2−𝑠)+2𝑐_𝛽+2^−𝑞]. According to the choice of 𝜇, we can easily check that 1𝑚^1/(2−𝑠)≤𝜇^𝛽,𝜇^(𝛽−1)/2𝑚≤𝜇^𝛽,1𝑚^𝑞𝜇^{𝑝(𝛽+𝑞)2/(2+𝑝)𝑞}≤1𝑚^𝑞𝜇^{𝑝(𝛽+𝑞)2/(4−2𝑠+𝑝𝑠)𝑞}≤𝜇^𝛽. (56) So, our theorem follows by taking 𝐶=5𝐶₂ and 𝑡=log(3/𝛿).

Remark 18. Theorem 17 provides an explicit learning rate for 𝑙^𝑞  (1≤𝑞<∞) coefficient-based regularized SVMR. This learning rate is independent of the dimension 𝑛 of the input space 𝑋. We do not require the marginal distribution 𝜌_𝑋 and the kernel 𝐾 to satisfy any additional regularity condition, such as the 𝐿_𝜏 condition.

Remark 19. Another advantage of coefficient-based regularization scheme is its flexibility in choosing the kernel. For instance, [26, 27] consider the least square regression with indefinite kernels and an 𝑙²-coefficient regularization, where they relax the requirement of the kernel to be only continuous and uniformly bounded bivariate function on 𝑋. It will be a very interesting topic in future work to extend the method in this paper to the indefinite kernel setting.

Let us end this paper by comparing our result with the learning rate presented in [7] in a special case 𝜀=0. To this end, we reformulate [7, Theorem 2.3] for 𝜀=0 as follows.

Proposition 20. If 𝐾∈𝐶^∞(𝑋×𝑋), 𝑓_𝜌,1/2∈ℋ_𝐾, and 𝑓_𝜌,1/2 is of 𝑎-average of type 2 for some 𝑎∈(0,∞], taking 𝜇=𝑚^{−(𝑎+1)/(𝑎+2)}, 0<𝜍<(𝑎+1)/2(𝑎+2), then, with 𝑎^∗=2𝑎/(𝑎+1), for any 0<𝛿<1, with confidence 1−𝛿, one has 𝜋(𝑓_𝐳,𝜇)−𝑓_{𝜌,1/2𝐿𝑎∗𝜌𝑋}≤𝐶₁3log𝛿𝑚^{𝜍−(𝑎+1)/2(𝑎+2)}, (57) where 𝐶₁ is a constant independent of 𝑚 or 𝛿.

By Theorem 17, we can see the following.

Corollary 21. Under the same conditions of Proposition 20, for 𝜀=0, by taking 𝜆=𝑚^{(𝑞−2𝑎−3)/(𝑎+2)}, one has, for any 0<𝛿<1, with confidence 1−𝛿, 𝜋(𝑓_𝐳,𝜆)−𝑓_{𝜌,1/2𝐿𝑎∗𝜌𝑋}≤𝐶₂3log𝛿𝑚^{−(𝑎+1)/2(𝑎+2)}, (58) where 𝐶₂ is a constant independent of 𝑚 or 𝛿.

Proof. Note that 𝑓^∗=𝑓_𝜌,1/2 when 𝜀=0. From [24, Theorem 2.7], we know that 𝜋(𝑓_𝐳,𝜆)−𝑓_{𝜌,1/2𝐿𝑎∗𝜌𝑋}≤𝑐_𝜌ℰ(𝜋(𝑓_𝐳,𝜆))−ℰ(𝑓_𝜌,1/2)^1/2, (59) and here 𝑐_𝜌 is a constant independent of 𝑚 or 𝛿.
Since 𝑓_𝜌,1/2∈ℋ_𝐾, we know that (20) holds with 𝛽=1.  𝑓_𝜌,1/2 is 𝑎-average type 2 which implies that (26) is satisfied with 𝑠=𝑎/(𝑎+1). Since 𝑋⊂ℝ^𝑛 and 𝐾∈𝐶^∞(𝑋×𝑋), we know from [28] that (31) holds true for any 𝑝>0. Therefore, let 𝑝→0; according to Theorem 17, by taking 𝜆=𝑚^𝑞−1𝜇^1+𝑞, 𝜇=𝑚^{−(𝑎+1)/(𝑎+2)}, we have, for any 0<𝛿<1, with confidence 1−𝛿, ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓_𝜌,1/23≤𝐶log𝛿1𝑚^{(𝑎+1)/(𝑎+2)}. (60) This together with (59) proves the corollary with the constant 𝐶₂=𝑐_𝜌√𝐶.

Corollary 21 shows us that the learning rate presented in Theorem 17 for 𝑙^𝑞-coefficient regularized SVMR is faster than the one given in [7] for RKHS norm regularized learning schemes at least in the case of 𝜀=0.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

The authors thank the referees for their valuable comments and helpful suggestions which greatly improved this work. The research is supported by NSF of China under Grants 11171014.