Learning Rates for 𝑙¹-Regularized Kernel Classifiers

Abstract

We consider a family of classification algorithms generated from a regularization kernel scheme associated with 𝑙¹-regularizer and convex loss function. Our main purpose is to provide an explicit convergence rate for the excess misclassification error of the produced classifiers. The error decomposition includes approximation error, hypothesis error, and sample error. We apply some novel techniques to estimate the hypothesis error and sample error. Learning rates are eventually derived under some assumptions on the kernel, the input space, the marginal distribution, and the approximation error.

1. Introduction

Let 𝑋 be a compact subset of ℝ^𝑛, 𝑌={−1,1}. Classification algorithms produce binary classifiers 𝒞∶𝑋→𝑌, such a classifier 𝒞 labels a class 𝒞(𝑥)∈𝑌 for each point 𝑥∈𝑋. The prediction power of the classifier 𝒞 is measured by its misclassification error. If 𝜌 is a probability distribution on 𝑍∶=𝑋×𝑌, then the misclassification error of 𝒞 is defined by ℛ(𝒞)∶=Prob𝒞(𝑥)≠𝑦=_𝑋𝜌𝑦≠𝒞(𝑥)∣𝑥𝑑𝜌_𝑋. (1) Here 𝜌_𝑋 is the marginal distribution on 𝑋 and 𝜌(⋅∣𝑥) is the conditional probability measure at 𝑥 induced by 𝜌. The classifier minimizing the misclassification error is called the Bayes rule 𝑓_𝑐 and is given by 𝑓_𝑐(𝑥)=1,if𝜌𝑦=1∣𝑥≥𝜌𝑦=−1∣𝑥,−1,otherwise. (2)

The classifiers considered in this paper have the form sgn(𝑓), defined as sgn(𝑓)(𝑥)=1, if 𝑓(𝑥)≥0, and sgn(𝑓)(𝑥)=−1, if 𝑓(𝑥)<0, induced by real-valued functions 𝑓∶𝑋→ℝ. Those functions are generated from a regularization scheme associated with convex loss function (see [1]).

Definition 1. A continuous function 𝑉: ℝ→ℝ⁺ is called a classifying loss (function) if it is convex, differentiable at 0 with 𝑉^′(0)<0, and 1 is the smallest real for which the value of 𝑉 is zero.

Typical examples of classifying loss include(1) hinge loss 𝑉_ℎ(𝑡)=(1−𝑡)₊=max{1−𝑡,0} for the classical support vector machines (SVM) classifier; see [2–5];(2) least square loss 𝑉_ls(𝑡)=(1−𝑡)²; see for example [6, 7];(3)𝑞-norm (𝑞>1) SVM loss 𝑉_𝑞(𝑡)=(1−𝑡)^𝑞₊; see [8, 9].

The following concept describes the increment of 𝑉.

Definition 2. One says that 𝑉 has a increment exponent 𝜃≥1, if there exists some 𝑐_𝜃>0 such that |𝑉(𝑡)|≤𝑐_𝜃(1+|𝑡|)^𝜃,𝑉^′_±(𝑡)≤𝑐_𝜃(1+|𝑡|)^𝜃−1,∀𝑡∈ℝ, (3) where 𝑉^′_± denotes the right and left derivatives of 𝑉.

It is easy to see that 𝑉_ℎ,𝑉_ls, and 𝑉_𝑞 satisfy Definition 2 with increment exponent 1, 2, and 𝑞, respectively.

Let 𝐳∶={𝑧_𝑖}^𝑚_𝑖=1={(𝑥_𝑖,𝑦_𝑖)}^𝑚_𝑖=1∈𝑍^𝑚 be a set of samples independently drawn according to 𝜌; we call ℰ_𝐳𝑓∶=ℰ^𝑉_𝐳𝑓=1𝑚^𝑚_𝑖=1𝑉𝑦_𝑖𝑓𝑥_𝑖 (4) the empirical error with respect to 𝐳. Regularized learning schemes are implemented by minimizing a penalized version of empirical error over a set of functions, called a hypothesis space.

Definition 3. Given a classifying loss 𝑉 and a hypothesis space ℋ, Ω∶ℋ→ℝ⁺ is a penalty functional called regularizer that reflects the constrains imposed on functions from ℋ. The regularized classifier is then defined as sgn(𝑓_𝐳), where 𝑓_𝐳 is a minimizer of the following regularization scheme: 𝑓_𝐳∶=argmin_𝑓∈ℋℰ_𝐳𝑓+𝜆Ω𝑓. (5) Here 𝜆 is a regularization parameter which may depend on the sample size 𝜆=𝜆(𝑚) with lim_𝑚→∞𝜆(𝑚)=0.

Choosing different hypothesis spaces and regularizers in (5) will lead to different regularization algorithms. These learning algorithms are often based on a kernel function 𝐾∶𝑋×𝑋→ℝ (see, e.g., [10]). One way appears naturally when 𝐾 is a Mercer kernel. Such a kernel is continuous, symmetric, and positive semidefinite on 𝑋×𝑋. The reproducing kernel Hilbert spaces (RKHS) ℋ_𝐾 associated with the Mercer kernel 𝐾 is defined [11] to be the completion of the linear span of functions {𝐾_𝑥∶=𝐾(𝑥,⋅)∶𝑥∈𝑋} with the inner product: 𝐾_𝑥,𝐾_𝑢𝐾=𝐾(𝑥,𝑢), (6) and the reproducing property is given by 𝐾_𝑥,𝑓_𝐾=𝑓(𝑥),∀𝑥∈𝑋,𝑓∈ℋ_𝐾. (7) By setting ℋ=ℋ_𝐾, Ω(𝑓)=‖𝑓‖²_𝐾, (5) becomes the classical regularized classification scheme: 𝑓_𝐳,𝜆∶=argmin_{𝑓∈ℋ𝐾}ℰ_𝐳𝑓+𝜆𝑓²_𝐾. (8) Its mathematical analysis has been well understood with various techniques in extensive literature (see, e.g., [4, 5, 8, 12–15]). In this paper we will consider a different regularization scheme in RKHS for classification; in our setting, the regularizer is 𝑙¹-norm of the coefficients in the kernel expansions over the sample points.

Definition 4. Let ℋ_𝐾,𝐳∶=^𝑚_𝑖=1𝑎_𝑖𝐾_𝑥𝑖∶𝑎_𝑖,Ω∈ℝ_𝐳𝑓∶=inf^𝑚_𝑖=1𝑎_𝑖∶𝑓=^𝑚_𝑖=1𝑎_𝑖𝐾_𝑥𝑖. (9) Then the 𝑙¹-regularized classification scheme is given as 𝑓_𝐳,𝜆∶=argmin_{𝑓∈ℋ𝐾,𝐳}ℰ_𝐳𝑓+𝜆Ω_𝐳𝑓. (10)

Algorithm (10) can be efficiently computed because it reduces to solve a convex optimization problem in a finite dimensional space ℋ_𝐾,𝐳, containing linear combinations of kernels centered on the training points.

In the last ten years, learning with 𝑙¹-regularization has attracted much attention. The increasing interest is mainly brought by the progress of the Lasso algorithm [16–18] and compressive sensing [19, 20], in which 𝑙¹-regularizer is able to yield sparse representation of the resulting minimizer. Kernel methods formulate learning and estimation problems in RKHS of functions expanded in terms of kernels. There have been a series of papers to investigate the learning ability of coefficient-based regularization kernel regression methods (see, e.g., [21–25]). However, as we know, there are currently a few results on classification based on kernel designing. For example, [26] studies classification problem with hinge loss 𝑉_ℎ and 𝑙¹ complexity regularization in a finite-dimensional hypothesis space spanned by a set of base functions. While it does not assume a kernel setting nor is it assumed that the expansion must be in terms of the sample points, so the problem of data-dependent hypothesis space is not present there. Although [27] provided an error analysis for linear programming SVM classifiers by means of a stepping-stone from quadratic programming SVM to linear programming SVM, no evidence shows that this method can still work for other classifying losses.

In this paper we will present an elaborate error analysis for algorithm (10), and we use a modified error decomposition technique that was firstly introduced in [28], by dealing with the approximation error, the hypothesis error, and the sample error, and we derive an explicit learning rate for classification scheme (10) under some assumptions.

2. Preliminaries

For a classifying loss 𝑉, we define the generalization error of 𝑓∶𝑋→ℝ as ℰ𝑓∶=ℰ^𝑉𝑓=_𝑍𝑉𝑦𝑓(𝑥)𝑑𝜌. (11) Let 𝑓^𝑉_𝜌 be a measurable function minimizing the generalization error: 𝑓^𝑉_𝜌∶=argminℰ𝑓, (12) where the minimum is taken over all measurable functions. According to Theorem 3(c) in [12], we may always choose an 𝑓^𝑉_𝜌 satisfying 𝑓^𝑉_𝜌(𝑥)∈[−1,1] for each 𝑥∈𝑋. This choice will be taken throughout the paper.

Estimating the excess misclassification error ℛsgn𝑓_𝐳,𝜆−ℛ𝑓_𝑐 (13) for classification scheme (10) is our main purpose. The following comparison theorem (see [7, 8, 12]) describes the relationship between excess misclassification error and excess generalization error.

Proposition 5. If 𝑉 is a classifying loss, then, for any measurable function 𝑓, ℛsgn𝑓−ℛ𝑓_𝑐≤ℰ𝑓−ℰ𝑓_𝑐𝑖𝑓𝑉(𝑡)=(1−𝑡)₊,𝑐_𝑉ℰ𝑓−ℰ𝑓^𝑉_𝜌𝑖𝑓𝑉^′′(0)≥0, (14) where 𝑐_𝑉 is some constant dependent on 𝑉.

Since 𝑓^𝑉_𝜌(𝑥)∈[−1,1], we can improve the error estimates by replacing values of 𝑓 by projections onto [−1,1]. The idea of the following projection operator was firstly introduced for this purpose in [29].

Definition 6. The projection operator 𝜋 is defined on the space of measurable functions 𝑓∶𝑋→ℝ as 𝜋𝑓(𝑥)=1,if𝑓(𝑥)>1,−1,if𝑓(𝑥)𝑓(𝑥),<−1,if−1≤𝑓(𝑥)≤1. (15)

The definition of classifying loss implies that 𝑉(𝑦𝜋(𝑓)(𝑥))≤𝑉(𝑦𝑓(𝑥)), so ℰ𝜋𝑓≤ℰ𝑓,ℰ_𝐳𝜋𝑓≤ℰ_𝐳𝑓. (16) It is trivial that sgn(𝜋(𝑓))=sgn(𝑓). By Proposition 5, ℛsgn𝑓−ℛ𝑓_𝑐≤ℰ𝜋𝑓−ℰ𝑓_𝑐if𝑉(𝑡)=(1−𝑡)₊,𝑐_𝑉ℰ𝜋𝑓−ℰ𝑓^𝑉_𝜌if𝑉^′′(0)≥0. (17)

So it is sufficient for us to bound (13) by means of ℰ(𝜋(𝑓_𝐳,𝜆))−ℰ(𝑓^𝑉_𝜌), which in turn can be estimated by an error decomposition technique. However, there are essential differences between algorithm (8) and (10). For example, the hypothesis space ℋ_𝐾,𝐳 and the regularizer Ω_𝐳(𝑓) in (10) are dependent on samples 𝐳. This causes that the standard error analysis methods of (8) (see, e.g., [8, 12, 13, 30]) cannot be applied to (10) any more. This difficulty was overcome in [28] by introducing a modified error decomposition with an extra hypothesis error term. In this paper we apply the same underlying idea to classification scheme (10). To this end, we need to consider a Banach space containing all of the possible hypothesis space ℋ_𝐾,𝐳.

Definition 7. The Banach space ℋ₀ is defined as the function set on 𝑋 containing all functions of the form 𝑓=^∞_𝑗=1𝑎_𝑗𝐾_𝑢𝑗,𝑎_𝑗^∞_𝑗=1∈𝑙¹,𝑢_𝑗^∞_𝑗=1⊂𝑋, (18) with the norm 𝑓∶=inf^∞_𝑗=1𝑎_𝑗∶𝑓=^∞_𝑗=1𝑎_𝑗𝐾_𝑢𝑗. (19)

Obviously, ℋ_𝐾,𝐳⊂ℋ₀,∀𝐳∈𝑍^𝑚. (20) By the continuity of 𝐾 and compactness of 𝑋, we have 𝜅∶=sup_𝑥∈𝑋𝐾(𝑥,𝑥)<∞. (21) It implies that ℋ₀ is a subset of the continuous function space 𝐶(𝑋), and 𝑓_∞≤𝜅𝑓,∀𝑓∈ℋ₀. (22) To formulate the error decomposition for scheme (10), we introduce a regularization function as 𝑓_𝜆∶=argmin_𝑓∈ℋ0ℰ𝑓+𝜆𝑓. (23)

Proposition 8. Let 𝑓_𝐳,𝜆 be defined in (10), 𝜆>0; then ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^𝑉_𝜌+𝜆𝑓_𝐳,𝜆≤𝑆(𝐳,𝜆)+𝑃(𝐳,𝜆)+𝐷(𝜆). (24) Here 𝑆(𝐳,𝜆)∶=ℰ𝜋𝑓_𝐳,𝜆−ℰ_𝐳𝜋𝑓_𝐳,𝜆+ℰ_𝐳𝑓_𝜆−ℰ𝑓_𝜆,𝑃(𝐳,𝜆)∶=ℰ_𝐳𝜋𝑓_𝐳,𝜆+𝜆Ω_𝐳𝑓_𝐳,𝜆−ℰ_𝐳𝑓_𝜆+𝜆𝑓_𝜆,𝐷(𝜆)∶=ℰ𝑓_𝜆−ℰ𝑓^𝑉_𝜌+𝜆𝑓_𝜆=inf_𝑓∈ℋ0ℰ𝑓−ℰ𝑓^𝑉_𝜌+𝜆𝑓. (25)(26)

Proof. ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^𝑉_𝜌+𝜆𝑓_𝐳,𝜆≤ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^𝑉_𝜌+𝜆Ω_𝐳𝑓_𝐳,𝜆=ℰ𝜋𝑓_𝐳,𝜆−ℰ_𝐳𝜋𝑓_𝐳,𝜆+ℰ_𝐳𝜋𝑓_𝐳,𝜆+𝜆Ω_𝐳𝑓_𝐳,𝜆+ℰ_𝐳𝑓_𝜆−ℰ𝑓_𝜆−ℰ_𝐳𝑓_𝜆+𝜆𝑓_𝜆+ℰ𝑓_𝜆−ℰ𝑓^𝑉_𝜌+𝜆𝑓_𝜆=𝑆(𝐳,𝜆)+𝑃(𝐳,𝜆)+𝐷(𝜆). (27)

𝑃(𝐳,𝜆) and 𝑆(𝐳,𝜆) are called hypothesis error and sample error, and they will be estimated, respectively, in next two sections. 𝐷(𝜆) is independent of samples and usually called approximation error, and it characterizes the approximation ability of the function space ℋ₀ with respect to target function 𝑓^𝑉_𝜌. We will assume that, for some constants 0<𝛽≤1 and 𝑐_𝛽>0, 𝐷(𝜆)≤𝑐_𝛽𝜆^𝛽,∀𝜆>0. (28)

3. Estimating the Hypothesis Error

In this section we bound the hypothesis error 𝑃(𝐳,𝜆) by a technique of scattered data interpolation which was firstly used in kernel regression context in [25]. To this end, we need some assumptions on input space 𝑋, margin distribution 𝜌_𝑋, and kernel 𝐾. Denote |⋅|₂ the Euclidean norm in ℝ^𝑛.

Definition 9. A subset 𝑋 of ℝ^𝑛 is said to satisfy an interior cone condition if there exist an angle 𝜗∈(0,𝜋/2), a radius 𝑅>0, and a unit vector 𝜁(𝑥) for every 𝑥∈𝑋 such that the cone 𝐶((𝑥))=𝑥,𝜁,𝜗,𝑅𝑥+𝑡𝑤∶𝑤∈ℝ^𝑛,|𝑤|₂=1,𝑤^𝑇𝜁(𝑥)≥cos𝜗,𝑡∈[0,𝑅] (29) is contained in 𝑋.

Definition 10. The margin distribution 𝜌_𝑋 is said to satisfy condition 𝐿_𝜏 with 0<𝜏<∞ if for some 𝑐_𝜏>0𝜌_𝑋𝑢∈𝑋∶|𝑢−𝑥|₂<𝑟≥𝑐_𝜏𝑟^𝜏,∀𝑥∈𝑋,0<𝑟≤1. (30)

Recall that, for 𝑠∈ℕ, the space 𝐶^𝑠(𝑋) consists of functions 𝑓 whose partial derivative 𝐷^𝑑𝑓 is continuous for every 𝑑=(𝑑₁,𝑑₂,…,𝑑_𝑛)∈ℕ^𝑛 with |𝑑|∶=𝑑₁+⋯+𝑑_𝑛≤𝑠, and ‖𝑓‖_𝐶𝑠∶=∑_{|𝑑|≤𝑠}‖𝐷^𝑑𝑓‖_∞. Throughout the paper we assume the kernel 𝐾∈𝐶^𝑠(𝑋×𝑋) with 𝑠>0.

Definition 11. a set {𝑥₁,𝑥₂,…,𝑥_𝑚}⊂𝑋 is said to be Δ-dense if for any 𝑥∈𝑋 there exists some 1≤𝑖≤𝑚 such that |𝑥−𝑥_𝑖|₂<Δ.

The following lemma derived from [31] describes a local polynomial reproduction and it is the key point to bound the hypothesis error.

Lemma 12. Suppose that 𝑋⊂ℝ^𝑛 is compact and satisfies an interior cone condition with some radius 𝑅>0 and angle 𝜗∈(0,𝜋/2). Fix 𝑠∈ℕ with 𝑠≥2. Assume that the point set {𝑥₁,𝑥₂,…,𝑥_𝑚}⊂𝑋 is Δ-dense with Δ≤𝑅/𝑐₀ for some constant 𝑐₀ depending on 𝑋 and 𝑠; then, for any 𝑢∈𝑋, there exist real number 𝑏_𝑖(𝑢), 𝑖=1,2,…,𝑚, satisfying that(1)∑^𝑚_𝑖=1𝑏_𝑖(𝑢)𝑝(𝑥_𝑖)=𝑝(𝑢),𝑝(⋅) is any polynomial of degree at most 𝑠 on ℝ^𝑛,(2)∑^𝑚_𝑖=1|𝑏_𝑖(𝑢)|≤2,(3)𝑏_𝑖(𝑢)=0 for those 𝑢 satisfying |𝑢−𝑥_𝑖|₂>𝑐₀Δ.

Proposition 13. Let 𝑉 be a classifying loss satisfying (3), 𝐾∈𝐶^𝑠(𝑋×𝑋) for some 𝑠∈ℕ, 𝑠≥2. If 𝑋 satisfies the conditions in Lemma 12 and {𝑥₁,𝑥₂,…,𝑥_𝑚} is Δ-dense in 𝑋 with Δ≤𝑅/𝑐₀, then 𝑃(𝐳,𝜆)≤𝐷(𝜆)+2𝑐_𝜃‖𝐾‖_𝐶^𝑠2^𝜃−1+(4𝜅)^𝜃−1×𝐷(𝜆)𝜆^𝜃+𝐷(𝜆)𝜆𝑐^𝑠₀Δ^𝑠. (31)

Proof. We know from (26) that ‖𝑓_𝜆‖≤𝐷(𝜆)/𝜆. So for any 𝜂>0, 𝑓_𝜆 can be written as 𝑓_𝜆=∑^∞_𝑗=1𝛽_𝑗𝐾_𝑢𝑗 with 𝑢_𝑗∈𝑋 and 𝑓_𝜆≤^∞_𝑗=1𝛽_𝑗<𝑓_𝜆𝐷(𝜆)+𝜂≤𝜆+𝜂. (32) At the same time, there exists some 𝑁∈ℕ such that ∑^∞_𝑗=𝑁|𝛽_𝑗|<𝜂, and thus ^𝑁_𝑗=1𝛽_𝑗𝐾_𝑢𝑗−𝑓_𝜆∞≤𝜅^∞_𝑗=𝑁𝛽_𝑗≤𝜅𝜂. (33) Fix 𝑥∈𝑋 and 𝑗∈{1,2,…,𝑁}, and we can take 𝑝_𝑥 as the Taylor polynomial of 𝐾_𝑥 of degree 𝑠−1 at 𝑢_𝑗. Then by Lemma 12, there exists {𝑏_𝑖(𝑢_𝑗)}^𝑚_𝑖=1∈ℝ^𝑚 such that _{𝑖∈𝐼(𝑢𝑗)}𝑏_𝑖𝑢_𝑗𝑝_𝑥𝑥_𝑖=𝑝_𝑥𝑢_𝑗=𝐾_𝑥𝑢_𝑗,_{𝑖∈𝐼(𝑢𝑗)}𝑏_𝑖𝑢_𝑗≤2, (34) where 𝐼(𝑢_𝑗)={𝑖∈{1,2,…,𝑚}∶|𝑥_𝑖−𝑢_𝑗|₂≤𝑐₀Δ}. Moreover, max_{𝑖∈𝐼(𝑢𝑗)}𝐾_𝑥𝑥_𝑖−𝑝_𝑥𝑥_𝑖≤‖𝐾‖_𝐶^𝑠𝑐₀Δ^𝑠. (35) It follows from (34) that 𝐾_𝑢𝑗(𝑥)−_{𝑖∈𝐼(𝑢𝑗)}𝑏_𝑖𝑢_𝑗𝐾_𝑥𝑖(𝑥)=𝐾_𝑥𝑢_𝑗−_{𝑖∈𝐼(𝑢𝑗)}𝑏_𝑖𝑢_𝑗𝐾_𝑥𝑥_𝑖=_{𝑖∈𝐼(𝑢𝑗)}𝑏_𝑖𝑢_𝑗𝑝_𝑥𝑥_𝑖−𝐾_𝑥𝑥_𝑖≤2‖𝐾‖_𝐶^𝑠𝑐₀Δ^𝑠. (36) The above bound holds for every 𝑥∈𝑋 and 𝑗∈{1,2,…,𝑁}, so ^𝑁_𝑗=1𝛽_𝑗𝐾_𝑢𝑗−_{𝑖∈𝐼(𝑢𝑗)}𝑏_𝑖𝑢_𝑗𝐾_𝑥𝑖∞≤2‖𝐾‖_𝐶^𝑠𝑐₀Δ^𝑠𝑁_𝑗=1𝛽_𝑗. (37) This together with (33) and (32) implies ^𝑁_𝑗=1𝛽_{𝑗𝑢𝑖∈𝐼𝑗}𝑏_𝑖𝑢_𝑗𝐾_𝑥𝑖−𝑓_𝜆∞≤𝜅𝜂+2‖𝐾‖_𝐶^𝑠𝑐₀Δ^𝑠𝐷(𝜆)𝜆+𝜂. (38) Denote 𝑓₀=∑^𝑁_𝑗=1𝛽_𝑗∑_{𝑖∈𝐼(𝑢𝑗)}𝑏_𝑖(𝑢_𝑗)𝐾_𝑥𝑖∈ℋ_𝐾,𝐳; we get from (16), (10), and (32) that ℰ_𝐳𝜋𝑓_𝐳,𝜆+𝜆Ω_𝐳𝑓_𝐳,𝜆≤ℰ_𝐳𝑓_𝐳,𝜆+𝜆Ω_𝐳𝑓_𝐳,𝜆≤ℰ_𝐳𝑓₀+𝜆Ω_𝐳𝑓₀≤ℰ_𝐳𝑓₀+2𝜆𝑓_𝜆+𝜂. (39) Since 𝑉 is convex and satisfies (3), we have, for any 𝑡₁,𝑡₂∈ℝ, 𝑉𝑡₁−𝑉𝑡₂≤max𝑉^′_±𝑡₁,𝑉^′_±𝑡₂𝑡₁−𝑡₂≤𝑐_𝜃1+max𝑡₁,𝑡₂^𝜃−1𝑡₁−𝑡₂. (40) This in connect with (22), (32), and (38) yields ℰ_𝐳𝑓₀−ℰ_𝐳𝑓_𝜆≤𝑐_𝜃𝐷(𝜆)1+2𝜅𝜆+𝜂^𝜃−1𝑓₀−𝑓_𝜆∞≤𝑐_𝜃𝐷(𝜆)1+2𝜅𝜆+𝜂^𝜃−1×𝜅𝜂+2‖𝐾‖_𝐶^𝑠𝑐₀Δ^𝑠𝐷(𝜆)𝜆+𝜂. (41) Therefore ℰ_𝐳𝜋𝑓_𝐳,𝜆+𝜆Ω_𝐳𝑓_𝐳,𝜆≤ℰ_𝐳𝑓_𝜆+𝜆𝑓_𝜆+𝐷(𝜆)+2𝜆𝜂+𝑐_𝜃𝐷(𝜆)1+2𝜅𝜆+𝜂^𝜃−1×𝜅𝜂+2‖𝐾‖_𝐶^𝑠𝑐₀Δ^𝑠𝐷(𝜆)𝜆+𝜂. (42) Let 𝜂→0, and we then derive 𝑃(𝐳,𝜆)≤𝐷(𝜆)+2𝑐_𝜃‖𝐾‖_𝐶^𝑠2^𝜃−1+(4𝜅)^𝜃−1×𝐷(𝜆)𝜆^𝜃+𝐷(𝜆)𝜆𝑐^𝑠₀Δ^𝑠. (43)

We can now bound 𝑃(𝐳,𝜆) by the following theorem.

Theorem 14. Let 𝑉 be a classifying loss satisfying (3), 𝐾∈𝐶^𝑠(𝑋×𝑋) for some 𝑠∈ℕ,𝑠≥2. Suppose that 𝑋 satisfies the conditions in Lemma 12 and 𝜌_𝑋 satisfies condition 𝐿_𝜏 with some 𝜏>0, and (28) is valid; then, for any 0<𝛿<1 and 𝑚 satisfying 𝑚≥𝐶₁2log𝛿+log(𝑚+1), (44) with confidence 1−𝛿/2, 𝑃(𝐳,𝜆)≤𝐶₂𝜆^𝛽+𝜆^𝛽−1+𝜆^{(𝛽−1)𝜃}×log(2/𝛿)+log(𝑚+1)𝑚^𝑠/𝜏, (45) where 𝐶₁,𝐶₂ are some constants independent of 𝜆, 𝑚, or 𝛿.

Proof. Applying Lemma  3 in [21], we get that, with confidence 1−𝛿/2, the point set {𝑥₁,𝑥₂,…,𝑥_𝑚} is 𝐴((log(2/𝛿)+log(𝑚+1))/𝑚)^1/𝜏-dense in 𝑋, where 𝐴 is a constant depending on 𝑋, 𝜏, and 𝑐_𝜏. Taking Δ=𝐴((log(2/𝛿)+log(𝑚+1))/𝑚)^1/𝜏, 𝐶₁=(𝑐₀𝐴/𝑅)^𝜏. If 𝑚≥𝐶₁(log(2/𝛿)+log(𝑚+1)), then we have Δ≤𝑅/𝑐₀. So Proposition 13 ensures us that, with confidence 1−𝛿/2, 𝑃(𝐳,𝜆)≤𝑐_𝛽𝜆^𝛽+2𝑐_𝜃‖𝐾‖_𝐶^𝑠2^𝜃−1+(4𝜅)^𝜃−1×𝑐^𝜃_𝛽𝜆^{(𝛽−1)𝜃}+𝑐_𝛽𝜆^(𝛽−1)𝑐^𝑠₀𝐴^𝑠log(2/𝛿)+log(𝑚+1)𝑚^𝑠/𝜏. (46) This proves the theorem by setting 𝐶₂∶=𝑐_𝛽+2𝑐_𝜃‖𝐾‖_𝐶^𝑠(2^𝜃−1+(4𝜅)^𝜃−1)𝑐^𝑠₀𝐴^𝑠(𝑐^𝜃_𝛽+𝑐_𝛽).

4. Estimating the Sample Error

In this section we focus on the sample error, it is the major improvement we make in this paper for the error analysis of algorithm (10).

Definition 15. Let ℱ be a class of functions on 𝑍 and 𝐳∶={𝑧_𝑖}^𝑚_𝑖=1∈𝑍^𝑚. The 𝑙²-metric 𝑑_2,𝐳 is defined on ℱ by 𝑑_2,𝐳1𝑓,𝑔∶=𝑚^𝑚_𝑖=1𝑓(𝑧_𝑖)−𝑔(𝑧_𝑖)^21/2. (47) For every 𝜀>0, the covering number of ℱ with respect to 𝑑_2,𝐳 is 𝒩_2,𝐳()𝑓ℱ,𝜀∶=inf𝑙∈ℕ∶∃_𝑖^𝑙_𝑖=1suchthatℱ=∪^𝑙_𝑖=1𝑓∈ℱ∶𝑑_2,𝐳𝑓,𝑓_𝑖≤𝜀. (48)

The function sets in our situation are balls of ℋ₀ in the form of ℬ_𝑅={𝑓∈ℋ₀∶‖𝑓‖≤𝑅}. We need the 𝑙²-empirical covering number of ℬ₁ defined as 𝒩₂ℬ₁,𝜀∶=sup_𝑚∈ℕsup_{𝐱∈𝑋𝑚}𝒩_2,𝐱ℬ₁,𝜀. (49) According to a bound for 𝑙²-empirical covering number derived in [32], we know that if 𝐾∈𝐶^𝑠(𝑋×𝑋), then log𝒩₂ℬ₁,𝜀≤𝑐_𝑝1𝜀^𝑝,∀𝜀>0, (50) where 𝑐_𝑝 is a constant independent of 𝜀>0, and 𝑝∈(0,2) is a power index defined dy 𝑝=2𝑛,2𝑛+𝑠when0<𝑠≤1,2𝑛,2𝑛+2when1<s𝑛≤1+2,𝑛𝑠,when𝑛𝑠>1+2. (51)

For a measurable function 𝑓∶𝑍→ℝ, denote 𝔼𝑓∶=∫_𝑍𝑓(𝑧)𝑑𝜌. The following definition is a variance-expectation condition for the pair (𝑉,𝜌), which is generally used to achieve tight bounds.

Definition 16. A variance power 𝛼 of the pair (𝑉,𝜌) is a number in [0,1] such that for any 𝑓∶𝑋→[−1,1], there exists some constant 𝑐_𝛼>0 satisfying 𝔼𝑉𝑦𝑓(𝑥)−𝑉𝑦𝑓^𝑉_𝜌(𝑥)²≤𝑐_𝛼ℰ𝑓−ℰ𝑓^𝑉_𝜌^𝛼. (52)

Remark 17. It is easy to see that (52) always holds for 𝛼=0 and 𝑐_𝛼=𝑉²(−1). Larger 𝛼 is possible when 𝑉 has strong convexity or 𝜌 satisfies some noise condition (see [1, 5]).

We are in a position to bound the sample error. Write 𝑆(𝐳,𝜆) as 𝑆()=ℰ𝜋𝑓𝐳,𝜆_𝐳,𝜆−ℰ𝑓^𝑉_𝜌−ℰ_𝐳𝜋𝑓_𝐳,𝜆−ℰ_𝐳𝑓^𝑉_𝜌+ℰ_𝐳𝑓_𝜆−ℰ_𝐳𝑓^𝑉_𝜌−ℰ𝑓_𝜆−ℰ𝑓^𝑉_𝜌∶=𝑆₁(𝐳,𝜆)+𝑆₂(𝐳,𝜆). (53) We will first bound 𝑆₂(𝐳,𝜆), and to this end we need the following one-side Bernstein inequality (see [33]).

Let 𝜉 be a random variable on a probability space 𝑍 with mean 𝔼𝜉=𝜇 and variance 𝜎²(𝜉)=𝜎². If |𝜉−𝜇|≤𝐵 almost everywhere, then for all 𝜂>0Prob_𝐳∈𝑍^𝑚1𝑚^𝑚_𝑖=1𝜉𝑧_𝑖−−𝜇≥𝜂≤exp𝑚𝜂²2𝜎²+().1/3𝐵𝜂 (54)

Proposition 18. Suppose that classifying loss 𝑉 satisfies (3). If (28) and (52) hold, then, for any 0<𝛿<1 with confidence 1−𝛿/4, 𝑆₂(𝐳,𝜆)≤𝐶₃8log𝛿1𝑚^1/(2−𝛼)+𝜆^𝛽+𝜆^{(𝛽−1)𝜃}𝑚, (55) where 𝐶₃ is a constant independent of 𝜆, 𝑚, or 𝛿.

Proof. Denote 𝜉₁∶=𝑉(𝑦𝑓_𝜆(𝑥))−𝑉(𝑦𝜋(𝑓_𝜆)(𝑥)), 𝜉₂∶=𝑉(𝑦𝜋(𝑓_𝜆)(𝑥))−𝑉(𝑦𝑓^𝑉_𝜌(𝑥)). Then 𝑆₂()=1𝐳,𝜆𝑚^𝑚_𝑖=1𝜉₁𝑧_𝑖−𝔼𝜉₁+1𝑚^𝑚_𝑖=1𝜉₂𝑧_𝑖−𝔼𝜉₂. (56) By (22) and (26), we can see that 𝑓_𝜆∞≤𝜅𝑓_𝜆𝐷(𝜆)≤𝜅𝜆. (57) We may assume |𝑓_𝜆(𝑥)|>1, since otherwise 𝜉₁=0. Then from (3), we can derive that 0≤𝜉₁≤𝑉𝑦𝑓_𝜆(𝑥)≤𝑐_𝜃2^𝜃𝜅𝐷(𝜆)𝜆^𝜃,𝜎²𝜉₁≤𝑐_𝜃2^𝜃𝜅𝐷(𝜆)𝜆^𝜃𝔼𝜉₁. (58)
Applying the one-side Bernstein inequality to 𝜉₁ we have, for any 𝑡>1 with confidence 1−𝑒^−𝑡, 1𝑚^𝑚_𝑖=1𝜉₁𝑧_𝑖−𝔼𝜉₁≤2𝑐_𝜃2^𝜃𝜅^𝜃𝑡𝐷(𝜆)3𝑚𝜆^𝜃+2𝑐_𝜃2^𝜃𝜅^𝜃𝑡𝑚𝐷(𝜆)𝜆^𝜃𝔼𝜉₁≤2𝑐_𝜃2^𝜃𝜅^𝜃𝑡𝐷(𝜆)3𝑚𝜆^𝜃+𝑐_𝜃2^𝜃𝜅^𝜃𝑡𝐷(𝜆)2𝑚𝜆^𝜃+𝔼𝜉₁=7𝑐_𝜃2^𝜃𝜅^𝜃𝑡𝐷(𝜆)6𝑚𝜆^𝜃+𝔼𝜉₁. (59)
On the other hand, both 𝜋(𝑓_𝜆)(𝑥) and 𝑓^𝑉_𝜌(𝑥) are contained in [−1,1], and we know from (3) and (52) that 𝜉₂≤𝑐_𝜃2^𝜃,𝜎²𝜉₂≤𝑐_𝛼𝔼𝜉₂^𝛼. (60) Applying the one-side Bernstein inequality again, we have, with confidence 1−𝑒^−𝑡, 1𝑚^𝑚_𝑖=1𝜉₂𝑧_𝑖−𝔼𝜉₂≤4𝑐_𝜃2^𝜃𝑡+3𝑚2𝑡𝑐_𝛼𝔼𝜉₂^𝛼𝑚≤4𝑐_𝜃2^𝜃𝑡+𝛼3𝑚2𝔼𝜉₂+𝛼1−22𝑐_𝛼𝑡𝑚^1/(2−𝛼)≤4𝑐_𝜃2^𝜃𝑡+3𝑚2𝑐_𝛼𝑡𝑚^1/(2−𝛼)+𝔼𝜉₂, (61) where in the second inequality we have used the elementary inequality 1𝜚+1𝜚^∗=1,with𝜚,𝜚^∗1>1⟹𝑎𝑏≤𝜚𝑎^𝜚+1𝜚^∗𝑏^𝜚∗,∀𝑎,𝑏>0. (62) Since 𝔼𝜉₁+𝔼𝜉₂=ℰ(𝑓_𝜆)−ℰ(𝑓^𝑉_𝜌)≤𝐷(𝜆), combining the estimates above, we can get that, under assumption (28) with confidence 1−2𝑒^−𝑡, 𝑆₂(𝐳,𝜆)≤7𝑐_𝜃2^𝜃𝜅^𝜃𝑐^𝜃_𝛽𝜆^{(𝛽−1)𝜃}𝑡+6𝑚4𝑐_𝜃2^𝜃𝑡+3𝑚2𝑐_𝛼𝑡𝑚^1/(2−𝛼)+𝑐_𝛽𝜆^𝛽. (63) Then we prove the proposition by setting 𝑡=log(8/𝛿), and 𝐶₃=7𝑐_𝜃2^𝜃𝜅^𝜃𝑐^𝜃_𝛽6+4𝑐_𝜃2^𝜃3+2𝑐_𝛼^1/(2−𝛼)+𝑐_𝛽. (64)

It is more difficult to bound 𝑆₁(𝐳,𝜆), because it involves the samples 𝐳 and thus runs over a set of functions. To get a better error estimation, An iteration technique is often used to shrink the radius of the ball containing 𝑓_𝐳,𝜆 (see, e.g., [5, 12, 30, 32]); however this process is rather tough and complicated. In this paper We succeed to avoid the prolix iteration by considering the following reweighted empirical process 𝜔_𝑟𝑓ℰ𝜋𝑓−ℰ𝑓^𝑉_𝜌−ℰ_𝐳𝜋𝑓−ℰ_𝐳𝑓^𝑉_𝜌∶𝑓∈ℋ₀. (65) Here 𝜔_𝑟(𝑓)=(𝑟+𝜔(𝑓))⁻¹ for a threshold 𝑟>0 and 𝜔(𝑓)=ℰ(𝜋(𝑓))−ℰ(𝑓^𝑉_𝜌)+𝜆‖𝑓‖. Different from the classical weight function, 𝜔(𝑓) contains the regularization term 𝜆‖𝑓‖ and thus makes it possible to control the variances and ‖𝑓‖ by the threshold 𝑟 simultaneously.

The following concentration inequality is a scaled version of Theorem  2.3 in [34], where the case 𝐵=1 is given.

Lemma 19. Assume that 𝑧₁,…,𝑧_𝑚 are identically distributed according to 𝜌. Let ℱ be a countable set of measurable functions from 𝑍 to [−𝐵,𝐵] and assume that all functions 𝑔 in ℱ satisfy 𝔼𝑔=0, 𝜎²(𝑔)≤𝜎² for some positive real number 𝜎. Denote 𝜉=sup_𝑔∈ℱ1𝑚^𝑚_𝑖=1𝑔𝑧_𝑖. (66) Then, for all 𝑡>0, one has Prob𝜉≥𝔼𝜉+2𝑡𝜎²+2𝐵𝔼𝜉𝑚+𝐵𝑡3𝑚≤𝑒^−𝑡. (67)

This lemma allows us to take care of the deviation of the supremum of a empirical process with respect to its expectation.

Proposition 20. Let 𝑟≥0 and Φ_𝑟∶=sup_𝑓∈ℋ0𝜔_𝑟𝑓ℰ𝜋𝑓−ℰ𝑓^𝑉_𝜌−ℰ_𝐳𝜋𝑓−ℰ_𝐳𝑓^𝑉_𝜌. (68) If (3) and (52) are satisfied, then, for any 𝑡>0 with confidence 1−𝑒^−𝑡, Φ_𝑟≤2𝔼Φ_𝑟+2𝑡𝑐_𝛼𝑟^𝛼−2𝑚+8𝑐_𝜃2^𝜃𝑡.3𝑚𝑟 (69)

Proof. Before presenting the proof, let us first introduce some additional notations: 𝑔_𝑓(𝑧)∶=𝑉𝑦𝜋𝑓(𝑥)−𝑉𝑦𝑓^𝑉_𝜌(𝑥,ℎ_𝑟𝑓)(𝑧)∶=𝜔_𝑟𝑓𝔼𝑔_𝑓−𝑔_𝑓(𝑧=1)𝔼𝑔_𝑓+𝜆𝑓+𝑟𝔼𝑔_𝑓−𝑔_𝑓(𝑧). (70) Then Φ_𝑟=sup_𝑓∈ℋ01𝑚^𝑚_𝑖=1ℎ^𝑟_𝑓𝑧_𝑖. (71) By (3) and (52), we can see that ℎ^𝑟_𝑓(𝑧)≤2𝑐_𝜃2^𝜃𝑟,𝜎²ℎ^𝑟_𝑓≤𝔼𝑔_𝑓²𝔼𝑔_𝑓+𝑟²≤𝑐_𝛼𝔼𝑔_𝑓^𝛼(2/𝛼)𝔼𝑔_𝑓^𝛼((2/(2−𝛼))𝑟)^2−𝛼≤𝑐_𝛼𝑟^𝛼−2. (72) Here in the second inequality of (72), we have used the elementary inequality (62) again with 𝜚=2/𝛼, 𝑎=(𝜚𝔼𝑔_𝑓)^1/𝜚 and 𝑏=(𝜚^∗𝑟)^1/𝜚∗. So applying Lemma 19 to Φ_𝑟, we get with confidence 1−𝑒^−𝑡Φ_𝑟≤𝔼Φ_𝑟+2𝑡𝑐_𝛼𝑟^𝛼−2+4𝑐_𝜃2^𝜃𝔼Φ_𝑟/𝑟𝑚+2𝑐_𝜃2^𝜃𝑡3𝑚𝑟≤𝔼Φ_𝑟+2𝑡𝑐_𝛼𝑟^𝛼−2𝑚+8𝑡𝑐_𝜃2^𝜃𝔼Φ_𝑟+𝑚𝑟2𝑐_𝜃2^𝜃𝑡3𝑚𝑟≤2𝔼Φ_𝑟+2𝑡𝑐_𝛼𝑟^𝛼−2𝑚+8𝑐_𝜃2^𝜃𝑡.3𝑚𝑟 (73)

So we can bound Φ_𝑟 through bounding its expectation. To this end, we need some preparations.

Definition 21. Let (𝑍,𝜌) be a probability space, and ℱ is a class of measurable functions from 𝑍 to ℝ. Set {𝑧_𝑖}^𝑚_𝑖=1 to be 𝑚 independent random variables distributed according to 𝜌 and {𝜖_𝑖}^𝑚_𝑖=1 to be 𝑚 independent Rademacher random variables. Then for 𝜂>0 the local Rademacher average of ℱ is defined by Radℱ,𝑚,𝜂∶=𝔼sup_{𝑓∈ℱ𝔼𝑓2≤𝜂}1𝑚^𝑚_𝑖=1𝜖_𝑖𝑓𝑧_𝑖. (74)

The following lemma was given in [35].

Lemma 22. Let ℱ be a class of measurable functions from 𝑍 to [−𝐵,𝐵]. If for some 𝑎>0 and 𝑝∈(0,2)sup_𝑚∈ℕsup_{𝐳∈𝑍𝑚}log𝒩_2,𝐳(ℱ,𝜀)≤𝑎𝜀^−𝑝,∀𝜀>0, (75) then there exists a constant 𝑐^′_𝑝 depending only on 𝑝 such that 𝑅𝑎𝑑ℱ,𝑚,𝜂≤𝑐^′_𝑝max𝜂^1/2−𝑝/4𝑎𝑚^1/2,𝐵^{(2−𝑝)/(2+𝑝)}𝑎𝑚^2/(2+𝑝). (76)

Lemma 23. Let 𝑧 be a nonnegative stochastic process indexed by 𝑇 and 𝜔(𝑡) a nonnegative nonrandom function defined on 𝑇. Define 𝑟^∗=inf_𝑡∈𝑇𝜔(𝑡). Let 𝜓∶[𝑟^∗,+∞)→[0,+∞) be a function such that 𝜓(4𝑟)≤4^𝜈𝜓(𝑟) for some 0<𝜈<1, and 𝔼sup_{𝑡∈𝑇𝜔(𝑡)≤𝑟}𝑧(𝑡)≤𝜓(𝑟),∀𝑟≥𝑟^∗. (77) Then, for any 𝑟≥𝑟^∗, we have 𝔼sup_𝑡∈𝑇𝑧(𝑡)𝜔(𝑡)≤4+𝑟1+^𝜈1−4^𝜈−1𝜓(𝑟)𝑟. (78)

Proof. For 𝑟≥𝑟^∗, we obtain by a standard peeling approach sup_𝑡∈𝑇𝑧(𝑡)𝜔(𝑡)+𝑟≤sup_{𝜔(𝑡)𝑡∈𝑇≤𝑟}𝑧(𝑡)𝑟+^∞_𝑖=0sup_{𝜔(𝑡)∈4𝑡∈𝑇𝑖𝑟,4𝑖+1𝑟}𝑧(𝑡)4^𝑖.𝑟+𝑟 (79) Therefore, 𝔼sup_𝑡∈𝑇𝑧(𝑡)𝜔(𝑡)≤𝜓(𝑟)+𝑟𝑟+^∞_𝑖=0𝜓4^𝑖+1𝑟1+4^𝑖𝑟≤𝜓(𝑟)𝑟+𝜓(𝑟)𝑟^∞_𝑖=04^(𝑖+1)𝜈1+4^𝑖≤𝜓(𝑟)𝑟1+^∞_𝑖=04^{(𝜈−1)𝑖+𝜈}=41+^𝜈1−4^𝜈−1𝜓(𝑟)𝑟. (80)

Now we can give a bound of Φ_𝑟.

Proposition 24. Let 𝑉 be a classifying loss that satisfies (3), 𝐾∈𝐶^𝑠(𝑋×𝑋) with 𝑠>0, and 𝑝∈(0,2) given by (51). Under assumption (52), for any 𝑡>0 and 𝑟>0 satisfying 𝑟≥max𝐶₄1𝑚𝜆^{𝑝1/24/(2−𝑝)(2−𝛼)},𝐶₄1𝑚𝜆^{𝑝2/(2+𝑝)(2+𝑝)/(2−𝑝)},32𝑡𝑐_𝛼𝑚^1/(2−𝛼),𝑐_𝜃2^𝜃+5𝑡(𝜆),3𝑚,𝐷 (81) where 𝐶₄∶=16(1+(4^𝜈/(1−4^𝜈−1)))𝑐^′_𝑝max{𝑐_𝛼^(2−𝑝)/4(𝑐_𝑝(𝑐_𝜃2^𝜃−1)^𝑝)^1/2, (𝑐_𝜃2^𝜃)^{(2−𝑝)/(2+𝑝)}(𝑐_𝑝(𝑐_𝜃2^𝜃−1)^𝑝)^2/(2+𝑝)} with 𝜈=max{((2−𝑝)𝛼/4)+𝑝/2,2𝑝/(2+𝑝)}. We have, with confidence 1−𝑒^−𝑡, Φ_𝑟≤34. (82)

Proof. Using the notations in the proof of Proposition 20, the weight function 𝜔𝑓=𝔼𝑔_𝑓𝑓,𝑟+𝜆_∗∶=inf_𝑓∈ℋ0𝜔𝑓=𝐷(𝜆). (83) Hence 𝑟≥𝑟^∗, and 𝔼sup_{𝑓∈ℋ0𝜔(𝑓)≤𝑟}𝔼𝑔_𝑓−1𝑚^𝑚_𝑖=1𝑔_𝑓𝑧_𝑖≤𝔼sup_{𝑓∈ℬ𝑟/𝜆𝔼𝑔𝑓≤𝑟}𝔼𝑔_𝑓−1𝑚^𝑚_𝑖=1𝑔_𝑓𝑧_𝑖. (84) Denote 𝒢∶={𝑔_𝑓∶𝑓∈ℬ_𝑟/𝜆}. Standard symmetrization argument (see Lemma 2.3.1 of [36]) and Assumption (52) then yield 𝔼sup_{𝑓∈ℬ𝑟/𝜆𝔼𝑔𝑓≤𝑟}𝔼𝑔_𝑓−1𝑚^𝑚_𝑖=1𝑔_𝑓𝑧_𝑖≤2Rad𝒢,𝑚,𝑐_𝛼𝑟^𝛼. (85) By (3) we know |𝑔_𝑓(𝑧)|≤𝑐_𝜃2^𝜃, and 𝑔_𝑓1(𝑧)−𝑔_𝑓2(𝑧)=𝑉𝑦𝜋𝑓₁(𝑥)−𝑉𝑦𝜋𝑓₂(𝑥)≤𝑐_𝜃2^𝜃−1𝜋𝑓₁(𝑥)−𝜋𝑓₂(𝑥)≤𝑐_𝜃2^𝜃−1𝑓₁(𝑥)−𝑓₂(𝑥), (86) for any 𝑓₁,𝑓₂∈ℬ_𝑟/𝜆, so 𝒩_2,𝐳(𝒢,𝜀)≤𝒩_2,𝐱ℬ_𝑟/𝜆,𝜀𝑐_𝜃2^𝜃−1=𝒩_2,𝐱ℬ₁,𝜀𝜆𝑐_𝜃2^𝜃−1𝑟. (87) This together with (50) implies that sup_𝑚∈ℕsup_{𝐳∈𝑍𝑚}log𝒩_2,𝐳(𝒢,𝜀)≤𝑐_𝑝𝑐_𝜃2^𝜃−1𝑝𝑟𝜆^𝑝𝜀^−𝑝. (88) By Lemma 22, we obtain Rad𝒢,𝑚,𝑐_𝛼𝑟^𝛼≤𝑐^′_𝑝max𝑐_𝛼^1/2−𝑝/4𝑟^{𝛼(1/2−𝑝/4)}𝑐_𝑝(𝑐_𝜃2^𝜃−1)^𝑝𝑚𝑟𝜆^𝑝1/2,𝑐_𝜃2^{𝜃(2−𝑝)/(2+𝑝)}𝑐_𝑝𝑐_𝜃2^𝜃−1𝑝𝑚𝑟𝜆^{𝑝2/(2+𝑝)}𝜓(𝑟)∶=2. (89) Setting 𝜈=max{((2−𝑝)𝛼/4)+𝑝/2,2𝑝/(2+𝑝)}∈(0,1), it is easy to see 𝜓(4𝑟)≤4^𝜈𝜓(𝑟). So Lemma 23 tells us 𝔼Φ_𝑟=𝔼sup_𝑓∈ℋ0𝔼𝑔_𝑓−(1/𝑚)∑^𝑚_𝑖=1𝑔_𝑓𝑧_𝑖𝜔𝑓≤4+𝑟1+^𝜈1−4^𝜈−1𝜓(𝑟)𝑟. (90) By the choice of 𝑟, we can easily check that 41+^𝜈1−4^𝜈−1𝜓(𝑟)𝑟≤18,2𝑡𝑐_𝛼𝑟^𝛼−2𝑚≤14,8𝑐_𝜃2^𝜃𝑡≤13𝑚𝑟4. (91) Then the conclusion follows from Proposition 20.

Corollary 25. Under the condition of Proposition 24, if (28) is satisfied, then for any 0<𝛿<1, with confidence 1−𝛿/4, there holds 𝑆₁()≤3𝐳,𝜆4ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^𝑉_𝜌+𝜆𝑓_𝐳,𝜆+𝐶₅4log𝛿1𝑚𝜆^{𝑝2/(2−𝑝)(2−𝛼)}+1𝑚𝜆^{𝑝2/(2−𝑝)}+1𝑚^1/(2−𝛼)+𝜆^𝛽, (92) where 𝐶₅ is a constant independent of 𝜆, 𝑚, or 𝛿.

Proof. Taking 𝑟=𝐶₄1𝑚𝜆^{𝑝1/24/(2−𝑝)(2−𝛼)}+𝐶₄1𝑚𝜆^{𝑝2/(2+𝑝)(2+𝑝)/(2−𝑝)}+32𝑡𝑐_𝛼𝑚^1/(2−𝛼)+𝑐_𝜃2^𝜃+5𝑡(𝜆)3𝑚+𝐷 (93) in Proposition 24, then, for any 𝑡>1 with confidence 1−𝑒^−𝑡, 𝜔_𝑟𝑓_𝐳,𝜆ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^𝑉_𝜌−ℰ_𝐳𝜋𝑓_𝐳,𝜆−ℰ_𝐳𝑓^𝑉_𝜌≤Φ_𝑟≤34. (94) It follows that 𝑆₁()≤3𝐳,𝜆4𝜔𝑓_𝐳,𝜆=3+𝑟4ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^𝑉_𝜌+𝜆𝑓_𝐳,𝜆+34𝑟. (95) Then the corollary is proved by setting 𝑡=log(4/𝛿), and 𝐶₅3∶=4𝐶₄^{4/(2−𝑝)(2−𝛼)}+𝐶₄^{(2+𝑝)/(2−𝑝)}+32𝑐_𝛼^1/(2−𝛼)+𝑐_𝜃2^𝜃+53+𝑐_𝛽. (96)

5. Deriving Learning Rates

We may now present the main results by combining the results obtained in the previous two sections. The following theorem gives the bounds for the excess generalization error.

Theorem 26. Let 𝑉 be a classifying loss satisfying (3), 𝐾∈𝐶^𝑠(𝑋×𝑋) for some 𝑠∈ℕ, 𝑠≥2, and 𝑝∈(0,2) given by (51). Suppose that 𝑋 satisfies an interior cone conditions, and 𝜌_𝑋 satisfies condition 𝐿_𝜏 with some 𝜏>0. If (28) is valid, then for any 0<𝛿<1 and 𝑚 satisfying (44), by taking 𝜆=𝑚^−𝛾 with 𝛾=min{𝑠/𝜏(𝛽+(1−𝛽)𝜃),1/(𝛽+(1−𝛽)𝜃),2/(2−𝛼)(2−𝑝)𝛽+2𝑝}, we have, with confidence 1−𝛿, ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^𝑉_𝜌8≤𝐶log𝛿+log(𝑚+1)^{max{1,𝑠/𝜏}}1𝑚^𝛾𝛽, (97) where 𝐶 is a constant independent of 𝑚 or 𝛿.

Proof. Putting the estimates of Theorem 14, Proposition 18, and Corollary 25 into the error decomposition (24), we see that, with confidence 1−𝛿, ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^𝑉_𝜌+𝜆𝑓_𝐳,𝜆≤𝐶₂𝜆^𝛽+𝜆^𝛽−1+𝜆^{(𝛽−1)𝜃}log(2/𝛿)+log(𝑚+1)𝑚^𝑠/𝜏+𝐶₃8log𝛿1𝑚^1/(2−𝛼)+𝜆^𝛽+𝜆^{(𝛽−1)𝜃}𝑚+34ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^𝑉_𝜌+𝜆𝑓_𝐳,𝜆+𝐶₅4log𝛿1𝑚𝜆^{𝑝2/(2−𝑝)(2−𝛼)}+1𝑚𝜆^{𝑝2/(2−𝑝)}+1𝑚^1/(2−𝛼)+𝜆^𝛽. (98) Therefore, with the same confidence ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^𝑉_𝜌≤ℰ𝜋𝑓_𝐳,𝜆−ℰ𝑓^𝑉_𝜌+𝜆𝑓_𝐳,𝜆≤4𝐶₂+𝐶₃+𝐶₅8log𝛿+log(𝑚+1)^{max{1,𝑠/𝜏}}×𝜆^𝛽+1𝑚^𝑠/𝜏𝜆^𝛽−1+𝜆^{(𝛽−1)𝜃}+1𝑚^1/(2−𝛼)+𝜆^{(𝛽−1)𝜃}𝑚+1𝑚𝜆^{𝑝2/(2−𝑝)(2−𝛼)}+1𝑚𝜆^{𝑝2/(2−𝑝)}. (99) By the choice of 𝜆, we can easily check that 1𝑚^𝑠/𝜏𝜆^𝛽−1≤1𝑚^𝑠/𝜏𝜆^{(𝛽−1)𝜃}≤𝜆^𝛽,1𝑚^1/(2−𝛼)≤𝜆^𝛽,𝜆^{(𝛽−1)𝜃}𝑚≤𝜆^𝛽,1𝑚𝜆^{𝑝2/(2−𝑝)(2−𝛼)}≤𝜆^𝛽,1𝑚𝜆^{𝑝2/(2−𝑝)}≤𝜆^𝛽. (100) So our theorem follows by taking 𝐶=28(𝐶₂+𝐶₃+𝐶₅).

Theorem 26 together with (17) allows us to give an explicit learning rate for misclassification error of scheme (10).

Corollary 27. If the conditions in Theorem 26 are satisfied, then for any 0<𝛿<1, with confidence 1−𝛿, there holds ℛsgn𝑓_𝐳,𝜆−ℛ𝑓_𝑐≤𝐶8log𝛿+log(𝑚+1)^{max{1,𝑠/𝜏}}1𝑚^𝛾𝛽𝑖𝑓𝑉(𝑡)=(1−𝑡)₊,𝐶8log𝛿+log(𝑚+1)^{max{1/2,𝑠/2𝜏}}1𝑚^𝛾𝛽/2𝑖𝑓𝑉^′′(0)≥0, (101) where 𝐶∶=𝑐_𝑉√𝐶.

Remark 28. For the hinge loss 𝑉_ℎ(𝑡)=(1−𝑡)₊, the increment exponent 𝜃=1. If 𝐾∈𝐶^∞(𝑋×𝑋), then one can take 𝑠→∞ and 𝑝→0. This is the case for polynomial kernel (see [13, 14]) or Gaussian kernel (see [5, 15]), usually used in practice. So Corollary 27 tells us the learning rate of the 1-norm SVM is 𝒪(𝑚^−𝜍) with 𝜍 arbitrarily close to min{𝛽,1/(2−𝛼)}.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgment

This work was supported by National Natural Science Foundation of China under Grants 11171014 and 11072274, the Program for Innovative Research Team in UIBE.