Approximation Analysis of Gradient Descent Algorithm for Bipartite Ranking

Abstract

We introduce a gradient descent algorithm for bipartite ranking with general convex losses. The implementation of this algorithm is simple, and its generalization performance is investigated. Explicit learning rates are presented in terms of the suitable choices of the regularization parameter and the step size. The result fills the theoretical gap in learning rates for ranking problem with general convex losses.

1. Introduction

In this paper we consider a gradient descent algorithm for bipartite ranking generated from Tikhonov regularization scheme with general convex losses and reproducing kernel Hilbert spaces (RKHS).

Let 𝒳 be a compact metric space and 𝒴={−1,1}. In bipartite ranking problem, the learner is given positive samples 𝑆⁺={𝑥⁺_𝑖}^𝑚_𝑖=1 and negative samples 𝑆⁻={𝑥⁻_𝑖}^𝑛_𝑖=1, which are randomly independent drawn from 𝜌⁺ and 𝜌⁻, respectively. Given training set 𝑆∶=(𝑆⁺,𝑆⁻), the goal of bipartite ranking is to learn a real-valued ranking function 𝑓∶𝒳→ℝ that ranks future positive samples higher than negative ones.

The expected loss incurred by a ranking function 𝑓 on a pair of instances (𝑥⁺,𝑥⁻) is 𝐼_{{𝑓(𝑥+)−𝑓(𝑥−)≤0}}, where 𝐼_{𝑡} is 1 if 𝑡 is true and 0 otherwise. However, due to the nonconvexity of 𝐼, the empirical minimization method based on 𝐼 is NP-hard. Thus, we consider replacing 𝐼 by a convex upper loss function 𝜙(𝑓(𝑥⁺)−𝑓(𝑥⁻)). Typical choices of 𝜙 include the hinge loss, the least square loss, and the logistic loss.

The expected convex risk is ℰ𝑓=_𝒳𝜙𝑓𝑥⁺−𝑓(𝑥⁻)𝑑𝜌⁺𝑥⁺𝑑𝜌⁻(𝑥⁻). (1.1) The corresponding empirical risk is ℰ_𝑆𝑓=1𝑚𝑛^{𝑚𝑖=1𝑛𝑗=1}𝜙𝑓𝑥⁺_𝑖−𝑓𝑥⁻_𝑗. (1.2)

Let 𝜗={𝑓∈𝔉∶𝑓=argmin_𝑓∈𝔉ℰ(𝑓)} be the target function set, where 𝔉 is the measurable function space. We can observe that the target function is not unique. In particular, for the least square loss, the regression function is one element in this set.

The ranking algorithm we investigate in this paper is based on a Tikhonov regularization scheme associated with a Mercer kernel. We usually call a symmetric and positive semidefinite continuous function 𝐾∶𝒳×𝒳→ℝ a Mercer kernel. The RKHS ℋ_𝐾 associated with the kernel 𝐾 is defined (see [1]) to be the closure of the linear span of the set of functions {𝐾_𝑥∶=𝐾(𝑥,⋅)∶𝑥∈𝒳} with the inner product ⟨⋅⟩_𝐾 given by ⟨𝐾_𝑥,𝐾_𝑥′⟩_𝐾=𝐾(𝑥,𝑥′). The reproducing property takes the form 𝑓(𝑥)=⟨𝑓,𝐾_𝑥⟩_𝐾, for all 𝑥∈𝒳,𝑓∈ℋ_𝐾. The reproducing property with the Schwartz inequality yields that |𝑓(𝑥)|≤𝐾(𝑥,𝑥)‖𝑓‖_ℋ𝐾. Then, ‖𝑓‖_∞≤𝜅‖𝑓‖_ℋ𝐾, where 𝜅∶=sup_𝑥∈𝒳𝐾(𝑥,𝑥).

The regularized ranking algorithm is implemented by an offline regularization scheme [2] in ℋ_𝐾𝑓_𝐳,𝜆=argmin_{𝑓∈ℋ𝐾}ℰ_𝑆𝑓+𝜆𝑓²_ℋ𝐾, (1.3) where 𝜆>0 is the regularization parameter. A data free-limit of (1.3) is 𝑓_𝜆=argmin_{𝑓∈ℋ𝐾}ℰ𝑓+𝜆𝑓²_ℋ𝐾. (1.4)

Though the offline algorithm (1.3) has been well understood in [2], it might be practically challenging when the sample size 𝑚 or 𝑛 is large. The same difficulty for classification and regression algorithms is overcome by reducing the computational complexity through a stochastic gradient descent method. Such algorithms have been proposed for online regression in [3, 4], online classification in [5, 6], and gradient learning in [7, 8]. In this paper, we use the idea of gradient descent to propose an algorithm for learning a target function in 𝜗.

Since 𝜙 is convex, we know that its left derivative 𝜙^′₋ is well defined and nondecreasing on ℝ. By taking functional derivatives in (1.3), we introduce the following algorithm for ranking.

Definition 1.1. The stochastic gradient descent ranking algorithm is defined for the sample 𝑆 by 𝑓^𝑆₁=0 and 𝑓^𝑆_𝑡+1=1−𝜂_𝑡𝜆𝑓^𝑆_𝑡−𝜂_𝑡𝑚𝑛^{𝑚𝑖=1𝑛𝑗=1}𝜙^′₋𝑓^𝑆_𝑡𝑥⁺_𝑖−𝑓^𝑆_𝑡𝑥⁻_𝑗𝐾_𝑥⁺_𝑖−𝐾_𝑥⁻_𝑗, (1.5) where 𝑡∈ℕ and {𝜂_𝑡} is the sequence of step sizes.

In fact, Burges et al. [9] investigate gradient descent methods for learning ranking functions and introducing a neural network to model the underlying ranking function. From the idea of maximizing the generalized Wilcoxon-Mann-Whitney statistic, a ranking algorithm using gradient approximation has been proposed in [10]. However, these approaches are different from ours and their analysis focuses on computational complexity. Recently, for least square loss, numerical experiments by gradient descent algorithm have been presented in [11]. The aim of this paper is to provide generalization bounds for the gradient descent ranking algorithm (1.5) with general convex losses. To the best of our knowledge, there is no error analysis in this case; This is why we conduct our study in this paper.

We mainly analyze the errors ‖𝑓^𝑆_𝑡−𝑓_𝜆‖_ℋ𝐾and inf_𝑓∈𝜗‖𝑓^𝑆_𝑡−𝑓‖_ℋ𝐾, which is different from previous error analysis for ranking algorithms based on uniform convergence (e.g., [12–16]) and stability analysis in [2, 17, 18]. Though the convergence rates of ℋ_𝐾 norm for classification and regression algorithms have been elegantly investigated in [19, 20], there is no such analysis in the ranking setting. The main difference in the formulation of the ranking problem as compared to the problems of classification and regression is that the performance or loss in ranking is measured on pairs of examples, rather than on individual examples. This means in particular that, unlike the empirical error in classification or regression, the empirical error in ranking cannot be expressed as a sum of independent random variables [17]. This makes the convergence analysis of ℋ_𝐾 norm difficult and previous techniques invalid. Fortunately, we observe that similar difficulty for gradient learning has been well overcome in [7, 21, 22] for gradient learning by introducing some novel techniques. In this paper, we will develop an elaborative analysis in terms of these analysis techniques.

2. Main Result

In this section we present our main results on learning rates of algorithm (1.5) for learning ranking functions. We assume that 𝜙∈𝐶¹(ℝ) satisfies 𝜙(𝑢)≤𝐶₀(1+|𝑢|)^𝑞,𝜙^′(𝑢)≤𝐶₀(1+|𝑢|)^𝑞−1,∀𝑢∈ℝ (2.1) for some 𝐶₀>0 and 𝑞≥1. Denote the constant Δ_∗=1+4𝜅²sup_|𝑢|≤1𝜙^′₋(𝑢)−𝜙^′(0)|𝑢|+𝐶₀𝜙^′(0)+8𝜅²𝜙^′(0)^𝑞−1. (2.2)

Theorem 2.1. Assume 𝜙 satisfies (2.1), and choose the step size as 𝜂_𝑡=𝜂_∗𝜆^{max{𝑞−2,0}}𝑡^−𝜃forsome0<𝜃<1,0<𝜂_∗≤1Δ_∗. (2.3) For 0<𝛾<(1−𝜃)/min{𝑞−1,1},𝑠>0, one takes 𝜆=𝑡^−𝛾 with (𝑚𝑛/(𝑚+𝑛)^3/2)^𝑠≤𝑡≤2(𝑚𝑛/(𝑚+𝑛)^3/2)^𝑠. Then, for any 0<𝛿<1, with confidence at least 1−𝛿, one has 𝑓^𝑆_𝑡−𝑓_𝜆²_ℋ𝐾≤𝐶𝑚𝑛(𝑚+𝑛)^3/2−𝛼, (2.4) where 𝐶 is a constant independent of 𝑚,𝑛, and 𝛼=min𝑠𝜃−𝑠𝛾min𝑞+1,2𝑞−1,1−𝑠𝛾1+𝑞. (2.5)

Theorem 2.1 will be proved in the next section where the constant 𝐶 can be obtained explicitly. The explicit parameters in Theorem 2.1 are described in Table 1 for some special loss functions 𝜙. Note that the iteration steps and iterative numbers depend on sample number 𝑚,𝑛. When 𝑚=𝑂(𝑛) and 𝑚→∞, we have 𝑡→∞ and 𝜂_𝑡→0.

Table 1 

The values of parameters for different convex losses.

From the results in Theorem 2.1, we know that the balance of samples is crucial to reach fast learning rates. For 𝑚=𝑂(𝑛) and the least square loss, the approximation order is 𝑂(𝑚^{(−1/2)min{𝑠𝜃−3𝑠𝛾,1−3𝑠𝛾}}). Moreover, when 𝑠𝜃→1 and 𝑠𝛾→0, we have ‖𝑓^𝑆_𝑡−𝑓_𝜆‖²_ℋ𝐾→0 with the order 𝑂(𝑚^(−1/2)).

Now we present the estimates of inf_𝑓∈𝜗‖𝑓^𝑆_𝑡−𝑓‖_ℋ𝐾 under some approximation conditions.

Corollary 2.2. Assume that there is 𝑓^∗∈𝜗 such that ‖𝑓_𝜆−𝑓^∗‖²_ℋ𝐾≤𝐶_𝛽𝜆^𝛽 for some 0<𝛽<1. Under the condition in Theorem 2.1, for any 0<𝛿<1, with confidence at least 1−𝛿, one has inf_𝑓∈𝜗𝑓^𝑆_𝑡−𝑓²_ℋ𝐾≤𝐶𝑚𝑛(𝑚+𝑛)^3/2−,𝛼 (2.6) where 𝐶 is a constant independent of 𝑚,𝑛, and 𝛼=min𝑠𝜃−𝑠𝛾min𝑞+1,2𝑞−1,1−𝑠𝛾1+𝑞,𝑠𝛾𝛽. (2.7)

For 𝑚=𝑂(𝑛) and the least square loss, by setting 𝑠=1/𝛾(3+𝛽), we can derive the learning rate 𝑂(𝑚^{(−1/𝛾(6+2𝛽))min{𝜃−3𝛾,𝛾𝛽}}). Moreover, if 𝛽<(𝜃−3𝛾)/𝛾, we get the approximation order 𝑂(𝑚^{−(𝛽)/(6+2𝛽)}).

For the least square loss, the regression function is an optimal predictor in 𝜗. Then, the bipartite ranking problem can be reduced as a regression problem. Based on the theoretical analysis in [19, 20], we know that the approximation condition in Corollary 2.2 can be achieved when the regression function lies in the (𝛽+1)/2th power of the integral operator associated with the kernel 𝐾.

The highlight of our theoretical analysis results is to provide the estimate of the distance between 𝑓^𝑆_𝑡 and the target function set 𝜗 in RKHS. This is different from the previous result on error analysis that focuses on establishing the estimate of |ℰ(𝑓)−ℰ_𝑆(𝑓)|. Compared with the previous theoretical studies, the approximation analysis in ℋ_𝐾-norm is new and fills the gap on learning rates for ranking problem with general convex losses.

We also note that the techniques of previous error estimate for ranking problem mainly include stability analysis in [2, 17], concentration estimation based on U-statistics in [14], and uniform convergence bounds based on covering numbers [15, 16]. Our analysis presents a novel capacity-independent procedure to investigate the generalization performance of ranking algorithms.

3. Proof of Main Result

We introduce a special property of ℰ(𝑓)+(𝜆/2)‖𝑓‖²_ℋ𝐾. Since the proof is the same as that in [5], we will omit it here.

Lemma 3.1. Let 𝜆>0. For any 𝑓∈ℋ_𝐾, there holds 𝜆2𝑓−𝑓_𝜆ℋ𝐾≤ℰ𝑓+𝜆2𝑓²_ℋ𝐾−ℰ𝑓_𝜆+𝜆2𝑓_𝜆²_ℋ𝐾. (3.1)

Denote 𝑓^𝜆_𝑡=1𝑚𝑛^{𝑚𝑖=1𝑛𝑗=1}𝜙^′₋𝑓𝑥⁺_𝑖−𝑓𝑥⁻_𝑗𝐾_𝑥⁺_𝑖−𝐾_𝑥⁻_𝑗+𝜆𝑓^𝑆_𝑡,𝑓^𝑆_𝑡+1=𝑓^𝑆_𝑡−𝜂_𝑡𝑓^𝜆_𝑡. (3.2)

Now we give the one-step analysis.

Lemma 3.2. For 𝑡≥0, one has 𝑓^𝑆_𝑡+1−𝑓_𝜆²_ℋ𝐾≤1−𝜂_𝑡𝜆𝑓^𝑆_𝑡−𝑓_𝜆²_ℋ𝐾+𝜂²_𝑡𝑓^𝜆_𝑡²_ℋ𝐾+2𝜂_𝑡𝜑(𝑆,𝑡), (3.3) where 𝜑(𝑆,𝑡)=ℰ_𝑆(𝑓_𝜆)−ℰ(𝑓_𝜆)+ℰ(𝑓^𝑆_𝑡)−ℰ_𝑆(𝑓^𝑆_𝑡).

Proof. Observe that 𝑓^𝑆_𝑡+1−𝑓_𝜆²_ℋ𝐾=𝑓^𝑆_𝑡−𝑓_𝜆²_ℋ𝐾+𝜂²_𝑡𝑓^𝜆_𝑡²_ℋ𝐾+2𝜂_𝑡𝑓_𝜆−𝑓^𝑆_𝑡,𝑓^𝜆_𝑡𝐾. (3.4) Note that 𝑓_𝜆−𝑓^𝑆_𝑡,𝑓^𝜆_𝑡𝐾=1𝑚𝑛^{𝑚𝑖=1𝑛𝑗=1}𝜙^′₋𝑓^𝑆_𝑡𝑥⁺_𝑖−𝑓^𝑆_𝑡𝑥⁻_𝑗𝑓_𝜆𝑥⁺_𝑖−𝑓_𝜆𝑥⁻_𝑗−𝑓^𝑆_𝑡𝑥⁺_𝑖+𝑓^𝑆_𝑡𝑥⁻_𝑗+𝜆𝑓_𝜆−𝑓^𝑆_𝑡,𝑓^𝑆_𝑡𝐾≤ℰ_𝑆𝑓_𝜆−ℰ_𝑆𝑓^𝑆_𝑡−𝜆𝑓^𝑆_𝑡²_ℋ𝐾+𝜆𝑓_𝜆,𝑓^𝑆_𝑡𝐾≤ℰ_𝑆𝑓_𝜆+𝜆2𝑓_𝜆²_ℋ𝐾−ℰ_𝑆𝑓^𝑆_𝑡+𝜆2𝑓^𝑆_𝑡²_ℋ𝐾, (3.5) where the first and the second inequalities are derived by the convexity of 𝜙 and the Schwartz inequality, respectively.
By Lemma 3.1, we know that ℰ_𝑆𝑓_𝜆+𝜆2𝑓_𝜆²_ℋ𝐾−ℰ_𝑆𝑓^𝑆_𝑡+𝜆2𝑓^𝑆_𝑡²_ℋ𝐾≤ℰ_𝑆𝑓_𝜆−ℰ𝑓_𝜆+ℰ𝑓^𝑆_𝑡−ℰ_𝑆𝑓^𝑆_𝑡−𝜆2𝑓^𝑆_𝑡−𝑓_𝜆²_ℋ𝐾. (3.6) Thus, the desired result follows by combining (3.5) and (3.6) with (3.4).

To deal with the sample error iteratively by applying (3.3), we need to bound the quantity 𝜑(𝑆,𝑡) by the theory of uniform convergence. To this end, a bound for the norm of 𝑓^𝑆_𝑡 is required.

Definition 3.3. One says that 𝜙^′₋ is locally Lipschitz at the origin if the local Lipschitz constant 𝑀(𝜆)𝜙=sup^′−(𝑢)−𝜙^′₋(0)|𝑢|∶|𝑢|≤4𝜅²𝜙^′₋(0)𝜆 (3.7) is finite for any 𝜆>0.

Now we estimate the bound of 𝑓^𝑆_𝑡 from the ideas given in [5].

Lemma 3.4. Assume that 𝜙^′₋ is locally Lipschitz at the origin. If the step size 𝜂_𝑡 satisfies 𝜂_𝑡(4𝜅²𝑀(𝜆)+𝜆)≤1 for each t, then ‖𝑓^𝑆_𝑡‖_ℋ𝐾≤2𝜅|𝜙^′₋(0)|/𝜆.

Proof. We prove by induction. It is trivial that 𝑓^𝑆₁=0 satisfies the bound.
Suppose that this bound holds true for 𝑓^𝑆_𝑡, ‖𝑓^𝑆_𝑡‖_ℋ𝐾≤2𝜅|𝜙^′₋(0)|/𝜆. Consider𝑓^𝑆_𝑡+1=1−𝜂_𝑡𝜆𝑓^𝑆_𝑡−𝜂_𝑡𝑚𝑛^{𝑚𝑖=1𝑛𝑗=1}𝜙^′₋𝑓^𝑆_𝑡𝑥⁺_𝑖−𝑓^𝑆_𝑡𝑥⁻_𝑗𝐾_𝑥⁺_𝑖−𝐾_𝑥⁻_𝑗=1−𝜂_𝑡𝜆𝑓^𝑆_𝑡−𝜂_𝑡𝑚𝑛^{𝑚𝑖=1𝑛𝑗=1}𝜙^′₋𝑓^𝑆_𝑡𝑥⁺_𝑖−𝑓^𝑆_𝑡𝑥⁻_𝑗−𝜙^′₋(0)𝑓^𝑆_𝑡𝑥⁺_𝑖−𝑓^𝑆_𝑡𝑥⁻_𝑗⋅𝑓^𝑆_𝑡𝑥⁺_𝑖−𝑓^𝑆_𝑡𝑥⁻_𝑗𝐾_𝑥⁺_𝑖−𝐾_𝑥⁻_𝑗−𝜂_𝑡𝑚𝑛^{𝑚𝑖=1𝑛𝑗=1}𝜙^′₋(0)𝐾_𝑥⁺_𝑖−𝐾_𝑥⁻_𝑗. (3.8) Let 𝐿_𝑖𝑗𝑓=⟨𝑓,𝐾_𝑥⁺_𝑖−𝐾_𝑥⁻_𝑗⟩_𝐾(𝐾_𝑥⁺_𝑖−𝐾_𝑥⁻_𝑗). Since 0≤𝐿_𝑖𝑗𝑓,𝑓_𝐾=𝑓,𝐾_𝑥⁺_𝑖−𝐾_𝑥⁻_𝑗𝐾²≤4𝜅²𝑓²_ℋ𝐾, (3.9) we have ‖𝐿_𝑖𝑗‖≤4𝜅².
Meanwhile, (𝜙^′₋𝑓^𝑆_𝑡(𝑥⁺_𝑖)−𝑓^𝑆_𝑡(𝑥⁻_𝑗)−𝜙^′₋(0)/𝑓^𝑆_𝑡(𝑥⁺_𝑖)−𝑓^𝑆_𝑡(𝑥⁻_𝑗)≤𝑀(𝜆)). Then, 𝜂_𝑡𝑚𝑛^{𝑚𝑖=1𝑛𝑗=1}𝜙^′₋𝑓^𝑆_𝑡𝑥⁺_𝑖−𝑓^𝑆_𝑡𝑥⁻_𝑗−𝜙^′₋(0)𝑓^𝑆_𝑡𝑥⁺_𝑖−𝑓^𝑆_𝑡𝑥⁻_𝑗𝐿_𝑖𝑗 (3.10) is a positive linear operator on ℋ_𝐾 and its norm is bounded by 4𝜅²𝑀(𝜆).
Since 𝜂_𝑡(4𝜅²𝑀(𝜆)+𝜆)≤1, the operator 𝐴∶=1−𝜂_𝑡𝜆𝜂𝐼−^𝑡𝑚𝑛^{𝑚𝑖=1𝑛𝑗=1}𝜙^′₋𝑓^𝑆_𝑡𝑥⁺_𝑖−𝑓^𝑆_𝑡𝑥⁻_𝑗−𝜙^′₋(0)𝑓^𝑆_𝑡𝑥⁺_𝑖−𝑓^𝑆_𝑡𝑥⁻_𝑗𝐿_𝑖𝑗 (3.11) on ℋ_𝐾 is positive and 𝐴≤(1−𝜂_𝑡𝜆)𝐼.
Thus, 𝑓^𝑆_{𝑡+1ℋ𝐾}≤1−𝜂_𝑡𝜆𝑓^𝑆_𝑡ℋ𝐾+𝜂_𝑡𝑚𝑛^{𝑚𝑖=1𝑛𝑗=1}𝜙^′₋(0)𝐾_𝑥⁺_𝑖−𝐾_𝑥⁻_𝑗ℋ𝐾=2𝜅𝜙^′₋(0)𝜆. (3.12) This proves the lemma.

For 𝑟>0, denote 𝔉_𝑟={𝑓∈ℋ_𝐾∶‖𝑓‖_ℋ𝐾≤𝑟}. Meanwhile, denote 𝐿_𝑟=max{|𝜙^′₋(2𝜅𝑟)|,|𝜙^′₋(−2𝜅𝑟)|} and 𝑀_𝑟=max{|𝜙(2𝜅𝑟)|,|𝜙(−2𝜅𝑟)|}.

Based on analysis techniques in [21, 23], we derive the capacity-independent bounds for 𝑊(𝑆,𝑟)∶=sup_{𝑓∈𝔉𝑟}|ℰ_𝑆(𝑓)−ℰ(𝑓)|.

Lemma 3.5. For every 𝑟>0 and 𝜀>0, one has Prob_𝑆{|𝑊(𝑆,𝑟)−𝐸𝑊(𝑆,𝑟)|>𝜀}≤exp−2𝑚²𝑛²𝜀²(𝑚+𝑛)³𝑀²_𝑟,𝐸𝑊(𝑆,𝑟)≤4𝐿_𝑟(0)√𝜅𝑟+2𝜙𝑚+√𝑛√.𝑚𝑛 (3.13)

Proof. Because of the feature of 𝑆, four cases of samples change should be taken into account to use McDiarmid's inequality. Denote by 𝑆_𝑘 the sample coinciding with 𝑆 except for 𝑥⁺_𝑘 (or 𝑥⁻_𝑘) replaced by 𝑥⁺_𝑘 (or 𝑥⁻_𝑘). It is easy to verify that 𝑊(𝑆,𝑟)−𝑊𝑆_𝑘,𝑟=sup_{𝑓∈𝔉𝑟}ℰ_𝑆𝑓−ℰ𝑓−sup_{𝑓∈𝔉𝑟}ℰ_𝑆𝑘𝑓−ℰ𝑓≤sup_{𝑓∈𝔉𝑟}ℰ_𝑆𝑓−ℰ_𝑆𝑘𝑓≤𝑚+𝑛𝑀𝑚𝑛^𝑟. (3.14) Based on MicDiarmid’s inequality in [24], we can derive the first result in Lemma 3.5. To derive the second result, we denote 𝜉(𝑥⁺,𝑢⁻)=𝜙(𝑓(𝑥⁺)−𝑓(𝑢⁻)). Then, ℰ(𝑓)=𝐸_𝑥+𝐸_𝑥−𝜉(𝑥⁺,𝑥⁻) and ℰ_𝑆(𝑓)=1/𝑚𝑛∑^𝑚_𝑖=1∑^𝑛_𝑗=1𝜉(𝑥⁺_𝑖,𝑥⁻_𝑗). Observe that 𝑊(𝑆,𝑟)≤sup_{𝑓∈𝔉𝑟}ℰ𝑓−1𝑛^𝑛𝑗=1𝐸_𝑥+𝜉𝑥⁺,𝑥⁻_𝑗+sup_{𝑓∈𝔉𝑟}1𝑛^𝑛𝑗=1𝐸_𝑥+𝜉𝑥⁺,𝑥⁻_𝑗−ℰ_𝑆𝑓≤𝐸_𝑥+sup_{𝑓∈𝔉𝑟}𝐸_𝑥−𝜉𝑥⁺,𝑥⁻−1𝑛^𝑛𝑗=1𝜉𝑥⁺,𝑥⁻_𝑗+1𝑛^𝑛𝑗=1sup_{𝑓∈𝔉𝑟}sup_𝑥−𝐸_𝑥+𝜉𝑥⁺,𝑥⁻−1𝑚^𝑚𝑖=1𝜉𝑥⁺_𝑖,𝑥⁻=𝑊₁+𝑊₂. (3.15)
Denote 𝐺_𝑥+={ℎ(𝑥⁻)=𝑓(𝑥⁺)−𝑓(𝑥⁻)∶𝑓∈𝔉}. Then, 𝐸𝑊₁=𝐸_𝑥𝐸sup_{ℎ∈𝐺𝑥+}𝐸_𝑥−𝜙(ℎ(𝑥⁻−1))𝑛^𝑛𝑗=1𝜙ℎ𝑥⁻_𝑗≤2sup_𝑥+𝐸sup_{ℎ∈𝐺𝑥+}1𝑛^𝑛𝑗=1𝜀_𝑗𝜙ℎ𝑥⁻_𝑗. (3.16)
Since 𝜙(ℎ(𝑥⁻_𝑗))−𝜙(0)≤𝐿_𝑟(ℎ(𝑥⁻_𝑗)−0), we have 𝐸𝑊₁≤2𝐿_𝑟sup_𝑥+𝐸sup_{ℎ∈𝐺𝑥+}1𝑛^𝑛𝑗=1𝜀_𝑗ℎ𝑥⁻_𝑗+2𝜙(0)𝑛𝐸^𝑛_𝑗=1𝜀_𝑗≤4𝐿_𝑟𝜅𝑟𝑛+2𝜙(0)𝑛𝐸^𝑛_𝑗=1𝜀_𝑗≤4𝐿_𝑟𝜅𝑟+2𝜙(0)𝑛𝐸^𝑛_𝑗=1𝜀_𝑗^21/2≤4𝐿_𝑟𝜅𝑟+2𝜙(0)𝑛𝐸^𝑛_{𝑗,𝑗′=1}𝜀_𝑗𝜀_𝑗′^1/2=4𝐿_𝑟𝜅𝑟+2𝜙(0)√𝑛. (3.17)
With the same fashion, we can also derive 𝐸𝑊₂≤4𝐿_𝑟𝜅𝑟+2𝜙(0)√𝑚. (3.18) Thus, the second desired result follows by combining (3.17) and (3.18).

Now we can derive the estimate of 𝜑(𝑆,𝑡).

Lemma 3.6. If 𝜂_𝑡 satisfies (1) for each 𝑡 and 𝑟=2𝜅|𝜙^′₋(0)|/𝜆+2𝜙(0)/𝜆, then with confidence at least 1−𝛿 one has 𝜑(𝑆,𝑡)≤𝐵_𝜆∶=4𝐿_𝑟𝜅𝑟+2𝜙(0)+𝑀_𝑟22log𝛿(𝑚+𝑛)^3/2.𝑚𝑛 (3.19)

Proof. By Lemma 3.5, we have, with confidence at least 1−𝛿, 𝑊(𝑆,𝑟)≤4𝐿_𝑟(0)√𝜅𝑟+2𝜙𝑚+√𝑛√+()𝑚𝑛𝑚+𝑛^3/2𝑀_𝑟2𝑚𝑛22log𝛿. (3.20) By taking 𝑓=0 in the definition of 𝑓_𝜆, we see that 𝜆2𝑓_𝜆²_ℋ𝐾≤ℰ(0)+0≤𝜙(0). (3.21) Then, for any 𝜆>0, we have ‖𝑓_𝜆‖_ℋ𝐾≤2𝜙(0)/𝜆. Thus, 𝑓^𝑆_𝑡,𝑓_𝜆∈𝔉_𝑟 for 𝑟=2𝜅|𝜙^′₋(0)|/𝜆+2𝜙(0)/𝜆. So, 𝜑(𝑆,𝑡)≤2𝑊(𝑆,𝑟) for each 𝑡. This completes the proof.

We are in a position to give bounds for the sample error. We need the following elementary inequalities that can be found in [3, 5].

Lemma 3.7. (1) For 𝛼∈(0,1] and 𝜃∈[0,1], _𝑡𝑖=11𝑖^𝜃𝑡_𝑗=𝑖+1𝛼1−𝑗^𝜃≤3𝛼. (3.22)
(2) Let 𝑣∈(0,1] and 𝜃∈(0,1]. Then _{𝑇−1𝑡=1}1𝑡^2𝜃exp−𝑣^𝑇_𝑗=𝑡+1𝑗^−𝜃≤18𝑣𝑇^𝜃+9𝑇^1−𝜃(1−𝜃)2^1−𝜃−𝑣exp1−2^𝜃−1()1−𝜃𝑇+1^1−𝜃8if𝜃<1,()1−𝑣𝑇+1^−𝑣if𝜃=1. (3.23)
(3) For any 𝑡<𝑇 and 𝜃∈(0,1], there holds ^𝑇_𝑗=𝑡+1𝑗^−𝜃≤1)1−𝜃(𝑇+1^1−𝜃−(𝑡+1)^1−𝜃()()if𝜃<1,log𝑇+1−log𝑡+1if𝜃=1. (3.24)

Proposition 3.8. Let 𝜂_𝑡=𝜂₁𝑡^−𝜃 for some 𝜃∈[0,1], and let 𝜂₁ satisfy 𝜂₁(4𝑀(𝜆)+𝜆)≤1. Set 𝑟 and 𝐵_𝜆 as in Lemma 3.6. Denote 𝐵_𝜆=2𝜅𝐿_𝑟+𝜅|𝜙^′₋(0)|. Then, with confidence at least 1−𝛿, the following bound holds for 𝑡≥1: when 𝜃<1, 𝑓^𝑆_𝑡−𝑓_𝜆²_ℋ𝐾≤𝑓_𝜆²_ℋ𝐾−𝜂exp¹𝜆𝑡1−𝜃^1−𝜃+𝐵−118²_𝜆𝜂₁𝜆𝑡^𝜃+2𝐵²_𝜆𝜂²₁𝑡^1−𝜃(1−𝜃)2^1−𝜃−𝜂exp¹𝜆1−2^𝜃−1()1−𝜃𝑡+1^1−𝜃+6𝐵_𝜆𝜆, (3.25) when 𝜃=1, 𝑓^𝑆_𝑡−𝑓_𝜆²_ℋ𝐾≤𝑓_𝜆²_ℋ𝐾𝑡^−𝜂1𝜆+8𝐵²_𝜆𝜂²₁1−𝜂₁𝜆(𝑡+1)^−𝜂1𝜆+6𝐵_𝜆𝜆. (3.26)

Proof. Since ‖𝑓^𝑆_𝑡‖_ℋ𝐾≤2𝜅|𝜙^′₋(0)|/𝜆, we have |𝑓^𝑆_𝑡(𝑥⁺_𝑖)−𝑓^𝑆_𝑡(𝑥⁻_𝑗)|≤2𝜅‖𝑓^𝑆_𝑡‖_ℋ𝐾≤2𝜅𝑟. From the definition of 𝑓^𝜆_𝑡, we know that ‖𝑓^𝜆_𝑡‖_ℋ𝐾≤2𝜅𝐿_𝑟+𝜅|𝜙^′₋(0)|=𝐵_𝜆. Thus, when 𝜑(𝑆,𝑡)≤𝐵_𝜆, we have from Lemma 3.2𝑓^𝑆_𝑡+1−𝑓_𝜆²_ℋ𝐾≤1−𝜂_𝑡𝜆𝑓^𝑆_𝑡−𝑓_𝜆²_ℋ𝐾+𝜂²_𝑡𝐵²_𝜆+2𝜂_𝑡𝐵_𝜆. (3.27) Applying this relation iteratively, we have 𝑓^𝑆_𝑡−𝑓_𝜆²_ℋ𝐾≤^𝑡−1_𝑖=11−𝜂_𝑖𝜆𝑓_𝜆²_ℋ𝐾+^𝑡−1_𝑖=1^𝑡−1_𝑗=𝑖+11−𝜂_𝑖𝜆𝜂²_𝑖𝐵²_𝜆+2𝜂_𝑖𝐵_𝜆. (3.28) Since 𝜂_𝑖=𝜂₁𝑖^−𝜃, by Lemma 3.7(2), we have for 𝜃<1^𝑡−1_𝑖=1^𝑡−1_𝑗=𝑖+11−𝜂_𝑖𝜆𝜂²_𝑖≤𝜂²₁^𝑡−1_𝑖=112𝜃exp−𝜂₁𝜆^𝑡−1_𝑗=𝑖+1𝑗^−𝜃≤18𝜂₁+𝜆𝑡9𝜂²₁𝑡^1−𝜃(1−𝜃)2^1−𝜃−𝜂exp¹𝜆1−2^𝜃−1()1−𝜃𝑡+1^1−𝜃 (3.29) and for 𝜃=1^𝑡−1_𝑖=1^𝑡−1_𝑗=𝑖+11−𝜂_𝑖𝜆𝜂²_𝑖≤8𝜂²₁1−𝜂₁𝜆(𝑡+1)^−𝜂1𝜆. (3.30) Lemma 3.7(1) yields ^𝑡−1_𝑖=1^𝑡−1_𝑗=𝑖+11−𝜂_𝑖𝜆𝜂_𝑖≤𝜂₁^𝑡−1_𝑖=11𝑖^𝜃𝑡−1_𝑗=𝑖+1𝜂1−¹𝜆𝑗^𝜃≤3𝜆. (3.31) By Lemma 3.7(3), we also have for 𝜃<1^𝑡−1_𝑖=11−𝜂_𝑖𝜆≤exp−^𝑡−1_𝑖=1𝜂_𝑖𝜆𝜂≤exp¹𝜆1−𝜃1−𝑡^1−𝜃 (3.32) and for 𝜃=1^𝑡−1_𝑖=11−𝜂_𝑖𝜆≤𝑡^−𝜂1𝜆. (3.33) Combining the above estimations with Lemma 3.6, we derive the desired results.

Now we present the proof of Theorem 2.1.

Proof of Theorem 2.1. First we derive explicit expressions for the quantities in Proposition 3.8. Since 𝜆=𝑡^−𝛾, we have 𝑟≤𝐶₃𝑡^𝛾, where 𝐶₃=2𝜅|𝜙^′₋(0)|+2𝜙(0). By (2.1), we find that 𝐿_𝑟≤𝐶₀(1+2𝜅𝑟)^𝑞−1≤𝐶₀(1+2𝜅)^𝑞−1𝐶₃^𝑞−1𝑡^{(𝑞−1)𝛾},𝑀_𝑟≤𝐶₀(1+2𝜅)^𝑞𝐶^𝑞₃𝑡^𝑞𝛾. (3.34) Then, 𝐵_𝜆≤𝐶₄2log𝛿𝑡^𝑞𝛾(𝑚+𝑛)^3/2,𝐵𝑚𝑛^𝜆≤𝐶₅𝑡^{(𝑞−1)𝛾}, (3.35) where 𝐶₄=4𝐶₀𝜅(1+2𝜅)^𝑞−1𝐶₃^𝑞−1+𝐶₀(1+2𝜅)^𝑞𝐶^𝑞₃+2𝜙(0) and 𝐶₅=2𝜅𝐶₀(1+2𝜅)^𝑞−1𝐶₃^𝑞−1+2𝜙(0).
Next, we bound 𝑀(𝜆). When 1≤|𝑢|≤4𝜅²|𝜙^′₋(0)|/𝜆, we have 𝜙^′₋(𝑢)−𝜙^′₋(0)|𝑢|≤𝜙^′₋(0)+2^𝑞−1𝐶₀4𝜅²𝜙^′₋(0)^𝑞−1𝜆^2−𝑞. (3.36) Hence, 𝑀(𝜆)≤sup_|𝑢|≤1𝜙^′₋(𝑢)−𝜙^′₋(0)|𝑢|+𝜙^′₋(0)+2^𝑞−1𝐶₀4𝜅²𝜙^′₋(0)^𝑞−1𝜆^{min{2−𝑞,0}}. (3.37) It follows that the condition 𝜂₁(4𝜅²𝑀(𝜆)+𝜆)≤1 in Proposition 3.8 holds true when 𝜂_𝑡=𝜂₁𝑡^−𝜃 and 𝜂₁=𝜂_∗𝜆^{max{𝑞−2,0}}. Based on Proposition 3.8 and (𝑚𝑛/𝑚+𝑛^3/2)^𝑠≤𝑡≤2(𝑚𝑛/𝑚+𝑛^3/2)^𝑠, we have, with confidence at least 1−𝛿, 𝑓^𝑆_𝑡−𝑓_𝜆²_ℋ𝐾≤𝑐₁̃𝑓_𝜆²_ℋ𝐾𝜂exp^∗+1−𝜃𝑐₂̃𝜂²_∗(1−𝜃)2^1−𝜃𝑚𝑛(𝑚+𝑛)^{3/2𝑠(1−𝜃−2(𝑞−1)𝛾)}−𝜂×exp^∗1−2^𝜃−1𝑡1−𝜃^{1−𝜃−𝛾max{𝑞−1,1}}+𝑐₃̃𝑚𝑛(𝑚+𝑛)^{3/2𝑠𝛾(1+𝑞)−1}+𝑐₄̃𝜂_∗𝑚𝑛(𝑚+𝑛)^{3/2𝑠𝛾min{𝑞+1,2𝑞−1}−𝑠𝜃}, (3.38) where 𝑐₁̃,𝑐₂̃,𝑐₃̃, and 𝑐₄̃ are constants independent of 𝑚,𝑛,and 𝑡.
Thus, when 1−𝜃−𝛾max{𝑞−1,1}>0, we can derive the desired result in Theorem 2.1.

Acknowledgments

This work was supported partially by the National Natural Science Foundation of China (NSFC) under Grant no. 11001092 and the Fundamental Research Funds for the Central Universities (Program no. 2011PY130, 2011QC022). The authors are indebted to the anonymous reviewers for their constructive comments.