Approximation Analysis of Learning Algorithms for Support Vector Regression and Quantile Regression

Abstract

We study learning algorithms generated by regularization schemes in reproducing kernel Hilbert spaces associated with an 𝜖-insensitive pinball loss. This loss function is motivated by the 𝜖-insensitive loss for support vector regression and the pinball loss for quantile regression. Approximation analysis is conducted for these algorithms by means of a variance-expectation bound when a noise condition is satisfied for the underlying probability measure. The rates are explicitly derived under a priori conditions on approximation and capacity of the reproducing kernel Hilbert space. As an application, we get approximation orders for the support vector regression and the quantile regularized regression.

1. Introduction and Motivation

In this paper, we study a family of learning algorithms serving both purposes of support vector regression and quantile regression. Approximation analysis and learning rates will be provided, which also helps better understanding of some classical learning methods.

Support vector regression is a classical kernel-based algorithm in learning theory introduced in [1]. It is a regularization scheme in a reproducing kernel Hilbert space (RKHS) ℋ_𝐾 associated with an 𝜖-insensitive loss 𝜓^𝜖∶ℝ→ℝ₊ defined for 𝜖≥0 by𝜓^𝜖(𝑢)=max{|𝑢|−𝜖,0}=|0𝑢,|−𝜖,iofth|e𝑢r|w≥is𝜖e,.(1.1) Here, for learning functions on a compact metric space 𝑋, 𝐾∶𝑋×𝑋→ℝ is a continuous, symmetric, and positive semidefinite function called a Mercer kernel. The associated RKHS ℋ_𝐾 is defined [2] as the completion of the linear span of the set of function {𝐾_𝑥=𝐾(𝑥,⋅)∶𝑥∈𝑋} with the inner product ⟨⋅,⋅⟩_𝐾 satisfying ⟨𝐾_𝑥,𝐾_𝑦⟩_𝐾=𝐾(𝑥,𝑦). Let 𝑌=ℝ and 𝜌 be a Borel probability measure on 𝑍∶=𝑋×𝑌. With a sample 𝐳={(𝑥_𝑖,𝑦_𝑖)}^𝑚𝑖=1∈𝑍^𝑚 independently drawn according to 𝜌, the support vector regression is defined as𝑓^SVR_𝐳=argmin_{𝑓∈ℋ𝐾}𝑚1^𝑚_𝑖=1𝜓^𝜖𝑓𝑥_𝑖−𝑦_𝑖+𝜆‖𝑓‖^2𝐾,(1.2) where 𝜆=𝜆(𝑚)>0 is a regularization parameter.

When 𝜖>0 is fixed, convergence of (1.2) was analyzed in [3]. Notice from the original motivation [1] for the insensitive parameter 𝜖 for balancing the approximation and sparsity of the algorithm that 𝜖 should change with the sample size and usually 𝜖=𝜖(𝑚)→0 as the sample size 𝑚 increases. Mathematical analysis for this original algorithm is still open. We will solve this problem in a special case of our approximation analysis for general learning algorithms. In particular, we show how 𝑓^SVR_𝐳 approximates the median function 𝑓_𝜌,1/2, which is one of the purposes of this paper. Here, for 𝑥∈𝑋, the median function value 𝑓_𝜌,1/2(𝑥) is a median of the conditional distribution 𝜌(⋅∣𝑥) of 𝜌 at 𝑥.

Quantile regression, compared with the least squares regression, provides richer information about response variables such as stretching or compressing tails [4]. It aims at estimating quantile regression functions. With a quantile parameter 0<𝜏<1, a quantile regression function 𝑓_𝜌,𝜏 is defined by its value 𝑓_𝜌,𝜏(𝑥) to be a 𝜏-quantile of 𝜌(⋅∣𝑥), that is, a value 𝑢∈𝑌 satisfying𝜌({𝑦∈𝑌∶𝑦≤𝑢}∣𝑥)≥𝜏,𝜌({𝑦∈𝑌∶𝑦≥𝑢}∣𝑥)≥1−𝜏.(1.3)

Quantile regression has been studied by kernel-based regularization schemes in a learning theory literature (e.g., [5–8]). These regularization schemes take the form𝑓^Q𝑅𝐳=argmin_{𝑓∈ℋ𝐾}𝑚1^𝑚_𝑖=1𝜓_𝜏𝑓𝑥_𝑖−𝑦_𝑖+𝜆‖𝑓‖^2𝐾,(1.4) where 𝜓_𝜏∶ℝ→ℝ₊ is the pinball loss shown in Figure 1 defined by𝜓_𝜏(𝑢)=(−1𝜏−𝑢,𝜏)𝑢,iiff𝑢𝑢>≤00,.(1.5)

Figure 1 

Motivated by the 𝜖-insensitive loss 𝜓^𝜖 and the pinball loss 𝜓_𝜏, we propose the 𝜖-insensitive pinball loss 𝜓^𝜖𝜏∶ℝ→ℝ₊ with an insensitive parameter 𝜖≥0 shown in Figure 1 defined as𝜓^𝜖𝜏(𝑢)=⎧⎪⎨⎪⎩(−01,𝜏−(𝑢𝜏+)(𝜖𝑢)−,𝜖),iioffth𝑢𝑢e>≤rw𝜖i−s,e𝜖.,(1.6) This loss function has been applied to online learning for quantile regression in our previous work [8]. It is applied here to a regularization scheme in the RKHS as𝑓^𝐳(𝜖)=𝑓^{(𝜖)𝐳,𝜆,𝜏}=argmin_{𝑓∈ℋ𝐾}𝑚1^𝑚_𝑖=1𝜓^𝜖𝜏𝑓𝑥_𝑖−𝑦_𝑖+𝜆‖𝑓‖^2𝐾.(1.7) The main goal of this paper is to study how the output function 𝑓^𝐳(𝜖) given by (1.7) converges to the quantile regression function 𝑓_𝜌,𝜏 and how explicit learning rates can be obtained with suitable choices of the parameters 𝜆=𝑚^−𝛼,𝜖=𝑚^−𝛽 based on a priori conditions on the probability measure 𝜌.

2. Main Results on Approximation

Throughout the paper, we assume that the conditional distribution 𝜌(⋅∣𝑥) is supported on [−1,1] for every 𝑥∈𝑋. Then, we see from (1.3) that we can take values of 𝑓_𝜌,𝜏 to be on [−1,1]. So to see how 𝑓^𝐳(𝜖) approximates 𝑓_𝜌,𝜏, it is natural to project values of the output function 𝑓^𝐳(𝜖) onto the same interval by the projection operator introduced in [9].

Definition 2.1. The projection operator 𝜋 on the space of function on 𝑋 is defined by 𝜋(𝑓)(𝑥)=⎧⎪⎨⎪⎩1−𝑓,1(𝑥,),iiifff𝑓𝑓−((1𝑥𝑥))≤<>𝑓1−(,1𝑥,)≤1.(2.1)

Our approximation analysis aims at establishing bounds for the error ‖𝜋(𝑓^𝐳(𝜖))−𝑓_𝜌,𝜏‖_{𝐿𝑝∗𝜌𝑋} in the space 𝐿^𝑝∗_𝜌𝑋 with some 𝑝^∗>0 where 𝜌_𝑋 is the marginal distribution of 𝜌 on 𝑋.

2.1. Support Vector Regression and Quantile Regression

Our error bounds and learning rates are presented in terms of a noise condition and approximation condition on 𝜌.

The noise condition on 𝜌 is defined in [5, 6] as follows.

Definition 2.2. Let 𝑝∈(0,∞] and 𝑞∈(1,∞). We say that 𝜌 has a 𝜏-quantile of 𝑝-average type 𝑞 if for every 𝑥∈𝑋, there exist a 𝜏-quantile 𝑡^∗∈ℝ and constants 𝑎_𝑥∈(0,2],𝑏_𝑥>0 such that for each 𝑢∈[0,𝑎_𝑥], 𝜌𝑦∈𝑡^∗−𝑢,𝑡^∗∣𝑥≥𝑏_𝑥𝑢^𝑞−1,𝜌𝑦∈𝑡^∗,𝑡^∗+𝑢∣𝑥≥𝑏_𝑥𝑢^𝑞−1,(2.2) and that the function on 𝑋 taking value (𝑏_𝑥𝑎^𝑥𝑞−1)⁻¹ at 𝑥∈𝑋 lies in 𝐿^𝑝𝜌_𝑋.

Note that condition (2.2) tells us that 𝑓_𝜌,𝜏(𝑥)=𝑡^∗ is uniquely defined at every 𝑥∈𝑋.

The approximation condition on 𝜌 is stated in terms of the integral operator 𝐿_𝐾∶𝐿^2𝜌_𝑋→𝐿^2𝜌_𝑋 defined by 𝐿_𝐾(𝑓)(𝑥)=∫_𝑋𝐾(𝑥,𝑢)𝑓(𝑢)𝑑𝜌_𝑋(𝑢). Since 𝐾 is positive semidefinite, 𝐿_𝐾 is a compact positive operator and its 𝑟-th power 𝐿^𝑟𝐾 is well-defined for any 𝑟>0. Our approximation condition is given as𝑓_𝜌,𝜏=𝐿^𝑟𝐾𝑔_𝜌,𝜏forsome0<𝑟≤12,𝑔^𝜌,𝜏∈𝐿^2𝜌_𝑋.(2.3)

Let us illustrate our approximation analysis by the following special case which will be proved in Section 5.

Theorem 2.3. Let 𝑋⊂ℝ^𝑛 and 𝐾∈𝐶^∞(𝑋×𝑋). Assume 𝑓_𝜌,𝜏∈ℋ_𝐾 and 𝜌 has a 𝜏-quantile of 𝑝-average type 2 for some 𝑝∈(0,∞]. Take 𝜆=𝑚^{−(𝑝+1)/(𝑝+2)} and 𝜖=𝑚^−𝛽 with (𝑝+1)/(𝑝+2)≤𝛽≤∞. Let 0<𝜂<(𝑝+1)/2(𝑝+2). Then, with 𝑝^∗=2𝑝/(𝑝+1)>0, for any 0<𝛿<1, with confidence 1−𝛿, one has ‖‖𝜋𝑓^𝐳(𝜖)−𝑓^𝜌,𝜏‖‖_{𝐿𝑝∗𝜌𝑋}≤𝐶log3𝛿𝑚^{𝜂−(𝑝+1)/2(𝑝+2)},(2.4) where 𝐶 is a constant independent of 𝑚 or 𝛿.

If 𝑝≥(1/2𝜂)−2 for 0<𝜂<1/4, we see that the power exponent for the learning rate (2.4) is at least (1/2)−2𝜂. This exponent can be arbitrarily close to 1/2 when 𝜂 is small enough.

In particular, if we take 𝜏=1/2, Theorem 2.3 provides rates for output function 𝑓^SVR_𝐳 of the support vector regression (1.2) to approximate the median function 𝑓_𝜌,1/2.

If we take 𝛽=∞ leading to 𝜖=0, Theorem 2.3 provides rates for output function 𝑓^Q𝑅𝐳 of the quantile regression algorithm (1.4) to approximate the quantile regression function 𝑓_𝜌,𝜏.

2.2. General Approximation Analysis

To state our approximation analysis in the general case, we need the capacity of the hypothesis space measured by covering numbers.

Definition 2.4. For a subset 𝑆 of 𝐶(𝑋) and 𝑢>0, the covering number 𝒩(𝑆,𝑢) is the minimal integer 𝑙∈ℕ such that there exist 𝑙 disks with radius 𝑢 covering 𝑆.

The covering numbers of balls 𝐵_𝑅={𝑓∈ℋ_𝐾∶‖𝑓‖_𝐾≤𝑅} with radius 𝑅>0 of the RKHS have been well studied in the learning theory literature [10, 11]. In this paper, we assume for some 𝑠>0 and 𝐶_𝑠>0 thatlog𝒩𝐵₁,𝑢≤𝐶_𝑠1𝑢^𝑠,∀𝑢>0.(2.5)

Now we can state our main result which will be proved in Section 5. For 𝑝∈(0,∞] and 𝑞∈(1,∞), we denote𝜃=min2𝑞,𝑝+𝑝1∈(0,1].(2.6)

Theorem 2.5. Assume (2.3) with 0<𝑟≤1/2 and (2.5) with 𝑠>0. Suppose that 𝜌 has a 𝜏-quantile of 𝑝-average type 𝑞 for some 𝑝∈(0,∞] and 𝑞∈(1,∞). Take 𝜆=𝑚^−𝛼 with 0<𝛼≤1 and 𝛼<(2+𝑠)/𝑠(2+𝑠−𝜃). Set 𝜖=𝑚^−𝛽 with 𝛼𝑟/(1−𝑟)≤𝛽≤∞. Let 0<𝜂<(1+𝑠)[2+2𝑠−𝑠𝛼(2+𝑠−𝜃)]𝑠(2+𝑠−𝜃)(2+𝑠).(2.7) Then, with 𝑝^∗=𝑝𝑞/(𝑝+1)>0, for any 0<𝛿<1, with confidence 1−𝛿, one has ‖‖𝜋𝑓^𝐳(𝜖)−𝑓^𝜌,𝜏‖‖_{𝐿𝑝∗𝜌𝑋}≤𝐶log3𝜂²log3𝛿𝑚^−𝜗,(2.8) where 𝐶 is a constant independent of 𝑚 or 𝛿 and the power index 𝜗 is given in terms of 𝑟,𝑠,𝑝,𝑞,𝛼, and 𝜂 by 𝜗=1𝑞min1𝛼−1𝑟𝑟,2+1𝑠−𝜃−𝑠([2𝛼+(2𝑠+−𝑠𝜃−)(𝜃2)+−𝑠1)]−1𝑠+𝜂𝑠,2+𝑠−𝜃−(1𝑠+𝛼𝑠(1)(−2−2𝑟2)𝑟),2+1𝑠−𝜃−1+𝑠𝑠𝛼2−2(21−𝜃).(2.9)

The index 𝜗 can be viewed as a function of variables 𝑟,𝑠,𝑝,𝑞,𝛼,𝜂. The restriction 0<𝛼<(2+𝑠)/𝑠(2+𝑠−𝜃) on 𝛼 and (2.7) on 𝜂 ensure that 𝜗 is positive, which verifies the valid learning rate in Theorem 2.5.

Assumption (2.5) is a measurement of regularity of the kernel 𝐾 when 𝑋 is a subset of ℝ^𝑛. In particular, 𝑠 can be arbitrarily small when 𝐾 is smooth enough. In this case, the power index 𝜗 in (2.8) can be arbitrarily close to (1/𝑞)min{𝛼𝑟/(1−𝑟),1/(2−𝜃)}. Again, when 𝛽=∞, 𝜖=0, algorithm (1.7) corresponds to algorithm (1.4) for quantile regression. In this case, Theorem 2.5 provides learning rates for quantile regression algorithm (1.4).

Error analysis has been done for quantile regression algorithm (1.4) in [5, 6]. Under the assumptions that 𝜌 satisfies (2.2) with some 𝑝∈(0,∞] and 𝑞>1 and the ℓ²-empirical covering number 𝒩_𝐳(𝐵₁,𝜂,ℓ²) (see [5] for more details) of 𝐵₁ is bounded assup_{𝐳∈𝑍𝑚}log𝒩_𝐳𝐵₁,𝜂,ℓ²≤𝑎1𝜂^𝑠with𝑠∈(0,2),𝑎≥1,(2.10) it was proved in [5] that with confidence 1−𝛿,‖‖𝜋𝑓^Q𝑅𝐳−𝑓^𝜌,𝜏‖‖_{𝐿𝑝∗𝜌𝑋}≤⎧⎪⎨⎪⎩𝒟_𝜏(𝜆)+𝒟_𝜏(𝜆)𝜆log𝑚(3/𝛿)+𝐾^{𝑠,𝐶𝑝}𝑎𝜆^𝑠/2𝑚^{(𝑝+1)/(𝑝+2−𝑠/2)}+𝐾^{𝑠,𝐶𝑝}𝑎𝜆^𝑠/2𝑚+532𝐶^𝑝log(3/𝛿)𝑚^{(𝑝+1)/(𝑝+2)}+145log(3/𝛿)𝑚⎫⎪⎬⎪⎭^1/𝑞,(2.11) where 𝐶_𝑝 and 𝐾_{𝑠,𝐶𝑝} are constants independent of 𝑚 or 𝜆. Here, 𝒟_𝜏(𝜆) is the regularization error defined as𝒟_𝜏(𝜆)=inf_{𝑓∈ℋ𝐾}ℰ_𝜏(𝑓)−ℰ_𝜏𝑓_𝜌,𝜏+𝜆‖𝑓‖^2𝐾,(2.12) and ℰ_𝜏(𝑓) is the generalization error associated with the pinball loss 𝜓_𝜏 defined byℰ_𝜏(𝑓)=_𝑍𝜓_𝜏(𝑓(𝑥)−𝑦)𝑑𝜌=_𝑋_𝑌𝜓_𝜏(𝑓(𝑥)−𝑦)𝑑𝜌(𝑦∣𝑥)𝑑𝜌_𝑋(𝑥).(2.13) Note that ℰ_𝜏(𝑓) is minimized by the quantile regression function 𝑓_𝜌,𝜏. Thus, when the regularization error 𝒟_𝜏(𝜆) decays polynomially as 𝒟_𝜏(𝜆)=𝑂(𝜆^{𝑟/(1−𝑟)}) (which is ensured by Lemma 2.6 below when (2.3) is satisfied) and 𝜆=𝑚^−𝛼, then ‖𝜋(𝑓^Q𝑅𝐳)−𝑓_𝜌,𝜏‖_{𝐿𝑝∗𝜌𝑋}=𝑂(log(3/𝛿)𝑚^−𝜗) with𝜗=1𝑞min1𝛼−𝑟𝑟,𝑝+𝑝2+−1(𝑠/2)1−𝛼2𝑠,𝑝𝑝++12.(2.14) Since (𝑝+1)/(𝑝+2)=1/(2−𝜃), we see that this learning rate is comparable to our result in (2.8).

2.3. Comparison with Least Squares Regression

There has been a large literature in learning theory (described in [12]) for the least squares algorithms:𝑓^L𝑆𝐳=argmin_{𝑓∈ℋ𝐾}𝑚1^𝑚_𝑖=1𝑓𝑥_𝑖−𝑦_𝑖²+𝜆‖𝑓‖^2𝐾.(2.15) It aims at learning the regression function 𝑓_𝜌(𝑥)=∫_𝑌𝑦𝑑𝜌(𝑦∣𝑥). A crucial property for its error analysis is the identity ℰ_𝑙𝑠(𝑓)−ℰ_𝑙𝑠(𝑓_𝜌)=‖𝑓−𝑓_𝜌‖^2𝐿_2𝜌𝑋 for the least squares generalization error ℰ_𝑙𝑠(𝑓)=∫_𝑍(𝑦−𝑓(𝑥))²𝑑𝜌. It yields a variance-expectation bound 𝐄(𝜉²)≤4𝐄(𝜉) for the random variable 𝜉=(𝑦−𝑓(𝑥))²−(𝑦−𝑓_𝜌(𝑥))² on (𝑍,𝜌) where 𝑓∶𝑋→𝑌 is an arbitrary measurable function. Such a variance-expectation bound with 𝐄(𝜉) possibly replaced by its positive power (𝐄(𝜉))^𝜃 plays an essential role for analyzing regularization schemes and the power exponent 𝜃 depends on strong convexity of the loss. See [13] and references therein. However, the pinball loss in the quantile regression setting has no strong convexity [6] and we would not expect a variance-expectation bound for a general distribution 𝜌. When 𝜌 has a 𝜏-quantile of 𝑝-average type 𝑞, the following variance-expectation bound with 𝜃 given by (2.6) can be found in [5, 7] (derived by means of Lemma 3.1 below).

Lemma 2.6. If 𝜌 has a 𝜏-quantile of 𝑝-average type 𝑞 for some 𝑝∈(0,∞] and 𝑞∈(1,∞), then 𝐄𝜓_𝜏(𝑓(𝑥)−𝑦)−𝜓_𝜏𝑓_𝜌,𝜏(𝑥)−𝑦²≤𝐶^𝜃ℰ_𝜏(𝑓)−ℰ_𝜏𝑓_𝜌,𝜏^𝜃,∀𝑓∶𝑋⟶𝑌,(2.16) where the power index 𝜃 is given by (2.6) and the constant 𝐶_𝜃 is 𝐶_𝜃=2^2−𝜃𝑞^𝜃‖𝛾⁻¹‖^𝜃𝐿_𝑝𝜌𝑋.

Lemma 2.6 overcomes the difficulty of quantile regression caused by lack of strong convexity of the pinball loss. It enables us to derive satisfactory learning rates, as in Theorem 2.5.

3. Insensitive Relation and Error Decomposition

An important relation for quantile regression observed in [5] assets that the error ‖𝜋(𝑓^𝐳(𝜖))−𝑓_𝜌,𝜏‖ taken in a suitable 𝐿^𝑝∗_𝜌𝑋 space can be bounded by the excess generalization error ℰ_𝜏(𝜋(𝑓^𝐳(𝜖)))−ℰ_𝜏(𝑓_𝜌,𝜏) when the noise condition is satisfied.

Lemma 3.1. Let 𝑝∈(0,∞] and 𝑞∈(1,∞). Denote 𝑝^∗=𝑝𝑞/(𝑝+1)>0. If 𝜌 has a 𝜏-quantile of 𝑝-average type 𝑞, then for any measurable function 𝑓 on 𝑋, one has ‖‖𝑓−𝑓_𝜌,𝜏‖‖_{𝐿𝑝∗𝜌𝑋}≤𝐶_𝑞,𝜌ℰ_𝜏(𝑓)−ℰ_𝜏𝑓_𝜌,𝜏^1/𝑞,(3.1) where 𝐶_𝑞,𝜌=2^1−(1/𝑞)𝑞^1/𝑞‖{(𝑏_𝑥a^𝑥𝑞−1)⁻¹}_𝑥∈𝑋‖^𝐿1/𝑞_𝑝𝜌𝑋.

By Lemma 3.1, to estimate ‖𝜋(𝑓^𝐳(𝜖))−𝑓_𝜌,𝜏‖_{𝐿𝑝∗𝜌𝑋}, we only need to bound the excess generalization error ℰ_𝜏(𝜋(𝑓^𝐳(𝜖)))−ℰ_𝜏(𝑓_𝜌,𝜏). This will be done by conducting an error decomposition which has been developed in the literature for regularization schemes [9, 13–15]. Technical difficulty arises for our problem here because the insensitive parameter 𝜖 changes with 𝑚. This can be overcome [16] by the following insensitive relation𝜓_𝜏(𝑢)−𝜖≤𝜓^𝜖𝜏(𝑢)≤𝜓_𝜏(𝑢),𝑢∈ℝ.(3.2)

Now, we can conduct an error decomposition. Define the empirical error ℰ_𝐳,𝜏(𝑓) for 𝑓∶𝑋→ℝ asℰ_𝐳,𝜏(𝑓)=1𝑚^𝑚_𝑖=1𝜓_𝜏𝑓𝑥_𝑖−𝑦_𝑖.(3.3)

Lemma 3.2. Let 𝜆>0, 𝑓^𝑧(𝜖) be defined by (1.7) and 𝑓^𝜆(0)=argmin_{𝑓∈ℋ𝐾}ℰ_𝜏(𝑓)−ℰ_𝜏𝑓_𝜌,𝜏+𝜆‖𝑓‖^2𝐾.(3.4) Then, ℰ_𝜏𝜋𝑓^𝐳(𝜖)−ℰ^𝜏𝑓_𝜌,𝜏+𝜆‖‖𝑓^𝐳(𝜖)‖‖^2𝐾≤𝒮₁+𝒮₂+𝜖+𝒟_𝜏(𝜆),(3.5) where 𝒮𝒮₁₂=ℰ_𝜏𝜋𝑓^𝐳(𝜖)−ℰ^𝜏𝑓_𝜌,𝜏−ℰ_𝐳,𝜏𝜋𝑓^𝐳(𝜖)−ℰ^𝐳,𝜏𝑓_𝜌,𝜏,=ℰ_𝐳,𝜏𝑓^𝜆(0)−ℰ^𝐳,𝜏𝑓_𝜌,𝜏−ℰ_𝜏𝑓^𝜆(0)−ℰ^𝜏𝑓_𝜌,𝜏,(3.6)

Proof. The regularized excess generalization error ℰ_𝜏(𝜋(𝑓^𝐳(𝜖)))−ℰ_𝜏(𝑓_𝜌,𝜏)+𝜆‖𝑓^𝐳(𝜖)‖^2𝐾 can be expressed as ℰ^𝜏𝜋𝑓^𝐳(𝜖)−ℰ^𝐳,𝜏𝜋𝑓^𝐳(𝜖)+ℰ^𝐳,𝜏𝜋𝑓^𝐳(𝜖)+𝜆‖‖𝑓^𝐳(𝜖)‖‖^2𝐾−ℰ^𝐳,𝜏𝑓^𝜆(0)+𝜆‖‖𝑓^𝜆(0)‖‖^2𝐾+ℰ_𝐳,𝜏𝑓^𝜆(0)−ℰ^𝜏𝑓^𝜆(0)+ℰ^𝜏𝑓^𝜆(0)−ℰ^𝜏𝑓_𝜌,𝜏+𝜆‖‖𝑓^𝜆(0)‖‖^2𝐾.(3.7) The fact |𝑦|≤1 implies that ℰ_𝐳,𝜏(𝜋(𝑓^𝐳(𝜖)))≤ℰ_𝐳,𝜏(𝑓^𝐳(𝜖)). The insensitive relation (3.2) and the definition of 𝑓^𝐳(𝜖) tell us that ℰ_𝐳,𝜏𝑓^𝐳(𝜖)+𝜆‖‖𝑓^𝐳(𝜖)‖‖^2𝐾≤1𝑚^𝑚_𝑖=1≤1𝜓^𝜖𝜏𝑓^𝐳(𝜖)𝑥_𝑖−𝑦_𝑖+𝜆‖‖𝑓^𝐳(𝜖)‖‖^2𝐾+𝜖𝑚^𝑚_𝑖=1𝜓^𝜖𝜏𝑓^𝜆(0)𝑥_𝑖−𝑦_𝑖+𝜆‖‖𝑓^𝜆(0)‖‖^2𝐾+𝜖≤ℰ_𝐳,𝜏𝑓^𝜆(0)+𝜆‖‖𝑓^𝜆(0)‖‖^2𝐾+𝜖.(3.8) Then, by subtracting and adding ℰ_𝜏(𝑓_𝜌,𝜏) and ℰ_𝐳,𝜏(𝑓_𝜌,𝜏) and noting 𝒟_𝜏(𝜆)=ℰ_𝜏(𝑓^𝜆(0))−ℰ_𝜏(𝑓_𝜌,𝜏)+𝜆‖𝑓^𝜆(0)‖^2𝐾, we see that the desired inequality in Lemma 3.2 holds true.

In the error decomposition (3.5), the first two terms are called sample error. The last term is the regularization error defined in (2.12). It can be estimated as follows.

Proposition 3.3. Assume (2.3). Define 𝑓^𝜆(0) by (3.4). Then, one has ‖‖𝑓𝒟_𝜏𝜆(0)(𝜆)≤𝐶₀𝜆^{𝑟/(1−𝑟)},‖‖_𝐾≤√C₀𝜆^{(2𝑟−1)/(2−2𝑟)},(3.9) where 𝐶₀ is the constant 𝐶₀=‖𝑔_𝜌,𝜏‖_𝐿^2𝜌𝑋+‖𝑔_𝜌,𝜏‖^2𝐿_2𝜌𝑋.

Proof. Let 𝜇=𝜆^1/(1−𝑟)>0 and 𝑓_𝜇=𝐿_𝐾+𝜇𝐼⁻¹𝐿_𝐾𝑓_𝜌,𝜏.(3.10) It can be found in [17, 18] that when (2.3) holds, we have ‖‖𝑓_𝜇−𝑓_𝜌,𝜏‖‖^2𝐿_2𝜌𝑋+𝜇‖‖𝑓_𝜇‖‖^2𝐾≤𝜇^2𝑟‖‖𝑔_𝜌,𝜏‖‖^2𝐿_2𝜌𝑋.(3.11) Hence, ‖𝑓_𝜇−𝑓_𝜌,𝜏‖_𝐿^2𝜌𝑋≤𝜇^𝑟‖𝑔_𝜌,𝜏‖_𝐿^2𝜌𝑋 and ‖𝑓_𝜇‖^2𝐾≤𝜇^2𝑟−1‖𝑔_𝜌,𝜏‖^2𝐿_2𝜌𝑋. Since 𝜓_𝜏 is Lipschitz, we know by taking 𝑓=𝑓_𝜇 in (2.12) that 𝒟_𝜏(𝜆)≤_𝑍𝜓_𝜏𝑓_𝜇(𝑥)−𝑦−𝜓_𝜏𝑓_𝜌,𝜏(𝑥)−𝑦+𝜆‖‖𝑓_𝜇‖‖^2𝐾≤‖‖𝑓_𝜇−𝑓_𝜌,𝜏‖‖_{𝐿1𝜌𝑋}+𝜆‖‖𝑓_𝜇‖‖^2𝐾≤‖‖𝑓_𝜇−𝑓_𝜌,𝜏‖‖_{𝐿2𝜌𝑋}+𝜆‖‖𝑓_𝜇‖‖^2𝐾≤𝜇^𝑟‖‖𝑔_𝜌,𝜏‖‖_{𝐿2𝜌𝑋}+𝜆𝜇^2𝑟−1‖‖𝑔_𝜌,𝜏‖‖^2𝐿_2𝜌𝑋=‖‖𝑔_𝜌,𝜏‖‖_{𝐿2𝜌𝑋}+‖‖𝑔_𝜌,𝜏‖‖^2𝐿_2𝜌𝑋𝜆^{𝑟/(1−𝑟)}.(3.12) This verifies the desired bound for 𝒟_𝜏(𝜆). By taking 𝑓=0 in (2.12), we have 𝜆‖‖𝑓^𝜆(0)‖‖^2𝐾≤𝒟_𝜏(𝜆)≤𝐶₀𝜆^{𝑟/(1−𝑟)}.(3.13) Then the bound for ‖𝑓^𝜆(0)‖_𝐾 is proved.

4. Estimating Sample Error

This section is devoted to estimating the sample error. This is conducted by using the variance-expectation bound in Lemma 2.6.

Denote 𝜅=sup_𝑥∈𝑋√𝐾(𝑥,𝑥). For 𝑅≥1, denote𝒲(𝑅)=𝐳∈𝑍^𝑚∶‖‖𝑓^𝐳(𝜖)‖‖_𝐾≤𝑅.(4.1)

Proposition 4.1. Assume (2.3) and (2.5). Let 𝑅≥1 and 0<𝛿<1. If 𝜌 has a 𝜏-quantile of 𝑝-average type 𝑞 for some 𝑝∈(0,∞] and 𝑞∈(1,∞), then there exists a subset 𝑉_𝑅 of 𝑍^𝑚 with measure at most 𝛿 such that for any 𝐳∈𝒲(𝑅)⧵𝑉_𝑅, ℰ_𝜏𝜋𝑓^𝐳(𝜖)−ℰ^𝜏𝑓_𝜌,𝜏+𝜆‖‖𝑓^𝐳(𝜖)‖‖^2𝐾≤2𝜖+𝐶^′₁log3𝛿𝜆^{𝑟/(1−𝑟)}max1,𝜆^{−(1/(2−2𝑟))}+𝐶_2𝑚log3𝛿𝑚^{−1/(2−𝜃)}+𝐶^3𝑚^{−1/(2+𝑠−𝜃)}𝑅^{𝑠/(1+𝑠)},(4.2) where 𝜃 is given by (2.6) and 𝐶^1,𝐶^2,𝐶^3 are constants given by 𝐶^1=4𝐶₀+2𝜅√𝐶₀,𝐶^2=245+324𝐶^{𝜃1/(2−𝜃)},𝐶^3=406𝐶_𝑠^1/(1+𝑠)+408𝐶_𝜃𝐶_𝑠^{1/(2+𝑠−𝜃)}.(4.3)

Proof. Let us first estimate the second part 𝒮₂ of the sample error. It can be decomposed into two parts 𝒮₂=𝒮_2,1+𝒮_2,2 where 𝒮𝒮_2,1=ℰ_𝐳,𝜏𝑓^𝜆(0)−ℰ^𝐳,𝜏𝜋𝑓^𝜆(0)−ℰ^𝜏𝑓^𝜆(0)−ℰ^𝜏𝜋𝑓^𝜆(0)_2,2,=ℰ_𝐳,𝜏𝜋𝑓^𝜆(0)−ℰ^𝐳,𝜏𝑓_𝜌,𝜏−ℰ_𝜏𝜋𝑓^𝜆(0)−ℰ^𝜏𝑓_𝜌,𝜏.(4.4)
For bounding 𝒮_2,1, we take the random variable 𝜉(𝑧)=𝜓_𝜏(𝑓^𝜆(0)(𝑥)−𝑦)−𝜓_𝜏(𝜋(𝑓^𝜆(0))(𝑥)−𝑦) on (𝑍,𝜌). It satisfies 0≤𝜉≤|𝜋(𝑓^𝜆(0))(𝑥)−𝑓^𝜆(0)(𝑥)|≤1+‖𝑓^𝜆(0)‖_∞. Hence, |𝜉−𝐄(𝜉)|≤1+‖𝑓^𝜆(0)‖_∞ and 𝐄(𝜉−𝐄(𝜉))²≤𝐄𝜉²≤(1+‖𝑓^𝜆(0)‖_∞)𝐄(𝜉). Applying the one-side Bernstein inequality [12], we know that there exists a subset 𝑍_1,𝛿 of 𝑍^𝑚 with measure at least 1−(𝛿/3) such that 𝒮_2,1≤71+‖‖𝑓^𝜆(0)‖‖_∞log(3/𝛿)6𝑚+ℰ^𝜏𝑓^𝜆(0)−ℰ^𝜏𝜋𝑓^𝜆(0),∀𝐳∈𝑍^1,𝛿.(4.5)
For 𝒮_2,2, we apply the one-side Bernstein inequality again to the random variable 𝜉(𝑧)=𝜓_𝜏(𝜋(𝑓^𝜆(0))(𝑥)−𝑦)−𝜓_𝜏(𝑓_𝜌,𝜏(𝑥)−𝑦), bound the variance by Lemma 2.6 with 𝑓=𝜋(𝑓^𝜆(0)), and find that there exists another subset 𝑍_2,𝛿 of 𝑍^𝑚 with measure at least 1−(𝛿/3) such that 𝒮_2,2≤4log(3/𝛿)3𝑚+𝜃2^{𝜃/(4−2𝜃)}1−𝜃22𝐶^𝜃log(3/𝛿)𝑚^1/(2−𝜃)+ℰ_𝜏𝜋𝑓^𝜆(0)−ℰ^𝜏𝑓_𝜌,𝜏,∀𝐳∈𝑍_2,𝛿.(4.6)
Next, we estimate the first part 𝒮₁ of the sample error. Consider the function set 𝒢=𝜓_𝜏(𝜋(𝑓)(𝑥)−𝑦)−𝜓_𝜏𝑓_𝜌,𝜏(𝑥)−𝑦∶‖𝑓‖_𝐾≤𝑅.(4.7) A function from this set 𝑔(𝑧)=𝜓_𝜏(𝜋(𝑓)(𝑥)−𝑦)−𝜓_𝜏(𝑓_𝜌,𝜏(𝑥)−𝑦) satisfies 𝐄(𝑔)≥0, |𝑔(𝑧)|≤2, and 𝐄(𝑔²)≤𝐶_𝜃(𝐄(𝑔))^𝜃 by (2.16). Also, the Lipschitz property of the pinball loss yields 𝒩(𝒢,𝑢)≤𝒩(𝐵₁,𝑢/𝑅). Then, we apply a standard covering number argument with a ratio inequality [12, 13, 19, 20] to 𝒢 and find from the covering number condition (2.5) that Prob_𝐳∈𝑍^𝑚⎧⎪⎨⎪⎩sup_{‖𝑓‖𝐾≤𝑅}ℰ_𝜏(𝜋(𝑓))−ℰ_𝜏𝑓_𝜌,𝜏−ℰ_𝐳,𝜏(𝜋(𝑓))−ℰ_𝐳,𝜏𝑓_𝜌,𝜏ℰ_𝜏(𝜋(𝑓))−ℰ_𝜏𝑓_𝜌,𝜏^𝜃+𝑢^𝜃≤4𝑢^1−(𝜃/2)≥1−𝒩𝐵₁⎫⎪⎬⎪⎭,𝑢𝑅exp−2𝐶_𝜃+𝑚(𝑢4/3^2−𝜃)𝑢^1−𝜃≥1−exp𝐶^𝑠𝑅𝑢^𝑠−𝑚𝑢^2−𝜃2𝐶_𝜃+(4/3)𝑢^1−𝜃.(4.8) Setting the confidence to be 1−(𝛿/3), we take 𝑢^∗(𝑅,𝑚,𝛿/3) to be the positive solution to the equation 𝐶_𝑠𝑅𝑢^𝑠−𝑚𝑢^2−𝜃2𝐶_𝜃+(4/3)𝑢^1−𝜃=log𝛿3.(4.9) Then, there exists a third subset 𝑍_3,𝛿 of 𝑍^𝑚 with measure at least 1−(𝛿/3) such that sup_{‖𝑓‖𝐾≤𝑅}ℰ_𝜏(𝜋(𝑓))−ℰ_𝜏𝑓_𝜌,𝜏−ℰ_𝐳,𝜏(𝜋(𝑓))−ℰ_𝐳,𝜏𝑓_𝜌,𝜏ℰ_𝜏(𝜋(𝑓))−ℰ_𝜏𝑓_𝜌,𝜏^𝜃+(𝑢^∗(𝑅,𝑚,𝛿/3))^𝜃≤4𝑢^∗𝑅,𝑚,𝛿3^1−(𝜃/2),∀𝐳∈𝑍_3,𝛿.(4.10) Thus, for 𝑧∈𝒲(𝑅)∩𝑍_3,𝛿, we have 𝒮₁=≤4ℰ𝑢_𝜏∗𝜋𝑓^𝐳(𝜖)−ℰ^𝜏𝑓_𝜌,𝜏−ℰ_𝐳,𝜏𝜋𝑓^𝐳(𝜖)−ℰ^𝐳,𝜏𝑓_𝜌,𝜏𝑅,𝑚,𝛿3^1−(𝜃/2)ℰ^𝜏𝜋𝑓^𝐳(𝜖)−ℰ^𝜏𝑓_𝜌,𝜏^𝜃+𝑢^∗𝑅,𝑚,𝛿3^𝜃≤1−𝜃24^2/(2−𝜃)𝑢^∗𝑅,𝑚,𝛿3+𝜃2ℰ^𝜏𝜋𝑓^𝐳(𝜖)−ℰ^𝜏𝑓_𝜌,𝜏+4𝑢^∗𝑅,𝑚,𝛿3.(4.11) Here, we have used the elementary inequality √𝑎+𝑏≤√𝑎+√𝑏 and Young’s inequality. Putting this bound and (4.5), (4.6) into (3.5), we know that for 𝐳∈𝒲(𝑅)∩𝑍_3,𝛿∩𝑍_1,𝛿∩𝑍_2,𝛿, there holds ℰ_𝜏𝜋𝑓_𝐳−ℰ_𝜏𝑓_𝜌,𝜏+𝜆‖‖𝑓^𝐳(𝜖)‖‖^2𝐾≤12ℰ^𝜏𝜋𝑓^𝐳(𝜖)−ℰ^𝜏𝑓_𝜌,𝜏+𝜖+2𝒟_𝜏+20𝑢_∗(𝜆)𝑅,𝑚,𝛿3+71+‖‖𝑓^𝜆(0)‖‖_∞log(3/𝛿)6𝑚+4log3𝑚(3/𝛿)+2𝐶^𝜃log(3/𝛿)𝑚^1/(2−𝜃),(4.12) which together with Proposition 3.3 implies ℰ_𝜏𝜋𝑓_𝐳−ℰ_𝜏𝑓_𝜌,𝜏+𝜆‖‖𝑓^𝐳(𝜖)‖‖^2𝐾≤2𝜖+4𝐶₀𝜆^{𝑟/(1−𝑟)}+5+22𝐶_𝜃^1/(2−𝜃)log3𝛿𝑚^{−1/(2−𝜃)}+40𝑢^∗𝑅,𝑚,𝛿3+2𝜅√𝐶₀log3𝛿𝜆𝑚^{(2𝑟−1)/(2−2𝑟)}.(4.13) Here, we have used the reproducing property in ℋ_𝐾 which yields [12] ‖𝑓‖_∞≤𝜅‖𝑓‖_𝐾,∀𝑓∈ℋ_𝐾.(4.14) Equation (4.9) can be expressed as 𝑢^{2+𝑠−𝜃}−43𝑚log𝛿3𝑢^{1+𝑠−𝜃}−2𝐶^𝜃𝑚log𝛿3𝑢^𝑠−4𝐶^𝑠𝑅^𝑠3𝑚𝑢^1−𝜃−2𝐶^𝜃𝐶_𝑠𝑅^𝑠𝑚=0.(4.15) By Lemma  7.2 in [12], the positive solution 𝑢^∗(𝑅,𝑚,𝛿/3) to this equation can be bounded as 𝑢^∗(𝑅,𝑚,𝛿/3)≤max6𝑚log𝛿3,8𝑚𝐶^𝜃log𝛿3^1/(2−𝜃),6𝐶^𝑠𝑅^𝑠𝑚^1/(1+𝑠),8𝐶^𝜃𝐶_𝑠𝑅^𝑠≤6+8𝐶_𝜃𝑚^{1/(2+𝑠−𝜃)}×𝑚^1/(2−𝜃)_{−1/(2+𝑠−𝜃)}log𝛿3𝑚^{−1/(2−𝜃)}+6𝐶_𝑠^1/(1+𝑠)+8𝐶_𝜃𝐶_𝑠^{1/(2+𝑠−𝜃)}𝑅^{𝑠/(1+𝑠)}.(4.16) Thus, for 𝐳∈𝒲(𝑅)∩𝑍_3,𝛿∩𝑍_1,𝛿∩𝑍_2,𝛿, the desired bound (4.2) holds true. Since the measure of the set 𝑍_3,𝛿∩𝑍_1,𝛿∩𝑍_2,𝛿 is at least 1−𝛿, our conclusion is proved.

5. Deriving Convergence Rates by Iteration

To apply Proposition 4.1 for error analysis, we need some 𝑅≥1 for 𝐳∈𝒲(𝑅). One may choose 𝑅=𝜆^−1/2 according to‖‖𝑓^𝐳(𝜖)‖‖_𝐾≤𝜆^−1/2,∀𝐳∈𝑍^𝑚(5.1) which is seen by taking 𝑓=0 in (1.7). This choice is too rough. Recall from Proposition 3.3 that ‖𝑓^𝜆(0)‖_𝐾≤√𝐶₀𝜆^{(2𝑟−1)/(2−2𝑟)} which is a bound for the noise-free limit 𝑓^𝜆(0) of 𝑓^𝐳(𝜖). It is much better than 𝜆^−1/2. This motivates us to try similar tight bounds for 𝑓^𝐳(𝜖). This target will be achieved in this section by applying Proposition 4.1 iteratively. The iteration technique has been used in [13, 21] to improve learning rates.

Lemma 5.1. Assume (2.3) with 0<𝑟≤1/2 and (2.5) with 𝑠>0. Take 𝜆=𝑚^−𝛼 with 0<𝛼≤1 and 𝜖=𝑚^−𝛽 with 0<𝛽≤∞. Let 0<𝜂<1. If 𝜌 has a 𝜏-quantile of 𝑝-average type 𝑞 for some 𝑝∈(0,∞] and 𝑞∈(1,∞), then for any 0<𝛿<1, with confidence 1−𝛿, there holds ‖‖𝑓^𝐳(𝜖)‖‖_𝐾≤4𝐶^′₃1+√2+𝐶^1+𝐶^2log3𝜂²log3𝛿𝑚^𝜃𝜂,(5.2) where 𝜃_𝜂 is given by 𝜃_𝜂=max𝛼−𝛽2,𝛼2(1−−22𝑟𝑟),𝛼2−2(21−𝜃),[𝛼(2(2++𝑠𝑠−−𝜃𝜃))−(21]+(1𝑠)+𝑠)+𝜂≥0.(5.3)

Proof. Putting 𝜆=𝑚^−𝛼 with 0<𝛼≤1 and 𝜖=𝑚^−𝛽 with 0<𝛽≤∞ into Proposition 4.1, we know that for any 𝑅≥1 there exists a subset 𝑉_𝑅 of 𝑍^𝑚 with measure at most 𝛿 such that ‖‖𝑓^𝐳(𝜖)‖‖_𝐾≤𝑎_𝑚𝑅^{𝑠/(2+2𝑠)}+𝑏_𝑚,∀𝐳∈𝒲(𝑅)⧵𝑉_𝑅,(5.4) where with 𝜁∶=max{(𝛼−𝛽)/2,𝛼(1−2𝑟)/(2−2𝑟),𝛼/2−1/2(2−𝜃)}≥0, the constants are given by 𝑎_𝑚=𝐶^3𝑚^{(𝛼/2)−(1/2(2+𝑠−𝜃))},𝑏_𝑚=√2+𝐶^1log3𝛿+𝐶^2log3𝛿𝑚^𝜁.(5.5) It follows that 𝒲(𝑅)⊆𝒲𝑎_𝑚𝑅^{𝑠/(2+2𝑠)}+𝑏_𝑚∪𝑉_𝑅.(5.6)
Let us apply (5.6) iteratively to a sequence {𝑅^(𝑗)}^𝐽𝑗=0 defined by 𝑅⁽⁰⁾=𝜆^−1/2 and 𝑅^(𝑗)=𝑎_𝑚(𝑅^(𝑗−1))^{𝑠/(2+2𝑠)}+𝑏_𝑚 where 𝐽∈ℕ will be determined later. Then, 𝒲(𝑅^(𝑗−1))⊆𝒲(𝑅^(𝑗))∪𝑉_𝑅^(𝑗−1). By (5.1), 𝒲(𝑅⁽⁰⁾)=𝑍^𝑚. So we have 𝑍^𝑚=𝒲𝑅⁽⁰⁾⊆𝒲𝑅⁽¹⁾∪𝑉_𝑅⁽⁰⁾⊆⋯⊆𝒲𝑅^(𝐽)∪∪^{𝐽−1𝑗=0}𝑉_𝑅^(𝑗).(5.7) As the measure of 𝑉_𝑅^(𝑗) is at most 𝛿, we know that the measure of ∪^{𝐽−1𝑗=0}𝑉_𝑅^(𝑗) is at most 𝐽𝛿. Hence, 𝒲(𝑅^(𝐽)) has measure at least 1−𝐽𝛿.
Denote Δ=𝑠/(2+2𝑠)<1/2. The definition of the sequence {𝑅^(𝑗)}^𝐽𝑗=0 tells us that 𝑅^(𝐽)=𝑎^{1+Δ+Δ2+⋯+Δ𝐽−1}_𝑚𝑅⁽⁰⁾^Δ𝐽+^𝐽−1_𝑗=1𝑎^{1+Δ+Δ2+⋯+Δ𝑗−1}_𝑚𝑏^Δ𝑗_𝑚+𝑏_𝑚.(5.8) Let us bound the two terms on the right-hand side.
The first term equals 𝐶^3^{(1−Δ𝐽)/2(1−Δ)}𝑚^{((𝛼(2+𝑠−𝜃)−1)/(4+2𝑠−2𝜃))((1−Δ𝐽)/(1−Δ))}𝑚^{(𝛼/2)Δ𝐽},(5.9) which is bounded by 𝐶^3𝑚^{(𝛼(2+𝑠−𝜃)−1)/(4+2𝑠−2𝜃)(1−Δ)}𝑚^{((𝛼/2)−(𝛼(2+𝑠−𝜃)−1)/(4+2𝑠−2𝜃)(1−Δ))Δ𝐽}≤𝐶^3𝑚^{[𝛼(2+𝑠−𝜃)−1](1+𝑠)/(2+𝑠−𝜃)(2+𝑠)}𝑚^{(1/(2+𝑠−𝜃))2−𝐽}.(5.10) Take 𝐽 to be the smallest integer greater than or equal to log(1/𝜂)/log2. The above expression can be bounded by 𝐶^3𝑚^{[𝛼(2+𝑠−𝜃)−1](1+𝑠)/(2+𝑠−𝜃)(2+𝑠)+𝜂}.
The second terms equals ^𝐽−1_𝑗=1𝑎^{1+Δ+Δ2+⋯+Δ𝑗−1}_𝑚𝑏^Δ𝑗_𝑚+𝑏_𝑚≤^𝐽−1_𝑗=1𝐶^3𝑚^{((𝛼/2)−1/2(2+𝑠−𝜃))((1−Δ𝑗)/(1−Δ))}𝑏^Δ𝑗₁𝑚^𝜁Δ𝑗+𝑏₁𝑚^𝜁,(5.11) where 𝑏₁∶=√2+𝐶^1log(3/𝛿)+𝐶^2log(3/𝛿). It is bounded by 𝑚^{[𝛼(2+𝑠−𝜃)−1](1+𝑠)/(2+𝑠−𝜃)(2+𝑠)}𝐶^3𝑏_1𝐽−1_𝑗=0𝑚^{(𝜁−([𝛼(2+𝑠−𝜃)−1](1+𝑠)/(2+𝑠−𝜃)(2+𝑠)))(𝑠𝑗/(2+2𝑠)𝑗)}.(5.12) When 𝜁≤[𝛼(2+𝑠−𝜃)−1](1+𝑠)/(2+𝑠−𝜃)(2+𝑠), the above expression is bounded by 𝐶^3𝑏₁𝐽𝑚^{[𝛼(2+𝑠−𝜃)−1](1+𝑠)/(2+𝑠−𝜃)(2+𝑠)}. When 𝜁≥[𝛼(2+𝑠−𝜃)−1](1+𝑠)/(2+𝑠−𝜃)(2+𝑠), it is bounded by 𝐶^3𝑏₁𝐽𝑚^𝜁.
Based on the above discussion, we obtain 𝑅^(𝐽)≤𝐶^3+𝐶^3𝑏₁𝐽𝑚^𝜃𝜂,(5.13) where 𝜃_𝜂=max{[𝛼(2+𝑠−𝜃)−1](1+𝑠)/(2+𝑠−𝜃)(2+𝑠)+𝜂,𝜁}. So with confidence 1−𝐽𝛿, there holds ‖‖𝑓^𝐳(𝜖)‖‖_𝐾≤𝑅^(𝐽)≤𝐶^31+√2+𝐶^1+𝐶^2𝐽log3𝛿𝑚^𝜃𝜂.(5.14) Then, our conclusion follows by replacing 𝛿 by 𝛿/𝐽 and noting 𝐽≤2log(3/𝜂).

Now, we can prove our main result, Theorem 2.5.

Proof of Theorem 2.5. Take 𝑅 to be the right side of (5.2). By Lemma 5.1, there exists a subset 𝑉^𝑅 of 𝑍^𝑚 with measure at most 𝛿 such that 𝑍^𝑚⧵𝑉^𝑅⊆𝒲(𝑅). Applying Proposition 4.1 to this 𝑅, we know that there exists another subset 𝑉_𝑅 of 𝑍^𝑚 with measure at most 𝛿 such that for any 𝐳∈𝒲(𝑅)⧵𝑉_𝑅, ℰ_𝜏𝜋𝑓^𝐳(𝜖)−ℰ^𝜏𝑓_𝜌,𝜏≤2𝑚^−𝛽+𝐶^1log3𝛿𝑚^{−𝛼𝑟/(1−𝑟)}+𝐶^2log3𝛿𝑚^{−1/(2−𝜃)}+𝐶^4log3𝜂²log3𝛿𝑚^{(𝑠/(1+𝑠))𝜃𝜂−(1/(2+𝑠−𝜃))},(5.15) where 𝐶^4=𝐶^34𝐶^3^{𝑠/(1+𝑠)}1+√2+𝐶^1+𝐶^2.(5.16) Since the set 𝑉_𝑅∪𝑉^𝑅 has measure at most 2𝛿, after scaling 2𝛿 to 𝛿 and setting the constant 𝐶 by 𝐶=2𝐶_𝑞,𝜌2+𝐶^1+𝐶^2+𝐶^4,(5.17) we see that the above estimate together with Lemma 3.1 gives the error bound ‖‖𝜋𝑓^𝐳(𝜖)−𝑓^𝜌,𝜏‖‖_{𝐿𝑝∗𝜌𝑋}≤𝐶log3𝜂²log3𝛿𝑚^−𝜗(5.18) with confidence 1−𝛿 and the power index 𝜗 give by 𝜗=1𝑞min𝛽,1𝛼−𝑟𝑟,2+1𝑠−𝜃−1+𝑠𝑠𝜃^𝜂,(5.19) provided that 𝜃_𝜂<1+𝑠𝑠(2+𝑠−𝜃).(5.20) Since 𝛽≥𝛼𝑟/(1−𝑟), we know that (𝛼−𝛽)/2≤𝛼(1−2𝑟)/(2−2𝑟). By the restriction 0<𝛼<(2+𝑠)/𝑠(2+𝑠−𝜃) on 𝛼, we find 𝛼(1−2𝑟)/(2−2𝑟)<(1+𝑠)/𝑠(2+𝑠−𝜃) and (𝛼/2)−(1/2(2−𝜃))<(1+𝑠)/𝑠(2+𝑠−𝜃). Moreover, restriction (2.7) on 𝜂 tell us that [𝛼(2+𝑠−𝜃)−1](1+𝑠)/(2+𝑠−𝜃)(2+𝑠)+𝜂<(1+𝑠)/𝑠(2+𝑠−𝜃). Therefore, condition (5.20) is satisfied. The restriction 𝛽≥𝛼𝑟/(1−𝑟) and the above expression for 𝜗 tells us that the power index for the error bound can be exactly expressed by formula (2.9). The proof of Theorem 2.5 is complete.

Finally, we prove Theorem 2.3.

Proof of Theorem 2.3. Since 𝑓_𝜌,𝜏∈ℋ_𝐾, we know that (2.3) holds with 𝑟=1/2. The noise condition on 𝜌 is satisfied with 𝑞=2 and 𝑝∈(0,∞]. Then, 𝜃=𝑝/(𝑝+1)∈(0,1]. Since 𝑋⊂ℝ^𝑛 and 𝐾∈𝐶^∞(𝑋×𝑋), we know from [11] that (2.5) holds true for any 𝑠>0. With 0<𝜂<((𝑝+1)/2(𝑝+2)), let us choose 𝑠 to be a positive number satisfying the following four inequalities: 𝑝+11𝑝+2<𝑠(22++𝑠𝑠−𝜃),𝑝+13<(1+𝑠)[2+2𝑠𝑠−(2(𝑠+(2𝑠−+𝜃𝑠)−(2𝜃+)(𝑝𝑠)+1))/(𝑝+2)],𝑝+2−2𝜂≤2+1𝑠−𝜃−𝑝+𝑠[1(2+𝑠(2−+𝜃)𝑠(−𝑝+𝜃)1()2/+(𝑝𝑠+)2−1)]−3(1𝑠+𝑠),𝑝+2−2𝜂≤2+1𝑠−𝜃.(5.21) The first inequality above tells us that the restrictions on 𝛼,𝛽 are satisfied by choosing 𝛼=(𝑝+1)/(𝑝+2)=1/(2−𝜃) and (𝑝+1)/(𝑝+2)≤𝛽≤∞. The second inequality shows that condition (2.7) for the parameter 𝜂 renamed now as 𝜂^∗ is also satisfied by taking 𝜂^∗=1/3. Thus, we apply Theorem 2.5 and know that with confidence 1−𝛿, (2.8) holds with the power index 𝜗 given by (2.9); but 𝑟=1/2,𝛼=1/(2−𝜃), and 𝜂^∗=1/3 imply that 𝜗=12min𝛼,2+1𝑠−𝜃,2+1𝑠−𝜃−𝑠([2𝛼+(2𝑠+−𝑠𝜃−)(𝜃2)+−𝑠1)]−3(1𝑠+𝑠).(5.22) The last two inequalities satisfied by 𝑠 yield 𝜗≥(𝛼/2)−𝜂. So (2.8) verifies (2.4). This completes the proof of the theorem.

Acknowledgment

The work described in this paper is supported by National Science Foundation of China under Grant 11001247 and by a grant from the Research Grants Council of Hong Kong (Project no. CityU 103709).