ERM Scheme for Quantile Regression

Abstract

This paper considers the ERM scheme for quantile regression. We conduct error analysis for this learning algorithm by means of a variance-expectation bound when a noise condition is satisfied for the underlying probability measure. The learning rates are derived by applying concentration techniques involving the ℓ²-empirical covering numbers.

1. Introduction

In this paper, we study empirical risk minimization scheme (ERM) for quantile regression. Let 𝑋 be a compact metric space (input space) and 𝑌=ℝ. Let 𝜌 be a fixed but unknown probability distribution on 𝑍∶=𝑋×𝑌 which describes the noise of sampling. The conditional quantile regression aims at producing functions to estimate quantile regression functions. With a prespecified quantile parameter 𝜏∈(0,1), a quantile regression function 𝑓_𝜏,𝜌 is defined by its value 𝑓_𝜏,𝜌(𝑥) to be a 𝜏-quantile of 𝜌(⋅∣𝑥), that is, a value 𝑡∈𝑌 satisfying 𝜌((−∞,𝑡]∣𝑥)≥𝜏,𝜌([𝑡,∞)∣𝑥)≥1−𝜏,𝑥∈𝑋, (1) where 𝜌(⋅∣𝑥) is the conditional distribution of 𝜌 at 𝑥∈𝑋.

We consider a learning algorithm generated by ERM scheme associated with pinball loss and hypothesis space ℋ. The pinball loss 𝐿_𝜏∶ℝ→[0,∞) is defined by 𝐿_𝜏(𝑟)=(𝜏−1)𝑟,if𝑟≤0,𝜏𝑟,if𝑟≥0. (2) The hypothesis space ℋ is a compact subset of 𝐶(𝑋). So there exists some 𝑀>0 such that ‖𝑓‖_𝐶(𝑋)≤𝑀 for any 𝑓∈ℋ. We assume without loss of generality ‖𝑓‖_𝐶(𝑋)≤1 for any 𝑓∈ℋ.

The ERM scheme for quantile regression is defined with a sample 𝐳={(𝑥_𝑖,𝑦_𝑖)}^𝑚_𝑖=1∈𝑍^𝑚 drawn independently from 𝜌 as follows: 𝑓_𝐳=argmin_𝑓∈ℋ1𝑚^𝑚_𝑖=1𝐿_𝜏𝑦_𝑖−𝑓𝑥_𝑖. (3)

A family of kernel based learning algorithms for quantile regression has been widely studied in a large literature [1–4] and references therein. The form of the algorithms is a regularized scheme in a reproducing kernel Hilbert space ℋ_𝐾 (RKHS, see [5] for details) associated with a Mercer kernel 𝐾. Given a sample 𝐳 the kernel based regularized scheme for quantile regression is defined by 𝑓_𝐳,𝜆=argmin_{𝑓∈ℋ𝐾}1𝑚^𝑚_𝑖=1𝐿_𝜏𝑦_𝑖−𝑓𝑥_𝑖+𝜆𝑓²_𝐾. (4) In [1, 3, 4], error analysis for general ℋ_𝐾 has been done. Learning with varying Gaussian kernel was studied in [2].

ERM scheme (3) is very different from kernel based regularized scheme (4). The output function 𝑓_𝐳 produced by the ERM scheme has a uniform bound, under our assumption, ‖𝑓_𝐳‖_𝐶(𝑋)≤1. However, we cannot expect it for 𝑓_𝐳,𝜆. It is easy to see that 𝜆‖𝑓_𝐳,𝜆‖²_𝐾≤∑^𝑚_𝑖=1|𝑦_𝑖| by choosing 𝑓=0. It happens often that ‖𝑓_𝐳,𝜆‖_𝐾→∞ as 𝜆→0. The lack of a uniform bound for 𝑓_𝐳,𝜆 has a serious negative impact on the learning rates. So in the literature of kernel based regularized schemes for quantile regression, values of the output function 𝑓_𝐳,𝜆 are always projected onto the interval [−1,1], and error analysis is conducted for the projected function, not 𝑓_𝐳,𝜆 itself.

In this paper, we aim at establishing convergence and learning rates for the error ‖𝑓_𝐳−𝑓_𝜏,𝜌‖_𝐿^𝑟_𝜌𝑋 in the space 𝐿^𝑟_𝜌𝑋. Here 𝑟>0 depends on the pair (𝜌,𝜏) which will be decided in Section 2 and 𝜌_𝑋 is the marginal distribution of 𝜌 on 𝑋. In the rest of this paper, we assume 𝑌=[−1,1] which in turn leads to values of the target function 𝑓_𝜏,𝜌 lie in the same interval.

2. Noise Condition and Main Results

There has been a large literature in learning theory (see [6] and references therein) devoted to the least square regression. It aims at learning the regression function 𝑓_𝜌(𝑥)=∫_𝑌𝑦𝑑𝜌(𝑦∣𝑥). The identity ℰ_ls(𝑓)−ℰ_ls(𝑓_𝜌)=‖𝑓−𝑓_𝜌‖²_{𝐿2𝜌𝑋} for the generalization error ℰ_ls(𝑓)=∫_𝑍(𝑦−𝑓(𝑥))²𝑑𝜌 leads to a variance-expectation bound with the form of 𝐄𝜉²≤4𝐄𝜉, where 𝜉=(𝑦−𝑓(𝑥))²−(𝑦−𝑓_𝜌(𝑥))² on (𝑍,𝜌). It plays an essential role in error analysis of kernel based regularized schemes.

However, this identity relation and expectation-variance bound fail in the setting of the quantile regression. The reason is that the pinball loss is lack of strong convexity. If we add some noise condition on distribution 𝜌 named 𝜏-quantile of 𝑝-average type 𝑞 (see Definition 1), we can also get a similar identity relation which in turn enables us to have a variance-expectation bound stated in the following which is proved by Steinwart and Christman [1].

Definition 1. Let 𝑝∈(0,∞] and 𝑞∈[1,∞). A distribution 𝜌 on 𝑋×[−1,1] is said to have a 𝜏-quantile of 𝑝-average type 𝑞 if for 𝜌_𝑋-almost every 𝑥∈𝑋, there exist a 𝜏-quantile 𝑡^∗∈ℝ and constants 𝛼_{𝜌(⋅∣𝑥)}∈(0,2],𝑏_{𝜌(⋅∣𝑥)}>0 such that for each 𝑠∈[0,𝛼_{𝜌(⋅∣𝑥)}], 𝜌𝑡^∗−𝑠,𝑡^∗∣𝑥≥𝑏_{𝜌(⋅∣𝑥)}𝑠^𝑞−1,𝜌𝑡^∗,𝑡^∗+𝑠∣𝑥≥𝑏_{𝜌(⋅∣𝑥)}𝑠^𝑞−1, (5) and that the function 𝛾 on 𝑋 defined by 𝛾(𝑥)=𝑏_{𝜌(⋅∣𝑥)}𝛼^𝑞−1_{𝜌(⋅∣𝑥)} satisfies 𝛾⁻¹∈𝐿^𝑝_𝜌𝑋.

We also need capacity of the hypothesis space ℋ for our learning rates. Here in this paper, we measure the capacity by empirical covering numbers.

Definition 2. Let (𝔐,𝑑) be a pseudometric space and 𝑆 be a subset of 𝔐. For every 𝜀>0, the covering number 𝒩(𝑆,𝜀,𝑑) of 𝑆 with respect to 𝜀 and 𝑑 is defined as the minimal number of balls of radius 𝜀 whose union covers 𝑆, that is, 𝒩(𝑆,𝜀,𝑑)=minℓ∈ℕ∶𝑆⊂∪^ℓ_𝑗=1𝐵𝑠_𝑗,𝜀forsome𝑠_𝑗^ℓ_𝑗=1⊂𝔐, (6) where 𝐵(𝑠_𝑗,𝜀)={𝑠∈𝔐∶𝑑(𝑠,𝑠_𝑗)≤𝜀} is a ball in 𝔐.

Definition 3. Let ℱ be a set of functions on 𝑋, 𝐱=(𝑥_𝑖)^𝑘_𝑖=1⊂𝑋^𝑘 and ℱ|_𝐱={(𝑓(𝑥_𝑖))^𝑘_𝑖=1∶𝑓∈ℱ}⊂ℝ^𝑘. Set 𝒩_2,𝐱(ℱ,𝜀)=𝒩(ℱ|_𝐱,𝜀,𝑑₂). The ℓ²-empirical covering number of ℱ is defined by 𝒩₂(ℱ,𝜀)=sup_𝑘∈ℕsup_{𝐱∈𝑋𝑘}𝒩_2,𝐱(ℱ,𝜀),𝜀>0. (7) Here 𝑑₂ is the normalized ℓ²-metric on the Euclidean space ℝ^𝑘 given by 𝑑₂()=1𝐚,𝐛𝑘^𝑘_𝑖=1𝑎_𝑖−𝑏_𝑖^21/2for𝐚=𝑎_𝑖^𝑘_𝑖=1,𝐛=𝑏_𝑖^𝑘_𝑖=1∈ℝ^𝑘. (8)

Assumption. Assume that the empirical covering number of the hypothesis space ℋ is bounded for some 𝑎>0 and 𝜄∈(0,2), log𝒩₂(ℋ,𝜀)≤𝑎𝜀^−𝜄,∀𝜀>0. (9)

Theorem 4. Assume that 𝜌 satisfies (5) with some 𝑝∈(0,∞] and 𝑞∈[1,∞). Denote 𝑟=𝑝𝑞/(𝑝+1). One further assumes that 𝑓_𝜏,𝜌 is uniquely defined. If 𝑓_𝜏,𝜌∈ℋ and ℋ satisfies (9) with 𝜄∈(0,2), then for any 0<𝛿<1, with confidence 1−𝛿, one has 𝑓_𝐳−𝑓_{𝜏,𝜌𝐿𝑟𝜌𝑋}≤𝐶𝑚^−𝜗2log𝛿, (10) where 2𝜗=𝑝+14𝑞𝑝+1−(2−𝜄)min2𝑝+1,𝑝𝑞 (11) and 𝐶 is a constant independent of 𝑚 and 𝛿.

Remark 5. In the ERM scheme, we can choose 𝑓_𝜏,𝜌∈ℋ which in turn makes the approximation error described by (23) equal to zero. However, it is impossible for the kernel based regularized scheme because of the appearance of the penalty term 𝜆‖𝑓‖²_𝐾.
If 𝑞≤2, all conditional distributions around the quantile behave similar to the uniform distribution. In this case 𝑟=𝑝𝑞/(𝑝+1)≤2 and 𝜃=min{2/𝑞,𝑝/(𝑝+1)}=𝑝/(𝑝+1) for all 𝑝>0. Hence, 𝜗=2(𝑝+1)/((2+𝜄)𝑝𝑞+4𝑞). Furthermore, when 𝑝 is large enough, the parameter 𝑟 tends to 𝑞 and the power index for the above learning rate arbitrarily approaches to 2/(2+𝜄)𝑞 which shows that the learning rate power index for ‖𝑓_𝐳−𝑓_𝜏,𝜌‖^𝑞_{𝐿𝑞𝜌𝑋} is arbitrarily close to 2/(2+𝜄) independent of 𝑞. In particular, 𝜄 can be arbitrarily small when ℋ is smooth enough. In this case, the power index of the learning rates 2/(2+𝜄) can be arbitrarily close to 1 which is the optimal learning rate for the least square regression.

Let us take some examples to demonstrate the above main result.

Example 6. Let ℋ be a unit ball of the sobolev space 𝐻^𝑠 with 𝑠>0. Observe that the empirical covering number is bounded above by the uniform covering number defined in Definition 2. Hence we have (see [6, 7]) log𝒩₂(ℋ,𝜀)≤𝐶_𝑠1𝜀^𝑛/𝑠, (12) where 𝑛 is the dimension of the input space 𝑋 and 𝐶_𝑠>0.
Under the same assumptions as Theorem 4, we get that by replacing 𝜄 by 𝑛/𝑠, for any 0<𝛿<1, with confidence 1−𝛿, 𝑓_𝐳−𝑓_{𝜏,𝜌𝐿𝑟𝜌𝑋}≤𝐶𝑚^−𝜗2log𝛿, (13) where 2𝜗=𝑝+14𝑞𝑝+1−(2−𝑛/𝑠)min2𝑝+1,𝑝𝑞 (14) and 𝐶 is a constant independent of 𝑚 and 𝛿.
We carry out the same discussions on the case of 𝑞≤2 and large enough 𝑝 as Remark 5. Therefore the power index of the learning rates for ‖𝑓_𝐳−𝑓_𝜏,𝜌‖^𝑞_{𝐿𝑞𝜌𝑋} is arbitrarily close to 2/(2+𝑛/𝑠) independent of 𝑞. Furthermore, 𝑠 can be arbitrarily large if the Sobolev space is smooth enough. In this special case, the learning rate power index arbitrarily approaches to 1.

Example 7. Let ℋ be a unit ball of the reproducing kernel Hilbert space ℋ_𝜎 generated by a Gaussian kernel (see [5]). Reference [7] tells us that log𝒩(ℋ,𝜀)≤𝐶_𝑛,𝜎1log𝜀^𝑛+1,∀𝜀>0, (15) where 𝐶_𝑛,𝜎>0 depends only on 𝑛 and 𝜎>0. Obviously, the right-hand side of (15) is bounded by 𝐶_𝑛,𝜎(1/𝜀)^𝑛+1.
So from Theorem 4, we can get different learning rates with power index 2𝜗=𝑝+14𝑞𝑝+1−(1−𝑛)min2𝑝+1,𝑝𝑞. (16) If 𝑞≤2 and 𝑝 is large enough, the power index of the learning rates for ‖𝑓_𝐳−𝑓_𝜏,𝜌‖^𝑞_{𝐿𝑞𝜌𝑋} is arbitrarily close to 2/(3+𝑛) which is very slow if 𝑛 is large. However, in most data sets the data are concentrated on a much lower dimensional manifold embedded in the high dimensional space. In this setting an analysis that replaces 𝑛 by the intrinsic dimension of the manifold would be of great interest (see [8] and references therein).

3. Error Analysis

Define the noise-free error called generalization error associated with the pinball loss 𝐿_𝜏 as ℰ_𝜏𝑓=_𝑍𝐿_𝜏𝑦−𝑓(𝑥)𝑑𝜌for𝑓∶𝑋→[−1,1]. (17) Then the measurable function 𝑓_𝜏,𝜌 is a minimizer of ℰ_𝜏. Obviously, 𝑓_𝜏,𝜌(𝑥)∈[−1,1].

We need the following results from [1] for our error analysis.

Proposition 8. Let 𝐿_𝜏 be the pinball loss. Assume that 𝜌 satisfies (5) with some 𝑝∈(0,∞] and 𝑞∈[1,∞). Then for all 𝑓∶𝑋→[−1,1] one has 𝑓−𝑓_{𝜏,𝜌𝐿𝑟𝜌𝑋}≤2^1−1/𝑞𝑞^1/𝑞𝛾⁻¹_𝐿^1/𝑞_𝑝𝜌𝑋ℰ_𝜏𝑓−ℰ_𝜏𝑓_𝜏,𝜌^1/𝑞. (18) Furthermore, with 2𝜃=min𝑞,𝑝∈(],𝐶𝑝+10,1_𝜃=2^2−𝜃𝑞^𝜃𝛾^−1𝜃_{𝐿𝑝𝜌𝑋}>0, (19) one has 𝐄𝐿_𝜏𝑦−𝑓(𝑥)−𝐿_𝜏𝑦−𝑓_𝜏,𝜌(𝑥)²≤𝐶_𝜃ℰ_𝜏(𝑓)−ℰ_𝜏(𝑓_𝜏,𝜌)^𝜃,∀𝑓∶𝑋→𝑌. (20)

The above result implies that we can get convergence rates of 𝑓_𝐳 in the space 𝐿^𝑟_𝜌𝑋 by bounding the excess generalization error ℰ_𝜏(𝑓_𝐳)−ℰ_𝜏(𝑓_𝜏,𝜌).

To bound ℰ_𝜏(𝑓_𝐳)−ℰ_𝜏(𝑓_𝜏,𝜌), we need a standard error decomposition procedure [6] and a concentration inequality.

3.1. Error Decomposition

Define the empirical error associated with the pinball loss 𝐿_𝜏 as ℰ_𝐳,𝜏𝑓=1𝑚^𝑚_𝑖=1𝐿_𝜏𝑦_𝑖−𝑓𝑥_𝑖for𝑓∶𝑋→[−1,1]. (21) Define 𝑓_ℋ=argmin_𝑓∈ℋℰ_𝜏𝑓=argmin_𝑓∈ℋℰ_𝜏𝑓−ℰ_𝜏𝑓_𝜏,𝜌. (22)

Lemma 9. Let 𝐿_𝜏 be the pinball loss, 𝑓_𝐳 be defined by (3) and 𝑓_ℋ∈ℋ by (22). Then ℰ_𝜏𝑓_𝐳−ℰ_𝜏𝑓_𝜏,𝜌≤ℰ_𝜏𝑓_𝐳−ℰ_𝜏𝑓_𝜏,𝜌−ℰ_𝐳,𝜏𝑓_𝐳−ℰ_𝐳,𝜏𝑓_𝜏,𝜌+ℰ_𝐳,𝜏𝑓_ℋ−ℰ_𝐳,𝜏𝑓_𝜏,𝜌−ℰ_𝜏𝑓_ℋ−ℰ_𝜏𝑓_𝜏,𝜌+ℰ_𝜏𝑓_ℋ−ℰ_𝜏𝑓_𝜏,𝜌. (23)

Proof. The excess generalization error can be written as ℰ_𝜏𝑓_𝐳−ℰ_𝜏𝑓_𝜏,𝜌=ℰ_𝜏𝑓_𝐳−ℰ_𝐳,𝜏𝑓_𝐳+ℰ_𝐳,𝜏𝑓_𝐳−ℰ_𝐳,𝜏𝑓_ℋ+ℰ_𝐳,𝜏𝑓_ℋ−ℰ_𝜏𝑓_ℋ+ℰ_𝜏𝑓_ℋ−ℰ_𝜏𝑓_𝜏,𝜌. (24) The definition of 𝑓_𝐳 implies that ℰ_𝐳,𝜏(𝑓_𝐳)−ℰ_𝐳,𝜏(𝑓_ℋ)≤0. Furthermore, by subtracting and adding ℰ_𝜏(𝑓_𝜏,𝜌) and ℰ_𝐳,𝜏(𝑓_𝜏,𝜌) in the first term and third term, we see that Lemma 9 holds true.

We call the term (23) approximation error. It has been studied in [9].

3.2. Concentration Inequality and Sample Error

Let us recall the one-sided Bernstein inequality as follows.

Lemma 10. Let 𝜉 be a random variable on a probability space 𝑍 with variance 𝜎² satisfying |𝜉−𝔼(𝜉)|≤𝑀_𝜉 for some constant 𝑀_𝜉. Then for any 0<𝛿<1, with confidence 1−𝛿, one has 1𝑚^𝑚_𝑖=1𝜉𝑧_𝑖−𝔼(𝜉)≤2𝑀_𝜉log(1/𝛿)+3𝑚2𝜎²log(1/𝛿)𝑚. (25)

Proposition 11. Let 𝑓_ℋ∈ℋ. Assume that 𝜌 on 𝑋×[−1,1] satisfies the variance bound (20) with index 𝜃 indicated in (19). For any 0<𝛿<1, with confidence 1−𝛿/2, (3.6) can be bounded as ℰ_𝐳,𝜏𝑓_ℋ−ℰ_𝐳,𝜏𝑓_𝜏,𝜌−ℰ_𝜏𝑓_ℋ−ℰ_𝜏𝑓_𝜏,𝜌≤4log(2/𝛿)+3𝑚2𝐶_𝜃log(2/𝛿)𝑚^1/(2−𝜃)+ℰ_𝜏𝑓_ℋ−ℰ_𝜏𝑓_𝜏,𝜌. (26)

Proof. Let 𝜉(𝑧)=𝐿_𝜏(𝑦−𝑓_ℋ(𝑥))−𝐿_𝜏(𝑦−𝑓_𝜏,𝜌(𝑥)) which satisfies |𝜉|≤2 and in turn |𝜉−𝐄(𝜉)|≤2. The variance bound (20) implies that 𝐄(𝜉−𝐄(𝜉))²≤𝐄𝜉²≤𝐶_𝜃ℰ_𝜏(𝑓_ℋ)−ℰ_𝜏𝑓_𝜏,𝜌^𝜃. (27) Using (25) on the random variable 𝜉(𝑧)=𝐿_𝜏(𝑦−𝑓_ℋ(𝑥))−𝐿_𝜏(𝑦−𝑓_𝜏,𝜌(𝑥)), we can get the desired bound (28) with the help of Young’s inequality.

Let us turn to estimate the sample error (3.5) involving the function 𝑓_𝐳 which runs over a set of functions since 𝐳 is a random sample itself. To estimate it, we use a concentration inequality below involving empirical covering numbers [10–12].

Lemma 12. Let ℱ be a class of measurable functions on 𝑍. Assume that there are constants 𝐵,𝑐>0 and 𝛼∈[0,1] and ‖𝑓‖_∞≤𝐵 and 𝔼𝑓²≤𝑐(𝔼𝑓)^𝛼 for every 𝑓∈ℱ. If (7) holds, then there exists a constant 𝑐^′_𝜄 depending only on 𝜄 such that for any 𝑡>0, with probability at least 1−𝑒^−𝑡, there holds 1𝔼𝑓−𝑚^𝑚_𝑖=1𝑓𝑧_𝑖≤12𝜂^1−𝛼𝔼𝑓^𝛼+𝑐^′_𝜄𝜂+2𝑐𝑡𝑚^1/(2−𝛼)+18𝐵𝑡𝑚,∀𝑓∈ℱ, (28) where 𝜂∶=max𝑐^{(2−𝜄)/(4−2𝛼+𝜄𝛼)}𝑎𝑚^{2/(4−2𝛼+𝜄𝛼)},𝐵^{(2−𝜄)/(2+𝜄)}𝑎𝑚^2/(2+𝜄). (29)

We apply Lemma 12 to a function set ℱ, where ℱ=𝐿_𝜏𝑦−𝑓(𝑥)−𝐿_𝜏𝑦−𝑓_𝜏,𝜌(𝑥)∶𝑓∈ℋ. (30)

Proposition 13. Assume 𝜌 on 𝑋×[−1,1] satisfies the variance bound (20) with index 𝜃 indicated in (19). If ℋ satisfies (9) with 𝜄∈(0,2), then for any 0<𝛿<1, with confidence 1−𝛿/2, one has ℰ_𝜏𝑓−ℰ_𝜏𝑓_𝜏,𝜌−ℰ_𝐳,𝜏𝑓−ℰ_𝐳,𝜏𝑓_𝜏,𝜌≤12ℰ_𝜏𝑓−ℰ_𝜏𝑓_𝜏,𝜌+𝐶_𝜄,𝜃2log𝛿1𝑚^{2/(4−2𝜃+𝜄𝜃)},∀𝑓∈ℋ, (31) where 𝐶_𝜄,𝜃=12+𝑐^′_𝜄max𝐶_𝜃^{(2−𝜄)/(4−2𝜃+𝜄𝜃)}𝑎^{2/(4−2𝜃+𝜄𝜃)},2^{(2−𝜄)/(2+𝜄)}𝑎^2/(2+𝜄)+2𝐶_𝜃^1/(2−𝜃)+36. (32)

Proof. Take 𝑔∈ℱ with the form 𝑔(𝑧)=𝐿_𝜏(𝑦−𝑓(𝑥))−𝐿_𝜏(𝑦−𝑓_𝜏,𝜌(𝑥)) where 𝑓∈ℋ. Hence 𝔼𝑔=ℰ_𝜏(𝑓)−ℰ_𝜏(𝑓_𝜏,𝜌) and (1/𝑚)∑^𝑚_𝑖=1𝑔(𝑧_𝑖)=ℰ_𝐳,𝜏(𝑓)−ℰ_𝐳,𝜏(𝑓_𝜏,𝜌).
The Lipschitz property of the pinball loss 𝐿_𝜏 implies that 𝑔(𝑧)≤𝑓(𝑥)−𝑓_𝜏,𝜌(𝑥)≤2. (33) For 𝑔₁,𝑔₂∈ℱ, we have 𝑔₁(𝑧)−𝑔₂(𝑧=𝐿_𝜏)𝑦−𝑓₁(𝑥)−𝐿_𝜏𝑦−𝑓₂(𝑥)≤𝑓₁(𝑥)−𝑓₂(𝑥), (34) where 𝑓₁,𝑓₂∈ℋ. It follows that 𝒩_2,𝐳(ℱ,𝜀)≤𝒩_2,𝐱(ℋ,𝜀). (35) Hence log𝒩₂(ℱ,𝜀)≤𝑎𝜀^−𝜄. (36)
Applying Lemma 12 with 𝐵=2,𝛼=𝜃, and 𝑐=𝐶_𝜃, we know that for any 0<𝛿<1, with confidence 1−𝛿/2, there holds 1𝔼𝑓−𝑚^𝑚_𝑖=1𝑔𝑧_𝑖≤12𝜂^1−𝜃𝔼𝑔^𝜃+𝑐^′_𝜄𝐶𝜂+2^𝜃log(2/𝛿)𝑚^1/(2−𝜃)+36log(2/𝛿)𝑚≤121𝔼𝑔+2+𝑐^′_𝜄𝐶𝜂+2^𝜃log(2/𝛿)𝑚^1/(2−𝜃)+36log(2/𝛿)𝑚. (37) Here 𝜂≤max𝐶_𝜃^{(2−𝜄)/(4−2𝜃+𝜄𝜃)}𝑎^{2/(4−2𝜃+𝜄𝜃)},2^{(2−𝜄)/(2+𝜄)}𝑎^2/(2+𝜄)×1𝑚^{2/(4−2𝜃+𝜄𝜃)}. (38) Note that 12+𝑐^′_𝜄𝐶𝜂+2^𝜃log(2/𝛿)𝑚^1/(2−𝜃)+36log(2/𝛿)𝑚≤𝐶_𝜄,𝜃2log𝛿1𝑚^{2/(4−2𝜃+𝜄𝜃)}, (39) where 𝐶_𝜄,𝜃 is indicated in (32). Then our desired bound holds true.

Proposition 14. Assume 𝜌 on 𝑋×[−1,1] satisfies the variance bound (20) with index 𝜃 indicated in (19). If ℋ satisfies (9) with 𝜄∈(0,2) Then for any 0<𝛿<1, with confidence 1−𝛿, there holds ℰ_𝜏𝑓_𝐳−ℰ_𝜏𝑓_𝜏,𝜌≤2ℰ_𝜏𝑓_ℋ−ℰ_𝜏𝑓_𝜏,𝜌+8log(2/𝛿)3𝑚+22𝐶_𝜃log(2/𝛿)𝑚^1/(2−𝜃)+2𝐶_𝜄,𝜃2log𝛿1𝑚^{2/(4−2𝜃+𝜄𝜃)}. (40)

The above bound follows directly from Propositions 11 and 13 with the fact that 𝑓_𝐳∈ℋ.

3.3. Bounding the Total Error

Now we are in a position to present our general result on error analysis for algorithm (3).

Theorem 15. Assume that 𝜌 satisfies (5) with some 𝑝∈(0,∞] and 𝑞∈[1,∞). Denote 𝛾=𝑝𝑞/(𝑝+1). Further assume that ℋ satisfies (9) with 𝜄∈(0,2) and 𝑓_𝜏,𝜌 is uniquely defined. Then for any 0<𝛿<1, with confidence 1−𝛿, one has 𝑓_𝐳−𝑓_{𝜏,𝜌𝐿𝑟𝜌𝑋}≤𝐶inf_𝑓∈ℋℰ_𝜏𝑓−ℰ_𝜏𝑓_𝜏,𝜌^1/𝑞+𝐶𝑚^−𝜗2log𝛿, (41) where 2𝜗=𝑝+14𝑞𝑝+1−(2−𝜄)min2𝑝+1,𝑝𝑞 (42) and 𝐶 are constant independent of 𝑚 and 𝛿.

Proof. Combining (18), (19), and (40), with confidence 1−𝛿, we have 𝑓_𝐳−𝑓_{𝜏,𝜌𝐿𝑟𝜌𝑋}≤𝐶ℰ_𝜏𝑓_ℋ−ℰ_𝜏𝑓_𝜏,𝜌^1/𝑞+2𝐶log𝛿1𝑚^{2/𝑞(4−2𝜃+𝜄𝜃)}≤𝐶inf_𝑓∈ℋℰ_𝜏𝑓−ℰ_𝜏𝑓_𝜏,𝜌^1/𝑞+𝐶𝑚^−𝜗2log𝛿, (43) where 2𝜗=𝑝+1−()2,4𝑞𝑝+12−𝜄min𝑝+1,𝑝𝑞𝐶=2𝑞^1/𝑞𝛾⁻¹_𝐿^1/𝑞_𝑝𝜌𝑋43+2𝐶_𝜃^1/(2−𝜃)+𝐶_𝜄,𝜃^1/𝑞. (44)

Proof of Theorem 4. The assumption 𝑓_𝜏,𝜌∈ℋ implies that inf_𝑓∈ℋℰ_𝜏𝑓−ℰ_𝜏𝑓_𝜏,𝜌=0. (45)
Therefore, our desired result comes directly from Theorem 15.

4. Further Discussions

In this paper, we studied ERM algorithm (3) for quantile regression and provide convergence and learning rates. We showed some essential differences between ERM scheme and kernel based regularized scheme for quantile regression. We also point out the difficulty to deal with quantile regression: the lack of strong convexity of the pinball loss. To overcome this difficulty, some noise condition on 𝜌 is proposed to enable us to get a variance-expectation bound similar to the one for the least square regression.

In our analysis we just consider 𝑓∈ℋ and ‖𝑓‖_𝐶(𝑋)≤1. The case ‖𝑓‖_𝐶(𝑋)≤𝑅 for 𝑅≥1 would be interesting in the future work. The approximation error involving 𝑅 can be estimated by the knowledge of interpolation space.

In our setting, the sample is drawn independently from the distribution 𝜌. However, in many practical problems, the i.i.d condition is a little demanding, so it would be interesting to investigate the ERM scheme for quantile regression with nonidentical distributions [13, 14] or dependent sampling [15].

Acknowledgment

This work described in this paper is supported by NSF of China under Grant 11001247 and 61170109.