Regularized Ranking with Convex Losses and ℓ¹-Penalty

Abstract

In the ranking problem, one has to compare two different observations and decide the ordering between them. It has received increasing attention both in the statistical and machine learning literature. This paper considers ℓ¹-regularized ranking rules with convex loss. Under some mild conditions, a learning rate is established.

1. Introduction

In the ranking problem, one has to compare two different observations and decide the ordering between them. The problem of ranking has become an interesting field for researchers in machine learning community. It has received increasing attention both in the statistical and machine learning literature.

The problem of ranking may be modeled in the framework of statistical learning (see [1, 2]). Let (𝑋,𝑌) be a pair of random variables taking values in 𝒳×ℝ. The random observation 𝑋 models some object and 𝑌 denotes its real-valued label. Let (𝑋^′,𝑌^′) denote a pair of random variables identically distributed with (𝑋,𝑌) (with respect to the probability ℙ) and independent of it. In the ranking problem one observes 𝑋 and 𝑋^′ but not their labels 𝑌 and 𝑌^′. 𝑋 is “better” than 𝑋^′ if 𝑌>𝑌^′. We are to construct a measurable function 𝑓∶𝒳×𝒳→ℝ, called a ranking rule, which predicts the ordering between objects in the following way: if 𝑓(𝑋,𝑋^′)≥0, we predict that 𝑋 is better than 𝑋^′. A ranking rule 𝑓 has the property 𝑓(𝑥,𝑥^′)𝑓(𝑥^′,𝑥)≤0. The performance of a ranking rule 𝑓 is measured by the ranking error: 𝐿𝑓=ℙsign𝑌−𝑌^′𝑓𝑋,𝑋^′<0, (1) that is, the probability that 𝑓 ranks two randomly drawn instances incorrectly. It is easily seen that 𝐿(𝑓) attains its minimum 𝐿^∗, over the class of all measurable functions, at the ranking rule 𝑓^∗𝑥,𝑥^′∶=sgn2𝜂𝑥,𝑥^′−1with𝜂𝑥,𝑥^′=ℙ𝑌>𝑌^′∣𝑋=𝑥,𝑋^′=𝑥^′. (2)

In practice, the best rule 𝑓^∗ is unknown since the probability ℙ is unknown. A widely used approach for estimating 𝐿^∗ is the empirical risk minimization with convex loss.

Definition 1. one says that 𝜙∶ℝ→[0,∞) is a ranking loss (function) if it is convex, differentiable at 0 with 𝜙^′(0)<0, and the smallest zero of 𝜙 is 1.

Examples of ranking loss include the least square loss 𝜙(𝑡)=(1−𝑡)² and 𝑞-norm SVM loss 𝜙_𝑞(𝑡)=(1−𝑡)^𝑞₊, where 𝑞≥1 and 𝑡₊=max{0,𝑡} for 𝑡∈ℝ.

The risk of a measurable function 𝑓 is defined as 𝑄(𝑓)=𝔼[𝜙(sign(𝑌−𝑌^′)𝑓(𝑋,𝑋^′))]. Denote by 𝑓_𝜙 a minimizer of 𝑄(𝑓) over the set of all measurable and antisymmetric functions. For example, as in the classification case (see [3, 4]), 𝑓_𝜙1=𝑓^∗, and for 𝑞>1,𝑓_𝜙𝑞𝑥,𝑥^′=1+𝑓^∗𝑥,𝑥^{′1/(𝑞−1)}−1−𝑓^∗𝑥,𝑥^{′1/(𝑞−1)}1+𝑓^∗𝑥,𝑥^{′1/(𝑞−1)}+1−𝑓^∗𝑥,𝑥^{′1/(𝑞−1)},𝑥,𝑥^′∈𝒳. (3)

The following inequality holds for any 𝑓: 𝐿𝑓−𝐿^∗≤𝑄𝑓−𝑄^∗,if𝜙(𝑡)=(1−𝑡)₊,𝑐𝑄𝑓−𝑄^∗,if𝜙^′′(0)≥0, (4) where 𝑄^∗=𝑄(𝑓_𝜙) and 𝑐 is some constant.

Before proceeding further, we introduce the notion of Reproducing Kernel Hilbert Space (RKHS). Recall that a continuous function 𝐾(𝜎,𝜎^′) is a Mercer kernel on a set Σ, if 𝐾(𝜎,𝜎^′)=𝐾(𝜎^′,𝜎),∀𝜎,𝜎^′∈Σ, and given an arbitrary finite set {𝜎₁,…,𝜎_𝑛}⊂Σ, the matrix 𝐊=(𝐾(𝜎_𝑖,𝜎_𝑗))^𝑛_𝑖,𝑗=1 is positive semidefinite. The RKHS ℋ_𝐾 associated with the Mercer kernel 𝐾 is the completion of span{𝐾_𝜎=𝐾(𝜎,⋅)∣𝜎∈Σ}, with respect to the inner product given by ⟨𝐾_𝜎,𝐾_𝜎′⟩_𝐾=𝐾(𝜎,𝜎^′). See [5] and ([6, Ch. 4]) for details.

For convenience, we assume hereafter that the Mercer kernels 𝐾 on 𝒳²×𝒳² are symmetric in the sense that 𝐾𝑢,𝑢^′,𝑥,𝑥^′=𝐾𝑢^′,𝑢,𝑥^′,𝑥,∀𝑢,𝑢^′,𝑥,𝑥^′∈𝒳. (5) Such examples are Mercer kernels 𝐾 of either form 𝐾(𝑠,𝑡)=𝑘(|𝑠−𝑡|₂) or 𝐾(𝑠,𝑡)=𝑘(⟨𝑠,𝑡⟩),𝑠,𝑡∈𝒳², where |⋅|₂ and ⟨⋅,⋅⟩ are the Euclidean norm and inner product, respectively.

Since the best ranking rule 𝑓^∗ is antisymmetric, it is reasonable that we restricted ourselves to the subspace ℋ^as of anti-symmetric functions in ℋ_𝐾; that is, ℋ^as_𝐾=𝑓∣𝑓∈ℋ_𝐾,𝑓𝑥,𝑥^′=−𝑓𝑥^′,𝑥,∀𝑥,𝑥^′∈𝒳. (6) For any 𝑥_𝑖∈𝒳,𝑖=1,…,𝑛, and 𝛼_𝑖𝑗∈ℝ with 𝛼_𝑖𝑗=−𝛼_𝑗𝑖,𝑖,𝑗=1,…,𝑛, it is easily seen that 𝑓=^𝑛_𝑖,𝑗=1𝛼_𝑖𝑗𝐾_{(𝑥𝑖,𝑥𝑗)}∈ℋ^as_𝐾. (7) Conversely, any anti-symmetric function 𝑓∈span{𝐾_{(𝑥𝑖,𝑥𝑗)}}^𝑛_𝑖,𝑗=1 with above expression should satisfy 𝛼_𝑖𝑗=−𝛼_𝑗𝑖,𝑖,𝑗=1,…,𝑛, provided det𝐊>0.

For a set of samples 𝐳={𝑍₁,…,𝑍_𝑛}⊂𝒳², let Ω_𝑧𝑓=inf^𝑛_𝑖,𝑗=1𝛼_𝑖𝑗∣𝑓=^𝑛_𝑖,𝑗=1𝛼_𝑖𝑗𝐾_{(𝑥𝑖,𝑥𝑗)};ℋ_𝐾,𝑧=^𝑛_𝑖,𝑗=1𝛼_𝑖𝑗𝐾_{(𝑥𝑖,𝑥𝑗)},ℋ^as_𝐾,𝑧=^𝑛_𝑖,𝑗=1𝛼_𝑖𝑗𝐾_{(𝑥𝑖,𝑥𝑗)}∣𝛼_𝑖𝑗=−𝛼_𝑗𝑖. (8)

For 𝜆>0, the ℓ¹-penalty regularized ranking rule 𝑓_𝑧,𝜆 is the minimizer of the minimization problem 𝑓_𝑧,𝜆=argmin_{𝑓∈ℋas𝐾,𝑧}𝑄_𝑛𝑓+𝜆Ω_𝑧𝑓, (9) where 𝑄_𝑛(𝑓), known as empirical risk, is given by 𝑄_𝑛𝑓=1𝑛(𝑛−1)_𝑖≠𝑗𝜙sign𝑦_𝑖−𝑦_𝑗𝑓𝑥_𝑖,𝑥_𝑗. (10)

Associated with any ranking rule 𝑓, we construct another ranking rule as follows: 𝜋𝑓𝑥,𝑥^′=sign𝑓𝑥,𝑥^′,if𝑓𝑥,𝑥^′𝑓>1,𝑥,𝑥^′,if𝑓𝑥,𝑥^′≤1. (11) Clearly, 𝜋(𝑓) gives the same ranking rule as 𝑓 and it satisfies 𝑄𝜋𝑓≤𝑄𝑓,𝑄_𝑛𝜋𝑓≤𝑄_𝑛𝑓. (12)

Hereafter, we denote 𝑔_𝑛=𝜋(𝑓_𝑧,𝜆). The goal of this paper is to bound the excess error 𝑄(𝑔_𝑛)−𝑄^∗, which in turn together with (4) up bounds the excess ranking error 𝐿(𝑔_𝑛)−𝐿^∗. The main result of this paper is, under mild conditions, to establish a learning rate for ℓ¹-penalty regularized ranking rules with convex loss.

Classification with convex loss, in particular for 𝑞-norm SVMs, has been the subject of many theoretical considerations in recent years. The 1-norm SVMs with regularizer being RKHS norm for ranking was investigated in [2, 7]. The ℓ¹-penalty has been used in [8, 9] for classification problems under the framework of SVMs. It is well known that ℓ¹-regularization usually leads to solution with sparse representation (see, e.g., [10–13]). In this paper, we consider ranking with convex loss and ℓ¹-penalty.

In [2], the RKHS-norm SVMs for ranking was proposed. But it was implemented over a ball 𝐵_𝑅={𝑓∈ℋ∣|‖𝑓‖_𝐾≤𝑅} of ℋ_𝐾, not the whole RKHS ℋ_𝐾; that is, it solves the minimization problem 𝑓_𝑛=argmin_{𝑓∈𝐵𝑅}𝑄_𝑛𝑓+𝜆𝑓²_𝐾. (13) A convergence rate for 𝑄(𝑓_𝑛)−inf_{𝑓∈𝐵𝑅}𝑄(𝑓) has been established for Gaussian kernel. The approximation error inf_{𝑓∈𝐵𝑅}𝑄(𝑓)−𝑄^∗ was not considered there. The asymptotic behavior of the same algorithm implemented over the whole RKHS ℋ_𝐾 was investigated in [7]. Moreover, a fast learning rate 𝑂(1/𝑛) is obtained under some conditions.

We would like to mention a recent paper [14], where the error ℰ(𝑓) of function 𝑓 is defined by ℰ(𝑓)=𝔼((𝑌−𝑌^′−𝑓(𝑋)+𝑓(𝑋^′))²). A convergence rate for the minimizer of the regularized empirical error was established. The author made use of the technique of estimation via integral operator developed in [15].

The rest of the paper is organized as follows. In Section 2, after making some assumptions, we state the main result, an upper bound for 𝑄(𝑔_𝑛)−𝑄^∗+𝜆Ω_𝑧(𝑓_𝑧,𝜆). As usual, it is decomposed as a sum of three terms, sample error, hypothesis error, and approximation error. Sections 3 and 4 are devoted to the estimations of hypothesis error and sample error, respectively. A proof of the main result is given in Section 5.

2. Assumptions and Main Results

For the statement of the main results, we need to introduce some notions and make some assumptions.

Denote 𝑔_𝑓=𝜙(sign(𝑦−𝑦^′)𝑓(𝑥,𝑥^′))−𝜙(sign(𝑦−𝑦^′)𝑓_𝜙(𝑥,𝑥^′)). The following assumption is a bound for variance of 𝑔_𝑓, which is adopted by many authors.

Assumption 2. There is a constant 𝛼∈[0,1] such that for any 𝑀, 𝔼𝑔²_𝑓≤𝑐₁𝔼𝑔_𝑓^𝛼,∀𝑓∶𝒳²⟶[−𝑀,𝑀], (14) where 𝑐₁=𝑐₁(𝑀) is a constant.

For 𝜙_𝑞(𝑡)=(1−𝑡)^𝑞₊, the assumption is satisfied [16] for 𝛼=1,if21<𝑞≤2,𝑞,if2<𝑞. (15)

It is known in [17] that if there is some positive constant 𝐶 such that ℙ2𝜂𝑋,𝑋^′−1≤𝜉≤𝐶𝜉^{𝛼/(1−𝛼)},∀𝜉>0, (16) then the assumption is satisfied for 𝜙₁(𝑡)=(1−𝑡)₊.

Suppose hereafter 𝜅∶=sup_𝑠∈𝒳²𝐾(𝑠,𝑠)<∞. We note that, for any Mercer kernel 𝐾, sup_𝑠∈𝒳2𝐾(𝑠,𝑠)=sup_{𝑠,𝑠′∈𝒳2}𝐾𝑠,𝑠^′. (17) We now construct a set of functions which contains ℋ_𝐾,𝑧 and is independent of the samples.

Definition 3. The Banach space ℋ is defined as the function set on 𝒳² containing all functions of the form 𝑓=^∞_𝑖=1𝑎_𝑖𝐾_𝑠𝑖,𝑎_𝑖^∞_𝑖=1∈ℓ¹,𝑠_𝑖∈𝒳², (18) with the norm 𝑓∶=inf^∞_𝑖=1𝑎_𝑖𝑓=^∞_𝑖=1𝑎_𝑖𝐾_𝑠𝑖. (19)

Obviously, ℋ_𝐾,𝑧⊂ℋ,∀𝐳∈𝐙^𝑛. By the definition of 𝜅 and (17), one has ^𝑁2_𝑖=𝑁1𝑎_𝑖𝐾_𝑠𝑖²_𝐾=^𝑁2_{𝑖,𝑗=𝑁1}𝑎_𝑖𝑎_𝑗𝐾𝑠_𝑖,𝑠_𝑗≤𝜅^𝑁2_𝑖=𝑁1𝑎_𝑖², (20) which implies that the series ∑^∞_𝑖=1𝑎_𝑖𝐾_𝑠𝑖 converges in ℋ_𝐾. Consequently, 𝑓∈ℋ_𝐾 and ‖𝑓‖_𝐾≤√𝜅‖𝑓‖. The following also holds: ‖𝑓‖_𝐶≤𝜅‖𝑓‖,∀𝑓∈ℋ, where ‖𝑓‖_𝐶=sup_𝑠∈𝒳²|𝑓(𝑠)| for 𝑓∈𝐶(𝒳²).

Denote ℋ^as={𝑓∈ℋ∣𝑓(𝑥,𝑥^′)=−𝑓(𝑥^′,𝑥)}. The approximation error of 𝑄^∗ by 𝑄(𝑓) with 𝑓∈ℋ^as is defined as 𝐷(𝜆)∶=inf_𝑓∈ℋas𝑄𝑓−𝑄^∗+𝜆𝑓. (21)

Denote the minimizer 𝑓_𝜆∶=argmin_𝑓∈ℋas𝑄𝑓+𝜆𝑓,𝜆>0. (22)

The next assumption is concerned with the approximation power of ℋ^as to 𝑓_𝜙.

Assumption 4. There are positive constants 𝑐₂ and 𝛽 such that 𝒟(𝜆)=𝑄𝑓_𝜆−𝑄^∗+𝜆𝑓_𝜆≤𝑐₂𝜆^𝛽,∀𝜆>0. (23)

Recall 𝑓_𝜙(𝑥,𝑥^′)=−𝑓_𝜙(𝑥^′,𝑥). The above assumption is not too restrict.

Assumption 5. (i) The kernel 𝐾 satisfies a Lipschitz condition of order 𝛾 with 0<𝛾<1; that is, there exists some 𝑐₃>0 such that 𝐾(𝑠,𝑡)−𝐾𝑠,𝑡^′≤𝑐₃𝑡−𝑡^′𝛾₂. (24) (ii) The ranking loss has an increment exponent 𝜃≥1; that is, there exist some constants 𝜃≥1,𝑐₄ such that 𝜙(𝑡)≤𝑐₄(1+|𝑡|)^𝜃,𝜙^′_±(𝑡)≤𝑐₄(1+|𝑡|)^𝜃−1,𝑡∈ℝ, (25)where 𝜙^′_± denotes the right- and left-sided derivatives of 𝜙, respectively.

Assumption 6. The margin distribution 𝜌_𝑋 satisfies condition 𝐿_𝜏 with 0<𝜏<∞; that is, for some 𝑐₅>0 and any ball 𝐵(𝑥,𝛿)∶={𝑢∈𝒳∣|𝑢−𝑥|₂<𝛿}, one has 𝜌_𝑋(𝐵(𝑥,𝛿))≥𝑐₅𝛿^𝜏,∀𝑥∈𝒳,0<𝛿≤1. (26)

The last assumption concerns covering numbers. For a subset 𝒮 of a space with pseudometric 𝜌 and 𝛿>0. The covering number 𝒩(𝛿,𝒮,𝜌) is defined to be the minimal number 𝑙 such that there exist 𝑙 disks with radius 𝛿 covering 𝒮. When Σ is compact this number is finite.

Assumption 7. (i) There are some 𝛼>0 and 𝑐_𝛼>0 such that 𝒩𝛿,𝒳,|⋅|₂≤𝑐₆1𝛿^𝛼,∀𝛿>0. (27) (ii) For 𝑅>0, let 𝐵_𝑅={𝑓∈ℋ^as∣‖𝑓‖≤𝑅}. There are some constant 𝑠∈(0,1),𝑐₇>0 such that log𝒩𝛿,𝐵₁,‖⋅‖_∞≤𝑐₇𝛿^−𝑠,∀𝛿>0. (28)

It was shown in [18], under Assumptions 5(i), 6, and 7(i), that the following holds: log𝒩𝛿,𝒳,|⋅|₂≤𝑐₆4𝑐₃𝛿^𝛼/𝛾log2+4𝜅𝛿,∀0<𝛿<1. (29) Therefor (ii) in Assumption 7 holds provided that 𝛼/𝛾<1.

We are in a position to state the main result of this paper. The proof is given in Section 5.

Theorem 8. For any 𝜀∈(0,𝛽), under the Assumptions 2–7, one has confidence at least 1−𝐶_𝜀𝑒^−𝑡𝑄𝑔_𝑛−𝑄^∗+𝜆Ω_𝑧𝑓_𝑧,𝜆≤𝐶^′maxlog𝑛+𝑡^𝛾/𝜏,𝑡^{(2−𝛼+𝑠)/(2−𝛼)}𝑛^{−(𝛽−𝜀)𝜇}, (30) where 𝜇=min{𝛾/𝜏(𝛽+(1−𝛽)𝜃),1/(𝛽+(1−𝛽)𝜃),1/((2−𝛼)𝛽+𝑠)} and 𝐶_𝜀, 𝐶^′ are constants independent of 𝑡 or 𝑛.

The first step of the proof is to decompose 𝑄(𝑔_𝑛)−𝑄^∗+𝜆Ω_𝑧(𝑓_𝑧,𝜆) into errors of different types as the following: 𝑄𝑔_𝑛−𝑄^∗+𝜆Ω_𝑧𝑓_𝑧,𝜆=𝑆(𝐳,𝜆)+𝑃(𝐳,𝜆)+𝐷(𝜆), (31) where 𝑆(𝐳,𝜆)=𝑄𝑔_𝑛−𝑄_𝑛𝑔_𝑛+𝑄_𝑛𝑓_𝜆−𝑄𝑓_𝜆, (32) referred to as sample error and 𝑃(𝐳,𝜆)=𝑄_𝑛𝑔_𝑛+𝜆Ω_𝑧𝑓_𝑧,𝜆−𝑄_𝑛𝑓_𝜆+𝜆𝑓_𝜆, (33) referred to as hypothesis error. We bound hypothesis error and sample error in the next two sections, respectively.

In the estimation of sample error, Hoeffding's decomposition of 𝑈-statistic, which breaks 𝑈-statistic into a sum of iid random variables and a degenerate 𝑈-statistic (see Section 4 for details), is a useful tool.

3. Hypothesis Error

In this section, we bound hypothesis error 𝑃(𝐳,𝜆). This error is caused as we switch from the minimizer 𝑓_𝑧,𝜆 of 𝑄_𝑛(𝑓)+𝜆Ω_𝑧(𝑓) in ℋ^as_𝐾,𝑧 to the minimizer 𝑓_𝜆 of 𝑄(𝑓)+𝜆||𝑓|| in ℋ^as. Such errors are estimated in some papers, for example, [7, 18], and so forth. We note that, different from [18, 19], the underlying spaces ℋ^as_𝐾,𝑧 and ℋ^as are sets of antisymmetric functions. We begin with the representations of the functions.

Lemma 9. Let 𝑓∈ℋ^𝑎𝑠. For any 𝜂>0, one has a representation: 1𝑓=2^∞_𝑖=1𝑎_𝑖𝐾_{(𝑥𝑖,𝑢𝑖)}−𝐾_{(𝑢𝑖,𝑥𝑖)},𝑥_𝑖,𝑢_𝑖∈𝒳,_∞𝑖=1𝑎_𝑖≤𝑓+𝜂. (34)

Proof. For any 𝜂>0, there are sequences {𝑠_𝑖}^∞_𝑖=1∈𝒳² and {𝑎_𝑖}^∞_𝑖=1∈ℓ¹ such that 𝑓=∑^∞_𝑖=1𝑎_𝑖𝐾_𝑠𝑖,𝑠_𝑖∈𝒳² and ∑^∞_𝑖=1|𝑎_𝑖|≤‖𝑓‖+𝜂.
Denote 𝑠_𝑖=(𝑥_𝑖,𝑢_𝑖),𝑥_𝑖,𝑢_𝑖∈𝒳,𝑖=1,2,…. It follows from (5) that 𝑓𝑥^′,𝑥=^∞_𝑖=1𝑎_𝑖𝐾𝑢_𝑖,𝑥_𝑖,𝑥,𝑥^′. (35) The proof is complete by 𝑓(𝑥,𝑥^′)=1/2(𝑓(𝑥,𝑥^′)−𝑓(𝑥^′,𝑥)).

A set {𝑥_𝑖}^𝑛_𝑖=1⊂𝒳 is said to be Δ-dense in 𝒳 if for any 𝑥∈𝒳 there exists some 1≤𝑗≤𝑛 such that |𝑥−𝑥_𝑗|₂<Δ.

Proposition 10 ([19], Proposition 3.1). Let {𝑋_𝑗}^𝑛_𝑗=1 be drawn independently according to 𝜌_𝑋. Then for any 𝑡>1, under Assumption 6 and (i) in Assumption 7, with confidence at least 1−𝑒^−𝑡,{𝑋_𝑗}^𝑛_𝑗=1 is 𝑐_𝛼,𝜏((log𝑛+𝑡)/𝑛)^𝛾/𝜏-dense in 𝒳, where 𝑐_𝛼,𝜏 is a constant that depends only on 𝛼 and 𝜏.

The hypothesis error 𝑃(𝐳,𝜆) is bounded by the following proposition.

Proposition 11. Assume Assumptions 5 and 6. Then for any 𝑡>1, with confidence at least 1−𝑒^−𝑡, there holds 𝑃(𝐳,𝜆)≤𝐶𝑓_𝜆∞log𝑛+𝑡𝑛^𝛾/𝜏×1+𝑓_𝜆∞+𝑓_𝜆∞log𝑛+𝑡𝑛^{𝛾/𝜏𝜃−1}, (36) where 𝐶^′ is a constant independent of 𝐳,𝜆,𝑚, and 𝑡. (hereafter, 𝐶 and 𝐶^′ are constants which are independent of 𝑅,𝑡,𝑛, or 𝜆, and may changes from line to line.)

Proof. The proof follows the line of [19, 20]. For any 𝜂>0, let the representation 𝑓_𝜆=∑^∞_𝑖=1𝑏_𝑖(𝐾_{(𝑥𝑖,𝑢𝑖)}−𝐾_{(𝑢𝑖,𝑥𝑖)}) with ∑^∞_𝑖=1|𝑏_𝑖|≤(‖𝑓_𝜆‖_∞+𝜂)/2 be given in Lemma 9.
By Proposition 11, with confidence at least 1−𝑒^−𝑡, for any 𝑖=1,2,…,𝑛, there are some 𝑋_𝑗𝑖,𝑋_𝑘𝑖∈{𝑋_𝑖}^𝑛_𝑖=1 such that max{|𝑥_𝑖−𝑋_𝑗𝑖|₂,|𝑢_𝑖−𝑋_𝑘𝑖|}≤𝑐_𝛼,𝜏((log𝑛+𝑡)/𝑛)^1/𝜏. For an integer 𝑁, which will be determined later, denote 𝑓=∑^𝑁_𝑖=1𝑏_𝑖(𝐾_{(𝑋𝑗𝑖,𝑋𝑘𝑖)}−𝐾_{(𝑋𝑘𝑖,𝑋𝑗𝑖)})∈ℋ^as_𝐾,𝑧. So by Assumption 7, 𝑓−^𝑁_𝑖=1𝑏_𝑖𝐾_{(𝑥𝑖,𝑢𝑖)}−𝐾_{(𝑢𝑖,𝑥𝑖)∞}≤𝐶𝑓_𝜆+𝜂log𝑛+𝑡𝑛^𝛾/𝜏, (37) where 𝐶=𝑐₃𝑐^𝛾_𝛼,𝜏.
Choose 𝑁 such that ∑^∞_𝑗=𝑁+1|𝑏_𝑗|<𝜂/2. Therefore ^𝑁_𝑖=1𝑏_𝑖𝐾_{(𝑥𝑖,𝑢𝑖)}−𝐾_{(𝑢𝑖,𝑥𝑖)}−𝑓_𝜆∞≤2𝜅^∞_𝑗=𝑁+1𝑏_𝑗≤𝜅𝜂. (38) Consequently, ‖𝑓−𝑓_𝜆‖_∞≤𝐷_𝜂∶=𝐶(‖𝑓_𝜆‖_∞+𝜂)((log𝑛+𝑡)/𝑛)^𝛾/𝜏+𝜅𝜂, which together with 𝑄_𝑛𝑓₁−𝑄_𝑛𝑓₂≤𝑐₄1+max𝑓_1∞,𝑓_2∞^𝜃−1𝑓₁−𝑓_2∞ (39) yields, with confidence at least 1−𝑒^−𝑡, 𝑄_𝑛𝑓≤𝑄_𝑛𝑓_𝜆+𝑐₄1+𝑓_𝜆∞+𝐷_𝜂^𝜃−1𝐷_𝜂. (40) On the other hand, since 𝑓∈ℋ^as_𝐾,𝑧, the following holds by (12) and (9), with confidence at least 1−𝑒^−𝑡: 𝑄_𝑛𝑔_𝑛+𝜆Ω_𝐳𝑓_𝑧,𝜆≤𝑄_𝑛𝑓_𝑧,𝜆+𝜆Ω_𝑧𝑓_𝑧,𝜆≤𝑄_𝑛𝑓+2𝜆^𝑁_𝑗=1𝑏_𝑗≤𝑄_𝑛𝑓+𝜆𝑓_𝜆+𝜂, (41) which together with (40) completes the proof by letting 𝜂→0.

The above bound for 𝑃(𝐳,𝜆) is the same as in [20], both using of the same density of {𝑋_𝑖}^𝑛_𝑖=1 in 𝒳. However, the functions considered there are defined on 𝒳, instead of 𝒳² as present.

4. Sample Error

As in [2, 7], Hoeffding's decomposition plays an important role in the estimation for sample error. For any 𝑓(𝑥,𝑥^′),𝑥,𝑥^′∈𝒳, denote 𝜑_𝑓𝑧,𝑧^′=𝜙sign𝑦−𝑦^′𝑓𝑥,𝑥^′,𝑧=𝑥,𝑦,𝑧^′=𝑥^′,𝑦^′. (42) By Hoeffding's decomposition of 𝑈-statistic, we have 𝑄𝑓−𝑓_𝜙−𝑄_𝑛𝑓−𝑓_𝜙=2𝑄𝑓−𝑓_𝜙−𝑃_𝑛𝑃𝜑_𝑓−𝑃𝜑_𝑓𝜙−𝑈_𝑛ℎ_𝑓−ℎ_𝑓𝜙, (43) where ℎ_𝑓(𝑧,𝑧^′)=𝜑_𝑓(𝑧,𝑧^′)−𝑃𝜑_𝑓(𝑧)−𝑃𝜑_𝑓(𝑧^′)+𝑄(𝑓) and, for any 𝑔(𝑧,𝑧^′), 𝑃𝑔(𝑧)=𝔼𝑔𝑍,𝑍^′,𝑃∣𝑍=𝑧_𝑛𝑔=1𝑛^𝑛_𝑖=1𝑔𝑍_𝑖,𝑈_𝑛𝑔=1𝑛(𝑛−1)_𝑖≠𝑗𝑔𝑍_𝑖,𝑍_𝑗. (44)

Moreover, for any ranking rule 𝑓, we denote 𝑔_𝑓(𝑧)=𝑃𝜑_𝑓(𝑧)−𝑃𝜑_𝑓𝜙(𝑧). Then 𝑄𝑓−𝑓_𝜙−𝑃_𝑛𝑃𝜑_𝑓−𝑃𝜑_𝑓𝜙=𝔼𝑔_𝑓−𝑃_𝑛𝑔_𝑓. (45) It is the deviation of sum of independent random variables from their mean.

As seen, in Hoeffding's decomposition (43), the first term is a sum of iid variables and the second term 𝑈_𝑛(ℎ_𝑓−ℎ_𝑓𝜙) is a degenerate 𝑈-statistic. The degeneration means 𝔼(𝑈_𝑛(ℎ_𝑓(𝑍,𝑍^′)−ℎ_𝑓𝜙(𝑍,𝑍^′))∣𝑍) = 0,∀𝑍.

Denote 𝑆₁=𝑄𝑔_𝑛−𝑄𝑓_𝜙−𝑄_𝑛𝑔_𝑛−𝑄_𝑛𝑓_𝜙=𝑄𝑔_𝑛−𝑓_𝜙−𝑄_𝑛𝑔_𝑛−𝑓_𝜙,𝑆₂=𝑄_𝑛𝑓_𝜆−𝑄_𝑛𝑓_𝜙−𝑄𝑓_𝜆−𝑄𝑓_𝜙=𝑄_𝑛𝑓_𝜆−𝑓_𝜙−𝑄𝑓_𝜆−𝑓_𝜙, (46) so that the sample error 𝑄(𝑔_𝑛)−𝑄_𝑛(𝑔_𝑛)+𝑄_𝑛(𝑓_𝜆)−𝑄(𝑓_𝜆)=𝑆₁+𝑆₂.

We first estimate 𝑆₁. Since 𝑔_𝑛 depends on 𝐳, by (43) and (45), we need to consider the suprema of the sets 𝔼𝑔_𝜋(𝑓)−𝑃_𝑛𝑔_𝜋(𝑓),𝑈∣𝐹∈ℱ_𝑛ℎ_𝜋(𝑓)−ℎ_𝑓𝜙∣𝐹∈ℱ, (47) where ℱ, containing 𝑔_𝑛, is (a subset of) a ball in ℋ^as.

With the above decomposition and the methods in [21], the following proposition is established for the ball {𝑓∈ℋ^as_𝐾∣‖𝑓‖_𝐾≤𝑅} and 𝜙(𝑡)=(1−𝑡)₊ in [7]. The Assumption 2 and the condition on the covering number of {𝑓∈ℋ^as_𝐾∣‖𝑓‖_𝐾≤𝑅} play a crucial role. The arguments also work in the present setting; that is, for the ball 𝐵_𝑅={𝑓∈ℋ^as∣‖𝑓‖≤𝑅}. For this we note that 𝜙 satisfies |𝜙(𝑡)−𝜙(𝑡^′)| ≤ 𝐶|𝑡−𝑡^′|,𝑡,𝑡^′∈[−1,1]. Therefore, under (ii) of Assumption 5 and (ii) of Assumption 7, the covering number 𝒩(𝛿,𝒢,‖⋅‖_∞) of 𝒢={𝑔_𝜋(𝑓)∣𝑓∈𝐵_𝑅} satisfies log𝒩(𝛿,𝒢,‖⋅‖_∞)≤𝐶(𝑅/𝛿)^𝑠. The interested reader may refer to [7, 21] for the details.

Proposition 12. Let 𝑅>0 and 𝑡>0. Under Assumption 2, (ii) of Assumption 5, and (ii) of Assumption 7, one has confidence at least 1−𝑒^−𝑡, 𝔼𝑔_𝜋(𝑓)−𝑃_𝑛𝑔_𝜋(𝑓)≤𝛿₀+𝛿₀^1−(𝛼/2)𝔼𝑔_𝜋(𝑓)^(𝛼/2),∀𝑓∈𝐵_𝑅, (48) where 𝛿₀ is bounded by 𝛿₀𝑡≤𝐶𝑛^1/(2−𝛼)+𝑅^𝑠𝑛^{1/(2−𝛼+𝑠)}, (49) with 𝐶 being a constant independent of 𝑅,𝑡, and 𝑛.

The estimation for the supremum sup_{𝑓∈ℱ𝑅}|𝑈_𝑛(ℎ_𝜋(𝑓)−ℎ_𝑓𝜙)| of 𝑈-process is much involved. The supremum of 𝑈-processes has been studied in a few papers. The following lemma follows from the proof of ([22, Theorem 3.2, 𝑚=2]).

Lemma 13. Suppose that a function class ℱ satisfies the following conditions. (i)For any 𝑓∈ℱ, 𝑓(𝑧,𝑧^′)=𝑓(𝑧^′,𝑧) and 𝔼(𝑓(𝑍,𝑍^′)∣𝑍)=0.(ii)ℱ is uniformly bounded by a universal constant 𝐶₀.(iii)𝐶_ℱ∶=∫^∞₀log𝒩(𝛿,ℱ,‖⋅‖_∞)𝑑𝛿<∞.Then 𝔼exp𝜆√Γ_𝑛≤𝐶exp𝐶^′𝜆²𝐶_ℱ,∀𝜆>0, (50) where 𝐶, 𝐶^′ are some constants, independent of 𝐶_ℱ, and Γ_𝑛=sup_𝑓∈ℱ(𝑛−1)𝑈_𝑛𝑓. (51)

Proposition 14. Suppose that (ii) in Assumption 7 holds. Then one has with confidence at least 1−𝐶𝑒^−𝑡sup_{𝑓∈ℱ𝑅}𝑈_𝑛ℎ_𝜋(𝑓)−ℎ_𝑓𝜙≤𝐶^′𝑅^𝑠+1𝑡𝑛, (52) where ℱ_𝑅={ℎ_𝜋(𝑓)∣𝑓∈𝐵_𝑅} and 𝐶,𝐶^′ are some positive constants.

Proof. We first claim that ℱ_𝑅 satisfies the conditions in Lemma 13. Indeed, condition (i) holds by definition of ℎ_𝜋(𝑓) and 𝑓(𝑥,𝑥^′)=−𝑓(𝑥^′,𝑥) for any 𝑓∈𝐵_𝑅. Also, by definition of ℎ_𝑓, for any 𝑓∈ℱ_𝑅, ||ℎ_𝜋(𝑓)||_∞≤4𝜙(−1), implying (ii). Moreover, by (ii) of Assumption 5ℎ_𝜋(𝑓1)−ℎ_{𝜋(𝑓2)∞}≤𝐶𝜋(𝑓₁)−𝜋(𝑓₂)_∞≤𝐶𝑓₁−𝑓_2∞ (53) we have log𝒩ℱ_𝑅,𝛿,‖⋅‖_∞≤log𝒩𝐵_𝑅,𝛿𝐶,‖⋅‖_∞≤𝑐₇(𝐶𝑅)^𝑠𝛿^−𝑠 (54) provided (ii) in Assumption 7. This establishes (iii), as claimed.
Applying Markov's inequality to exp(𝜆Γ_𝑛), with 𝜆=√𝑡/2𝐶^′𝑅^𝑠 and appealing to Lemma 13, we have ℙ(Γ_𝑛>𝑡)≤𝐶𝑒^{−𝑡/(4𝐶′𝑅𝑠)},∀𝑡>0; that is, ℙΓ_𝑛>4𝐶^′𝑅^𝑠𝑡≤𝐶𝑒^−𝑡,∀𝑡>0. (55) For single function ℎ_𝑓𝜙, it is known in ([23, Proposition 2.3]) that ℙ𝑈_𝑛ℎ_𝑓𝜙≥𝐶^′𝑡𝑛−1≤𝐶𝑒^−𝑡, (56) which together with (55) completes the proof.

The estimation for 𝑆₂ is easy since 𝑓_𝜆 does not change with the set 𝐳 of samples.

Proposition 15. Assume Assumption 2. For any 𝑡>0, one has with confidence at least 1−𝐶𝑒^−𝑡𝑆₂≤𝐶^′𝑡1+𝜅𝑓_𝜆∞^𝜃+13𝑛𝑛^1/(2−𝛼)+𝐷(𝜆), (57) where 𝐶,𝐶^′ are some constants, independent of 𝜆,𝑡 and 𝑛.

Proof. Clearly, the function 𝑔_𝑓𝜆(𝑍) satisfies ‖𝑔_𝑓𝜆‖_∞≤𝐶(1+‖𝑓_𝜆‖_∞)^𝜃. Then, by Assumption 2, we conclude, as in [20, 21], with confidence at least 1−𝑒^−𝑡, that 𝑃_𝑛𝑔_𝑓𝜆−𝑄𝑓_𝜆−𝑄^∗≤𝐶^′𝑡1+𝑓_𝜆∞^𝜃+13𝑛𝑛^1/(2−𝛼)+𝐷(𝜆). (58)
It remains to estimate 𝑈_𝑛(ℎ_𝑓𝜆−ℎ_𝑓𝜙). For any single function 𝑔 with 𝑔(𝑧,𝑧^′)=𝑔(𝑧^′,𝑧) and 𝔼[𝑔(𝑍,𝑍^′)∣𝑍]=0,∀𝑍, we have by [23, Proposition 2.3] ℙ𝑈_𝑛𝑔≥𝐶^′𝑡𝑔_∞𝑛−1≤𝐶𝑒^−𝑡, (59) which together with ‖ℎ_𝑓𝜆−ℎ_𝑓𝜙‖_∞≤𝐶^′(1+‖𝑓_𝜆‖_∞)^𝜃 implies, with confidence at least 1−𝐶𝑒^−𝑡, that 𝑈_𝑛ℎ_𝑓𝜆−ℎ_𝑓𝜙≤𝐶^′1+𝑓_𝜆∞^𝜃𝑡𝑛, (60) where 𝐶 and 𝐶^′ are constants. The proof is complete.

5. Proof of Theorem 8

Theorem 16. For 𝑅 such that Ω_𝑧(𝑓_𝑧,𝜆)≤𝑅 and any 𝑡>1, under Assumptions 2–7, one has with confidence at least 1−𝐶𝑒^−𝑡𝑄𝑔_𝑛−𝑄^∗+𝜆Ω_𝑧𝑓_𝑧,𝜆≤𝐶^′log𝑛+𝑡𝑛^𝛾/𝜏𝜆^{(𝛽−1)𝜃}+𝑡𝑛^1/(2−𝛼)+𝑅^𝑠𝑛^{1/(2−𝛼+𝑠)}+𝑅^𝑠+1𝑡𝑛+𝑡𝜆^{(𝛽−1)𝜃}𝑛+𝜆^𝛽, (61) where 𝐶,𝐶^′ are constant independent of 𝑅,𝑡, or 𝑛.

Proof. We note that 𝑓_𝜆∞≤𝜅𝑓_𝜆≤𝜅𝐷(𝜆)𝜆≤𝐶𝜆^𝛽−1. (62) By Ω_𝑧(𝑓_𝑧,𝜆)≤𝑅, a combination of Propositions 11, 12, 14, and 15 yields that, with confidence at least 1−𝐶𝑒^−𝑡, 𝑄𝑔_𝑛−𝑄^∗+𝜆Ω_𝑧𝑓_𝑧,𝜆≤𝐶₁log𝑛+𝑡𝑛^𝛾/𝜏𝜆^{(𝛽−1)𝜃}+𝛿₀+𝛿₀^1−𝛼/2𝑄𝑓−𝑄^∗𝛼/2+𝐶₂𝑅^𝑠+1𝑡𝑛+𝐶₃𝑡𝜆^{(𝛽−1)𝜃}+13𝑛𝑛^1/(2−𝛼)+2𝜆^𝛽, (63) where 𝐶_𝑖,𝑖=1,2,3, are constants, and 𝛿₀ is bounded by (49).
Putting 𝑥=𝑄(𝑔_𝑛)−𝑄^∗+𝜆Ω_𝑧(𝑓_𝑧,𝜆) and 𝜈=𝛼/2 into the implication relation 𝑥≤𝑎𝑥^𝜈+𝑏,𝑎,𝑏,𝑥>0⟹𝑥≤max(2𝑎)^1/(1−𝜈),2𝑏, (64) we obtain (61) from (63). Therefore, the conditional probability of the event that inequality (61) holds, given the event Ω_𝑧(𝑓_𝑧,𝜆)≤𝑅, is at least 1−𝐶𝑒^−𝑡. The proof is complete.

For any 𝑅, denote the random event Ω_𝑧(𝑓_𝑧,𝜆)≤𝑅 by 𝜉_𝑅. Obviously, ℙ(𝜉_𝑅)=1 for 𝑅=𝜙(0)/𝜆. However, to prove Theorem 8, a smaller 𝑅 with ℙ(𝜉_𝑅)=1 is desired. To this end, we apply the iteration technique for estimation of Ω_𝑧(𝑓_𝑧,𝜆) introduced in [17].

Recall that 𝜇 is given in Theorem 8. It is easily seen that, for 𝜆=𝑛^−𝜇,𝑅≤𝑛^1/𝑠, 1𝑛^𝛾/𝜏𝜆^{(𝛽−1)𝜃}≤𝜆^𝛽,1𝑛𝜆^{(𝛽−1)𝜃}≤𝜆^𝛽,1𝑛^1/(2−𝛼)≤𝑅^𝑠+1𝑛𝑅≤(^𝑠𝑛)^{1/(2−𝛼+𝑠)}≤𝜆^𝛽𝜆^1−𝛽𝑅^{𝑠/(2−𝛼+𝑠)}. (65) Therefore, we have, by Theorem 16, with the conditional probability at least 1−𝐶𝑒^−𝑡, given 𝜉_𝑅, 𝑄𝑔_𝑛−𝑄^∗+𝜆Ω_𝑧𝑓_𝑧,𝜆≤𝐶^′maxlog𝑛+𝑡^𝛾/𝜏,𝑡𝜆^𝛽𝜆^1−𝛽𝑅^{𝑠/(2−𝛼+𝑠)}+2. (66) If 𝜆^𝛽−1𝑅⁻¹=𝑂(1), the above inequality becomes 𝑄𝑔_𝑛−𝑄^∗+𝜆Ω_𝑧𝑓_𝑧,𝜆≤𝐶^′maxlog𝑛+𝑡^𝛾/𝜏,𝑡𝜆^𝛽𝜆^1−𝛽𝑅^{𝑠/(2−𝛼+𝑠)}. (67) Consequently, given event 𝜉_𝑅 with 𝜆^𝛽−1𝑅⁻¹=𝑂(1),𝑅≤𝑛^1/𝑠, we have with confidence at least 1−𝐶𝑒^−𝑡, 𝑓_𝑧,𝜆≤Ω_𝑧𝑓_𝑧,𝜆≤𝑟(𝑅)∶=𝐶^′maxlog𝑛+𝑡^𝛾/𝜏,𝑡×𝜆^{(𝛽−1)((1−𝑠)/(2−𝛼+𝑠))}𝑅^{𝑠/(2−𝛼+𝑠)}. (68)

Let 𝑅⁽⁰⁾=𝜙(0)/𝜆,𝑅^(𝑘)=𝑟(𝑅^(𝑘−1)),𝑘=1,2,…. By induction, it is easy to prove 𝜆^𝛽−1(𝑅^(𝑘))⁻¹ = 𝑂(1),𝑅^(𝑘)≤𝑛^1/𝑠. Since ℙ{𝜉_𝑅0}=1, we have with confidence at least 1−𝑘𝐶𝑒^−𝑡Ω_𝑧(𝑓_𝑧,𝜆) ≤ 𝑅^(𝑘),𝑘=1,2,…,. Clearly, 𝑅^(𝑘)≤𝐶^′maxlog𝑛+𝑡^𝛾/𝜏,𝑡^{(1−𝜈𝑘+1)/(1−𝜈)}×𝜆^𝛽−1𝜆^{−(1/(2−𝛼+𝑠))𝑘+1},𝑘=1,2,…, (69) where 𝜈=𝑠/(2−𝛼+𝑠)<1.

For any 𝜀>0, let 𝑘 be the smallest integer such that (1/(2−𝛼+𝑠))^𝑘+1<𝜀. Substituting 𝑅=𝑅^(𝑘) into (67), we bound the right hand side of (67) by (𝐶^′max{(log𝑛+𝑡)^𝛾/𝜏,𝑡})^1/(1−𝜈)𝜆^𝛽−𝜀, with confidence at least 1−𝐶_𝜀𝑒^−𝑡, where 𝐶_𝜀=𝑘𝐶. This completes the proof.

Acknowledgment

The authors thank Professor Di-Rong Chen for his help.