On the Convergence Rate of Kernel-Based Sequential Greedy Regression

Abstract

A kernel-based greedy algorithm is presented to realize efficient sparse learning with data-dependent basis functions. Upper bound of generalization error is obtained based on complexity measure of hypothesis space with covering numbers. A careful analysis shows the error has a satisfactory decay rate under mild conditions.

1. Introduction

Kernel methods have been extensively utilized in various learning tasks, and its generalization performance has been investigated from the viewpoint of approximation theory [1, 2]. Among these methods, a family of them can be considered as coefficient-based regularized framework in data-dependent hypothesis spaces; see, for example, [3–8]. For given samples {(𝑥_𝑖,𝑦_𝑖)}^𝑛_𝑖=1, the solution of these kernel methods has the following expression ∑^𝑛_𝑖=1𝛼_𝑖𝐾(𝑥_𝑖,⋅), where 𝛼_𝑖∈ℛ and 𝐾 is a Mecer kernel. The aim of these coefficient-based algorithms is to search a set of coefficients {𝛼_𝑖} with good predictive performance.

Inspired by greedy approximation methods in [9–12], we propose a sparse greedy algorithm for regression. The greedy approximation has two advantages over the regularization methods: one is that the sparsity is directly controlled by a greedy approximation algorithm, rather than by the regularization parameter; the other is that greedy approximation does not change the objective optimization function, while the regularized methods usually modify the objective function by including a sparse regularization term [13].

Before introducing the greedy algorithm, we recall some preliminary background for regression. Let the input space 𝒳⊂ℝ^𝑑 be a compact subset and 𝒴=[−𝑀,𝑀] for some constant 𝑀>0. In the regression model, the learner gets a sample set 𝐳={(𝑥_𝑖,𝑦_𝑖)}^𝑛_𝑖=1, where (𝑥_𝑖,𝑦_𝑖)∈𝒵∶=𝒳×𝒴,1≤𝑖≤𝑛, are randomly independently drawn from an unknown distribution 𝜌 on 𝒵. The goal of learning is to pick a function 𝑓∶𝒳→𝒴 with the expected error ℰ𝑓=_𝒵𝑓(𝑥)−𝑦²𝑑𝜌 (1.1) as small as possible. Note that the regression function 𝑓_𝜌(𝑥)=_𝒴𝑦𝑑𝜌𝑦∣𝑥,𝑥∈𝒳, (1.2) is the minimizer of ℰ(𝑓), where 𝜌(⋅∣𝑥) is the conditional probability measure at 𝑥 induced by 𝜌.

The empirical error is defined as ℰ_𝐳𝑓=1𝑛^𝑛_𝑖=1𝑓𝑥_𝑖−𝑦_𝑖². (1.3)

We call a symmetric and positive semidefinite continuous function 𝐾∶𝒳×𝒳→ℝ a Mercer kernel. The reproducing kernel Hilbert space (RKHS) ℋ_𝐾 is defined to be the closure of the linear span of the set of functions {𝐾_𝑥∶=𝐾(𝑥,⋅)∶𝑥∈𝒳} with the inner product ⟨⋅,⋅⟩_𝐾 defined by ⟨𝐾_𝑥,𝐾_𝑥′⟩_𝐾=𝐾(𝑥,𝑥^′). For all 𝑥∈𝑋 and 𝑓∈ℋ_𝐾, the reproducing property is given by ⟨𝐾_𝑥,𝑓⟩_𝐾=𝑓(𝑥). We can see 𝜅∶=sup_𝑥∈𝒳𝐾(𝑥,𝑥)<∞ because of the continuity of 𝐾 and the compactness of 𝒳.

Different from the coefficient-based regularized method [3–6], we use the idea of sequential greedy approximation to realize sparse learning in this paper. Denote ℋ={ℎ_𝑖}^2𝑛_𝑖=1, where ℎ_2𝑖−1=𝐾_𝑥𝑖 and ℎ_2𝑖=−𝐾_𝑥𝑖. The hypothesis space (depending on 𝐳) is defined as 𝐶𝑂_2𝑛ℋ=𝑓∶𝑓(𝑥)=^2𝑛_𝑖=1𝛼_𝑖ℎ_𝑖(𝑥),𝛼_𝑖≥0,^2𝑛_𝑖=1𝛼_𝑖≤1. (1.4) For any hypothesis function space 𝒢, we denote 𝛽𝒢={𝑓∶𝑓=𝛽𝑔,𝑔∈𝒢}.

The definition of 𝑓_𝜌 tells us |𝑓_𝜌(𝑥)|≤𝑀, so it is natural to restrict the approximating functions to [−𝑀,𝑀]. The projection operator has been used in error analysis of learning algorithms (see, e.g., [2, 14]).

Definition 1.1. The projection operator 𝜋=𝜋_𝑀 is defined on the space of measurable functions 𝑓∶𝒳→ℛ as 𝜋𝑓(𝑥)=(𝑥)(𝑥)𝑓(𝑥)𝑀,if𝑓>𝑀;−𝑀,if𝑓<−𝑀;,otherwise. (1.5)

The kernel-based greedy algorithm can be summarized as below. Let 𝑡 be a stopping time and let 𝛽 be a positive constant. Set 𝑓⁰_𝛽=0. And then for 𝜏=1,2,…,𝑡, define ℎ_𝜏,𝛼_𝜏,𝛽_𝜏=argmin_ℎ∈ℋ^{,0≤𝛼≤1,0≤𝛽′≤𝛽}ℰ_𝐳(1−𝛼)𝑓_𝛽^𝜏−1+𝛼𝛽^′ℎ,𝑓^𝜏_𝛽=1−𝛼_𝜏𝑓_𝛽^𝜏−1+𝛼_𝜏𝛽_𝜏ℎ_𝜏. (1.6) Different from the regularized algorithms in [6, 12, 14–18], the above learning algorithm tries to realize efficient learning by greedy approximation. The study for its generalization performance can enrich the learning theory of kernel-based regression. In the remainder of this paper, we focus on establishing the convergence rate of 𝜋(𝑓^𝑡_𝛽) to the regression function 𝑓_𝜌 under choice of suitable parameters. The theoretical result is dependent on weaker conditions than the previous error analysis for kernel-based regularization framework in [4, 5].

2. Main Result

Define a data-free basis function set ℋ=ℎ_𝑖∶ℎ_2𝑖−1=𝐾_𝑢𝑖,ℎ_2𝑖=−𝐾_𝑢𝑖,𝑢_𝑖,(ℋ)=(𝑥)=⊂𝒳,𝑖=1,…,∞𝐶𝑂𝑓∶𝑓^∞_𝑖=1𝛼_𝑖ℎ_𝑖(𝑥),𝛼_𝑖≥0,^∞_𝑖=1𝛼_𝑖≤1. (2.1)

To investigate the approximation of 𝜋(𝑓^𝑡_𝛽) to 𝑓_𝜌, we introduce a data-independent function 𝑓^∗_𝛽=argmin_{(ℋ)𝑓∈𝛽𝐶𝑂}ℰ𝑓. (2.2)

Observe that ℰ𝜋𝑓^𝑡_𝛽−ℰ𝑓_𝜌≤ℰ𝜋𝑓^𝑡_𝛽−ℰ_𝐳𝜋𝑓^𝑡_𝛽+ℰ_𝐳𝑓^∗_𝛽−ℰ𝑓^∗_𝛽+ℰ_𝐳𝜋𝑓^𝑡_𝛽−ℰ_𝐳𝑓^∗_𝛽+ℰ𝑓^∗_𝛽−ℰ𝑓_𝜌. (2.3) Here, the three terms on the right-hand side are called as the sample error, the hypothesis error, and the approximation error, respectively.

To estimate the sample error, we usually need the complexity measure of hypothesis function space ℋ_𝐾. For this reason, we introduce some definitions of covering numbers to measure the complexity.

Definition 2.1. Let (𝒰,𝑑) be a pseudometric space and denote a subset 𝑆⊂𝒰. For every 𝜖>0, the covering number 𝒩(𝑆,𝜖,𝑑) of 𝑆 with respect to 𝜖,𝑑 is defined as the minimal number of balls of radius 𝜖 whose union covers 𝑆, that is, 𝒩(𝑆,𝜖,𝑑)=min𝑙∈ℕ∶𝑆⊂^𝑙_𝑗=1𝐵𝑠_𝑗,𝜖forsome𝑠_𝑗^𝑙_𝑗=1⊂𝒰, (2.4) where 𝐵(𝑠_𝑗,𝜖)={𝑠∈𝒰∶𝑑(𝑠,𝑠_𝑗)≤𝜖} is a ball in 𝒰.

The empirical covering number with ℓ₂ metric is defined as below.

Definition 2.2. Let ℱ be a set of functions on 𝒳, 𝐮=(𝑢_𝑖)^𝑘_𝑖=1 and ℱ|_𝐮={(𝑓(𝑢_𝑖))^𝑘_𝑖=1∶𝑓∈ℱ}⊂ℝ^𝑘. Set 𝒩_2,𝐮(ℱ,𝜖)=𝒩_2,𝐮(ℱ|_𝐮,𝜖,𝑑₂). The ℓ₂ empirical covering number of ℱ is defined by 𝒩₂(ℱ,𝜖)=sup_𝑘∈ℕsup_{𝐮∈𝒳𝑘}𝒩_2,𝐮(ℱ,𝜖),𝜖>0, (2.5) where ℓ₂ metric 𝑑₂()=1𝐚,𝐛𝑘^𝑘_𝑖=1𝑎_𝑖−𝑏_𝑖^21/2,∀𝐚=𝑎_𝑖^𝑘_𝑖=1∈ℝ^𝑘,𝐛=𝑏_𝑖^𝑘_𝑖=1∈ℝ^𝑘. (2.6)

Denote ℬ_𝑟 as the ball of radius 𝑟 with 𝑟>0, where ℬ_𝑟={𝑓∈ℋ_𝐾∶‖𝑓‖_ℋ𝐾≤𝑟}. We need the following capacity assumption on ℋ_𝐾, which has been used in [5, 6, 18].

Assumption 2.3. There exist an exponent 𝑝, with 𝑝∈(0,2) and a constant 𝑐_𝑝,𝐾>0 such that log𝒩₂ℬ₁,𝜖≤𝑐_𝑝,𝐾𝜖^−𝑝. (2.7)

We now formulate the generalization error bounds for 𝜋(𝑓^𝑡_𝛽). The result follows from Propositions 3.2–3.5 in the next section.

Theorem 2.4. Under Assumption 2.3, for any 0<𝛿<1, the following inequality holds with confidence 1−𝛿ℰ𝜋𝑓^𝑡_𝛽−ℰ𝑓_𝜌≤4ℰ𝑓^∗_𝛽−ℰ𝑓_𝜌+32𝛽²𝑡+43𝑀+𝜅²𝛽²𝑛2log𝛿+1280𝑀²𝑐_𝑝,𝐾4𝑀𝜅𝛽^{𝑝2/(2+𝑝)}2log𝛿𝑛^{−2/(2+𝑝)}. (2.8)

From the result, we know there exists a constant 𝐶 independent of 𝑛,𝑡,𝛿 such that with confidence 1−𝛿ℰ𝜋𝑓^𝑡_𝛽−ℰ𝑓_𝜌≤4ℰ𝑓^∗_𝛽−ℰ𝑓_𝜌𝛽+𝐶max²𝑡,𝛽²𝑛,𝛽^𝑝𝑛^2/(2+𝑝)2log𝛿. (2.9) In particular, if 𝑓_𝜌∈𝛽𝐶𝑂(ℋ) for some fixed constant 𝛽 and 𝑡≥𝑛, we have ℰ(𝜋(𝑓^𝑡_𝛽))−ℰ(𝑓_𝜌)→0 with decay rate 𝑂(𝑛^{−2/(2+𝑝)}). The learning rate is satisfactory as 𝑝→0.

Here, the estimate of the hypothesis error is simple and does not need the strict condition on 𝜌 and 𝒳 in [3–5] for learning with data-dependent hypothesis spaces.

If there are some additional conditions on approximation error with the increasing of 𝛽, we can obtain the explicit learning rates with suitable parameter selection.

Corollary 2.5. Assume that the RKHS ℋ_𝐾 satisfies (2.7) and ℰ(𝑓^∗_𝛽)−ℰ(𝑓_𝜌)≤𝑐_𝛾𝛽^−𝛾 for some 𝛾>0. Choose 𝛽=𝑛^{𝑝/4(2+𝑝)}. For any 0<𝛿<1 and 𝑡=𝑛, one has ℰ𝜋𝑓^𝑡_𝛽−ℰ𝑓_𝜌≤𝐶𝑛^{−min{(4+𝑝)/(4+2𝑝),𝑝𝛾/(8+4𝑝)}}2log𝛿 (2.10) with confidence 1−𝛿. Here 𝐶 is a constant independent of 𝑛,𝛿.

Observe that the learning rate depends closely on the approximation condition between 𝑓_𝜌 and 𝑓^∗_𝛽. This means that only the target function can be well described by the functions from the hypothesis space, the learning algorithm can achieve good generalization performance. In fact, similar approximation assumption is extensively studied for error analysis in learning theory; see, for example, [1, 2, 4, 17].

From Corollary 2.5, when the kernel 𝐾∈𝐶^∞, 𝑝>0 can be arbitrarily small, one can easily see that the learning rate is quite low. Future research direction may be furthered to improve the estimate by introducing some new analysis techniques.

3. Proof of Theorem 2.4

In this section, we provide the proof of Theorem 2.4 based on the upper bound estimates of sample error and hypothesis error. Denote 𝑆₁=ℰ_𝐳𝑓^∗_𝛽−ℰ_𝐳𝑓_𝜌−ℰ𝑓^∗_𝛽−ℰ𝑓_𝜌,𝑆₂=ℰ𝜋𝑓^𝑡_𝛽−ℰ𝑓_𝜌−ℰ_𝐳𝜋𝑓^𝑡_𝛽−ℰ_𝐳𝑓_𝜌. (3.1) We can observe that the sample error ℰ𝜋𝑓^𝑡_𝛽−ℰ_𝐳𝜋𝑓^𝑡_𝛽+ℰ_𝐳𝑓^∗_𝛽−ℰ𝑓^∗_𝛽=𝑆₁+𝑆₂. (3.2)

Here 𝑆₁ can be bounded by applying the following one-side Bernstein type probability inequality; see, for example, [1, 2, 14].

Lemma 3.1. Let 𝜉 be a random variable on a probability space 𝑍 with mean 𝐸𝜉 and variance 𝜎²(𝜉)=𝜎². If |𝜉(𝑧)−𝐸𝜉|≤𝐵 for almost all 𝑧∈𝑍, then for all 𝜀>0, Prob_𝐳∈𝑍^𝑛1𝑛^𝑛_𝑖=1𝜉𝑧_𝑖−𝐸𝜉≥𝜀≤exp−𝑛𝜀²2𝜎².+𝐵𝜀/3 (3.3)

Proposition 3.2. For any 0<𝛿<1, with confidence 1−𝛿, one has 𝑆₁≤ℰ𝑓^∗_𝛽−ℰ𝑓_𝜌+23𝑀+𝜅²𝛽²𝑛1log𝛿. (3.4)

Proof. Following the definition of 𝑆₁, we have 𝑆₁=(1/𝑛)∑^𝑛_𝑖=1𝜉(𝑥_𝑖)−𝐸𝜉, where random variable 𝜉(𝑥)=(𝑦−𝑓^∗_𝛽(𝑥))²−(𝑦−𝑓_𝜌(𝑥))².
From the definition of 𝑓^∗_𝛽, we know ‖𝑓^∗_𝛽‖_𝐾≤𝜅𝛽 and ‖𝑓^∗_𝛽‖_∞≤𝜅‖𝑓^∗_𝛽‖_𝐾≤𝜅²𝛽. Then 𝜉(𝑥)=𝑓^∗_𝛽(𝑥)−𝑓_𝜌(𝑥)𝑓^∗_𝛽(𝑥)−𝑦+𝑓_𝜌(𝑥)−𝑦≤3𝑀+𝜅²𝛽²∶=𝑐₁ (3.5) and |𝜉−𝐸𝜉|≤2𝑐₁. Moreover, 𝜎²≤𝐸𝜉²=_𝒵𝑓^∗_𝛽(𝑥)−𝑓_𝜌(𝑥)²𝑓^∗_𝛽(𝑥)−𝑦+𝑓_𝜌(𝑥)−𝑦²𝑑𝜌≤𝑐₁ℰ𝑓^∗_𝛽−ℰ𝑓_𝜌. (3.6)
Applying Lemma 3.1 with 𝐵=2𝑐₁ and 𝜎²=𝑐₁{ℰ(𝑓^∗_𝛽)−ℰ(𝑓_𝜌)}, we get 1𝑛^𝑛_𝑖=1𝜉𝑥_𝑖−𝐸𝜉≤𝑡 (3.7) with confidence at least 1−exp{𝑛𝑡²/(2𝑐₁(ℰ(𝑓^∗_𝛽)−ℰ(𝑓_𝜌)+(2/3)𝑡))}. By setting −𝑛𝑡²/(2𝑐₁(ℰ(𝑓^∗_𝛽)−ℰ(𝑓_𝜌)+(2/3)𝑡))=log(𝛿), we derive the solution 𝑡^∗=2𝑐₁/3log(1/𝛿)+2𝑐₁/3log(1/𝛿)²+2𝑐₁log(1/𝛿)ℰ𝑓^∗_𝛽−ℰ𝑓_𝜌𝑛≤2𝑐₁𝑛1log𝛿+ℰ𝑓^∗_𝛽−ℰ𝑓_𝜌. (3.8) Thus, with confidence 1−𝛿, we have 1𝑛^𝑛_𝑖=1𝜉𝑥_𝑖−𝐸𝜉≤2𝑐₁𝑛1log𝛿+ℰ𝑓^∗_𝛽−ℰ𝑓_𝜌. (3.9) This completes the proof.

To establish the uniform upper bound of 𝑆₂, we introduce a concentration inequality established in [18].

Lemma 3.3. Assume that there are constants 𝐵,𝑐>0 and 𝛼∈[0,1] such that ‖𝑓‖_∞≤𝐵 and 𝐸𝑓≤𝑐(𝐸𝑓)^𝛼 for every 𝑓∈ℱ. If for some 𝑎>0 and 𝑝∈(0,2), log𝒩₂(ℱ,𝜖)≤𝑎𝜖^−𝑝,∀𝜖>0, (3.10) then there exists a constant 𝑐^′_𝑝 depending only on 𝑝 such that for any 𝑡>0, with probability at least 1−𝑒^−𝑡, there holds 1𝐸𝑓−𝑛^𝑛_𝑖=1𝑓𝑧_𝑖≤12𝜂^1−𝛼𝐸𝑓^𝛼+𝑐^′_𝑝𝜂+2𝑐𝑡𝑛^1/(2−𝛼)+18𝐵𝑡𝑛,∀𝑓∈ℱ, (3.11) where 𝜂∶=max𝑐^{(2−𝑝)/(4−2𝛼+𝑝𝛼)}𝑎𝑛^{2/(4−2𝛼+𝑝𝛼)},𝐵^{(2−𝑝)/(2+𝑝)}𝑎𝑛^2/(2+𝑝). (3.12)

Proposition 3.4. Under Assumption 2.3, for any 0<𝛿<1, one has with confidence at least 1−𝛿𝑆₂≤12ℰ𝜋𝑓^𝑡_𝛽−ℰ𝑓_𝜌+640𝑀²𝑐_𝑝,𝐾4𝑀𝜅𝛽^{𝑝2/(2+𝑝)}1log𝛿𝑛^{−2/(2+𝑝)}. (3.13)

Proof. From the definition of 𝑓^𝑡_𝛽, we have ‖𝑓^𝑡_𝛽‖_𝐾≤𝜅𝛽. Denote ℱ_𝜅𝛽=𝑔(𝑧)=𝑦−𝜋𝑓(𝑥)²−𝑦−𝑓_𝜌(𝑥)²∶𝑓∈ℬ_𝜅𝛽. (3.14) We can see that 𝐸𝑔=ℰ(𝜋(𝑓))−ℰ(𝑓_𝜌) and (1/𝑛)∑^𝑛_𝑖=1𝑔(𝑧_𝑖)=ℰ_𝐳(𝜋(𝑓))−ℰ_𝐳(𝑓_𝜌). Since ‖𝜋(𝑓)‖_∞≤𝑀 and |𝑓_𝜌(𝑥)|≤𝑀, we have 𝑔(𝑧)=𝜋𝑓(𝑥)−𝑓_𝜌(𝑥)𝜋𝑓(𝑥)−𝑦+𝑓_𝜌(𝑥)−𝑦≤8𝑀²,𝐸𝑔²=_𝒵𝜋𝑓(𝑥)−𝑓_𝜌(𝑥)²𝜋𝑓(𝑥)−𝑦+𝑓_𝜌(𝑥)−𝑦²𝑑𝜌≤16𝑀²𝐸𝑔. (3.15) For 𝑔₁,𝑔₂∈ℱ_𝜅𝛽, we have 𝑔₁(𝑧)−𝑔₂(𝑧)=𝑦−𝜋𝑓₁(𝑥)²−𝑦−𝜋𝑓₂(𝑥)²≤4𝑀𝜋𝑓₁(𝑥)−𝜋𝑓₂(𝑥𝑓)≤4𝑀₁(𝑥)−𝑓₂(𝑥). (3.16) Then, from Assumption 2.3, 𝒩_2,𝐳ℱ_𝜅𝛽,𝜖≤𝒩_2,𝐱ℬ_𝜅𝛽,𝜖4𝑀≤𝒩_2,𝐱ℬ₁,𝜖4𝑀𝜅𝛽≤𝑐_𝑝,𝐾4𝑀𝜅𝛽^𝑝𝜖^−𝑝. (3.17)
Applying Lemma 3.3 with 𝐵=𝑐=16𝑀² and 𝑎=𝑐_𝑝,𝐾(4𝑀𝜅𝛽)^𝑝, for any 𝛿∈(0,1) and for all 𝑔∈ℱ_𝜅𝛽, 1𝐸𝑔−𝑛^𝑛_𝑖=1𝑔𝑧_𝑖≤12𝐸𝑔+𝑐^′_𝑝16𝑀^{2(2−𝑝)/(2+𝑝)}𝑐_𝑝,𝐾4𝑀𝜅𝛽^𝑝𝑛^2/(2+𝑝)+320𝑀²log(1/𝛿)𝑛,≤12𝐸𝑔+640𝑀²𝑐_𝑝,𝐾4𝑀𝜅𝛽^{𝑝2/(2+𝑝)}1log𝛿𝑛^{−2/(2+𝑝)} (3.18) holds with confidence 1−𝛿. This completes the proof.

Different from the previous studies related with regularized framework [3–5], we introduce the estimate of hypothesis error ℰ_𝐳(𝑓^𝑡_𝛽)−ℰ_𝐳(𝑓^∗_𝛽) based on Theorem  4.2 in [11] for sequential greedy approximation.

Proposition 3.5. For a fixed sample 𝐳, one has ℰ_𝐳𝑓^𝑡_𝛽−ℰ_𝐳𝑓^∗_𝛽≤16𝛽²𝑡. (3.19)

The desired result in Theorem 2.4 can be derived directly by combining Propositions 3.2–3.5.

Acknowledgments

This work was supported partially by the National Natural Science Foundation of China under Grant no. 11001092, Humanities and Social Science Projects of the Ministry of Education of China (Program no. 11y3jc630197), and the Fundamental Research Funds for the Central Universities (Programs nos. 2011PY130, and 2011QC022).