Error Bounds for 𝑙^𝑝-Norm Multiple Kernel Learning with Least Square Loss

Abstract

The problem of learning the kernel function with linear combinations of multiple kernels has attracted considerable attention recently in machine learning. Specially, by imposing an 𝑙^𝑝-norm penalty on the kernel combination coefficient, multiple kernel learning (MKL) was proved useful and effective for theoretical analysis and practical applications (Kloft et al., 2009, 2011). In this paper, we present a theoretical analysis on the approximation error and learning ability of the 𝑙^𝑝-norm MKL. Our analysis shows explicit learning rates for 𝑙^𝑝-norm MKL and demonstrates some notable advantages compared with traditional kernel-based learning algorithms where the kernel is fixed.

1. Introduction

1.1. Overview of Multiple Kernel Learning

Kernel methods such as Support Vector Machines (SVMs) have been extensively applied to supervised learning tasks such as classification and regression. The performance of a kernel machine largely depends on the data representation via the choice of kernel function. Hence, one central issue in kernel methods is the problem of kernel selection; a great many approaches to selecting the right kernel have been studied in the literature [1–4] and other references therein.

We begin with reviewing the classical supervised learning setup. Let (𝑋,𝑑) be a compact metric space and 𝑌⊆ℝ, given a labeled sample 𝐳={(𝑥_𝑖,𝑦_𝑖)}^𝑚_𝑖=1⊆𝑍∶=𝑋×𝑌, sampled 𝑖.𝑖.𝑑. according to an unknown distribution 𝜌 supported on 𝑍, the goal is to estimate a real-valued function 𝑓_𝐳 depending on the sample, that generalizes well on new and unseen data. A widely used approach to estimate a function from empirical data consists in minimizing a regularization functional in a Hilbert space ℋ of real-valued functions: 𝑋→ℝ. Typically, a regularization scheme estimates 𝑓 as a minimizer of the functional ℰ_𝐳𝑓+𝜆Ω𝑓, (1.1) where ℰ_𝐳(𝑓)=(1/𝑚)∑^𝑚_𝑖=1𝑉(𝑓(𝑥_𝑖),𝑦_𝑖) is the empirical risk of hypothesis 𝑓, measured by a nonnegative loss function 𝑉∶ℝ×𝑌→ℝ⁺. In addition, Ω∶ℋ→ℝ is a regularizer and 𝜆>0 is a trade-off regularization parameter.

In this paper, we assume that ℋ is a reproducing kernel Hilbert space (RKHS) ℋ_𝐾 with kernel 𝐾, see [5]. Every kernel 𝐾 corresponds to a feature mapping Ψ_𝐾∶𝑋→ℋ_𝐾 satisfying 𝐾(𝑥,𝑦)=⟨Ψ_𝐾(𝑥),Ψ_𝐾(𝑦)⟩_𝐾, and each element of ℋ_𝐾 has the following form: 𝑓_𝑤(𝑥)=𝑤,Ψ_𝐾(𝑥)_𝐾,∀𝑤∈ℋ_𝐾. (1.2)

By restricting the regularization to be the form Ω(𝑓)=‖𝑓‖²_𝐾, there is a lot of studies from different perspectives such as statistics, optimal recovery and machine learning [6–9], and other references therein. Regularization in an RKHS has a number of attractive features, including the availability of effective error bounds and stability analysis relative to perturbations of the data (see Cucker and Smale [7]; Wu et al. [10]; Bousquet and Elisseeff [6]). Moreover, the optimization problem (1.1) in an RKHS can be reduced to seek for solution in a finite-dimensional space. Although it is simple to prove, this result shows that the variational problem (1.1) can be computational easily.

Because of their simplicity and generality, kernels and associated RKHS play an increasingly important role in Machine Learning, Pattern Recognition and Artificial Intelligence. When the kernel is fixed, an immediate concern is the choice of the regularization parameter 𝜆. This is typically solved by means of cross validation or generalized cross validation [11]. However, the performance of kernel methods critically relies on the choice of the kernel function. A natural question is how to choose the optimal kernel in a collection of candidate kernels.

Kernel learning can range from the width parameter selection of Gaussian kernels [9, 12] to obtaining an optimal linear combination from a set of finite candidate kernels. The latter is often referred to as multiple kernel learning in machine learning and nonparametric group Lasso in statistics [13]. Lanckriet et al. [3] pioneered work on MKL and proposed a semidefinite programming approach to automatically learn a linear combination of candidate kernels for the cases of SVMS. To improve computation efficiency, the multikernel class further is restricted to only convex combinations of kernels [2, 14, 15]. Most learning kernel algorithms are based on considering linear kernel mixtures 𝐾_𝜃=∑𝜃_𝑘𝐾_𝑘,(𝜃_𝑘≥0) with a prescribed kernels 𝐾₁,…,𝐾_𝑀. For notational simplicity, we will frequently use Ψ_𝑘 instead of the standard feature Ψ_𝐾𝑘. Compared to (1.1), the primal model for learning with multiple kernels is extended to 𝑓_𝑤,𝜃(𝑥)=^𝑀_𝑘=1𝜃_𝑘𝑤_𝑘,Ψ_𝑘(𝑥)_𝐾𝑘,∀𝑤=𝑤₁,…,𝑤_𝑀,where𝑤_𝑘∈ℋ_𝐾𝑘. (1.3)

In this paper, we mainly focus on the 𝑙^𝑝-norm MKL, consisting in minimizing the regularized empirical risk with respect to the optimal kernel mixture ∑^𝑀_𝑘=1𝜃_𝑘𝐾_𝑘, in addition to 𝑙^𝑝-regularizer on 𝜃 to avoid overfitting. This leads to the following optimization problem: inf𝑤_{,𝜃∶𝜃≥0}1𝑚^𝑚_𝑖=1𝑉^𝑀_𝑘=1𝜃_𝑘𝑤_𝑘,Ψ_𝑘𝑥_𝑖𝐾𝑘,𝑦_𝑖+𝜆^𝑀_𝑘=1𝑤_𝑘²_𝐾𝑘+𝜇^𝑀_𝑘=1𝜃_𝑘^𝑝. (1.4) This scheme was introduced in [2] and the existence of its minimum has been discussed in [4].

The optimization problem subsumes state-of-the-art approaches to multiple kernel learning, covering sparse and nonsparse MKL by arbitrary 𝑙^𝑝-norm regularization (1≤𝑝≤∞) on the mixing coefficients as well as the incorporation of prior knowledge by allowing for nonisotropic regularizer. Kloft et al. [2] developed two efficient interleaved optimization strategies for the 𝑙^𝑝-norm multiple kernel learning, and this interleaved optimization is much faster than the commonly used wrapper approaches, as demonstrated on real-world problems from computational biology. An analysis of this model, based on Rademacher complexities, was first developed by Cortes et al. [1]. Later improved rates of convergence were derived based on the theory of local Rademacher complexities [15]. However, the estimate on local Rademacher complexities with 1≤𝑝<2 strictly depends on no-correlation assumption of the 𝑀 different features, which is too strong condition in theory and practice. In this paper, we employ the notion of empirical covering number to present a theoretical analysis of its generalization error. Besides no-correlation condition is not necessary, empirical covering number is one tight upper bound of local Rademacher complexities [16], also independent of the underlying distribution. We will see that some satisfying learning rates are established when the regularization parameter is appropriately chosen. The interaction between the sample error and the approximation error plays an important role in our analysis, and our new methodology mainly depends on the complexity of hypothesis class measured by empirical covering number and the regularity of a target function.

It should be pointed out that the Tikhonov Regularization in (1.4) has two regularization parameter (𝜆, 𝜇), which may be hard to deal with in practice. Fortunately, an alternative approach has been studied by Rakotomamonjy et al. [14] and Kloft et al. [2]. More precisely, this approach employs the regularizer ‖𝜃‖_𝑙𝑝≤1 as an additional constraint into the optimization problem. By substituting 𝑤_𝑘 for √𝜃𝑤_𝑘, they arrive at the following problem: inf𝑤_{,𝜃∶𝜃≥0}1𝑚^𝑚_𝑖=1𝑉^𝑀_𝑘=1𝑤_𝑘,Ψ_𝑘𝑥_𝑖𝐾𝑘,𝑦_𝑖+𝜆2^𝑀_𝑘=1𝑤_𝑘²_𝐾𝑘𝜃_𝑘subjectto‖𝜃‖_𝑙𝑝≤1. (1.5)

1.2. Algorithm and Main Consequence

The following Lemma (see [4]) indicates that the above multikernel class can equivalently be represented as a block-norm regularized linear class in the product Hilbert space ℋ=ℋ_𝐾1×⋯×ℋ_𝐾𝑀.

Lemma 1.1. If 𝑝>0, 𝑞=1+(1/𝑝), and {𝑎_𝑗,𝑗∈ℕ_𝑛}⊆ℝ, then min_{𝑗∈ℕ𝑛}𝑎²_𝑗𝜆_𝑗^1/2∶𝜆_𝑙≥0,𝑙∈ℕ_𝑛,_{𝑗∈ℕ𝑛}𝜆^𝑝_𝑗≤1=_{𝑗∈ℕ𝑛}𝑎_𝑗^2/𝑞𝑞/2, (1.6) and the equality occurs for ∑_{𝑗∈ℕ𝑛}|𝑎_𝑗|>0 at 𝜆_𝑗𝑎∶=^{𝑗2/(𝑝+1)}∑_{𝑗∈ℕ𝑛}𝑎_𝑗^{2𝑝/(𝑝+1)1/𝑝}. (1.7)

Hence, Lemma 1.1 can be applied to define the feature mapping: Ψ∶𝑥∈𝑋→(Ψ₁(𝑥),...,Ψ_𝑀(𝑥))∈ℋ_𝐾 associated with a kernel 𝐾; the class of functions defined above coincides with 𝐻_{𝑝,𝐷,𝑀}=𝑓_𝑤∶𝑥⟶⟨𝑤,Ψ(𝑥)⟩_ℋ𝐾∣𝑤=𝑤₁,…,𝑤_𝑀,‖𝑤‖_2,𝑞≤𝐷, (1.8) when 𝑝∈[1,∞], 𝑞∈[1,2] holds from 𝑞=2𝑝/(𝑝+1). The 𝑙_2,𝑞-norm is defined here as ‖𝑤‖_2,𝑞=(∑^𝑀_𝑗=1‖𝑤_𝑗‖^𝑞_𝐾𝑗)^1/𝑞. For simplicity, we write ‖𝑓_𝑤‖_2,𝑞=‖𝑤‖_2,𝑞. Clearly learning the complexity of (1.8) will be greater than one that is based on a single kernel only, further it provides greater learning ability while the computational complexity increases accordingly. The sample complexity of the above hypothesis space has been studied by Cortes et al. [1] and Kloft and Blanchard [15]. Thus the primal MKL optimization problem (1.5) is equivalent to the following regularization scheme, which is the primary object of investigation in this paper 𝑓_𝐳=𝑤_𝐳,Ψ(𝑥)_ℋ𝐾,where𝑤_𝐳=argmin_{𝑤∈ℋ𝐾}1𝑚^𝑚_𝑖=1𝑉𝑤,Ψ𝑥_𝑖ℋ𝐾,𝑦_𝑖+𝜆‖𝑤‖²_2,𝑞. (1.9) Here we use the symbol “min” instead of “inf,” since (1.4) is equivalent to (1.9) and the solution of (1.4) exists and is unique. Remark that the above algorithm is a standard regularized empirical risk minimization; this implies that 𝑙^𝑝-norm multiple kernel learning scheme can be free of over-fitting, a phenomenon which occurs when the empirical error is zero but the expected error in far from zero.

In the following, we assume that {𝐾_𝑗}_{𝑗=1,...,𝑀} is uniformly bounded, that is, 𝜅=sup_{𝑗∈{1,...,𝑀}}sup_𝑥∈𝑋𝐾_𝑗(𝑥,𝑥)<∞. (1.10) Also suppose that each 𝐾_𝑗 is continuous. In other words, each 𝐾_𝑗 is a Mercer kernel with bound 𝜅; we refer to [17] for more properties and discussions on Mercer kernel.

In this paper, we only focus on the least square loss: 𝑉(𝑓(𝑥),𝑦)=(𝑓(𝑥)−𝑦)². Accordingly, the target function is given by 𝑓_𝜌(𝑥)=argmin𝔼𝑓(𝑥)−𝑦²=_𝑌𝑦𝑑𝜌(⋅∣𝑥),∀𝑥∈𝑋, (1.11) where we denote by 𝜌(⋅∣𝑥) the conditional distribution of 𝜌. Through this paper we assume that 𝜌(⋅∣𝑥) is supported on [−𝑇,𝑇], it follows that |𝑓_𝜌(𝑥)|≤𝑇 for 𝑥∈𝑋 almost surely. Since the learner 𝑓_𝐳 may be much larger than 𝑓_𝜌, it is natural to apply a projection operator on 𝑓_𝐳, which was introduced into learning algorithms to improve learning rates.

Definition 1.2. The projection operator 𝜋 is defined on the space of measurable functions 𝑓∶𝑋→ℝ as 𝜋𝑓(𝑥)=𝑇,if𝑓(𝑥)𝑓(𝑥),>𝑇,if𝑓(𝑥)≤𝑇,−𝑇,if𝑓(𝑥)<−𝑇, (1.12) where 𝑇>0 is called the projection level.

The target of error analysis is to understand how 𝜋(𝑓_𝐳) approximates the regression function 𝑓_𝜌. More precisely, we aim to estimate the excess generalization errorℰ𝜋𝑓_𝐳−ℰ𝑓_𝜌 (1.13) for the 𝑙^𝑝-norm MKL algorithm (1.4), where ℰ(𝑓)=𝔼(𝑓(𝑥)−𝑦)² denotes the expect error of 𝑓.

To show some ideas of our error analysis, we first state learning rates of (1.4) in a special case when 𝑓_𝜌∈ℋ_𝐾 and 𝐾 is 𝐶^∞ on 𝑋⊂ℝ^𝑛.

Theorem 1.3. Let 𝑓_𝐳 be defined by (1.9). Assume 𝐾 is 𝐶^∞ on 𝑋⊂ℝ^𝑛 and 𝑓_𝜌∈ℋ_𝐾. For any 0<𝛿<1 and 𝜖>0, with confidence 1−𝛿, there holds ℰ𝜋𝑓_𝐳−ℰ𝑓_𝜌=𝜋𝑓_𝐳−𝑓_𝜌²_𝜌≤2𝐶log𝛿1𝑚^1−𝜖,with1𝜆=𝑚^1−2𝜖, (1.14) where 𝐶 is some constant independent of 𝑚 or 𝛿.

Theorem 1.3 can be viewed as a corollary of our main result presented in Section 5. It can be arbitrary close to 𝒪(𝑚⁻¹) by choosing 𝜖 to be small enough, which is the best convergence rate in learning theory literature.

2. Key Error Analysis

Our main result is about learning rates of (1.4) stated under conditions on the approximation ability of ℋ_𝐾 with respect to 𝑓_𝜌 and capacity of ℋ_𝐾.

The approximation ability of the hypothesis space ℋ_𝐾 with respect to 𝑓_𝜌 in the space 𝐿²_𝜌𝑋 is reflected by regularization error.

Definition 2.1. The regularization error of the triple (ℋ_𝐾,𝑓_𝜌,𝜌_𝑋) is defined as 𝒜_𝑞(𝜆)=min_{𝑓∈ℋ𝐾}ℰ𝑓−ℰ𝑓_𝜌+𝜆𝑓²_2,𝑞,𝜆>0. (2.1) We will assume that for some 0<𝛽≤1 and 𝐶_𝛽>0, 𝒜_𝑞(𝜆)≤𝐶_𝛽𝜆^𝛽. (2.2)

Remark 2.2. Our assumption implies that when 𝑓_𝜌 is replaced by ℋ_𝐾, 𝒜_𝑞(𝜆) tends to zero by polynomial order decay as 𝜆 goes to zero. Note [7] that 𝒜_𝑞(𝜆)=𝑜(𝜆) would imply 𝑓_𝜌=0. So 𝛽=1 in (2.2) is the best we can expect. This case is equivalent to 𝑓_𝜌∈ℋ_𝐾 when ℋ_𝐾 is dense in 𝐿²_𝜌𝑋, see [18]. Assumption (2.2) with 0<𝛽<1 can be characterized in terms of interpolation spaces [7].

If 𝜌_𝑋 is the Lebesgue measure on 𝑋 and the target function 𝑓_𝜌∈𝐻^𝑠, a Sobolev space with power 𝑠. When Gaussian kernel (𝐺_𝜎(𝑥,𝑦)=exp(−𝜎‖𝑥−𝑦‖²)) is taken with a fixed variance 𝜎, a polynomial decay of 𝒜_𝑞(𝜆) is impossible. However, Example 1 of [19] successfully obtains a polynomial decay under the multikernel setting, allowing for varying variances of Gaussian kernels. This shows that multikernel learning can improve the approximation power and learning ability. More interestingly, we will take a special example to show the impact of the multikernel class on the regularization error in Section 5 below. In particular, a proper multikernel class can be applied to improve the regularization error if the regularity of 𝑓_𝜌 is rather high.

Next we define the truncated sample error as 𝒮_𝐳𝜆,𝑓,𝑇=ℰ𝜋𝑓_𝐳−ℰ_𝑧𝜋𝑓_𝐳+ℰ_𝐳𝑓−ℰ𝑓, (2.3) and the sample error as 𝒮_𝐳𝜆,𝑓=ℰ𝑓_𝐳−ℰ_𝐳𝑓_𝐳+ℰ_𝐳𝑓−ℰ𝑓. (2.4)

The function 𝑓 in the above equation can be arbitrarily chosen; however, only proper choices lead to good estimates of the regularization error. A good choice is 𝑓=𝑓_𝜆 where 𝑓_𝜆=argmin_{𝑓∈ℋ𝐾}ℰ𝑓+𝜆𝑓²_2,𝑞. (2.5)

A useful approach for regularization schemes with sample independent hypothesis spaces such as RKHS is an error decomposition, which decomposes the total error ℰ(𝜋(𝑓_𝐳))−ℰ(𝑓_𝜌) into the sum of the truncated sample error and the regularization error stated as follows.

Proposition 2.3. Let 𝑓_𝜆 be defined by (2.5); there holds ℰ𝜋𝑓_𝐳−ℰ𝑓_𝜌≤𝒮_𝐳𝜆,𝑓_𝜆,𝑇+𝒜_𝑞(𝜆). (2.6)

Proof. We can decompose ℰ(𝜋(𝑓_𝐳))−ℰ(𝑓_𝜌) into ℰ𝜋𝑓_𝐳−ℰ_𝐳𝜋𝑓_𝐳+ℰ_𝐳𝜋𝑓_𝐳+𝜆𝑓_𝐳²_2,𝑞−ℰ_𝐳𝑓_𝜆+𝜆𝑓_𝜆²_2,𝑞+ℰ_𝐳𝑓_𝜆−ℰ𝑓_𝜆+ℰ𝑓_𝜆−ℰ𝑓_𝜌+𝜆𝑓_𝜆²_2,𝑞−𝜆𝑓_𝐳²_2,𝑞. (2.7) To bound the second term, by Definition of 𝑓_𝐳, ℰ_𝐳(𝜋(𝑓_𝐳))+𝜆‖𝑓_𝐳‖²_2,𝑞 can be bounded by ℰ_𝐳𝑓_𝐳+𝜆𝑓_𝐳²_2,𝑞≤ℰ_𝐳𝑓_𝜆+𝜆𝑓_𝜆²_2,𝑞, (2.8) since |𝜋(𝑓)(𝑥)−𝑦|≤|𝑓(𝑥)−𝑦| holds for any function 𝑓 on 𝑍. The conclusion follows by combining these two inequalities.

3. Estimation on Sample Error

We are in a position to estimate the sample error 𝒮_𝐳(𝜆,𝑓_𝜆,𝑇). The sample error 𝒮_𝐳(𝜆,𝑓_𝜆,𝑇) can be written as ℰ𝜋𝑓_𝐳−ℰ𝑓_𝜌−ℰ_𝐳𝜋𝑓_𝐳−ℰ_𝐳𝑓_𝜌+ℰ_𝐳𝑓_𝜆−ℰ_𝐳𝑓_𝜌−ℰ𝑓_𝜆−ℰ𝑓_𝜌∶=𝒮₁+𝒮₂. (3.1)

𝒮₂ can be easily bounded by applying the following one-side Bernstein-type probability inequality.

Lemma 3.1. Let 𝜉 be a random variable on a probability space 𝑍 with variance 𝜎² satisfying |𝜉−𝔼(𝜉)|≤𝑀_𝜉 for some constant 𝑀_𝜉. Then for any 0<𝛿<1, we have 1𝑚^𝑚_𝑖=1𝜉𝑧_𝑖−𝔼(𝜉)≤2𝑀_𝜉log(1/𝛿)+3𝑚2𝜎²log(1/𝛿)𝑚. (3.2)

Proposition 3.2. Define the random variable 𝜉₂(𝑧)=(𝑓_𝜆(𝑥)−𝑦)²−(𝑓_𝜌(𝑥)−𝑦)². For every 0<𝛿<1, with confidence at least 1−𝛿/2, there holds 𝒮₂=1𝑚^𝑚_𝑖=1𝜉₂𝑧_𝑖−𝔼𝜉₂≤𝒜_𝑞(𝜆)1+3𝜅²𝑀^2/𝑞∗log(2/𝛿)+𝑚𝜆6𝑇²+log(2/𝛿)𝑚. (3.3)

Proof. From the definition of 𝒜_𝑞(𝜆), we see that 𝜆𝑓_𝜆²_2,𝑞≤ℰ𝑓_𝜆−ℰ𝑓_𝜌+𝜆𝑓_𝜆²_2,𝑞=𝒜_𝑞(𝜆). (3.4) Note that 𝑓_𝜆(𝑥)=⟨𝑤_𝜆,Ψ(𝑥)⟩ for some 𝑤_𝜆=(𝑤_𝜆⁽¹⁾,…,𝑤_𝜆^(𝑀))∈ℋ_𝐾, by the Cauchy-Schwarz inequality (𝐶.−𝑆.), we have for any 𝑥∈𝑋: 𝑤_𝜆,Ψ(𝑥)=^𝑀_𝑗=1𝑤_𝜆^(𝑗),Ψ_𝑗(𝑥)≤^{𝐶.−𝑆.𝑀}_𝑗=1𝑤_𝜆^(𝑗)_𝐾𝑗Ψ_𝑗(𝑥)_{𝐾𝑗Ḧ‌older}≤𝑤_𝜆2,𝑞^𝑀_𝑗=1Ψ_𝑗(𝑥)^𝑞∗_𝐾𝑗^1/𝑞∗, (3.5) where (1/𝑞)+(1/𝑞^∗)=1. Using Assumption (1.10), it follows that 𝑓_𝜆∞≤𝜅𝑀^1/𝑞∗𝑤_𝜆2,𝑞≤𝜅𝑀^1/𝑞∗𝒜_𝑞(𝜆)𝜆. (3.6)
Observe that 𝜉₂(𝑧)=𝑓_𝜆(𝑥)−𝑓_𝜌(𝑥)𝑓_𝜆(𝑥)+𝑓_𝜌(𝑥)−2𝑦. (3.7) Since that |𝑓_𝜌(𝑥)|≤𝑇 almost surely, we have 𝜉₂≤𝑓_𝜆∞+𝑇𝑓_𝜆∞+3𝑇≤𝑐∶=𝜅𝑀^1/𝑞∗𝒜_𝑞(𝜆)𝜆+3𝑇². (3.8) Hence |𝜉₂−𝔼(𝜉₂)|≤𝑀_𝜉2∶=2𝑐. Moreover, 𝔼(𝜉₂)² equals 𝔼𝑓_𝜆(𝑥)−𝑓_𝜌(𝑥)²𝑓_𝜆(𝑥)+𝑓_𝜌(𝑥)−2𝑦²≤𝑓_𝜆∞+3𝑇²𝑓_𝜆−𝑓_𝜌²_𝜌, (3.9) which implies that 𝜎²(𝜉₂)≤𝔼(𝜉²₂)≤𝑐𝒜_𝑞(𝜆). The desired result follows from Lemma 3.1.

Next we estimate the first term 𝒮₁. It is more difficult to deal with because it involves a set of random variables 𝑓_𝐳 varying with 𝐳, requiring to consider the functional complexity. For this purpose, we introduce the notion of empirical covering numbers, which often lead to sharp error estimates [16].

Definition 3.3. Let (𝒰,𝑑) be a pseudometric space and 𝑆⊂𝒰. For every 𝜖>0, the covering number 𝒩(𝑆,𝜖,𝑑) of 𝑆 with respect to 𝜖 and 𝑑 is defined as the minimal number of balls of radius 𝜖 whose union covers 𝑆, that is, 𝒩(𝑆,𝜖,𝑑)=min𝑙∈ℕ∶𝑆⊂^𝑙_𝑗=1𝐵𝑠_𝑗,𝜖forsome𝑠_𝑗^𝑙_𝑗=1⊂𝒰, (3.10) where 𝐵(𝑠_𝑗,𝜖)={𝑠∈𝒰∶𝑑(𝑠,𝑠_𝑗)≤𝜖} is a ball in 𝒰.

The 𝑙²-empirical covering number of a function set is defined by means of the normalized 𝑙²-metric 𝑑₂ on the Euclidian space ℝ^𝑘 given by 𝑑₂()=1𝐚,𝐛𝑘^𝑘_𝑖=1𝑎_𝑖−𝑏_𝑖^21/2for𝐚=𝑎_𝑖^𝑘_𝑖=1,𝐛=𝑏_𝑖^𝑘_𝑖=1∈ℝ^𝑘. (3.11)

Definition 3.4. Let ℱ be a set of function on 𝑋, 𝐱=(𝑥_𝑖)^𝑘_𝑖=1⊂𝑋^𝑘 and ℱ|_𝐱={(𝑓(𝑥_𝑖))^𝑘_𝑖=1∶𝑓∈ℱ}⊂ℝ^𝑘. Set 𝒩_2,𝐱(ℱ,𝜖)=𝒩(ℱ|_𝐱,𝜖,𝑑₂). The 𝑙²-empirical covering number of ℱ is defined by 𝒩₂(ℱ,𝜖)=sup_𝑘∈ℕsup_{𝐱∈𝑋𝑘}𝒩_2,𝐱(ℱ,𝜖),𝜖>0. (3.12)

Denote by 𝐵_𝑅 the ball of radius 𝑅 with 𝑅>0, 𝐵_𝑅={𝑓∈ℋ_𝐾∶‖𝑓‖_𝐾≤𝑅}. We need the following capacity assumption on ℋ_𝐾.

Assumption 3.5. There exists an exponent 𝜐, with 0<𝜐<2 and a constant 𝐶_𝜐,𝐾>0 such that log𝒩₂𝐵₁,𝜖≤𝐶_𝜐,𝐾𝜖^−𝜐,∀𝜖>0, (3.13) where 𝐵₁ is the unit ball of ℋ_𝐾 defined as above.

For any function 𝑓_𝑤=⟨𝑤,Ψ(𝑥)⟩_ℋ𝐾, by the hölder inequality, we have 𝑓_𝑤𝐾=^𝑀_𝑗=1𝑤_𝑗²_𝐾𝑗^1/2≤𝑀^1/𝑞∗𝑓_𝑤2,𝑞, (3.14) and it follows from (3.13) log𝒩₂𝐵^𝑞₁,𝜖≤𝐶_𝜐,𝐾𝑀^𝜐/𝑞∗𝜖^−𝜐, (3.15) where 𝐵^𝑞₁ is called the generalized unit ball of ℋ_𝐾 associated with 𝑞, defined by 𝐵^𝑞₁=𝑓∈ℋ_𝐾,𝑓_2,𝑞≤1. (3.16)

Note that for any function set ℱ⊆𝐶(𝑋), the empirical covering number 𝒩_2,𝐱(ℱ,𝜖) is bounded by 𝒩(ℱ,𝜖), the (uniform) covering number of ℱ under the metric ‖⋅‖_∞, since 𝑑₂(𝑓,𝑔)≤‖𝑓−𝑔‖_∞. It was shown in [20] that the quantity log(𝒩(𝐵₁,𝜖))≤𝐶₀(1/𝜖)^𝑠 holds for some 𝐶₀>0 if 𝐾 is 𝐶^2𝑛/𝑠 on a subset of ℝ^𝑛, hence log(𝒩₂(𝐵₁,𝜖))≤𝐶₀(1/𝜖)^𝑠 also holds. In particular, 𝑠 is arbitrarily small for a 𝐶^∞ kernel ( such as Gaussian kernel). Now we give a concrete example in ℝ^𝑛 to reveal relationship between the regularity of function class and its corresponding empirical covering number.

Example 3.6. Let 𝑋 be a bounded domain in ℝ^𝑛 and 𝐻^𝑠 the Sobolev space of index 𝑠. When 𝑠>𝑛, the classical Embedding Theorem tells us that 𝐻^𝑠(𝑋) is an RKHS and its unit ball 𝐵₁ is embedded in a finite ball of the function space 𝐶^{𝑠−(𝑛/2)−𝜁}(𝑋) with inclusion bounded where 0<𝜁<𝑠−𝑛. From the classical bounds for covering numbers of the unit ball of 𝐶^{𝑠−(𝑛/2)−𝜁}(𝑋), we see that log𝒩₂𝐵₁,𝜖≤𝐶_𝑠𝜖^{−𝑛/(𝑠−𝑛/2−𝜁)},∀𝜖>0. (3.17) Hence Assumption (3.13) below holds with 𝜐=𝑛/(𝑠−𝑛/2−𝜁)<2.

Our concentration estimate for the sample error dealing with 𝒮₁ is based on the following concentration inequality, which can be found in [12].

Lemma 3.7. Let ℱ be a class of measurable functions on 𝑍. Assume that there are constants 𝐵,𝑐>0 and 𝜂∈[0,1] such that ‖𝑓‖_∞≤𝐵 and 𝔼𝑓²≤𝑐(𝔼𝑓)^𝜂 for every 𝑓∈ℱ. If for some 𝑎>0 and 𝜐∈(0,2), log𝒩₂(ℱ,𝜖)≤𝑎𝜖^−𝜐,∀𝜖>0, (3.18) then there exists a constant 𝑐_𝜐 such that for any 𝑡>0, with confidence at least 1−𝑒^−𝑡, there holds 1𝔼𝑓−𝑚^𝑚_𝑖=1𝑓𝑧_𝑖≤12𝛾^1−𝜂𝔼𝑓^𝜂+𝑐_𝜐𝛾+2𝑐𝑡𝑚^1/(2−𝜂)+18𝐵𝑡𝑚,∀𝑓∈ℱ, (3.19) where 𝛾∶=max𝑐^{(2−𝜐)/(4−2𝜂+𝜐𝜂)}𝑎𝑚^{2/(4−2𝜂+𝜐𝜂)},𝐵^{(2−𝜐)/(2+𝜐)}𝑎𝑚^2/(2+𝜐). (3.20)

Denote the set of function ℱ^𝑞_𝑅 with 𝑅>0, where ℱ^𝑞_𝑅=𝜋𝑓(𝑥)−𝑦²−𝑓_𝜌(𝑥)−𝑦²∶𝑓∈𝐵^𝑞_𝑅. (3.21)

Proposition 3.8. If 𝐵^𝑞₁ satisfies the capacity condition (3.13) with some 0<𝜐<2, then for any 𝛿∈(0,1), with confidence 1−𝛿/2, there holds 𝒮₁≤12𝜋𝑓_𝐳−𝑓_𝜌²_𝜌+80𝑇²log(2/𝛿)𝑚+𝐶_𝜐,𝑇1𝜆^{𝜐/(2+𝜐)}1𝑚^2/(2+𝜐) (3.22) with constant 𝐶_𝜐,𝑇=4𝑐_𝜐(𝐶_𝜐,𝐾𝑇²)^{𝜐/(2+𝜐)}𝑀^{2𝜐/(2+𝜐)𝑞∗}.

Proof. Consider the set ℱ^𝑞_𝑅. Each function 𝑔∈ℱ^𝑞_𝑅 can be expressed as 𝑔(𝑧) = (𝑦−𝜋(𝑓)(𝑥))²−(𝑦−𝑓_𝜌(𝑥))² with some 𝑓∈𝐵^𝑞_𝑅. Then 𝔼𝑔=ℰ(𝜋(𝑓))−ℰ(𝑓_𝜌)=‖𝜋(𝑓)−𝑓_𝜌‖²_𝜌 and (1/𝑚)∑^𝑚_𝑖=1𝑔(𝑧_𝑖) = ℰ_𝐳(𝜋(𝑓))−ℰ_𝐳(𝑓_𝜌). Note that 𝑔(𝑧)=𝜋𝑓(𝑥)−𝑓_𝜌(𝑥)𝜋𝑓(𝑥)+𝑓_𝜌(𝑥)−2𝑦. (3.23) Since |𝜋(𝑓)(𝑥)|≤𝑇 and |𝑓_𝜌(𝑥)|≤𝑇 for any 𝑥∈𝑋, we see that for any 𝑧∈𝑍, 𝑔(𝑧)≤8𝑇²,𝔼𝑔²=_𝑍𝜋𝑓(𝑥)−𝑓_𝜌(𝑥)²𝜋𝑓(𝑥)+𝑓_𝜌(𝑥)−2𝑦²≤4𝑇²𝔼𝑔. (3.24) On the other hand, for any 𝑔₁,𝑔₂∈ℱ^𝑞_𝑅 at point 𝑧=(𝑥,𝑦), we have 𝑔₁(𝑧)−𝑔₂(𝑧)≤4𝑇𝑓₁(𝑥)−𝑓₂(𝑥), (3.25) since the projector operator 𝜋 is a contractive map. It follows that 𝒩_2,𝐳ℱ^𝑞_𝑅,𝜖≤𝒩_2,𝐱𝐵^𝑞_𝑅,𝜖4𝑇≤𝒩_2,𝐱𝐵^𝑞₁,𝜖.4𝑇𝑅 (3.26) It follows from the capacity condition (3.15) log𝒩₂ℱ^𝑞_𝑅,𝜖≤𝐶_𝜐,𝐾4𝑇𝑅𝑀^{1/𝑞∗𝜐}𝜖^−𝜐. (3.27)
Applying Lemma 3.7 with 𝐵=𝑐=4𝑇², 𝜂=1, and 𝑎=𝐶_𝜐,𝐾(4𝑇𝑅𝑀^1/𝑞∗)^𝜐, we see that for any 𝛿∈(0,1), with confidence 1−𝛿/2, there holds 1𝔼𝑔−𝑚^𝑚_𝑖=1𝑔𝑧_𝑖≤12𝔼𝑔+80𝑇²log(2/𝛿)𝑚+𝐶_𝜐,𝑇𝑅^{2𝜐/(2+𝜐)}1𝑚^2/(2+𝜐),∀𝑓∈ℱ^𝑞_𝑅. (3.28) Besides, following the definition of 𝑓_𝐳 (1.9), we have 𝜆𝑓_𝐳²_2,𝑞=𝜆𝑤≤_𝐳²_2,𝑞^𝑤=01𝑚^𝑚_𝑖=1𝑦²_𝑖≤𝑇², (3.29) that is, ‖𝑓_𝐳‖_2,𝑞≤𝑇/√𝜆. Hence we can replace 𝑅 with 𝑇/√𝜆. Thus we derive our desired result.

4. Total Learning Rates

We are now in a position to obtain the learning rates of projected algorithm (1.9). Main results of this paper will be presented in Theorem 4.1.

Following the error decomposition scheme in Proposition 2.3 and combining Propositions 3.2 and 3.8, we derive the following bounds on the total error.

Theorem 4.1. Suppose that 𝐵^𝑞₁ satisfies the capacity condition (3.13) with some 0<𝜐<2, and 𝒜_𝑞(𝜆)≤𝐶_𝛽𝜆^𝛽. For any 𝛿∈(0,1), with confidence 1−𝛿, there holds 𝜋(𝑓_𝐳)−𝑓_𝜌²_𝜌≤𝐶^′2log𝛿1𝑚^{min{2𝛽/(2𝛽+(1+𝛽)𝜐),𝛽}},bytaking1𝜆=𝑚^{min{2/(2𝛽+(1+𝛽)𝜐),1}}, (4.1) where 𝐶′=3𝜅²𝑀^2/𝑞∗+86𝑇²+1+𝐶_𝜐,𝑇2^{𝜐/(2+𝜐)}+𝐶_𝛽 and 𝐶_𝜐,𝑇 is defined as in Proposition 3.8.

Proof. Following Propositions 3.2 and 3.8, with confidence at least 1−𝛿, (1/2)‖𝜋(𝑓_𝐳)−𝑓_𝜌‖²_𝜌 can be bounded by 3𝜅²𝑀^2/𝑞∗𝒜_𝑞(𝜆)2𝑚𝜆log𝛿+86𝑇²+1log(2/𝛿)𝑚+𝐶_𝜐,𝑇2𝜆^{𝜐/(2+𝜐)}1𝑚^{𝜐/(2+𝜐)}+2𝒜_𝑞(𝜆). (4.2) Firstly, we set 𝒜_𝑞(𝜆)𝑚𝜆=𝒜_𝑞(𝜆), (4.3) which implies that 𝜆=(1/𝑚). On the other hand, from the assumption 𝒜_𝑞(𝜆)≤𝐶_𝛽𝜆^𝛽, we set 2𝜆^{𝜐/(2+𝜐)}1𝑚^2/(2+𝜐)=𝜆^𝛽,thatis,𝜆=𝑚^{−(2/(2𝛽+(1+𝛽)𝜐))}. (4.4) Hence our assertion follows by taking 𝜆=(1/𝑚)^{min{2/(2𝛽+(1+𝛽)𝜐),1}}.

Proof of Theorem 1.3. When 𝐾∈𝐶^∞, it follows that condition (3.13) holds for arbitrary small 𝜐>0. Moreover, 𝑓_𝜌∈ℋ_𝐾 implies that 𝛽=1, the conclusion follows easily from Theorem 4.1.
Our learning rates below in Corollary 4.2 will be achieved under the regularity assumption on the regression function that 𝑓_𝜌 lies in the range of 𝐿^𝑟_𝐾 for some 𝑟>0. Given any kernel 𝐾, 𝐿_𝐾 is the integral operator on 𝐿²_𝜌𝑋 defined by 𝐿_𝐾𝑓(𝑥)=_𝑋𝐾𝑥,𝑦𝑓𝑦𝑑𝜌_𝑋𝑦,𝑥∈𝑋,𝑓∈𝐿²_𝜌𝑋. (4.5) The operator 𝐿_𝐾 is linear, compact, positive and can be also regarded as a self-adjoint operator on ℋ_𝐾. Hence the fractional power operator 𝐿^𝑟_𝐾∶𝐿²_𝜌𝑋→𝐿²_𝜌𝑋 is well defined and is given by 𝐿^𝑟_𝐾𝑓=_𝑘𝜆^𝑟_𝑘𝑓,𝜑_{𝑘𝐿2𝜌𝑋}𝜑_𝑘,𝑓∈𝐿²_𝜌𝑋(𝑋), (4.6) where {𝜆_𝑘}_𝑘 are eigenvalues of the operator 𝐿_𝐾 arranged in a decreasing order and {𝜑_𝑘}_𝑘 are the corresponding eigenfunctions, which form an orthonormal basis of 𝐿²_𝜌𝑋. In fact, the image of 𝐿^𝑟_𝐾 is contained in ℋ_𝐾 if 𝑟≥1/2. So 𝐿_𝐾^−𝑟𝑓_𝜌∈𝐿²_𝜌𝑋 indicates that 𝑓_𝜌 lies in the range of 𝐿^𝑟_𝐾, measuring the regularity of the regression function.

Corollary 4.2. Suppose that 𝐵^𝑞₁ satisfies the capacity condition (3.13) with some 0<𝜐<2, and 𝐿_𝐾^−𝑟𝑓_𝜌∈𝐿²_𝜌𝑋 (0<𝑟≤1). For any 0<𝛿<1, with confidence 1−𝛿, there holds 𝜋(𝑓_𝐳)−𝑓_𝜌²_𝜌≤2𝐶log𝛿1𝑚^{min{2𝛽/(2𝛽+(1+𝛽)𝜐),𝛽}},with𝛽=min{2𝑟,1}, (4.7) where 𝐶 is some constant independent of 𝑚 or 𝛿.

Proof. Recall a result from [18], if 𝐿_𝐾^−𝑟𝑓_𝜌∈𝐿²_𝜌𝑋 (0<𝑟≤1/2), there holds 𝒜₂(𝜆)=𝑓_𝜆−𝑓_𝜌²_𝜌+𝜆𝑓_𝜆²_𝐾≤𝜆^2𝑟𝐿_𝐾^−𝑟𝑓_{𝜌𝐿2𝜌𝑋}. (4.8) If 𝑟≥1/2, this shows that 𝑓_𝜌∈ℋ_𝐾 as mentioned above, then we have 𝑓_𝜆=𝑓_𝜌 and 𝒜₂(𝜆)≤𝜆𝑓_𝜌²_𝐾. (4.9) Hence for any 𝑟∈[0,1], we have 𝐴₂(𝜆)≤𝜆^𝛽,with𝛽=min{2𝑟,1}. (4.10)
On the other hand, observe Lemma 5.1 below, and we have 𝒜_𝑞(𝜆)≤𝐶^∗𝜆^𝛽,with𝛽=min{2𝑟,1}, (4.11) where 𝐶^∗ is some constant and 𝑞∈[1,2]. The conclusion follows immediately from Theorem 4.1.

Let us compare our learning rates with the existing results.

In [10], a uniform covering number technique was used to derive the expected value of learning schemes (1.1) where Ω(𝑓_𝑤)=‖𝑤‖²_𝐾. If all the kernels {𝐾_𝑖} are the same with some specialized 𝐾 and 𝐿_𝐾^−𝑟𝑓_𝜌∈𝐿²_𝜌𝑋 for some 0<𝑟≤1/2. For any 0<𝜁<1/(1+𝜐) and any 0<𝛿<1, with confidence 1−𝛿, then 𝑓_𝐳−𝑓_𝜌²_𝜌2≤log𝛿𝒪𝑚^−2𝑟𝜁. (4.12) Clearly the learning rates derived from Corollary 4.2 are better than that in [10] since 2𝑟𝜁<2𝑟/(1+𝜐)≤4𝑟/(4𝑟+(1+2𝑟)𝜐).

In [21], an operator monotonic technique was used to improve the kernel independent error bounds in comparison with the result in [17]. If 𝐿_𝐾^−𝑟𝑓_𝜌∈𝐿²_𝜌𝑋 for some 0<𝑟≤1. For any 0<𝛿<1, with confidence 1−𝛿, there holds 𝑓_𝐳−𝑓_𝜌²_𝜌2≤log𝛿𝒪𝑚^{−3𝑟/(2𝑟+2)}. (4.13)

When 𝑟≥1/2 and 𝜐≤(2−𝑟)/3𝑟, the learning rate given by Corollary 4.2 is better than the above result.

As for empirical risk minimization (ERM), classical results on analysis of ERM schemes give error bounds between the empirical target function and the regression function. In particular, learning rates of type 𝑚^−𝜀 with 𝜀 arbitrarily close to 1 can be achieved by ERM schemes (see [15]). However, the ERM setting is different from the one on Tikhonov regularization. How to choose the regularization parameter 𝜆=𝜆(𝑚), depending on the sample size 𝑚, is the essential difficulty for the regularization scheme, even when 𝑓_𝜌 lies in ℋ_𝐾. On the other hand, it is obvious that our result is more general than that of [15] since the case for 𝑓_𝜌∉ℋ_𝐾 (𝑟<1/2) is also covered.

5. Discussion on Regularization Error

By our assumptions on M different kernels {𝐾_𝑗}^𝑀_𝑗=1, we see that ℋ_𝐾 is an RKHS generated by the Mercer kernel 𝐾. There are several standard approximation results on regularization error 𝒜₂(𝜆) in learning theory (see [17]). Next we establish a tight connection between 𝒜₂(𝜆) and 𝒜_𝑞(𝜆) with 1≤𝑞≤2.

Lemma 5.1. Let ℋ_𝐾 be a separable 𝑅𝐾𝐻𝑆 over 𝑋 associated with a bounded measurable kernel, 𝜌 be a distribution on 𝑋×[−𝑇,𝑇], and 1≤𝑞≤2. If there exist constants 𝐶>0 and 𝛽>0 such that 𝒜₂(𝜆)≤𝐶𝜆^𝛽, then for all 𝜆>0 we have 𝒜_𝑞(𝜆)≤𝐶𝑀^(2−𝑞)/2+1𝜆^𝛽. (5.1)

Proof. If there exists a function 𝑓_𝜆 satisfying 𝜆𝑓_𝜆²_2,2+𝑓_𝜆−𝑓_𝜌²_𝜌≤𝐶𝜆^𝛽, (5.2) we see that 𝜆‖𝑓_𝜆‖²_2,2≤𝐶𝜆^𝛽. We write 𝑓_𝜆(𝑥)=⟨𝑤,Ψ(𝑥)⟩ with some 𝑤=(𝑤⁽¹⁾,...,𝑤^(𝑀)), by the Hö‌lder inequality, we have 𝑓_𝜆^𝑞_2,𝑞=^𝑀_𝑗=1𝑤≤^(𝑗)𝑞_𝐾𝑗^Ḧ‌older𝑀^(2−𝑞)/2𝑓_𝜆^𝑞_2,2. (5.3) Then we obtain 𝒜_𝑞(𝜆)≤𝜆𝑓_𝜆²_2,𝑞+𝑓_𝜆−𝑓_𝜌²_𝜌≤𝐶𝑀^(2−𝑞)/2𝜆^𝛽+𝐶𝜆^𝛽=𝐶𝑀^(2−𝑞)/2+1𝜆^𝛽. (5.4)

In other words, if 𝒜₂(𝜆) has a polynomial behavior in 𝜆, then this behavior completely determines the behavior of all 𝒜_𝑞(𝜆). Thus it suffices to assume that the standard 2-approximation error function satisfies (2.2).

From statistical effective dimension point of view, we will discuss the impact of the multikernel class ℋ_𝐾 on the approximation error ‖𝑓_𝜆−𝑓_𝜌‖_{𝐿2𝜌𝑋}. To estimate this error, note that the regularizing function of 𝒜₂(𝜆) exists, is unique, and given by [7] 𝑓_𝜆=𝜆𝐼+𝐿_𝐾⁻¹𝐿_𝐾𝑓_𝜌. (5.5) For simplicity, let 𝑀=2 and take a Mercer kernel 𝐾_𝑜 as the original one, by the classical Mercer theorem, 𝐾_𝑜 can be expressed as 𝐾_𝑜(𝑥,𝑦)=∑^∞_𝑘=1𝜆_𝑘𝜑_𝑘(𝑥)𝜑_𝑘(𝑦). Another kernel we take is 𝐾^𝑁_𝑜=∑^∞_𝑘=1𝜆^𝑁_𝑘𝜑_𝑘(𝑥)𝜑_𝑘(𝑦) (2≤𝑁∈ℕ⁺). In this case, 𝐾=𝐾_𝑜+𝐾^𝑁_𝑜=∑^∞_𝑘=1(𝜆_𝑘+𝜆^𝑁_𝑘)𝜑_𝑘(𝑥)𝜑_𝑘(𝑦). By the fact that 𝑓_𝜆−𝑓_𝜌=𝜆(𝜆𝐼+𝐿_𝐾)⁻¹𝑓_𝜌 and assumption 𝐿_𝐾^−𝑟_𝑜𝑓_𝜌∈𝐿²_𝜌𝑋, we have 𝑓_𝜆−𝑓_{𝜌𝐿2𝜌𝑋}=𝜆𝜆𝐼+𝐿_𝐾⁻¹𝐿^𝑟_𝐾𝑜𝐿_𝐾^−𝑟_𝑜𝑓_{𝜌𝐿2𝜌𝑋}=𝜆^∞_𝑘=1𝛼_𝑘𝜆^𝑟_𝑘𝜆_𝑘+𝜆^𝑁_𝑘𝜑+𝜆^𝑘_{𝐿2𝜌𝑋}≤𝜆^{min{𝑟/𝑁,1}}𝐿_𝐾^−𝑟_𝑜𝑓_{𝜌𝐿2𝜌𝑋}for‖𝛼‖_𝑙2=𝐿_𝐾^−𝑟_𝑜𝑓_{𝜌𝐿2𝜌𝑋}. (5.6)

Let us compare the multikernel class regularization with Tikhonov regularization in ℋ_𝐾𝑜 when the Mercer kernel 𝐾_𝑜 is employed. Denote the saturation index as the maximal 𝑟 so that the approximation error achieves fastest decay rate under the condition 𝐿_𝐾^−𝑟_𝑜𝑓_𝜌∈𝐿²_𝜌𝑋. Then (5.6) shows the saturation index for multikernel class regularization is 𝑁 while it is 1 for Tikhonov regularization in ℋ_𝐾𝑜, as shown in [17].

In this case, our analysis implies that we should use an alternative kernel with faster eigenvalue decay when the spectral coefficients of the target function decay faster: for example, using 𝐾^𝑁_𝑜 instead of 𝐾_𝑜. This has a dimension reduction effect. Essentially, we effectively project the data into the principal components of data. The intuition is also quite clear: if the dimension of the target function is small (spectral coefficient 𝑟 decays fast), then we should project data to those dimensions by reducing the remaining noisy dimensions (corresponding to fast kernel eigenvalue decay 𝑁). In fact, the similar idea under the framework of semisupervised learning has been shown in spectral kernel design methods [22, 23].

In general, for the sample error, there exist rates of convergence which hold independently of the underlying distribution 𝜌. This is important, as it tells us that we can give convergence guarantees no matter what a kernel is used, even we do not know the underlying distribution. In fact, this is very common in statistical analysis of various machine learning algorithms (see [24]). This decay is usually fast enough for practical use where amounts of samples are available. For the approximation error, however, it is impossible to give rates of convergence which hold for all probability distributions 𝜌. Hence what determines the learning accuracy is the approximation error. In kernel regression setup, this is determined by the choice of the kernel and enhances the importance of learning kernels [4] and constructing refined kernels [25].

6. Further Study

In the last section, we exclusively discuss sparsity in the case of the square loss regularization functional in (1.1) with the regularizer Ω(𝑓)=‖𝑓‖²_𝐾 in RKHS. We can derive the explicit expression for this functional from [4], in turn which provides improvement and simplification of our algorithm (1.4).

Lemma 6.1. For any kernel 𝐾 and positive constant 𝜆, we have that 𝒮_𝜆(𝐾)1∶=min𝑚^𝑚_𝑖=1𝑓𝑥_𝑖−𝑦_𝑖²+𝜆𝑓²_𝐾=𝜆𝐲,(𝜆𝐼+𝐾(𝐱))⁻¹𝐲, (6.1) where the vector 𝐲=(𝑦_𝑖,…,𝑦_𝑚)^𝑇 and 𝐾(𝐱) denotes the 𝑚×𝑚 Gram matrix (𝐾(𝑥_𝑖,𝑥_𝑗)∶𝑖,𝑗=1,…,𝑚) and ⟨⋅⟩ denotes the standard inner product in Euclidean space.

According to Lemma 6.1, the least square algorithm of (1.4) can be rewritten as a one-layer minimization problem 𝜃_𝐳∶=argmin_𝜃≥0⟨𝐲,𝜆𝐼+𝐾_𝜃(𝐱)⁻¹𝜇𝐲⟩+𝜆^𝑀_𝑘=1𝜃_𝑘^𝑝. (6.2)

We assume that 𝑓_𝜌∈ℋ_𝜃0 for some 𝜃₀∈ℝ^𝑀. Define 𝐽_𝜃∶=supp(𝜃)={𝑘∶𝜃_𝑘≠0}. We say that 𝑓_𝜌 is sparse relative to 𝜃₀ if the cardinally of 𝐽_𝜃0 is far smaller than 𝑀.

For 𝑝∈[1,𝑝_𝑀] (but 𝑝_𝑀 is close to 1), it would be interesting to show that the solution 𝜃_𝐳 of (6.2) is approximately sparse following the path of 𝜃₀. In some sense, ∑_{𝑘∉𝐽𝜃0}(𝜃_𝐳)_𝑘 is very small with a very high probability. A refined analysis of 𝑙^𝑝-regularized methods was done by Koltchinskii [26] in the case of combination of 𝑀 basis functions, mainly taking into account the soft sparsity pattern of the Bayes function and establishing several oracle inequalities in statistical sense. Extending the ideas into the kernels learning setting would be of a great significance, because it can provide theoretical support showing that the 𝑙^𝑝-norm MKL can automatically select good kernels, which coincide with the underlying right kernels.

Acknowledgments

The authors would like to thank the two anonymous referees for their valuable comments and suggestions which have substantively improved this paper. The research of S.-G. Lv is supported partially by 211 project for the Southwestern University of Finance and Economics, China [Project no. 211QN2011028] as well as Annual Cultivation of the Fundamental Research Funds for the Central Universities [Project no. JBK120940].