Consistency Analysis of Spectral Regularization Algorithms

Abstract

We investigate the consistency of spectral regularization algorithms. We generalize the usual definition of regularization function to enrich the content of spectral regularization algorithms. Under a more general prior condition, using refined error decompositions and techniques of operator norm estimation, satisfactory error bounds and learning rates are proved.

1. Introduction

In this paper, we study the consistency analysis of spectral regularization algorithms in regression learning.

Let (𝑋,𝑑) be a compact metric space and 𝜌 a probability distribution on 𝑍=𝑋×𝑌 with 𝑌=ℝ. The regression learning aims at estimating or approximating the regression function𝑓_𝜌(𝑥)=_𝑌𝑦𝑑𝜌𝑦∣𝑥 (1.1) through a set of samples 𝐳={(𝑥_𝑖,𝑦_𝑖)}^𝑚_𝑖=1∈𝑍^𝑚 drawn independently and identically according to 𝜌 from 𝑍.

In learning theory, a reproducing kernel Hilbert space (RKHS) associated with a Mercer kernel 𝐾(𝑥,𝑦) is usually taken as the hypothesis space. Recall that a function 𝐾∶𝑋×𝑋→𝑅 is called a Mercer kernel if it is continuous, symmetric, and positive semidefinite. The reproducing kernel Hilbert space ℋ_𝐾 is defined to be the closure of the linear span of 𝐾_𝑥∶=𝐾(⋅,𝑥),𝑥∈𝑋. The reproducing property takes the form𝑓(𝑥)=𝑓,𝐾_𝑥𝐾,∀𝑓∈ℋ_𝐾,∀𝑥∈𝑋. (1.2) For the Mercer kernel 𝐾(𝑥,𝑦), we denote that𝜅=max_𝑥∈𝑋𝐾(𝑥,𝑥). (1.3)

Our first contribution is to generalize the definition of regularization in [1] such that many more learning algorithms can be included in the scope of spectral algorithms.

Definition 1.1. We say that a family of continuous functions 𝑔_𝜆∶[0,𝜅²]→𝑅,𝜆∈(0,1] is regularization, if the following conditions hold.(i)There exists a constant 𝐷 such that sup_{0<𝜎≤𝜅2}𝜎𝑔_𝜆(𝜎)≤𝐷. (1.4)(ii)There exists a constant 𝐵>0,0<𝛼≤1 such that sup_{0<𝜎≤𝜅2}𝑔_𝜆(𝜎≤𝐵)𝜆^𝛼. (1.5)(iii)There exists a constant 𝛾 such that sup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎≤𝛾. (1.6)(iv)The qualification 𝜈₀ of the regularization 𝑔_𝜆 is the maximal 𝜈 such that sup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜎^𝜈≤𝛾_𝜈𝜆^𝛼𝜈, (1.7) where 𝛾_𝜈 does not depend on 𝜆.
Our definition for regularization is different from that in [1]. In fact, the definition given by [1] is the special case when taking 𝛼=1 in (1.5) and (1.7). So from this viewpoint, our assumption is more mild and it is fit for more general situations, for example, coefficient regularization algorithms correspond to spectral algorithms with 𝛼=1/2, the relation between coefficient regularization algorithms and spectral algorithms had been explored in [2].
Let 𝐱={𝑥_𝑖}^𝑚_𝑖=1 and 𝐲={𝑦_𝑖}^𝑚_𝑖=1. The sample operator 𝑆_𝐱∶ℋ_𝐾→ℝ^𝑚 is defined as 𝑆_𝐱𝑓={𝑓(𝑥_𝑖)}^𝑚_𝑖=1. The adjoint of 𝑆_𝐱 under 1/𝑚 times the Euclidean norm is 𝑆^𝑇_𝐱𝑐=(1/𝑚)∑^𝑚_𝑖=1𝑐_𝑖𝐾_𝑥𝑖. For simplicity, we use 𝑇_𝐱 to stand for 𝑆^𝑇_𝐱𝑆_𝐱.
The spectral regularization algorithm considered here is given by𝑓^𝜆_𝑧=𝑔_𝜆𝑇_𝐱𝑆^𝑇_𝐱𝐲. (1.8)
The regularization 𝑔_𝜆,𝜆∈(0,1] in (1.8) was proposed originally to solve ill-posed inverse problems. The relation between learning theory and regularization of linear ill-posed problems has been well discussed in a series of articles, see [1, 3] and the references therein. The analysis made in previous literatures provides us with a deep understanding of the connection between learning theory and regularization.
A large class of learning algorithms can be considered as spectral regularization algorithms in accordance with different regularizations.

Example 1.2. The regularized least square algorithm is given as 𝑓^𝜆_𝐳=argmin_{𝑓∈ℋ𝐾}1𝑚^𝑚_𝑖=1𝑦_𝑖−𝑓𝑥_𝑖²+𝜆𝑓²_𝐾. (1.9) It has been well understood due to a lot of literatures [4–11], and so forth. It is proved in [7] that 𝑓^𝜆_𝐳=𝑇_𝐱+𝜆𝐼⁻¹𝑆^𝑇_𝐱𝑦, (1.10) which corresponds to algorithm (1.8) with the regularization 𝑔_𝜆(𝜎)=(𝜎+𝜆)⁻¹. (1.11) In this case, we have 𝐵=𝐷=𝛾=𝛾_𝜈0=𝛼=1, the qualification 𝜈₀=1.

Example 1.3. In regression learning, the coefficient regularization with 𝑙² norm becomes 𝑓^𝜆_𝐳=𝑓_𝛼𝐳,𝛼_𝐳=argmin_{𝛼∈ℝ𝑚}1𝑚^𝑚_𝑖=1𝑦_𝑖−𝑓_𝛼𝑥_𝑖²+𝜆𝑚^𝑚_𝑖=1𝛼²_𝑖, (1.12) where 𝑓_𝛼=^𝑚_𝑖=1𝛼_𝑖𝐾_𝑥𝑖,∀𝛼∈ℝ^𝑚. (1.13)
The coefficient regularization was first introduced by Vapnik [12] to design linear programming support vector machines. The consistency of this algorithm has been studied in literatures [2, 13, 14]. In [2], it is proved that the sample error has 𝑂(1/√𝑚) decay, even for nonpositive semidefinite kernels, and𝑓^𝜆_𝐳=𝜆𝐼+𝑇²_𝐱⁻¹𝑇_𝐱𝑆^𝑇_𝐱𝐲. (1.14) Thus, it corresponds to algorithm (1.8) with the regularization 𝑔_𝜆(𝜎)=𝜎²+𝜆⁻¹𝜎. (1.15) In this case, we have 𝐵=𝐷=𝛾=𝛾_𝜈0=1, the qualification 𝜈₀=2 and 𝛼=1/2.

Example 1.4. Landweber iteration is defined by 𝑔_𝜆(𝜎)=∑^{⌊1/𝜆⌋−1}_𝑖=0(1−𝜎)^𝑖 where ⌊𝑎⌋=max{𝑚∶𝑚∈Z,𝑚≤𝑎}. This corresponds to the gradient descent algorithm in Yao et al. [15] with constant step-size. In this case, we have that any 𝜈∈[0,+∞) can be considered as qualification of this method and 𝛾_𝜈=1 if 0<𝜈≤1 and 𝛾_𝜈=𝜈^𝜈 otherwise.
Let 𝑓⁺_ℋ be the projection of 𝑓_𝜌 onto ℋ_𝐾, here ℋ_𝐾 denotes the closure of ℋ_𝐾 in 𝐿²_𝜌𝑋(𝑋). The generalization error of 𝑓^𝜆_𝑧 isℰ𝑓^𝜆_𝑧=_𝑍𝑓^𝜆_𝑧(𝑥)−𝑦²𝑑𝜌=_𝑋𝑓^𝜆_𝑧(𝑥)−𝑓⁺_ℋ(𝑥)²𝑑𝜌_𝑋+_𝑋𝑓⁺_ℋ(𝑥)−𝑓_𝜌(𝑥)²𝑑𝜌_𝑋+𝜎², (1.16) where 𝜌_𝑋 is the marginal distribution of 𝜌 on 𝑋, 𝜎² is the variance of random variable 𝑦−𝑓_𝜌(𝑥). So the goodness of the approximation 𝑓^𝜆_𝑧 is measured by ‖𝑓^𝜆_𝑧−𝑓⁺_ℋ‖_𝜌𝑋, where we take the 𝐿² norm defined as 𝑓_𝜌𝑋=_𝑋𝑓(𝑥)²𝑑𝜌_𝑋^1/2,∀𝑓∈𝐿²_𝜌𝑋(𝑋). (1.17)
The integral operator 𝐿_𝐾 associated with kernel 𝐾 from 𝐿²_𝜌𝑋(𝑋) to 𝐿²_𝜌𝑋(𝑋) is defined by𝐿_𝐾𝑓(𝑥)=_𝑋𝐾(𝑥,𝑡)𝑓(𝑡)𝑑𝜌_𝑋(𝑡),∀𝑓∈𝐿²_𝜌𝑋(𝑋). (1.18)𝐿_𝐾 is a nonnegative self-adjoint compact operator [4]. If the domain of 𝐿_𝐾 is restricted to ℋ_𝐾, it also is a nonnegative self-adjoint compact operator from ℋ_𝐾 to ℋ_𝐾, with norm ‖𝐿_𝐾‖_{ℋ𝐾→ℋ𝐾}≤𝜅² [16]. In the sequel, we simply write ‖𝐿_𝐾‖ instead of ‖𝐿_𝐾‖_{ℋ𝐾→ℋ𝐾} and assume that |𝑦|≤𝑀 almost surely.
As usual, we use the following error decomposition:𝑓^𝜆_𝑧−𝑓⁺_ℋ𝜌𝑋≤𝑓^𝜆_𝑧−𝑓_𝜆𝜌𝑋+𝑓_𝜆−𝑓⁺_ℋ𝜌𝑋, (1.19) where 𝑓_𝜆=𝑔_𝜆𝐿_𝐾𝐿_𝐾𝑓⁺_ℋ. (1.20)
The first term on the right-hand side of (1.19) is called sample error, and the second one is approximation error. Sample error depends on the sampling, and the law of large numbers would lead to its estimation; approximation error is independent of the sampling, and its estimation is mainly through the method of operator approximation.
In order to deduce the error bounds and learning rates, we have to set restriction on the class of possible probability measures that is usually called prior condition. In previous literatures, prior conditions are usually described through the smoothness of regression function 𝑓_𝜌. We suppose the following prior condition:𝑓⁺_ℋ=𝜑𝐿_𝐾ℎ₀,ℎ₀∈𝐿²_𝜌𝑋(𝑋),ℎ_0𝜌𝑋≤𝑅. (1.21) Here, 𝜑 called the index function is some continuous nondecreasing function defined on [0,𝜅²] with 𝜑(0)=0.
In the sequel, we request the qualification 𝜈₀>1/2, and there exists 𝜇₀>0 covering 𝜑, which means that there is 𝑐>0 such that𝑐𝜆^𝜇0𝜑(𝜆)≤inf_{𝜆≤𝜎≤𝜅2}𝜎^𝜇0𝜑(𝜎),0<𝜆≤𝜅². (1.22) It is easy to see that, for any 𝜇≥𝜇₀, 𝜇 covers 𝜑.
Furthermore, we request that 𝜑(𝑡) is operator monotone on [0,𝜅²], that is, there is a constant 𝑐_𝜑<∞, such that for any pair 𝑈,𝑉 of nonnegative self-adjoint operators on some Hilbert space with norm less than 𝜅², it holds𝜑(𝑈)−𝜑(𝑉)≤𝑐_𝜑𝜑(‖𝑈−𝑉‖), (1.23) and, there is 𝑑_𝜑>0 such that 𝑑_𝜑𝜆𝜑(𝜆)≤𝜎𝜑(𝜎),0<𝜆<𝜎≤𝜅². (1.24) It is proved that 𝜑(𝑡)=𝑡^𝛼 for 0≤𝛼≤1 is operator monotone [8].
In [1], Bauer et al. consider the following prior condition:𝑓⁺_ℋ∈Ω_𝜑,𝑅,Ω_𝜑,𝑅=𝑓∈ℋ_𝐾∶𝑓=𝜑𝐿_𝐾𝑣,‖𝑣‖_𝐾≤𝑅. (1.25) This condition is somewhat restrictive, since it asks that 𝑓⁺_ℋ must belong to ℋ_𝐾.
Our result shows that satisfactory error bound is available with a more general prior condition, this is our second main contribution. So from this view point, our work is meaningful. The main result of this paper is the following theorem.

Theorem 1.5. Suppose the index function 𝜑 with covering number 𝜇₀>0 is operator monotone on [0,𝜅²]. The qualification 𝜈₀ satisfies 𝜈₀>max{1/2,𝜇₀} and that 𝑚≥2log(4/𝛿) for 0<𝛿<1. Then, with confidence 1−𝛿, there holds 𝑓^𝜆_𝑧−𝑓⁺_ℋ𝜌𝑋≤𝐶₁1+𝜆^−𝛼/2𝜁^1/2𝜑(𝜆)𝜆^{(𝛼−1)𝜇0}+𝜑(𝜁)+𝜆^−𝛼/2𝜂+𝜆^(𝛼−1)𝜑(𝜆)^{1/𝜇0min{𝜇0,𝜈0−1/2}}, (1.26) where 𝜁=2𝜅²2log(4/𝛿)𝑚,𝜂=𝜑(𝜆)𝜆^{−𝜇0+min{𝛼(𝜇0−1/2),0}}𝑚⁻¹4log𝛿+1+𝜑(𝜆)𝜆^{(𝛼−1)𝜇0}𝑚^−1/24log𝛿+𝜆^{min{𝜇0,𝜈0−1/2}(𝛼−1)+𝛼/2}𝜑(𝜆)^{min{(2𝜈0−1)/2𝜇0,1}}, (1.27) and 𝐶₁ is a constant independent of 𝜆,𝑚,𝛿.

This theorem shows the consistency of the spectral algorithms, gives the error bound, and also can lead to satisfactory learning rates by the explicit expression of 𝜑.

This paper is prepared as follows. In Section 2, we will prove a basic lemma about estimation of operator norms related to the regularization and two concentration inequalities with vector value random variables. In Section 3, we give the proof of Theorem 1.5. In Section 4, we derive learning rate under the setting of several specific regularization.

2. Some Lemmas

We simply write 𝛾₀ instead of 𝛾_𝜈0 in (1.7) for qualification 𝜈₀. To estimate the error ‖𝑓^𝜆_𝑧−𝑓⁺_ℋ‖_𝜌𝑋, we need the following lemma to bound the norms of some operators.

Lemma 2.1. Let 𝜑 be an index function and 𝜈₀>max{1/2,𝜇₀}. Then, the following inequalities hold true: sup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜎^𝑠≤𝛾^{1−𝑠/𝑣0}𝛾^𝑠/𝑣0₀𝜆^𝛼𝑠,∀0<𝑠≤𝜈₀,sup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜑^𝑠(𝜎)≤𝛼_𝑠𝜑^𝑠(𝜆)𝜆^{(𝛼−1)𝜇0𝑠}𝜈,∀0<𝑠≤⁰𝜇₀,sup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜑(𝜎)√𝜎≤𝛽₁𝜆^{min{𝜇0,𝜈0−1/2}(𝛼−1)+𝛼/2}𝜑(𝜆)^{min{(2𝜈0−1)/2𝜇0,1}},sup_{0<𝜎≤𝜅2}𝑔_𝜆(𝜎)𝜎^1/2𝜑(𝜎)≤𝛽₂𝜑(𝜆)𝜆^{min{𝛼(𝜇0−1/2),0}−𝜇0}. (2.1)(2.2)(2.3)(2.4) Here, 𝛼_𝑠,𝛽₁,𝛽₂ are constants only dependent on 𝜈₀,𝜇₀,𝛾,𝛾₀,𝑐,𝜑(𝜅²).

Proof. By (1.6) and (1.7), for any 0<𝑠≤𝜈₀, we have sup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜎^𝑠≤sup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜎^{𝜈0𝑠/𝜈0}×1−𝑔_𝜆(𝜎)𝜎^{1−𝑠/𝜈0}≤𝛾^{1−𝑠/𝑣0}𝛾^𝑠/𝑣0₀𝜆^𝛼𝑠. (2.5)
Since 𝜇₀𝑠≤𝜈₀ and 𝜑 is covered by 𝜇₀, by (2.1) and (1.6), we getsup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜑^𝑠(𝜎)=maxsup_0<𝜎<𝜆1−𝑔_𝜆(𝜎)𝜎𝜑^𝑠(𝜎),sup_{𝜆≤𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜑^𝑠(𝜎)𝜎^𝜇0𝑠𝜎^𝜇0𝑠≤max𝛾𝜑^𝑠(𝜆),1𝑐^𝑠𝜑^𝑠(𝜆)𝛾^{1−𝜇0𝑠/𝑣0}𝛾^{𝜇0𝑠/𝑣0}₀𝜆^{(𝛼−1)𝜇0𝑠}1=max𝛾,𝑐^𝑠𝛾^{1−𝜇0𝑠/𝑣0}𝛾^{𝜇0𝑠/𝑣0}₀𝜑^𝑠(𝜆)𝜆^{(𝛼−1)𝜇0𝑠}≐𝛼_𝑠𝜑^𝑠(𝜆)𝜆^{(𝛼−1)𝜇0𝑠}. (2.6)
In order to prove the third inequality, let 𝜏=min{2𝜈₀/(2𝜈₀−1),𝜈₀/𝜇₀} and 𝜏(1−1/2𝜈₀)=(1/𝜇₀)min{𝜇₀,𝜈₀−1/2}, by (2.2), we havesup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎^1−1/2𝜈0𝜑(𝜎)=sup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜑^𝜏(𝜎)^1−1/2𝜈0𝜑(𝜎)^{1−𝜏(1−1/2𝜈0)}≤𝜑𝜅^{21−𝜏(1−1/2𝜈0)}𝛼^1−1/2𝜈0_𝜏𝜆^{min{𝜇0,𝜈0−1/2}(𝛼−1)}×𝜑(𝜆)^{min{(2𝜈0−1)/2𝜇0,1}}. (2.7) Thus, sup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜑(𝜎)√𝜎=sup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜎^{𝜈01/2𝜈0}×1−𝑔_𝜆(𝜎)𝜎^1−1/2𝜈0𝜑(𝜎)≤𝛽₁𝜆^{min{𝜇0,𝜈0−1/2}(𝛼−1)+𝛼/2}𝜑(𝜆)^{min{(2𝜈0−1)/2𝜇0,1}}, (2.8) where 𝛽₁ is a constant only dependent on 𝜈₀,𝜇₀,𝛾,𝛾₀,𝑐,𝜑(𝜅²).
If 0<𝜇₀≤1/2, we havesup_{0<𝜎≤𝜅2}𝑔_𝜆(𝜎)𝜎^1/2𝜑(𝜎)=maxsup_0<𝜎<𝜆𝑔_𝜆(𝜎)𝜎^1/2𝜑(𝜎),sup_{𝜆≤𝜎≤𝜅2}𝑔_𝜆(𝜎)𝜎^𝜇0+1/2𝜑(𝜎)𝜎^𝜇0≤maxsup_0<𝜎<𝜆𝑔_𝜆(𝜎)𝜎^1/2𝑔_𝜆(𝜎)^1/2𝜑(𝜆),𝜑(𝜆)𝑐𝜆^𝜇0𝐷^𝜇0+1/2𝐵^1/2−𝜇0𝜆^{−(1/2−𝜇0)𝛼}≤max√𝐵𝐷,𝑐⁻¹𝐷^𝜇0+1/2𝐵^1/2−𝜇0𝜑(𝜆)𝜆^{𝛼(𝜇0−1/2)−𝜇0}. (2.9) Similarly computation shows that, for 𝜇₀≥1/2, sup_{0<𝜎≤𝜅2}𝑔_𝜆(𝜎)𝜎^1/2𝜑(𝜎)≤max√𝐵𝐷,𝑐⁻¹𝐷𝜅^2𝜇0−1𝜑(𝜆)𝜆^−𝜇0. (2.10) Thus, the last inequality holds, and we complete the proof.

By taking 𝑠=1/2 in (2.1), we havesup_{0<𝜎≤𝜅2}1−𝑔_𝜆(𝜎)𝜎𝜎^1/2≤𝛾^1−1/2𝑣0𝛾^1/2𝑣0₀𝜆^𝛼/2. (2.11)

The estimates of operator norm mainly adopt the following classical argument in operator theory. Argument: let 𝐴 be a positive operator in a Hilbert space, for 𝑓∈𝐶[0,‖𝐴‖], then 𝑓(𝐴) is self-adjoint by [17, Proposition  4.4.7] and 𝜎(𝑓(𝐴))={𝑓(𝑡)∶𝑡∈𝜎(𝐴)} by [17, Theorem  4.4.8] where 𝜎(𝐴) is the spectral set of 𝐴. Consequently, ‖𝑓(𝐴)‖≤‖𝑓‖_∞.

The following probability inequality concerning random variables with values in a Hilbert space is proved in [18].

Lemma 2.2. Let 𝐻 be a Hilbert space and 𝜉 a random variable on (𝑍,𝜌) with values in 𝐻. Assume ‖𝜉‖≤𝑀<∞ almost surely. Denote 𝜎²(𝜉)=𝐸(‖𝜉‖²). Let {𝑧_𝑖}^𝑚_𝑖=1 be independent random drawers of 𝜌. For any 0<𝛿<1, with confidence 1−𝛿, there holds 1𝑚^𝑚_𝑖=1𝜉𝑧_𝑖(𝜉≤2()−𝐸)𝑀log2/𝛿𝑚+2𝜎²(𝜉)log(2/𝛿)𝑚. (2.12)

Let HS(ℋ_𝐾) be the class of all the Hilbert Schmidt operators on ℋ_𝐾. It forms a Hilbert space with inner product⟨𝑇,𝑆⟩_HS∶=^∞_𝑖=1𝑇𝜑_𝑖,𝑆𝜑_𝑖𝐾, (2.13) where 𝜑_𝑖 is an orthonormal basis of ℋ_𝐾 and this definition does not depend on the choice of the basis. The integral operator 𝐿_𝐾, as an operator on ℋ_𝐾, belongs to HS(ℋ_𝐾) and ‖𝐿_𝐾‖_HS≤𝜅² (see [9]). By Lemma 2.2, we can estimate the following operator norms.

Lemma 2.3. Let 𝐱={𝑥_𝑖}^𝑚_𝑖=1 be a sample set i.i.d drawn from (𝑋,𝜌_𝑋). With confidence 1−𝛿, we have 𝐿_𝐾−𝑆^𝑇_𝐱𝑆_𝐱≤𝜅²2log(2/𝛿)𝑚+2log(2/𝛿)𝑚. (2.14)

Proof. Observe that 𝑆^𝑇_𝐱𝑆_𝐱=(1/𝑚)∑^𝑚_𝑖=1𝐾_𝑥𝑖⟨⋅,𝐾_𝑥𝑖⟩_𝐾. Denote 𝑆^𝑇_𝐱𝑆_𝐱=(1/𝑚)∑^𝑚_𝑖=1𝜉(𝑥_𝑖). Here 𝜉 is the random variable on (𝑋,𝜌_𝑋) given by 𝜉(𝑥)=𝐾_𝑥⟨⋅,𝐾_𝑥⟩_𝐾.
Consider𝜉(𝑥),𝜉(𝑥)_HS=^∞_𝑖=1𝐾_𝑥𝜑_𝑖,𝐾_𝑥𝐾,𝐾_𝑥𝜑_𝑖,𝐾_𝑥𝐾𝐾=^∞_𝑖=1𝜑_𝑖,𝐾_𝑥²_𝐾𝐾(𝑥,𝑥)≤𝜅⁴. (2.15) For 𝑥∈𝑋 and 𝑓∈ℋ_𝐾, the reproducing property insures that 𝜉(𝑥)𝑓=𝐾_𝑥𝑓,𝐾_𝑥𝐾=𝑓(𝑥)𝐾_𝑥. (2.16) Hence, 𝐸(𝜉)=𝐿_𝐾, and thereby 𝐿_𝐾−𝑆^𝑇_𝐱𝑆_𝐱1=𝐸𝜉−𝑚^𝑚_𝑖=1𝜉𝑥_𝑖. (2.17) According to (2.15), there holds 𝜎²(𝜉)=𝐸‖𝜉‖²_HS≤𝜅⁴. Inequality (2.14) then follows from (2.12) and the fact that ‖𝐿_𝐾−𝑆^𝑇_𝐱𝑆_𝐱‖≤‖𝐿_𝐾−𝑆^𝑇_𝐱𝑆_𝐱‖_HS.

Lemma 2.4. Under the assumption of Lemma 2.1. Let 𝐳={𝑧_𝑖}^𝑚_𝑖=1 be a sample set i.i.d drawn from (𝑍,𝜌). With confidence 1−𝛿, we have

𝑆^𝑇_𝐱𝑦−𝑇_𝐱𝑓_𝜆𝐾≤2𝜅𝑀+𝜅𝛽₂𝑅𝜑(𝜆)𝜆^{−𝜇0+min{𝛼(𝜇0−1/2),0}}log(2/𝛿)𝑚+𝛽₁𝑅𝜆^{min{𝜇0,𝜈0−1/2}(𝛼−1)+𝛼/2}×𝜑(𝜆)^{min{(2𝜈0−1)/2𝜇0,1}}+𝜅𝛼₁𝜆^{(𝛼−1)𝜇0}𝜑(𝜆)𝑅+𝑐_𝜌2log(2/𝛿)𝑚. (2.18)

Proof. Define 𝜍=(𝑓_𝜆(𝑥)−𝑦)𝐾_𝑥, so 𝜍 is a random variable from 𝑍 to ℋ_𝐾. Combing the reproducing property with Cauchy-Schwartz inequality, we get 𝑓_𝜆∞=sup_𝑥∈𝑋𝑓_𝜆,𝐾_𝑥𝐾≤𝜅𝑓_𝜆𝐾. (2.19) Since 𝐿_𝐾^1/2 is an isometric isomorphism from (ℋ_𝐾,‖⋅‖_𝜌𝑋) onto (ℋ_𝐾,‖⋅‖_𝐾) (see [16]), we obtain 𝑓_𝜆𝐾=𝑔_𝜆𝐿_𝐾𝐿_𝐾^1/2𝜑𝐿_𝐾𝐿_𝐾^1/2ℎ_0𝐾≤𝑔_𝜆𝐿_K𝐿_𝐾^1/2𝜑𝐿_𝐾×ℎ_0𝜌𝑋≤sup_{0<𝑡≤𝜅2}𝑔_𝜆(𝑡)𝑡^1/2𝜑(𝑡)×𝑅≤𝛽₂𝜑(𝜆)𝜆^{min{𝛼(𝜇0−1/2),0}−𝜇0}𝑅, (2.20) where the last inequality follows from (2.4).
By |𝑦|≤𝑀 almost surely, there holds‖𝜍‖²_𝐾=𝑓_𝜆(𝑥)−𝑦𝐾_𝑥,𝑓_𝜆(𝑥)−𝑦𝐾_𝑥𝐾≤𝜅²𝑀+𝜅𝛽₂𝜑(𝜆)𝜆^{min{𝛼(𝜇0−1/2),0}−𝜇0}𝑅². (2.21) By (2.3) and 𝐿_𝐾𝑓_𝜌=𝐿_𝐾𝑓⁺_ℋ, we get ‖𝐸𝜍‖_𝐾=𝐿_𝐾𝑓_𝜌−𝑓_𝜆𝐾=𝐿_𝐾𝑓⁺_ℋ−𝑓_𝜆𝐾=𝐿_𝐾𝜑𝐿_𝐾𝐼−𝑔_𝜆𝐿_𝐾𝐿_𝐾ℎ_0𝐾≤𝐼−𝑔_𝜆𝐿_𝐾𝐿_𝐾𝜑𝐿_𝐾𝐿_𝐾^1/2×𝐿_𝐾^1/2ℎ_0𝐾≤𝛽₁𝑅𝜆^{min{𝜇0,𝜈0−1/2}(𝛼−1)+𝛼/2}𝜑(𝜆)^{min{(2𝜈0−1)/2𝜇0,1}},𝐸‖𝜍‖²_𝐾=𝐸𝑦−𝑓_𝜆(𝑥)²𝐾(𝑥,𝑥)≤𝜅²𝐸𝑦−𝑓_𝜆(𝑥)²≤𝜅²_𝑍𝑦−𝑓_𝜌(𝑥)²𝑑𝜌+_𝑋𝑓_𝜌(𝑥)−𝑓⁺_ℋ(𝑥)²𝑑𝜌_𝑋+_𝑋𝑓⁺_ℋ(𝑥)−𝑓_𝜆(𝑥)²𝑑𝜌_𝑋≤𝜅²𝛼²₁𝜆^{2(𝛼−1)𝜇0}𝜑²(𝜆)𝑅²+_𝑍𝑦−𝑓_𝜌(𝑥)²𝑑𝜌+𝑓_𝜌−𝑓⁺_ℋ²_𝜌𝑋, (2.22) where, in the last step, we used the result of Proposition 3.1 in Section 3. For simplicity, we write 𝑐²_𝜌 for ∫_𝑍(𝑦−𝑓_𝜌(𝑥))²𝑑𝜌+‖𝑓_𝜌−𝑓⁺_ℋ‖²_𝜌𝑋. Applying Lemma 2.2, there holds 1𝑚^𝑚_𝑖=1[𝜍(𝑧_𝑖)−𝐸𝜍]_𝐾≤2𝜅𝑀+𝜅𝛽₂𝑅𝜑(𝜆)𝜆^{−𝜇0+min{𝛼(𝜇0−1/2),0}}log(2/𝛿)𝑚+𝜅𝛼₁𝜆^{(𝛼−1)𝜇0}𝜑(𝜆)𝑅+𝑐_𝜌2log(2/𝛿)𝑚. (2.23) Then, we can use the following inequality to get the desired error bound, 𝑆^𝑇_𝐱𝑦−𝑇_𝐱𝑓_𝜆𝐾≤1𝑚^𝑚_𝑖=1𝜍𝑧_𝑖−𝐸𝜍_𝐾+‖𝐸𝜍‖_𝐾. (2.24) This completes the proof of Lemma 2.4.

3. Error Analysis

Proposition 3.1. Let 𝜑 be an index function with 𝜇₀>0 covering 𝜑 and 𝜈₀>max{1/2,𝜇₀}, so under the assumptions of (1.21), there holds ‖𝑓_𝜆−𝑓⁺_ℋ‖_𝜌𝑋≤𝛼₁𝜆^{(𝛼−1)𝜇0}𝜑(𝜆)𝑅.

Proof. From the definition of 𝑓_𝜆 and 𝑓⁺_ℋ, we have 𝑓_𝜆−𝑓⁺_ℋ=𝑔_𝜆𝐿_𝐾𝐿_𝐾𝑓⁺_ℋ−𝑓⁺_ℋ=𝑔_𝜆𝐿_𝐾𝐿_𝐾−𝐼𝜑𝐿_𝐾ℎ₀. (3.1) So the following error estimation holds 𝑓_𝜆−𝑓⁺_ℋ𝜌𝑋≤𝑔_𝜆𝐿_𝐾𝐿_𝐾−𝐼𝜑𝐿_𝐾×ℎ_0𝜌𝑋≤sup_{0<𝜎≤𝜅2}𝑔_𝜆(𝜎)𝜎−1𝜑(𝜎)×ℎ_0𝜌𝑋≤𝛼₁𝜆^{(𝛼−1)𝜇0}𝜑(𝜆)𝑅, (3.2) where the last inequality follows from (2.2).

Let us focus on the estimation of sample error.

Consider𝑓^𝜆_𝑧−𝑓_𝜆𝜌𝑋=𝐿_𝐾^1/2(𝑓^𝜆_𝑧−𝑓_𝜆)_𝐾≤𝐿_𝐾^1/2𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝑓_𝜆−𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝐿_𝐾^1/2𝑓_𝜆𝐾+𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝑔_𝜆𝐿_𝐾𝐿_𝐾𝜑𝐿_𝐾𝐿_𝐾^1/2ℎ_0𝐾+𝐿_𝐾^1/2𝑔_𝜆𝑇_𝐱𝑆^𝑇_𝐱𝑦−𝑇_𝐱𝑓_𝜆𝐾∶=𝐼_1𝐾+𝐼_2𝐾+𝐼_3𝐾. (3.3) The idea is to separately bound each term in ℋ_𝐾. We start dealing with the first term of (3.3).

Consider𝐼₁=𝐿_𝐾^1/2−𝑇_𝐱^1/2𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝜑𝐿_𝐾−𝜑𝑇_𝐱𝑔_𝜆𝐿_𝐾𝐿_𝐾^1/2𝐿_𝐾^1/2ℎ₀+𝐿_𝐾^1/2−𝑇_𝐱^1/2𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝜑𝑇_𝐱𝑔_𝜆𝐿_𝐾L_𝐾^1/2𝐿_𝐾^1/2ℎ₀+𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝑇_𝐱^1/2−𝐿_𝐾^1/2𝜑𝐿_𝐾𝑔_𝜆𝐿_𝐾𝐿_𝐾^1/2𝐿_𝐾^1/2ℎ₀=𝐽₁+𝐽₂+𝐽₃. (3.4) According to (1.4) and (1.5), we derive the following bound:𝑔_𝜆𝐿_𝐾𝐿_𝐾^1/2≤sup_{0<𝜎≤𝜅2}𝑔_𝜆(𝜎)𝜎^1/2=sup_{0<𝜎≤𝜅2}𝑔_𝜆(𝜎)𝜎×𝑔_𝜆(𝜎)≤𝜆^−𝛼/2√𝐷𝐵. (3.5) Now, we are in the position to bound (3.4).

Suppose that 𝑚≥2log(4/𝛿), then𝜅²2log(4/𝛿)𝑚+2log(4/𝛿)𝑚≤2𝜅²2log(4/𝛿)𝑚∶=𝜁,2𝜅𝑀+𝜅𝛽₂𝑅𝜑(𝜆)𝜆^{−𝜇0+min{𝛼(𝜇0−1/2),0}}log(4/𝛿)𝑚+𝜅𝛼₁𝜆^{(𝛼−1)𝜇0}𝜑(𝜆)𝑅+𝑐_𝜌2log(4/𝛿)𝑚+𝛽₁𝑅𝜆^{min{𝜇0,𝜈0−1/2}(𝛼−1)+𝛼/2}𝜑(𝜆)^{min{(2𝜈0−1)/2𝜇0,1}}≤2𝜅²𝛽₂𝑅𝜑(𝜆)𝜆^{−𝜇0+min{𝛼(𝜇0−1/2),0}}log(4/𝛿)𝑚+𝜅𝑀+𝛼₁𝑅𝜑(𝜆)𝜆^{(𝛼−1)𝜇0}+𝑐_𝜌2log(4/𝛿)𝑚+𝛽₁𝑅𝜆^{min{𝜇0,𝜈0−1/2}(𝛼−1)+𝛼/2}𝜑(𝜆)^{min{(2𝜈0−1)/2𝜇0,1}}≤𝐶₀𝜑(𝜆)𝜆^{−𝜇0+min{𝛼(𝜇0−1/2),0}}𝑚⁻¹4log𝛿+1+𝜑(𝜆)𝜆^{(𝛼−1)𝜇0}𝑚^−1/24log𝛿+𝜆^{min{𝜇0,𝜈0−1/2}(𝛼−1)+𝛼/2}𝜑(𝜆)^{min{(2𝜈0−1)/2𝜇0,1}}∶=𝐶₀𝜂. (3.6) By Lemmas 2.3 and 2.4, with confidence 1−𝛿, the following inequalities hold simultaneously:𝐿_𝐾−𝑆^𝑇_𝐱𝑆_𝐱≤𝜁,𝑆^𝑇_𝐱𝑦−𝑇_𝐱𝑓_𝜆𝐾≤𝐶₀𝜂. (3.7)

Combing (1.6), (3.5) together with the operator monotonicity property of 𝜑(𝑡) and 𝑡^1/2, we obtain𝐽_1𝐾≤𝐿_𝐾^1/2−𝑇_𝐱^1/2×𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼×𝜑𝐿_𝐾−𝜑𝑇_𝐱×𝑔_𝜆𝐿_𝐾𝐿_𝐾^1/2×𝐿_𝐾^1/2ℎ_0𝐾≤𝑐_𝜑𝛾√𝐷𝐵𝑅𝜆^−𝛼/2𝐿_𝐾−𝑇_𝐱^1/2×𝜑𝐿_𝐾−𝑇_𝐱≤𝑐_𝜑𝛾√𝐷𝐵𝑅𝜆^−𝛼/2𝜁^1/2𝜑(𝜁). (3.8) By Lemma 2.1 and (3.5),𝐽_2𝐾≤𝐿_𝐾^1/2−𝑇_𝐱^1/2×𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝜑𝑇_𝐱×𝑔_𝜆𝐿_𝐾𝐿_𝐾^1/2×𝐿_𝐾^1/2ℎ_0𝐾≤𝑅√𝐷𝐵𝐿_𝐾−𝑇_𝐱^1/2×𝛼₁𝜆^{(𝛼−1)𝜇0−𝛼/2}𝜑(𝜆)≤𝛼₁𝑅√𝐷𝐵𝜁^1/2𝜑(𝜆)𝜆^{(𝛼−1)𝜇0−𝛼/2}. (3.9) For the purpose of bounding ‖𝐽₃‖_𝐾, we rewritten 𝐽₃ as the following form:𝐽₃=𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝑇_𝐱^1/2𝜑𝐿_𝐾−𝜑𝑇_𝐱𝑔_𝜆𝐿_𝐾𝐿_𝐾^1/2𝐿_𝐾^1/2ℎ₀+𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝑇_𝐱^1/2𝜑𝑇_𝐱𝑔_𝜆𝐿_𝐾𝐿_𝐾^1/2𝐿_𝐾^1/2ℎ₀−𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝜑𝐿_𝐾−𝜑𝑇_𝐱𝑔_𝜆𝐿_𝐾𝐿_𝐾𝐿_𝐾^1/2ℎ₀−𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝜑𝑇_𝐱𝑔_𝜆𝐿_𝐾𝐿_𝐾𝐿_𝐾^1/2ℎ₀. (3.10) In the same way, we have that𝐽_3𝐾≤𝛾^1−1/2𝑣0𝛾^1/2𝑣0₀𝑅√𝐷𝐵𝜑(𝜁)+𝛽₁𝜆^{min{𝜇0,𝜈0−1/2}(𝛼−1)}𝜑(𝜆)^{min{(2𝜈0−1)/2𝜇0,1}}√(𝜁)𝐷𝐵𝑅+𝛾𝑅𝐷𝜑+𝑅𝐷𝛼₁𝜑(𝜆)𝜆^{(𝛼−1)𝜇0}. (3.11) Thus, we can get the bound for ‖𝐼₁‖_𝐾 by combining (3.8), (3.9), and (3.11). What left is to estimate ‖𝐼₂‖_𝐾 and ‖𝐼₃‖_𝐾, we can employ the same way used in the estimation of ‖𝐼₁‖_𝐾.

Consider𝐼_2𝐾≤𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝜑𝐿_𝐾−𝜑𝑇_𝐱𝑔_𝜆𝐿_𝐾𝐿_𝐾𝐿_𝐾^1/2ℎ_0𝐾+𝑔_𝜆𝑇_𝐱𝑇_𝐱−𝐼𝜑𝑇_𝐱𝑔_𝜆𝐿_𝐾𝐿_𝐾𝐿_𝐾^1/2ℎ_0𝐾≤𝛾𝑅𝐷𝜑(𝜁)+𝛼₁𝑅𝐷𝜑(𝜆)𝜆^{(𝛼−1)𝜇0},𝐼_3𝐾≤𝐿_𝐾^1/2−𝑇_𝐱^1/2𝑔_𝜆𝑇_𝐱𝑆^𝑇_𝐱𝑦−𝑇_𝐱𝑓_𝜆𝐾+𝑇_𝐱^1/2𝑔_𝜆𝑇_𝐱𝑆^𝑇_𝐱𝑦−𝑇_𝐱𝑓_𝜆𝐾≤𝐶₀𝐵𝜆^−𝛼𝜁^1/2𝜂+𝐶₀√𝐷𝐵𝜂𝜆^−𝛼/2. (3.12)(3.13) Lastly, combining (3.8) to (3.13) with Proposition 3.1, we have Theorem 1.5 holds.

4. Learning Rates

Significance of this paper lies in two facts; firstly, we generalize the definition of regularization and enrich the content of spectral regularization algorithms; secondly, analysis of this paper is able to undertake on the very general prior condition (1.21). Thus, our results can be applied to many different kinds of regularization, such as regularized least square learning, coefficient regularization learning, and (accelerate) landweber iteration and spectral cutoff. In this section, we will choose a suitable index function and apply Theorem 1.5 to some specific algorithms mentioned in Section 1.

4.1. Least Square Regularization

In this case, the regularization is 𝑔_𝜆(𝜎)=1/(𝜎+𝜆),𝜆∈(0,1] with 𝐵=𝐷=𝛾=𝛾₀=𝛼=1. The qualification of this algorithm is 𝜈₀=1. Suppose 𝜑(𝑡)=𝑡^𝑟 with 0<𝑟≤1, that means 𝑓⁺_ℋ=𝐿^𝑟_𝐾ℎ₀, ℎ₀∈𝐿²_𝜌𝑋(𝑋). Thus, we have that 𝜇₀=𝑟 covering 𝜑(𝑡).

Using the result of Theorem 1.5, we obtain the following corollary.

Corollary 4.1. Under the assumptions of Theorem 1.5, we have the following.(i)For 0<𝑟≤1/2, with confidence 1−𝛿, there holds 𝑓^𝜆_𝑧−𝑓⁺_ℋ𝜌𝑋≤𝑂𝜆^𝑟+𝑚^−𝑟/2+𝜆⁻¹𝑚^−3/41+𝜆^−1/2𝑚^−1/44log𝛿^5/4. (4.1) By taking 𝜆=𝑚^−1/2, we have the following learning rate: 𝑓^𝜆_𝑧−𝑓⁺_ℋ𝜌𝑋≤𝑂𝑚^−𝑟/24log𝛿^5/4. (4.2)(ii)For 1/2≤𝑟<1, with confidence 1−𝛿, there holds 𝑓^𝜆_𝑧−𝑓⁺_ℋ𝜌𝑋≤𝑂𝜆^1/2+𝜆^−1/2𝑚^−1/2+𝜆⁻¹𝑚^−3/4+𝑚^−1/44log𝛿^5/4. (4.3) By taking 𝜆=𝑚^−1/2, we have the following learning rate: 𝑓^𝜆_𝑧−𝑓⁺_ℋ𝜌𝑋≤𝑂𝑚^−1/44log𝛿^5/4. (4.4)

4.2. Coefficient Regularization with the 𝑙² Norm

In this case, the regularization is 𝑔_𝜆(𝜎)=𝜎/(𝜎²+𝜆),𝜆∈(0,1] with 𝐵=𝐷=𝛾=𝛾₀=1,𝛼=1/2. The qualification is 𝜈₀=2. We also consider the index function 𝜑(𝑡)=𝑡^𝑟 with 0<𝑟≤1 and 𝜇₀=𝑟.

Corollary 4.2. Under the assumptions of Theorem 1.5, we have the following.(i)For 0<𝑟≤1/2, with confidence 1−𝛿, there holds 𝑓^𝜆_𝑧−𝑓⁺_ℋ𝜌𝑋≤𝑂1+𝜆^−1/4𝑚^−1/4𝜆^𝑟/2+𝑚^−𝑟/2+𝑚^−1/2𝜆^−1/4+𝑚⁻¹𝜆^{(1/2)(𝑟−1)}4log𝛿^5/4. (4.5) By taking 𝜆=𝑚⁻¹, we have the following learning rate: 𝑓^𝜆_𝑧−𝑓⁺_ℋ𝜌𝑋≤𝑂𝑚^−𝑟/24log𝛿^5/4. (4.6)(ii)For 1/2≤𝑟≤1, with confidence 1−𝛿, there holds 𝑓^𝜆_𝑧−𝑓⁺_ℋ𝜌𝑋≤𝑂1+𝜆^−1/4𝑚^−1/4𝜆^𝑟/2+𝑚^−𝑟/2+𝑚^−1/2𝜆^−1/44log𝛿^5/4. (4.7) By taking 𝜆=𝑚^{−2/(2𝑟+1)}, we have the following learning rate: 𝑓^𝜆_𝑧−𝑓⁺_ℋ𝜌𝑋≤𝑂𝑚^{−𝑟/(2𝑟+1)}4log𝛿^5/4. (4.8)

For coefficient regularization, the learning rates derived by Theorem 1.5 are almost the same, see Corollary  5.2 in [2]. For least square regularization, the learning rates in Corollary 4.1 are weak, the analysis in [8] by integral operator method gives learning rate 𝑂(𝑚^{−3𝑟/4(1+𝑟)}) for 0<𝑟≤1/2; leave one out analysis in [11] gives the rate 𝑂(𝑚^{−𝑟/(1+2𝑟)}).

Our analysis is influenced by both the prior condition and the regularization. Under the weaker prior condition (1.21), some techniques for error analysis in [1] are inapplicable; we take more complicated error decomposition and refined analysis to estimate error bounds and learning rates.

Acknowledgment

This work is supported by the Natural Science Foundation of China (Grant no. 11071276).