The Learning Rates of Regularized Regression Based on Reproducing Kernel Banach Spaces

Abstract

We study the convergence behavior of regularized regression based on reproducing kernel Banach spaces (RKBSs). The convex inequality of uniform convex Banach spaces is used to show the robustness of the optimal solution with respect to the distributions. The learning rates are derived in terms of the covering number and 𝐾-functional.

1. Introduction

Recently, there is an increasing research interest in learning with abstract functional spaces, and considerable work has been done in [1–3] and so on.

Let(ℬ,‖⋅‖_ℬ)be a normed vector space consisting of real functions on a compact distance space(𝑋,𝑑(⋅,⋅)), and let 𝑀>0be a given positive number. Let 𝑧={𝑧_𝑖}^𝑚_𝑖=1={(𝑥_𝑖,𝑦_𝑖)}^𝑚_𝑖=1∈𝑍^𝑚=(𝑋×[−𝑀,𝑀])^𝑚be a finite set of samples drawn independently and identically (i.i.d.) according to a distribution𝜌(𝑥,𝑦)on𝑍.Then, the regularized learning scheme associating with a given hypothesis spaceℬand the least square loss is 𝑓_𝑧∶=argmin_𝑓∈ℬ1𝑚^𝑚_𝑖=1𝑦_𝑖−𝑓𝑥_𝑖²+𝜆𝑞𝑓^𝑞_ℬ, (1)

where𝑞≥1is a given real number. The unknown Borel probability distribution𝜌(𝑥,𝑦)can be decomposed into𝜌(𝑦∣𝑥)and𝜌_𝑋(𝑥), where 𝜌(𝑦∣𝑥) is the conditional probability of 𝜌 at 𝑥∈𝑋 and 𝜌_𝑋(𝑥) is the marginal probability on 𝑋.

The regression function corresponding to the least square loss is 𝑓_𝜌(𝑥)=_{[−𝑀,𝑀]}𝑦𝑑𝜌𝑦∣𝑥, (2)

which satisfies 𝑓_𝜌(𝑥)∶=argmin_𝑓ℰ_𝜌𝑓=argmin_𝑓𝑍𝑦−𝑓(𝑥)²𝑑𝜌. (3)

When the hypothesis spacesℬin (1) are reproducing kernel Banach spaces, we call it the RKBSs based on regularized regression learning defined by [4, 5] recently. The represented theorem related closely to regularized learning is studied in case thatℬis an RKBS, and the discussions are extended to the generalized semi-inner-product RKBSs in [6].

In the present paper, we will provide an investigation on the learning rates of scheme (1) whenℬis an RKBS with uniform convexity. The paper is organized as follows. In Section 2, we show the main results of the present paper. The robustness is studied in Section 3, and the sample errors are bounded in Section 4. The approximation error boils down to a 𝐾-functional. The learning rates are bounded in Section 5.

For a given real number 𝑝≥1, we denote by𝐿^𝑝(𝜌_𝑋)the class of𝜌_𝑋-measurable functions𝑓satisfying ‖𝑓‖_{𝐿𝑝(𝜌𝑋)}=(∫_𝑋|𝑓(𝑥)|^𝑝𝑑𝜌_𝑋)<+∞.

We say 𝐴=𝑂(𝐵) if there is a constant𝐶>0such that𝐴/𝐵≤𝐶. We say 𝐴∼𝐵if both𝐴=𝑂(𝐵)and𝐵=𝑂(𝐴).

2. Notions and Results

To state the results of the present paper, we first introduce some notions as follows.

2.1. The RKBSs

We denote byℬthe Banach space with dual spaceℬ^∗and norm‖⋅‖_ℬ. For𝑓∈ℬand 𝑓^∗∈ℬ^∗, we write⟨𝑓,𝑓^∗⟩_ℬ=𝑓^∗(𝑓).

A reproducing kernel Banach space (RKBS) on𝑋is a reflexive Banach space of real functions on𝑋whose dual spaceℬ^∗is isometric to a Banach spaceℬ^#of functions on 𝑋, and the point evaluations are continuous functions on bothℬand ℬ^#. It was shown by Theorem  2 of [4] that if𝐵is an RKBS on𝑋, then, there exists uniquely a function𝐾∶𝑋×𝑋→ℜ called the reproducing kernel ofℬsatisfying the following:(i)𝐾(𝑥,⋅)∈ℬ, 𝐾(⋅,𝑥)∈ℬ^∗, 𝑥∈𝑋;(ii)𝑓(𝑥)=⟨𝑓,𝐾(⋅,𝑥)⟩_ℬ, 𝑓^∗(𝑥)=⟨𝐾(𝑥,⋅),𝑓^∗⟩_ℬ, 𝑓^∗∈ℬ^∗,  𝑥∈𝑋.(iii)The linear span of{𝐾(𝑥,⋅)∶𝑥∈𝑋}is dense inℬ, namely, span{𝐾(𝑥,⋅)∶𝑥∈𝑋}=ℬ. (4)

(iv)The linear span of{𝐾(⋅,𝑥)∶𝑥∈𝑋}is dense inℬ^∗, namely,span{𝐾(⋅,𝑥)∶𝑥∈𝑋}=ℬ^∗. (5)

(v)For all𝑥,𝑦∈𝑋there holds𝐾(𝑥,𝑦)=⟨𝐾(𝑥,⋅),𝐾(⋅,𝑦)⟩_ℬ.

Whenℬis an RKHS,𝐾is indeed the reproducing kernel in the usual sense (see [7]).

Sinceℬis a reflective Banach space, we have ⟨𝑓,𝑓^∗⟩_ℬ≤𝑓_ℬ×𝑓^∗_ℬ∗,𝑓∈ℬ,𝑓^∗∈ℬ^∗,𝑓^∗_ℬ∗=sup_𝑓ℬ≤1⟨𝑓,𝑓^∗⟩_ℬ. (6)(7)

A way of producing reproducing kernel spaces in 𝐿^𝑝 spaces by the idempotent integral operators was provided in [8]. In the present paper, we provide a method to construct RKBSs by orthogonal function series.

Example 1. Let𝑋=[𝑎,𝑏]be a given closed interval and let,{𝜑_𝑘(𝑥)}^+∞_𝑘=0be a sequence of continuous functions on[𝑎,𝑏]satisfying the following:(i)𝜑_𝑘∈𝐿^𝑝(𝜌)for𝑘=0,1,2,…;(ii)𝜑_𝑘and𝜑_𝑙are orthonormal (in𝐿²(𝜌)) when𝑙≠𝑘;(iii)span({𝜑_𝑘(𝑥)}^+∞_𝑘=0)is dense in𝐿^𝑝(𝜌)for1<𝑝<+∞.
Let{𝜆_𝑙}^+∞_𝑙=0be a given positive real number sequence satisfying ∑^+∞_𝑙=0𝜆_𝑙≤1. Define 𝐾𝑥,𝑦=^∞_𝑙=0𝜆_𝑙𝜑_𝑙(𝑥)𝜑_𝑙𝑦,𝑥,𝑦∈𝑋, (8)
and the functional classℬ_𝐾^(𝑝)on𝑋by ℬ_𝐾^(𝑝)=𝑓∈𝐶(𝑋)∶𝑓_ℬ𝐾^(𝑝)=^∞_𝑙=0𝑎_𝑙𝑓^𝑝′𝜆_𝑙^1/𝑝′<+∞, (9)
where𝑎_𝑘(𝑓)=∫_[𝑎,𝑏]𝑓(𝑦)𝜑_𝑘(𝑦)𝑑𝜌(𝑦),𝑘∈𝑁. We define the spaceℬ^(𝑝′)_𝐾for𝑝^′=𝑝/(𝑝−1)in an analogous way.

We have the following proposition.

Proposition 2. Define a bivariate operation onℬ_𝐾^(𝑝)andℬ^(𝑝′)_𝐾by ⟨𝑓,𝑔⟩=^∞_𝑙=0𝑎_𝑙𝑓𝑎_𝑙𝑔𝜆_𝑙,𝑓∈ℬ_𝐾^(𝑝),𝑔∈ℬ^(𝑝′)_𝐾. (10)
Then, ℬ_𝐾^(𝑝)is a reproducing kernel Banach space with reproducing kernel𝐾(𝑥,𝑦).

Proof. Let𝑙^𝑝_𝐾={𝑎={𝑎_𝑘}^+∞_𝑘=0∶‖𝑎‖_𝑙^𝑝_𝐾=(∑^∞_𝑙=0(|𝑎_𝑙|^𝑝/𝜆_𝑙))^1/𝑝<+∞} and𝑙^𝑝′_𝐾be defined in an analogous way. Then, both𝑙^𝑝_𝐾and𝑙^𝑝′_𝐾are Banach spaces and(𝑙^𝑝_𝐾)^∗=𝑙^𝑝′_𝐾and(𝑙^𝑝′_𝐾)^∗=𝑙^𝑝_𝐾.

By (9) we knowℬ_𝐾^(𝑝)and𝑙^𝑝′_𝐾are isometric isomorphisms. Therefore,ℬ_𝐾^(𝑝)are Banach spaces.

Since 𝑎_𝑙(𝐾(⋅,𝑦))=𝜆_𝑙𝜑_𝑙(𝑦), we have for 𝑓∈ℬ_𝐾^(𝑝) that 𝑓,𝐾⋅,𝑦=^∞_𝑙=0𝑎_𝑙𝑓𝑎_𝑙𝐾⋅,𝑦𝜆_𝑙=^∞_𝑙=0𝑎_𝑙𝑓𝜑_𝑙𝑦=𝑓𝑦. (11)

By the same way, we have for any 𝑔∈ℬ^(𝑝′)_𝐾 that ⟨𝐾(𝑥,⋅),𝑔⟩=𝑔(𝑥); that is, the reproducing property holds.

2.2. The Uniform Convexity

In this subsection, we focus on some notions in convex analysis and Banach geometry theory.

Let𝐹(𝑓)∶ℬ→ℜ be a convex function. Then, 𝜕𝐹𝑓=𝜉∈ℬ^∗∶𝐹𝑓^′−𝐹𝑓≥⟨𝑓^′−𝑓,𝜉⟩_ℬ,∀𝑓^′∈ℬ≠𝜙. (12)

We call𝜕𝐹(𝑓)the subdifferential of𝐹(𝑓)at𝑓∈ℬ. If𝜉∈𝜕𝐹(𝑓), then, we call𝜉a subgradient of𝐹at𝑓.

A well-known result is that𝑓₀is a minimal value point of a convex function𝐹(𝑓)onℬif and only if0∈𝜕𝐹(𝑓₀)(see [9]).

A Banach spaceℬis called𝑞-uniform convex if there are constants𝑞>0,  𝑐>0such that the modulus defined by 𝛿_ℬ(𝜀)=inf1−𝑓+𝑔_ℬ2∶𝑓_ℬ=𝑔_ℬ=1,𝑓−𝑔_ℬ,(]=𝜀𝜀∈0,2 (13)

satisfies𝛿_ℬ(𝜀)≥𝑐𝜀^𝑞.In particular, any Hilbert spaces are 2-uniform convex Banach spaces.

Define 𝐽_𝑞(𝑓)_ℬ=𝜕((1/𝑞)‖𝑓‖^𝑞_ℬ).Then, by (28) in Corollary  1 of [10] we knowℬis𝑞-uniform convex if and only if there is a positive constant𝑐_𝑞>0such that for all𝑓,𝑔∈ℬ and all 𝑗_𝑞(𝑓)∈𝐽_𝑞(𝑓)_ℬ there holds 1𝑞𝑓^𝑞_ℬ−𝑔^𝑞_ℬ≥⟨𝑗_𝑞𝑔,𝑓−𝑔⟩_ℬ+𝑐_𝑞𝑓−𝑔^𝑞_ℬ,∀𝑓,𝑔∈ℬ. (14)

In [11–14] we know that, for a given 1<𝑝<+∞, the space 𝑙_𝑝, the Lebesgue spaces𝐿_𝑝and the Sobolev space𝑊^𝑚_𝑝aremax{2,𝑝}-uniform convex. Also, letℬ_𝐾^(𝑝)andℬ^(𝑝′)_𝐾be defined as in Section 2.1. Then, by the fact thatℬ_𝐾^(𝑝)and𝑙^𝑝′_𝐾 are isometric isomorphisms, we knowℬ_𝐾^(𝑝)is2-uniform convex if𝑝>2and𝑝^′-uniform convex if1<𝑝≤2. Therefore, we knowℬ_𝐾^(𝑝)is a𝑞-uniform convex Banach space, where𝑞is 2 if𝑝>2and its value is𝑝/(𝑝−1)if1<𝑝≤2.

2.3. Main Results

Let𝑆be a distance space and𝜂>0. The covering number𝒩(𝑆,𝜂) is defined to be the minimal positive integer number𝑙such that there exists𝑙disk in𝑆with radius𝜂covering𝑆.

We say a compact subset𝐸in a distance space(ℬ,‖⋅‖_ℬ) has logarithmic complexity exponent𝑠≥0if there is a constant𝑐_𝑠>0such that the closed ball of radius𝑅centered at origin, that is, ℬ_𝑅={𝑓∈𝐸∶‖𝑓‖_ℬ≤𝑅}, satisfies log𝒩ℬ_𝑅,𝜂≤𝑐_𝑠𝑅𝜂^𝑠,∀𝜂>0. (15)

Now we are in a position to present the main results of this paper.

Theorem 3. Letℬbe an RKBS with𝑞-uniform convexity and a reproducing kernel 𝐾(⋅,𝑥)which is uniform continuous on𝑋in terms of the norm‖⋅‖_ℬ^∗, that is, ‖𝐾(⋅,𝑥)‖_ℬ^∗. ‖𝐾(⋅,𝑥)‖_ℬ^∗ is a uniform continuous function on𝑋, and there is a constant𝑘>0such that‖𝐾(⋅,𝑥)‖_ℬ^∗≤𝑘holds for all𝑥∈𝑋. Let𝑓_𝑧be the unique minimizer of scheme (1). If𝑓_𝜌∈𝐿²(𝜌_𝑋), then for any𝜖>0there holds Prob_𝑧∈𝑍^𝑚𝑓_𝑧−𝑓_{𝜌𝐿2(𝜌𝑋)}≤𝑘𝜖+𝐷_𝑞𝑓_𝜌,𝜆𝑞≥1−2𝒩ℬ₁,𝜆𝑐_𝑞𝜖^𝑞−18𝛾_𝑞×exp−𝑚𝜆²𝑐²_𝑞𝜖^2(𝑞−1)32𝑘²𝛾²_𝑞, (16)
where 𝐷_𝑞𝑓_𝜌,𝜆=inf_𝑓∈ℬ𝑓−𝑓_𝜌²_{𝐿2(𝜌𝑋)}+𝜆𝑓^𝑞_ℬ (17)
is a𝐾-functional,𝛾_𝑞∶=𝑀+𝑘(𝑞𝐷_𝑞(𝑓_𝜌,𝜆/𝑞)/𝜆)^1/𝑞and ℬ₁=𝑓(𝑥)∶𝑓∈ℬ,𝑓_ℬ≤1. (18)

The covering number involved in (16) has been studied widely (see [15–19]). In this paper, we assume 𝒩(ℬ_𝑅,𝜂)has the logarithmic complexity.

Theorem 4. Under the conditions of Theorem 3, if 𝑓_𝜌∈𝐿²(𝜌_𝑋) and (ℬ,‖⋅‖_ℬ)has logarithmic complexity with exponent 𝑠≥0, then for any 𝛿∈(0,1), with confidence 1−𝛿, there holds 𝑓_𝑧−𝑓_{𝜌𝐿2(𝜌𝑋)}≤𝑘64𝛾²_𝑞log(2/𝛿)𝑚𝜆²𝑐²_𝑞^{1/2(𝑞−1)}+8^2+𝑠𝛾_𝑞^2+𝑠𝑐_𝑠𝑚𝜆^2+𝑠𝑐_𝑞^{2+𝑠1/(2+𝑠)(𝑞−1)}+𝐷_𝑞𝑓_𝜌,𝜆𝑞, (19)
where𝑐_𝑠is defined in (15).

We now give some remarks on Theorems 3 and 4.(i)In Theorem 3, we require that the kernel𝐾(𝑥,𝑦)is uniform continuous and uniform bounded on𝑋. In fact, a large class of real bivariate functions satisfies these conditions. For example, if the function sequence{𝜑_𝑙(𝑥)}^+∞_𝑙=0defined in Example 1 is uniformly bounded, that is, |𝜑_𝑙(𝑥)|≤1holds for all𝑙and all𝑥∈[𝑎,𝑏], then, kernel𝐾(𝑥,𝑦) is continuous on [𝑎,𝑏]×[𝑎,𝑏]which turns out that 𝐾(𝑥,𝑦) is uniform continuous on [𝑎,𝑏]×[𝑎,𝑏]. Therefore, |𝐾(𝑥,𝑦)|≤∑^+∞_𝑙=0𝜆_𝑙≤1 shows that 𝐾(𝑥,𝑦) is uniform continuous and bounded with norm ‖⋅‖_𝐿^𝑝_(𝜌).(ii)By the definition of 𝛾_𝑞, we know that if 𝐷_𝑞(𝑓_𝜌,𝜆)=𝑂(𝜆^𝛽),0<𝛽≤1, then, 𝛾_𝑞=𝑀+𝑂(𝑞𝜆^𝛽−1)^1/𝑞. It is bounded if 𝛽=1.(iii)If ℬ is a reproducing kernel Hilbert space, then, 𝑞=2,  𝑐_𝑞=1. Moreover, if 𝐷_𝑞(𝑓_𝜌,𝜆)=𝑂(𝜆^𝛽),0<𝛽≤1, then, we have by (19) that 𝑓_𝑧−𝑓_{𝜌𝐿2(𝜌𝑋)}()=𝑂log2/𝛿𝑚𝜆^3−𝛽1/2+1𝑚𝜆^{(2+𝑠)(3−𝛽)/21/(2+𝑠)}+𝜆^𝛽/2. (20)

(iv)We can show a way of bounding the decay rates of𝐷_𝑞(𝑓_𝜌,𝜆) for 1<𝑝≤2. Let𝑓∈𝐿^𝑝(𝜌). Then, we have the following Fourier expansion:𝑓(𝑥)∼^∞_𝑙=0𝑎_𝑙𝑓𝜑_𝑙(𝑥). (21) Define an operator sequence by 𝑉_𝑛𝑓,𝑥=^𝑛_𝑙=0𝜆_𝑙𝑎_𝑙𝑓𝜑_𝑙(𝑥),𝑥∈𝑋. (22) Then, for a given positive integer𝑛we have𝑎_𝑙(𝑉_𝑛(𝑓))=𝜆_𝑙𝑎_𝑙(𝑓)and 𝑉_𝑛𝑓_ℬ𝐾^(𝑝)=^∞_𝑙=0𝑎_𝑙𝑉_𝑛𝑓^𝑝′𝜆_𝑙^1/𝑝′=^𝑛_𝑙=0𝜆^𝑝′_𝑙⁻¹𝑎_𝑙𝑓^{𝑝′1/𝑝′}≤max_{0≤𝑙≤𝑛}𝜆^{1−(1/𝑝′)}_𝑙^+∞_𝑙=0𝑎_𝑙𝑓^{𝑝′1/𝑝′}≤max_{0≤𝑙≤𝑛}𝜆^{1−(1/𝑝′)}_𝑙𝑓_{𝐿𝑝(𝜌)}, (23) where we have used the generalized Bessel inequality (see [20]): ^+∞_𝑙=0𝑎_𝑙𝑓^{𝑝′1/𝑝′}≤𝑓_{𝐿𝑝(𝜌)}. (24) Also, 𝑉_𝑛𝑓,𝐾⋅,𝑦=^∞_𝑙=0𝑎_𝑙𝑉_𝑛𝑓𝑎_𝑙𝐾⋅,𝑦𝜆_𝑙=^𝑛_𝑙=0𝜆_𝑙𝑎_𝑙𝑓𝜑_𝑙𝑦=𝑉_𝑛𝑓,𝑦. (25) By (25) and (23) we know𝑉_𝑛(𝑓)∈ℬ_𝐾^(𝑝)holds for all positive integers𝑛and, in this case, 𝐷_𝑞𝑓_𝜌,𝜆≤𝑉_𝑛𝑓_𝜌−𝑓_𝜌²_{𝐿2𝜌𝑋}+𝜆𝑉_𝑛𝑓_𝜌^𝑞_{ℬ𝐾(𝑝)}≤𝑉_𝑛𝑓_𝜌−𝑓_𝜌²_{𝐿2𝜌𝑋}+𝜆max_{0≤𝑙≤𝑛}𝜆^{1−(1/𝑝′)}_𝑙^𝑞𝑓_{𝐿𝑝(𝜌)}. (26) One can choose suitable𝑛such that it depends upon the sample number𝑚and obtain the decay rates when𝑚→+∞.There are many choices for the type of operator (22). For example, the Bernstein-Durrmeyer operators (see, e.g., [21–23]) and the de la Vallée-Poussin sum operators are such types (see [24]). This method was first provided by [25] and was extended in [26, 27].(v)We know from [19] that the RKHSs with logarithmic complexity with exponent𝑠≥0 exist. By Corollary  4.1 and Theorem  2.1 of [16] we know that if𝜆_𝑙satisfy𝜆_𝑙∼1/(1+𝑙)^𝛼,𝛼>1, then, the covering number ofℬ_𝐾⁽²⁾may attain the decay of complexity exponent. In a recent paper (see [28]), Guntuboyina and Sen showed that the set of all convex functions defined on[𝑎,𝑏]^𝑑that are uniform bounded has the logarithmic complexity exponent𝑑/2in the𝐿_𝑝-metric.

3. Robustness

Robustness is a quantitative description of the solutions on the distributions.

Define the𝜌-control integral regularized model corresponding to (1) by 𝑓^(𝜌)∶=argmin_𝑓∈ℬℰ_𝜌𝑓+𝜆𝑞𝑓^𝑞_ℬ, (27)

whereℰ_𝜌(𝑓)is defined in (3). Then,𝑓^(𝜌)is influenced by the distributions𝜌. For any bounded𝜌-measurable function𝑓(𝑥,𝑦)on𝑍, we define the empirical measure𝛾_𝑧(𝑥,𝑦)as follows: 𝐸_𝛾𝑧𝑓𝑥,𝑦=_𝑍𝑓𝑥,𝑦𝑑𝛾_𝑧=1𝑚^𝑚_𝑖=1𝑓𝑥_𝑖,𝑦_𝑖. (28)

Then,𝑓_𝑧=𝑓^(𝛾𝑧).We give the following theorem.

Theorem 5. Letℬbe an RKBS with𝑞-uniform convexity and the reproducing kernel𝐾(𝑥,𝑦), and let 𝑓^(𝜌)and𝑓^(𝛾)be the solutions of scheme (27) with respect to distributions𝜌and𝛾,   respectively. Then, 𝑓^(𝜌)−𝑓^(𝛾)_ℬ≤2𝜆𝑐_𝑞×_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾()−⋅,𝑥𝑑𝜌_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾(⋅,𝑥)𝑑𝛾_ℬ∗^1/(𝑞−1), (29)
where𝑐_𝑞is the constant defined in (14).

Theorem 5 shows how𝜌influences the unique solution𝑓^(𝜌).

To prove Theorem 5, we need the following lemmas.

Lemma 6. Under the conditions of Theorem 5, there holds 𝜕ℰ_𝜌𝑓₀=−2_𝑍𝑦−𝑓₀(𝑥)𝐾(⋅,𝑥)𝑑𝜌,𝑓₀∈ℬ, (30)
where the point⋅in𝐾(⋅,𝑥)means𝐾(⋅,𝑥)∈ℬfor any𝑥∈𝑋.

Proof. We restate the following statement.
Let(ℬ,‖⋅‖_ℬ)be a Banach space, 𝐹(𝑓)∶ℬ→ℜ⋃{∓∞}be a real function. We say𝐹is Gateaux differentiable at𝑓₀∈ℬif there is an𝜉∈ℬ^∗such that for any𝑔∈ℬthere holds lim_𝑡→0𝐹𝑓₀+𝑡𝑔−𝐹𝑓₀𝑡=⟨𝑔,𝜉⟩_ℬ, (31)
and write𝐹^′_𝐺(𝑓₀)=𝜉.By [29] we know that if𝐹is convex onℬand is Gateaux differentiable at𝑓₀∈ℬ, then,𝜕𝐹(𝑓₀)={𝐹^′_𝐺(𝑓₀)}.
By equality 𝑥²−𝑦²=2𝑦𝑥−𝑦+𝑥−𝑦²,𝑥,𝑦∈ℜ, (32)
we have for any𝑔(𝑥)=⟨𝑔,𝐾(⋅,𝑥)⟩_ℬ∈ℬ that lim_𝑡→0ℰ_𝜌𝑓₀+𝑡𝑔−ℰ_𝜌𝑓₀𝑡=lim_𝑡→0∫_𝑍𝑦−𝑓₀(𝑥)−𝑡𝑔(𝑥)²𝑑𝜌−∫_𝑍𝑦−𝑓₀(𝑥)²𝑑𝜌𝑡=lim_𝑡→0−2𝑡∫_𝑍𝑦−𝑓₀(𝑥)𝑔(𝑥)𝑑𝜌+𝑡²∫_𝑍𝑔(𝑥)²𝑑𝜌𝑡=−2_𝑍𝑦−𝑓₀(𝑥)𝑔(𝑥)=𝑑𝜌𝑔,−2_𝑍𝑦−𝑓₀(𝑥)𝐾(⋅,𝑥)𝑑𝜌_ℬ. (33)
Sinceℰ_𝜌(𝑓)is a convex function onℬ, we know (30) holds.

Lemma 7. Take𝐽_𝑞(𝑓)=𝜕((1/𝑞)‖𝑓‖^𝑞_ℬ). Then, under the conditions of Theorem 5, there hold the following.(i)There exists uniquely a minimizer𝑓^(𝜌)of the problem (27) and 𝑓^(𝜌)_ℬ≤𝑞𝐷_𝑞𝑓_𝜌,𝜆/𝑞𝜆^1/𝑞. (34)(ii)There is a𝑗_𝑞(𝑓^(𝜌))∈𝐽_𝑞(𝑓^(𝜌))such that 𝜆𝑗_𝑞𝑓^(𝜌)=2_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾(⋅,𝑥)𝑑𝜌. (35)

Proof. The uniqueness of the minimizer can be obtained by the fact that (27) is a strict convex optimization problem. By the definition of𝑓^(𝜌), we have 𝜆𝑞𝑓^(𝜌)𝑞_ℬ≤𝜆𝑞𝑓^(𝜌)𝑞_ℬ+ℰ_𝜌𝑓^(𝜌)−ℰ_𝜌𝑓_𝜌=𝐷_𝑞𝑓_𝜌,𝜆𝑞. (36)
We then have (34).

Proof of (35). Since𝑓^(𝜌)is the unique solution of (27), we have 0∈𝜕_𝑓ℰ_𝜌𝑓+𝜆𝑞𝑓^𝑞_{ℬ𝑓=𝑓(𝜌)}. (37)
Notice that bothℰ_𝜌(𝑓)and‖𝑓‖^𝑞_ℬare convex functions about𝑓onℬ. We have 𝜕_𝑓ℰ_𝜌𝑓+𝜆𝑞𝑓^𝑞_{ℬ𝑓=𝑓(𝜌)}=𝜕_𝑓ℰ_𝜌𝑓_{𝑓=𝑓(𝜌)}+𝜕_𝑓𝜆𝑞𝑓^𝑞_{ℬ𝑓=𝑓(𝜌)}. (38)
By (30), we know that (37) leads to 0∈−2_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾(⋅,𝑥)𝑑𝜌+𝜆𝐽_𝑞𝑓^(𝜌). (39)
Therefore, there is𝑗_𝑞(𝑓^(𝜌))∈𝐽_𝑞(𝑓^(𝜌))such that (35) holds.

Lemma 8. Letℬbe an RKBS satisfying the conditions of Theorem 3. Then, 𝑓(𝑥)≤𝑘𝑓_ℬ,𝑥∈𝑋,𝑓∈ℬ. (40)

Proof. The reproducing property and (16) give 𝑓(𝑥)=𝑓,𝐾(⋅,𝑥)_ℬ≤𝑓_ℬ×‖𝐾(⋅,𝑥)‖_ℬ^∗. (41)
Then, the fact‖𝐾(⋅,𝑥)‖_ℬ^∗≤𝑘gives (40).

Lemma 9. Let 𝐾(𝑥,𝑦) be the reproducing kernel of ℬ,   and 𝐾(⋅,𝑥) is uniform continuous about 𝑥 on 𝑋 in norm‖⋅‖_ℬ^∗, 𝑅>0 be a given real number. Then, the ball ℬ_𝑅={𝑓∈ℬ∶‖𝑓‖_ℬ≤𝑅} is a compact subset of 𝐶(𝑋).

Proof. Since𝑋is a compact distance space, so is 𝑋×𝑋. Since𝐾(⋅,𝑥)is uniform continuous about𝑥in norm‖⋅‖_ℬ^∗, we know that for any𝜖>0there is a𝛿>0such that for all𝑥,𝑥^′∈𝑋 with 𝑑(𝑥,𝑥^′)<𝛿, we have 𝐾(⋅,𝑥)−𝐾(⋅,𝑥^′)_ℬ∗<𝜖 (42)
and for any𝑓∈ℬ_𝑅holds 𝑓(𝑥)−𝑓𝑥^′=𝑓,𝐾(⋅,𝑥)_ℬ−𝑓,𝐾⋅,𝑥^′_ℬ=𝑓,𝐾(⋅,𝑥)−𝐾⋅,𝑥^′_ℬ≤𝑓_ℬ×𝐾(⋅,𝑥)−𝐾⋅,𝑥^′_ℬ∗≤𝑅𝜖. (43)
By (43), we know that ℬ_𝑅is a closed, bounded, and equicontinuous set. Therefore, ℬ_𝑅 is a compact set of 𝐶(𝑋).

Proof of Theorem 5. By the definition of𝜕ℰ_𝛾(𝑓^(𝜌))and (30) we know ℰ_𝛾𝑓^(𝛾)−ℰ_𝛾𝑓^(𝜌)≥𝑓^(𝛾)−𝑓^(𝜌),−2_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾(⋅,𝑥)𝑑𝛾_ℬ. (44)
Also, by (44) and the definitions of 𝑓^(𝜌) and 𝑓^(𝛾) we have 0≥_𝑍𝑦−𝑓^(𝛾)(𝑥)²𝜆𝑑𝛾+𝑞𝑓^(𝛾)𝑞_ℬ−_𝑍𝑦−𝑓^(𝜌)(𝑥)²𝜆𝑑𝛾+𝑞𝑓^(𝜌)𝑞_ℬ=ℰ_𝛾𝑓^(𝛾)−ℰ_𝛾𝑓^(𝜌)+𝜆𝑞𝑓^(𝛾)𝑞_ℬ−𝑓^(𝜌)𝑞_ℬ≥2𝑓^(𝛾)−𝑓^(𝜌),−_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾(⋅,𝑥)𝑑𝛾_ℬ+𝜆𝑞𝑓^(𝛾)𝑞_ℬ−𝑓^(𝜌)𝑞_ℬ. (45)
Sinceℬis𝑞-uniform convex, we have by (14) and the definition of𝑗_𝑞(𝑓^(𝜌))that 1𝑞𝑓^(𝛾)𝑞_ℬ−𝑓^(𝜌)𝑞_ℬ≥𝑓^(𝛾)−𝑓^(𝜌),𝑗_𝑞𝑓^(𝜌)_ℬ+𝑐_𝑞𝑓^(𝛾)−𝑓^(𝜌)𝑞_ℬ. (46)
Combining (46) with (45), we have 0≥_𝑍𝑦−𝑓^(𝛾)(𝑥)²𝜆𝑑𝛾+𝑞𝑓^(𝛾)𝑞_ℬ−_𝑍𝑦−𝑓^(𝜌)(𝑥)²𝜆𝑑𝛾+𝑞𝑓^(𝜌)𝑞_ℬ≥2𝑓^(𝜌)−𝑓^(𝛾),_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾(⋅,𝑥)𝑑𝛾_ℬ+𝜆𝑓^(𝛾)−𝑓^(𝜌),𝑗_𝑞𝑓^(𝜌)_ℬ+𝜆𝑐_𝑞𝑓^(𝛾)−𝑓^(𝜌)𝑞_ℬ()35=2𝑓^(𝜌)−𝑓^(𝛾),_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾(⋅,𝑥)𝑑𝛾_ℬ−2𝑓^(𝜌)−𝑓^(𝛾),_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾(⋅,𝑥)𝑑𝜌_ℬ+𝜆𝑐_𝑞𝑓^(𝛾)−𝑓^(𝜌)𝑞_ℬ=2_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾()−⋅,𝑥𝑑𝛾_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾(⋅,𝑥)𝑑𝜌,𝑓^(𝜌)−𝑓^(𝛾)_ℬ+𝜆𝑐_𝑞𝑓^(𝛾)−𝑓^(𝜌)𝑞_ℬ. (47)
It follows that 𝜆𝑐_𝑞𝑓^(𝛾)−𝑓^(𝜌)𝑞_ℬ≤2𝑓^(𝛾)−𝑓^(𝜌),_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾()−⋅,𝑥𝑑𝛾_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾(⋅,𝑥)𝑑𝜌_ℬ≤2𝑓^(𝛾)−𝑓^(𝜌)_ℬ×_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾()−⋅,𝑥𝑑𝜌_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾(⋅,𝑥)𝑑𝛾_ℬ∗. (48)
We then have (29).

4. Sample Error

We give the following sample error bounds.

Theorem 10. Letℬbe an RKBS satisfying the conditions of Theorem 3.𝑓^(𝜌)is the solution of scheme (27) with respect to𝜌and𝑓_𝑧is the solution of (1). Then, for all 𝜖>0there hold Prob_𝑧∈𝑍^𝑚𝑓^(𝜌)−𝑓_𝑧ℬ≤𝜖≥1−2𝒩ℱ,𝜆𝑐_𝑞𝜖^𝑞−18exp−𝑚𝜆²𝑐²_𝑞𝜖^2(𝑞−1)32𝛾²_𝑞, (49)
where ℱ=ℎ(𝑧)𝑓(𝑥)∶𝑓∈ℬ,𝑓_ℬ≤1,‖ℎ‖_𝐶(𝑍)≤𝛾_𝑞. (50)

To show Theorem 10, we first give a lemma.

Lemma 11 (see [15]). Letℱbe a family of functions from a probability space𝑍toℜand 𝑑(⋅,⋅) a distance onℱ. Let𝒰⊂𝑍be of full measure and constants𝐵,𝐿>0such that(i)|𝜉(𝑧)|≤𝐵 for all 𝜉∈ℱ and all 𝑧∈𝒰,(ii)|𝐿_𝑧(𝜉₁)−𝐿_𝑧(𝜉₂)|≤𝐿𝑑(𝜉₁,𝜉₂) for all 𝜉₁,𝜉₂∈ℱ and all 𝑧∈𝒰^𝑚, where 𝐿_𝑧(𝜉)=_𝑍𝜉(𝑧)−1𝑚^𝑚_𝑖=1𝜉𝑧_𝑖. (51)
Then, for all𝜖>0, Prob_𝑧∈𝑍^𝑚sup_𝜉∈ℱ𝐿_𝑧(𝜉𝜖)≤𝜖≥1−𝒩ℱ,−2𝐿×2exp𝑚𝜖²8𝐵². (52)

Proof of Theorem 10. Take 𝛾=𝛾_𝑧 into (29). Then, 𝑓^(𝜌)−𝑓_𝑧ℬ≤2𝜆𝑐_𝑞×_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾()−1⋅,𝑥𝑑𝜌𝑚^𝑚_𝑖=1𝑦_𝑖−𝑓^(𝜌)𝑥_𝑖𝐾⋅,𝑥_𝑖ℬ∗^1/(𝑞−1). (53)
By (7) and the reproducing property, we have _𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾()−1⋅,𝑥𝑑𝜌𝑚^𝑚_𝑖=1𝑦_𝑖−𝑓^(𝜌)𝑥_𝑖𝐾⋅,𝑥_𝑖ℬ∗=sup_𝑓ℬ≤1𝑓,_𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾()−1⋅,𝑥𝑑𝜌𝑚^𝑚_𝑖=1𝑦_𝑖−𝑓^(𝜌)𝑥_𝑖𝐾⋅,𝑥_𝑖ℬ=sup_{𝑓ℬ≤1𝑍}𝑦−𝑓^(𝜌)(𝑥)𝑓,𝐾(⋅,𝑥)_ℬ−1𝑑𝜌𝑚^𝑚_𝑖=1𝑦_𝑖−𝑓^(𝜌)𝑥_𝑖𝑓,𝐾⋅,𝑥_𝑖ℬ=sup_{𝑓ℬ≤1𝑍}𝑦−𝑓^(𝜌)(𝑥)𝑓(𝑥)−1𝑑𝜌𝑚^𝑚_𝑖=1𝑦_𝑖−𝑓^(𝜌)𝑥_𝑖𝑓𝑥_𝑖. (54)
Since 𝑦−𝑓^(𝜌)(𝑥)≤𝑦+𝑓^(𝜌)(𝑥)≤𝑀+𝑘𝑞𝐷_𝑞𝑓_𝜌,𝜆/𝑞𝜆^1/𝑞 (55)
and (40), we have _𝑍𝑦−𝑓^(𝜌)(𝑥)𝐾()−1⋅,𝑥𝑑𝜌𝑚^𝑚_𝑖=1𝑦_𝑖−𝑓^(𝜌)𝑥_𝑖𝐾⋅,𝑥_𝑖ℬ∗≤sup_{𝑓∈ℱ𝑍}𝑓(𝑧)1𝑑𝜌−𝑚^𝑚_𝑖=1𝑓𝑧_𝑖. (56)
Define 𝐿_𝑧𝑓=_𝑍𝑓(𝑧)1𝑑𝜌−𝑚^𝑚_𝑖=1𝑓𝑧_𝑖. (57)
Then, 𝐿_𝑧𝑓₁−𝐿_𝑧𝑓₂≤_𝑍𝑓₁(𝑧)−𝑓₂(𝑧)+1𝑑𝜌𝑚^𝑚_𝑖=1𝑓₁𝑧_𝑖−𝑓₂𝑧_𝑖≤2𝑓₁−𝑓_2𝐶(𝑍). (58)
By (52), we have for all𝜖>0that Prob_𝑧∈𝑍^𝑚sup_𝑓∈ℱ𝐿_𝑧𝑓𝜖≥𝜖≤2𝒩ℱ,4exp−𝑚𝜖²8𝛾²_𝑞. (59)
By (53), (56), and (59), we know 𝜆𝑐_𝑞2𝑓^(𝜌)−𝑓_𝑧ℬ^𝑞−1≤sup_{𝑓∈ℱ𝑍}𝑓(𝑧)1𝑑𝜌−𝑚^𝑚_𝑖=1𝑓𝑧_𝑖, (60)
which gives 𝑧∈𝑍^𝑚∶𝜆𝑐_𝑞2𝑓^(𝜌)−𝑓_𝑧ℬ^𝑞−1⊂≥𝜖𝑧∈𝑍^𝑚∶sup_𝑓∈ℱ𝐿_𝑧𝑓≥𝜖. (61)
It follows that Prob_𝑧∈𝑍^𝑚𝜆𝑐_𝑞2𝑓^(𝜌)−𝑓_𝑧ℬ^𝑞−1≥𝜖≤Prob_𝑧∈𝑍^𝑚sup_𝑓∈ℱ𝐿_𝑧𝑓𝜖≥𝜖≤2𝒩ℱ,4exp−𝑚𝜖²8𝛾²_𝑞. (62)
That is, Prob_𝑧∈𝑍^𝑚𝑓^(𝜌)−𝑓_𝑧ℬ≥𝜖≤2𝒩ℱ,𝜆𝑐_𝑞𝜖^𝑞−18exp−𝑚𝜆²𝑐²_𝑞𝜖^2(𝑞−1)32𝛾²_𝑞. (63)
We then have (49).

5. Learning Rates

Proof of Theorem 3. We know from [30] that for any𝑓∈𝐿²(𝜌_𝑋)there holds 𝑓−𝑓_𝜌²_{𝐿2(𝜌𝑋)}=ℰ_𝜌𝑓−ℰ_𝜌𝑓_𝜌. (64)
Since𝑋is a compact set, we have by (40) that‖𝑓‖_{𝐿2(𝜌𝑋)}≤𝑘‖𝑓‖_ℬ.Therefore, 𝑓_𝑧−𝑓_{𝜌𝐿2(𝜌𝑋)}≤𝑓_𝑧−𝑓^(𝜌)_{𝐿2(𝜌𝑋)}+ℰ_𝜌𝑓^(𝜌)+𝜆𝑞𝑓^(𝜌)𝑞_ℬ−ℰ_𝜌𝑓_𝜌≤𝑘𝑓_𝑧−𝑓^(𝜌)_ℬ+inf_𝑓∈ℬ𝑓−𝑓_𝜌²_{𝐿2𝜌𝑋}+𝜆𝑞𝑓^𝑞_ℬ=𝑘𝑓_𝑧−𝑓^(𝜌)_ℬ+𝐷_𝑞𝑓_𝜌,𝜆𝑞. (65)
By (65) we have 1𝑘𝑓_𝑧−𝑓_{𝜌𝐿2(𝜌𝑋)}−𝐷_𝑞𝑓_𝜌,𝜆𝑞≤𝑓_𝑧−𝑓^(𝜌)_ℬ, (66)
which gives for any ℎ>0 that 𝑧∈𝑍^𝑚∶1𝑘𝑓_𝑧−𝑓_{𝜌𝐿2(𝜌𝑋)}−𝐷_𝑞𝑓_𝜌,𝜆𝑞⊂>ℎ𝑧∈𝑍^𝑚∶𝑓_𝑧−𝑓^(𝜌)_ℬ>ℎ. (67)
By (49) and above inequality we have Prob_𝑧∈𝑍^𝑚1𝑘𝑓_𝑧−𝑓_{𝜌𝐿2(𝜌𝑋)}−𝐷_𝑞𝑓_𝜌,𝜆𝑞>ℎ≤Prob_𝑧∈𝑍^𝑚𝑓_𝑧−𝑓^(𝜌)_ℬ>ℎ≤2𝒩ℱ,𝜆𝑐_𝑞ℎ^𝑞−18exp−𝑚𝜆²𝑐²_𝑞ℎ^2(𝑞−1)32𝛾²_𝑞 (68)
or Prob_𝑧∈𝑍^𝑚𝑓_𝑧−𝑓_{𝜌𝐿2(𝜌𝑋)}≤𝑘ℎ+𝐷_𝑞𝑓_𝜌,𝜆𝑞≥1−2𝒩ℱ,𝜆𝑐_𝑞ℎ^𝑞−18×exp−𝑚𝜆²𝑐²_𝑞ℎ^2(𝑞−1)32𝛾²_𝑞. (69)
Sinceℱ⊂ℬ₁,we know 𝒩ℱ,𝜆𝑐_𝑞ℎ^𝑞−18≤𝒩ℬ₁,𝜆𝑐_𝑞ℎ^𝑞−18. (70)
By (69) and above inequality we have (16).

To show Theorem 4, we need two lemmas.

Lemma 12 (see [31]). Let𝑐₁>0,  𝑐₂>0and 𝑢>𝑡>0.Then, the equation 𝑥^𝑢−𝑐₁𝑥^𝑡−𝑐₂=0 (71)
has a unique positive zero𝑥^∗. In addition, 𝑥^∗≤max2𝑐₁^{1/(𝑢−𝑡)},2𝑐₂^1/𝑢. (72)

Proof of Theorem 4. Since(ℬ,‖⋅‖_ℬ)has logarithmic complexity exponent𝑠≥0, we have by (15) a constant𝑐_𝑠>0such that log𝒩ℬ₁,𝜆𝑐_𝑞ℎ^𝑞−18≤𝑐_𝑠8𝛾_𝑞𝜆𝑐_𝑞𝜖^𝑞−1𝑠. (73)
Then, by (16) we have Prob_𝑧∈𝑍^𝑚𝑓_𝑧−𝑓_{𝜌𝐿2(𝜌𝑋)}≤𝑘𝜖+𝐷_𝑞𝑓_𝜌𝑐,𝜆≥1−2exp_𝑠8𝛾_𝑞𝜆𝑐_𝑞𝜖^𝑞−1𝑠−𝑚𝜆²𝑐²_𝑞𝜖^2(𝑞−1)32𝛾²_𝑞. (74)
Take 2exp𝑐_𝑠8𝛾_𝑞𝜆𝑐_𝑞𝜖^𝑞−1𝑠−𝑚𝜆²𝑐²_𝑞𝜖^2(𝑞−1)32𝛾²_𝑞=𝛿. (75)
Then, 𝜖^{(2+𝑠)(𝑞−1)}−32𝛾²_𝑞log(2/𝛿)𝑚𝜆²𝑐²_𝑞𝜖^{𝑠(𝑞−1)}−32𝛾²_𝑞𝑐_𝑠𝑚𝜆²𝑐²_𝑞×8𝛾_𝑞𝜆𝑐_𝑞^𝑠=0. (76)
By Lemma 12, we know that the unique solution𝜖^∗of (75) satisfies 𝜖^∗≤max64𝛾²_𝑞log(2/𝛿)𝑚𝜆²𝑐²_𝑞^{1/2(𝑞−1)},64𝛾²_𝑞𝑐_𝑠𝑚𝜆²𝑐²_𝑞×8𝛾_𝑞𝜆𝑐_𝑞^{𝑠1/(2+𝑠)(𝑞−1)}≤64𝛾²_𝑞log(2/𝛿)𝑚𝜆²𝑐²_𝑞^{1/2(𝑞−1)}+8^2+𝑠𝛾_𝑞^2+𝑠𝑐_𝑠𝑚𝜆^2+𝑠𝑐_𝑞^{2+𝑠1/(2+𝑠)(𝑞−1)}. (77)By (74) and (77), we have (19).

Acknowledgments

This work was supported partially by the National Natural Science Foundation of China under Grant nos. 10871226, 61179041, 11271199. The authors thank the reviewers for giving many valuable suggestions and comments which make the paper presented in a better form.