Regularized Least Square Regression with Unbounded and Dependent Sampling

Abstract

This paper mainly focuses on the least square regression problem for the 𝛼-mixing and 𝜙-mixing processes. The standard bound assumption for output data is abandoned and the learning algorithm is implemented with samples drawn from dependent sampling process with a more general output data condition. Capacity independent error bounds and learning rates are deduced by means of the integral operator technique.

1. Introduction and Main Results

The aim of this paper is to study the least square regularized regression learning algorithm. The main novelty of this problem here is the unboundedness and dependence of the sampling process. Let 𝑋 be a compact metric space (usually a subset of ℝ^𝑛) and 𝑌=ℝ. Suppose that 𝜌 is a probability distribution defined on 𝑍=𝑋×𝑌. In regression learning, one wants to learn or approximate the regression function 𝑓_𝜌∶𝑋→𝑌 given by 𝑓_𝜌(𝑥)=𝔼𝑦∣𝑥=_𝑌𝑦𝑑𝜌𝑦∣𝑥,𝑥∈𝑋, (1) where 𝜌(𝑦∣𝑥) is the conditional distribution of 𝑦 for given 𝑥. 𝑓_𝜌 is not directly computable because 𝜌 is unknown in fact. Instead we learn a good approximation of 𝑓_𝜌 from a set of observations 𝐳={(𝑥_𝑖,𝑦_𝑖)}^𝑚_𝑖=1∈𝑍^𝑚 drawn according to 𝜌.

The learning algorithm studied here is based on a Mercer kernel 𝐾∶𝑋×𝑋→ℝ which is a continuous, symmetric, and positive semidefinite function. The RKHS 𝓗_𝐾 associated with the Mercer kernel 𝐾 is the completion of span {𝐾_𝑥=𝐾(⋅,𝑥)∶𝑥∈𝑋} with the inner product satisfying ⟨𝐾(𝑥,⋅),𝐾(𝑥^′,⋅)⟩_𝐾= 𝐾(𝑥,𝑥^′). The learning algorithm is a regularization scheme in 𝓗_𝐾 given by 𝑓_𝐳,𝜆=argmin_{𝑓∈𝓗𝐾}1𝑚^𝑚_𝑖=1𝑓𝑥_𝑖−𝑦_𝑖²+𝜆𝑓²_𝐾, (2) where 𝜆>0 is a regularization parameter.

Error analysis for learning algorithm (2) has been studied in a lot of literatures [1–4], which focused on independent samples. In recent years, there are some studies relaxing the independent restriction and turning to the dependent sampling learning [5–8]. In [8] the learning performance of regularized least square regression was studied with the mixing sequences, and the result for this setting was refined by an operator monotone inequality in [7].

For a stationary real-valued sequence {𝑧_𝑖}_𝑖≥1, the 𝜎-algebra generated by the random variables 𝑧_𝑎,𝑧_𝑎+1, …, 𝑧_𝑏 is denoted by ℳ^𝑏_𝑎. The uniformly mixing condition (or 𝜙-mixing condition) and the strongly mixing condition (or 𝛼-mixing condition) are defined as follows.

Definition 1 (𝜙-mixing). The 𝑙th 𝜙-mixing coefficient for the sequence is defined as 𝜙_𝑙=sup_𝑘≥1sup_{𝐴∈ℳ𝑘1,𝐵∈ℳ∞𝑘+𝑙}|𝑃(𝐴∣𝐵)−𝑃(𝐴)|. (3) The process {𝑧_𝑖}_𝑖≥1 is said to satisfy a uniformly mixing condition (or 𝜙-mixing condition) if 𝜙_𝑙→0, as 𝑙→∞.

Definition 2 (𝛼-mixing). The 𝑙th 𝛼-mixing coefficient for random sequence {𝑧_𝑖}_𝑖≥1 is defined as 𝛼_𝑙=sup_𝑘≥1sup_{𝐴∈ℳ𝑘1,𝐵∈ℳ∞𝑘+𝑙}|𝑃(𝐴∩𝐵)−𝑃(𝐴)𝑃(𝐵)|. (4) The random process {𝑧_𝑖}_𝑖≥1 is said to satisfy a strongly mixing condition (or 𝛼-mixing condition) if 𝛼_𝑙→0, as 𝑙→∞.

By the fact 𝑃(𝐴∩𝐵)=𝑃(𝐴∣𝐵)𝑃(𝐵), 𝛼-mixing condition is weaker than 𝜙-mixing condition. Many random processes satisfy the strongly mixing condition, for example, the stationary Markov process which is uniformly pure nondeterministic, the stationary Gaussian sequence with a continuous spectral density that is bounded away from 0, certain ARMA processes, and some aperiodic, Harris-recurrent Markov processes; see [5, 9] and the references therein.

In this paper we follow [7, 8] to consider 𝛼-mixing and 𝜙-mixing processes, estimate the error bounds, and derive the learning rates of algorithm (2), where the output data satisfy the following unbounded condition.

Unbounded Hypothesis. There exist two constants 𝑀>0 and 𝑝≥2 such that 𝔼𝑦^𝑝≤𝑀. (5)

The error analysis for the algorithm (2) was usually presented under the standard assumption that |𝑦|≤𝑀 almost surely with some constant 𝑀>0. This standard assumption was abandoned in [10–14]. In [10] the authors introduced the condition exp−_𝑌𝑦−𝑓_ℋ²𝑀−𝑦−𝑓_ℋ(𝑥)𝑀≤Σ−1𝑑𝜌𝑦∣𝑥²2𝑀² (6) for almost every 𝑥∈𝑋 and some constants 𝑀,Σ>0, where 𝑓_ℋ is the orthogonal projection of 𝑓_𝜌 onto the closure of 𝓗_𝐾 in 𝐿²_𝜌𝑋(𝑋). In [11–13] the error analysis was conducted in another setting satisfying the following moment hypothesis; that is, there exist constants 𝑀^∼>0 and 𝐶>0 such that ∫_𝑌|𝑦|^𝑙𝑑𝜌(𝑦∣𝑥)≤𝐶𝑙!𝑀^∼𝑙 for all 𝑙∈ℕ,𝑥∈𝑋. Notice that with different constants the moment hypothesis and (6) are equivalent in the case 𝑓_ℋ∈𝐿^∞(𝑋)  [13]. Obviously, our unbounded hypothesis is a natural generalization of the moment hypothesis. An example for which unbounded hypothesis (5) is satisfied but moment hypothesis failed has been given in [15]. It mainly studies the half supervised coefficient regularization with indefinite kernels and unbounded sampling, where the unbounded condition is ∫_𝑍𝑦²𝑑𝜌≤𝑀² for some constant 𝑀>0.

Since ℰ(𝑓_𝜌)=minℰ(𝑓), where the generalization error ℰ(𝑓)=∫_𝑍(𝑓(𝑥)−𝑦)²𝑑𝜌, the goodness of the approximation of 𝑓_𝜌 by 𝑓_𝐳,𝜆 is usually measured by the excess generalization error ℰ(𝑓_𝐳,𝜆)− ℰ(𝑓_𝜌)= ‖𝑓_𝐳,𝜆−𝑓_𝜌‖²_𝜌𝑋. Denoting 𝜅∶=sup_𝑥∈𝑋𝐾(𝑥,𝑥)<∞, (7) the reproducing property in RKHS 𝓗_𝐾 yields that ‖𝑓‖_∞≤𝜅‖𝑓‖_𝐾 for any 𝑓∈𝓗_𝐾. Thus, the distance between 𝑓_𝐳,𝜆 and 𝑓_𝜌 in 𝓗_𝐾 can be applied to measure this approximation as well when 𝑓_𝜌∈𝓗_𝐾.

The noise-free limit of algorithm (2) takes the form 𝑓_𝜆∶=argmin_{𝑓∈𝓗𝐾}𝑓−𝑓_𝜌²_𝜌𝑋+𝜆𝑓²_𝐾, (8) thus the error analysis can be divided into two parts. The difference between 𝑓_𝐳,𝜆 and 𝑓_𝜆 is called the sample error, and the distance between 𝑓_𝜆 and 𝑓_𝜌 is called the approximation error. We will bound the error in 𝐿²_𝜌𝑋(𝑋) and 𝓗_𝐾, respectively. Estimate of the sample error is more difficult because 𝑓_𝐳,𝜆 changes with the sample 𝐳 and cannot be considered as a fixed function. The approximation error does not depend on the samples, which has been studied in the literature [2, 3, 7, 16, 17].

We mainly devote the next two sections to estimating the sample error with more general sampling processes. Our main results can be stated as follows.

Theorem 3. Suppose that the unbounded hypothesis holds, 𝐿_𝐾^−𝑟𝑓_𝜌∈𝐿²_𝜌𝑋(𝑋) for some 𝑟>0, and the 𝜙-mixing coefficients satisfy a polynomial decay, that is, 𝜙_𝑖≤𝑎𝑖^−𝑡 for some 𝑎>0 and 𝑡>0. Then, for any 0<𝜂<1, one has with confidence 1−𝜂, 𝑓_𝐳,𝜆−𝑓_𝜌𝜌𝑋=𝑂𝑚^{−𝜃min{𝑡/2,1}}log𝑚^3/4, (9) where 𝜃 is given by 𝜃=3𝑟4()1𝑟+1𝑖𝑓0<𝑟<2,𝑟12𝑟+1𝑖𝑓21≤𝑟<1,3𝑖𝑓𝑟≥1. (10)
Moreover, when 𝑟>1/2, one has with confidence 1−𝜂, 𝑓_𝐳,𝜆−𝑓_𝜌𝐾=𝑂𝑚^{−𝜃′min{𝑡/2,1}}log𝑚^1/2, (11) where 𝜃^′ is given by 𝜃^′=2𝑟−12()12𝑟+1𝑖𝑓23<𝑟<2,143𝑖𝑓𝑟≥2. (12)

Theorem 3 proves the asymptotic convergence of algorithm (2) with the samples satisfying a uniformly mixing condition. Our second main result considers this algorithm with 𝛼-mixing process.

Theorem 4. Suppose that the unbounded hypothesis with 𝑝>2 holds, 𝐿_𝐾^−𝑟𝑓_𝜌∈𝐿²_𝜌𝑋(𝑋) for some 𝑟>0, and the 𝛼-mixing coefficients satisfy a polynomial decay, that is, 𝛼_𝑙≤𝑏𝑙^−𝑡 for some 𝑏>0 and 𝑡>0. Then, for any 0<𝜂<1, one has with confidence 1−𝜂, 𝑓_𝐳,𝛾−𝑓_𝜌𝜌𝑋=𝑂𝑚^{−𝜗min{(𝑝−2)𝑡/𝑝,1}}log𝑚^1/2, (13) where 𝜗 is given by 𝜗=𝑝𝑟212𝑟+𝑝−1𝑖𝑓0<𝑟<2𝑝,0<𝑡<;𝑝−23𝑝𝑟214𝑟+3𝑝−2𝑖𝑓0<𝑟<2𝑝,𝑡≥;𝑟𝑝−212𝑟+1𝑖𝑓21≤𝑟<1;3𝑖𝑓𝑟≥1. (14)
Moreover, when 𝑟>1/2, with confidence 1−𝜂, 𝑓_𝐳,𝛾−𝑓_𝜌𝐾=𝑂𝑚^{−𝜗′min{(𝑝−2)𝑡/𝑝,1}}log𝑚^1/2, (15) where 𝜗^′ is given by 𝜗^′=2𝑟−114𝑟+2𝑖𝑓23<𝑟<2,143𝑖𝑓𝑟≥2. (16)

The proof of these two theorems will be given in Sections 2, 3, and 4, and notice that the log term can be dropped when 𝑡≠2. Our error analysis reveals some interesting phenomena for learning with unbounded and dependent sampling.(i)Smoother target function 𝑓_𝜌 (i.e., 𝑟 becomes larger) implies better learning rates. Stronger dependence between samples (i.e., 𝑡 becomes smaller) implies that they contain less information and hence lead to worse rates.(ii) The learning rates are improved as the dependence between samples becomes weaker and 𝑟 becomes larger but they are no longer improved after some constant 𝑡,𝑟. This phenomenon is called saturation effect, which was discussed in [18–20]. In our setting, saturation effects include saturation for smoothness of function 𝑓_𝜌 mainly relative to the approximation error and saturation for dependence between samples. An interesting phenomenon revealed here is that when 𝛼-mixing coefficients satisfy 𝛼_𝑙≤𝑂(𝑙^−𝑡),𝑙∈ℕ for some 𝑡>0, the saturation for dependence between samples is 𝑡=𝑝/(𝑝−2) for 𝑝>2, which is dependent on the unbounded condition parameter 𝑝.(iii) For 𝜙-mixing process, the learning rates have nothing to do with unbound condition parameter 𝑝 since 𝔼(𝑦−𝑓_𝜆(𝑥))² is bounded by 𝔼𝑦²<∞. But for 𝛼-mixing process, to derive the learning rate, we have to estimate 𝔼|𝑦−𝑓_𝜆(𝑥)|^𝑝 with 𝑝>2.(iv) Under 𝛼-mixing condition, when 𝑡>𝑝/(𝑝−2) and 𝑟≥1/2, the influence of the unbounded condition becomes weak. Recall that the learning rate derived in [8] is 𝑂(𝑚^{−𝑟/(1+2𝑟)}) for 1/2≤𝑟≤1, 𝑡≥1. It implies that when 𝑡 is large enough, our learning rate for unbounded samples is as sharp as that for the uniform bounded sampling.

2. Sampling Satisfying 𝜙-Mixing Condition

In this section, we would apply the integral operator technique in [7] to handle the sample error with 𝜙-mixing condition. However, different from the uniform bounded case the learning performance of the unbounded sampling is not measured directly. Instead, the expectations are estimated first and then the bound for the sample error can be obviously deduced by Markov inequality:

To this end, define the sampling operator 𝑆_𝐱∶𝓗_𝐾→𝑙²(𝐱) as 𝑆_𝐱(𝑓)=(𝑓(𝑥_𝑖))^𝑚_𝑖=1, where 𝐱 is the set of input data {𝑥₁,…,𝑥_𝑚}. Then its adjoint is 𝑆^𝐓_𝐱𝑐=∑^𝑚_𝑖=1𝑐_𝑖𝐾_𝑥𝑖 for 𝑐∈𝑙²(𝐱). The analytic expression of optimization solution 𝑓_𝐳,𝜆,𝑓_𝜆 was given in [3], 𝑓_𝐳,𝜆=1𝑚𝑆^𝑇_𝐱𝑆_𝐱+𝜆𝐼⁻¹1𝑚𝑆^𝑇_𝐱𝑓𝑦,_𝜆=𝐿_𝐾+𝜆𝐼⁻¹𝐿_𝐾𝑓_𝜌, (17) where 𝐿_𝐾∶𝐿²_𝜌𝑋(𝑋)→𝐿²_𝜌𝑋(𝑋) is the integral operator defined as 𝐿_𝐾𝑓(𝑥)=_𝑋𝐾(𝑥,𝑡)𝑓(𝑡)𝑑𝜌_𝑋(𝑡),forany𝑥∈𝑋. (18)

For a random variable 𝜉 with values in a Hilbert space ℋ and 0≤𝑢≤+∞, denote the 𝑢th moment as ‖𝜉‖_𝑢=(𝔼‖𝜉‖^𝑢_ℋ)^1/𝑢 if 1≤𝑢<∞ and ‖𝜉‖_∞=sup‖𝜉‖_ℋ. Lemma 5 is due to Billingsley [21].

Lemma 5. Let 𝜉 and 𝜂 be random variables with values in a separable Hilbert space ℋ measurable 𝜎-field 𝒥 and 𝒟 and having finite 𝑝th and 𝑞th moments, respectively, where 𝑝,𝑞≥1 with 𝑝⁻¹+𝑞⁻¹=1. Then 𝔼𝜉,𝜂−𝔼𝜉,𝔼𝜂≤2𝜙^1/𝑝(𝒥,𝒟)𝜉_𝑝𝜂_𝑞. (19)

Lemma 6. For an 𝜙-mixing sequence {𝑥_𝑖}, one has 𝔼𝐿_𝐾−1𝑚𝑆^𝑇_𝐱𝑆_𝐱²≤𝜅⁴𝑚1+4^𝑚−1_𝑖=1𝜙_𝑖^1/2. (20)

Proof. With the definition of the sample operator, we have 𝐿_𝐾−1𝑚𝑆^𝑇_𝐱𝑆_𝐱=𝐿_𝐾−1𝑚^𝑚_𝑖=1𝐾_𝑥𝑖⊗𝐾_𝑥𝑖. (21) Letting 𝜂(𝑥)=𝐾_𝑥⊗𝐾_𝑥, then 𝜂(𝑥) is an HS(ℋ_𝐾)-valued random variable defined on 𝑋. Note that 𝔼𝜂(𝑥)=𝐿_𝐾∈HS(ℋ_𝐾), and ‖𝐿_𝐾‖_HS≤𝜅²,‖𝜂(𝑥)‖_HS≤𝜅². We have 𝔼𝐿_𝐾−1𝑚𝑆^𝑇_𝐱𝑆_𝐱²1≤𝔼𝔼𝜂−𝑚^𝑚_𝑖=1𝜂𝑥_𝑖²_HS=1𝑚𝜂²₂+1𝑚²_𝑖≠𝑗𝔼𝜂𝑥_𝑖,𝜂𝑥_𝑗HS−𝐿_𝐾²_HS. (22)
By Lemma 5 with 𝑝=𝑞=2, for 𝑖≠𝑗, 𝔼𝜂𝑥_𝑖,𝜂𝑥_𝑗HS≤𝔼𝜂𝑥_𝑖,𝔼𝜂𝑥_𝑗HS+2𝜙^1/2_{|𝑖−𝑗|}𝜂²₂≤𝐿_𝐾²_HS+2𝜅⁴𝜙^1/2_{|𝑖−𝑗|}. (23) Thus the desired estimate can be obtained by plugging (23) into (22).

Proposition 7. Suppose that the unbounded hypothesis holds with some 𝑝≥2 and that the sample sequence {(𝑥_𝑖,𝑦_𝑖)}^𝑚_𝑖=1 satisfies an 𝜙-mixing condition and 𝐿_𝐾^−𝑟𝑓_𝜌∈𝐿²_𝜌𝑋(𝑋) with 𝑟>0. Then one has 𝔼𝑓_𝐳,𝜆−𝑓_𝜆𝜌𝑋≤𝐶𝜆^−1/2𝑚^−1/2+𝜆⁻¹𝑚^−3/41+4^𝑚−1_𝑖=1𝜙_𝑖^1/21/4×1+4^𝑚−1_𝑙=1𝜙_𝑖^1/2, (24) where 𝐶 is a constant only dependent on 𝜅,𝑀.

Proof. By [7, Theorem 3.1], we have 𝔼𝑓_𝐳,𝜆−𝑓_𝜆𝜌𝑋≤𝜆^−1/2+𝜆⁻¹𝔼𝐿_𝐾−1𝑚𝑆^𝑇_𝐱𝑆_𝐱^21/4×𝔼1𝑚^𝑚−1_𝑙=1𝜉(𝑧_𝑖)−𝐿_𝐾𝑓_𝜌−𝑓_𝜆²_𝐾, (25) where 𝜉(𝑧)=(𝑦−𝑓_𝜆(𝑥))𝐾_𝑥 is a random variable with values in 𝓗_𝐾, and 𝔼𝜉=𝐿_𝐾(𝑓_𝜌−𝑓_𝜆). A similar computation together with the result of Lemma 6 leads to 𝔼1𝑚^𝑚_𝑖=1𝜉𝑧_𝑖−𝐿_𝐾𝑓_𝜌−𝑓_𝜆²_𝐾≤1𝑚1+4^𝑚−1_𝑖=1𝜙_𝑖^1/2𝜉²₂. (26)
It suffices to estimate ‖𝜉‖₂. By Hölder inequality, there is 𝔼𝑦²≤𝔼𝑦^𝑝2/𝑝≤𝑀^2/𝑝,𝑓_𝜌²_𝜌𝑋=_𝑋𝑓²_𝜌(𝑥)𝑑𝜌_𝑋=_𝑋𝑌𝑦𝑑𝜌(𝑦∣𝑥)²𝑑𝜌_𝑋≤_𝑍𝑦²𝑑𝜌≤𝑀^2/𝑝. (27) Thus 𝔼(𝑦−𝑓_𝜌(𝑥))²=𝔼𝑦²−‖𝑓_𝜌‖²_𝜌𝑋≤𝑀^2/𝑝 and 𝔼𝑓_𝜌(𝑥)−𝑓_𝜆(𝑥)²=𝜆𝜆𝐼+𝐿_𝐾⁻¹𝑓_𝜌²_𝜌𝑋≤𝑓_𝜌²_𝜌𝑋≤𝑀^2/𝑝, (28) which implies 𝜉²₂=𝔼𝑦−𝑓_𝜆(𝑥)²𝐾(𝑥,𝑥)≤𝜅²𝔼𝑦−𝑓_𝜆(𝑥)²=𝜅²𝔼𝑦−𝑓_𝜌(𝑥)²+𝔼𝑓_𝜌(𝑥)−𝑓_𝜆(𝑥)²≤2𝜅²𝑀^2/𝑝. (29)
Plugging (29) into (26), there holds 𝔼1𝑚^𝑚_𝑖=1𝜉(𝑧_𝑖)−𝐿_𝐾𝑓_𝜌−𝑓_𝜆²_𝐾≤2𝑀^2/𝑝𝑘²𝑚⁻¹1+4^𝑚−1_𝑖=1𝜙_𝑖^1/2. (30)
Combining (25), (22), and (30) and taking the constant 𝐶=√2𝑘(𝑘+1)𝑀^1/𝑝, we complete the proof.

The following proposition provides the bound of the difference between 𝑓_𝐳,𝜆 and 𝑓_𝜆 in 𝓗_𝐾 with 𝜙-mixing process.

Proposition 8. Under the assumption of Proposition 7, there holds 𝔼𝑓_𝐳,𝜆−𝑓_𝜆𝐾≤√2𝑀^1/𝑝𝜅𝜆⁻¹𝑚^−1/21+4^𝑚−1_𝑖=1𝜙_𝑖^1/2. (31)

Proof. The representations of 𝑓_𝐳,𝜆 and 𝑓_𝜆 imply that 𝔼𝑓_𝐳,𝜆−𝑓_𝜆𝐾1=𝔼𝑚𝑆^𝑇_𝐱𝑆_𝐱+𝜆𝐼⁻¹1𝑚^𝑚−1_𝑙=1𝜉𝑧_𝑖−𝐿_𝐾𝑓_𝜌−𝑓_𝜆𝐾≤𝜆⁻¹𝔼1𝑚^𝑚−1_𝑙=1𝜉(𝑧_𝑖)−𝐿_𝐾𝑓_𝜌−𝑓_𝜆²_𝐾. (32) Then the desired bound follows from (30) and (32).

3. Samples Satisfying 𝛼-Mixing Condition

Now we turn to bound the sample error when the sampling process satisfies strongly mixing condition, and unbounded hypothesis holds. In Section 2, the key point is to estimate ‖𝜉‖₂ with the lack of uniform boundedness. For the sampling satisfying 𝛼-mixing condition, we have to deal with ‖𝜉‖_𝑝 for some 𝑝>2.

Proposition 9. Suppose that the unbounded hypothesis holds with some 𝑝>2 and that the sample sequence {(𝑥_𝑖,𝑦_𝑖)}^𝑚_𝑖=1 satisfies an 𝛼-mixing condition and 𝐿_𝐾^−𝑟𝑓_𝜌∈𝐿²_𝜌𝑋(𝑋) with 𝑟>0. Then one gets 𝔼𝑓_𝐳,𝜆−𝑓_𝜆𝜌𝑋≤𝐶^∼𝜆^{min{(𝑝−2)(2𝑟−1)/2𝑝,0}}1+^𝑚−1_𝑙=1𝛼_𝑙^{(𝑝−2)/𝑝}×𝜆^−1/2𝑚^−1/2+𝜆⁻¹𝑚^−3/41+^𝑚−1_𝑙=1𝛼_𝑙^1/4, (33) where 𝐶^∼ is a constant only depending on 𝜅,𝑀 and ‖𝐿_𝐾^{−min{𝑟,1/2}}𝑓_𝜌‖_𝜌𝑋.

Proof. For the strongly mixing process, by [8, Lemma 5.1], 𝔼𝐿_𝐾−1𝑚𝑆^𝑇_𝐱𝑆_𝐱²≤𝑘⁴𝑚1+30^𝑚−1_𝑙=1𝛼_𝑙. (34) Taking 𝛿=𝑝−2 in [8, Lemma 4.2], we have 𝔼1𝑚^𝑚_𝑖=1𝜉(𝑧_𝑖)−𝐿_𝐾𝑓_𝜌−𝑓_𝜆²_𝐾≤1𝑚𝜉²₂+30𝑚^𝑚−1_𝑙=1𝛼_𝑙^{(𝑝−2)/𝑝}𝜉²_𝑝. (35)
The estimation of ‖𝜉‖₂ has been obtained in Section 2, and now we mainly devote to estimating ‖𝜉‖_𝑝. To get this estimation, the bound of 𝑓_𝜆 is needed which can be stated as follows ([3, Lemma 3] or [8, Lemma 4.3]): 𝑓_𝜆(𝑥)≤𝜅𝑓_𝜆𝐾≤𝐶₁𝜅𝜆^{min{(2𝑟−1)/2,0}}, (36) where 𝐶₁=‖𝐿_𝐾^{−min{𝑟,1/2}}𝑓_𝜌‖_𝜌𝑋. Observe that ‖𝑓_𝜆‖²_𝜌𝑋≤‖𝑓_𝜌‖²_𝜌𝑋≤𝔼𝑦²≤𝑀^2/𝑝. Hence, 𝔼𝑓_𝜆(𝑥)^𝑝2/𝑝≤𝑓_𝜆²_𝜌𝑋𝐶₁^𝑝−2𝜅^𝑝−2𝜆^{(𝑝−2)min{(2𝑟−1)/2,0}2/𝑝}≤𝑀^4/𝑝2𝐶²₁𝜅²+1𝜆^{min{(𝑝−2)(2𝑟−1)/𝑝,0}}. (37) Now we can deduce that 𝜉²_𝑝=𝔼𝑦−𝑓_𝜆(𝑥)²𝐾(𝑥,𝑥)^𝑝/22/𝑝≤𝜅²𝔼𝑦−𝑓_𝜆(𝑥)^𝑝2/𝑝≤4𝜅²𝔼max𝑦^𝑝,𝑓_𝜆(𝑥)^𝑝2/𝑝≤2𝜅²𝔼𝑦^𝑝2/𝑝+𝔼𝑓_𝜆(𝑥)^𝑝2/𝑝≤2𝜅²𝑀^2/𝑝+𝑀^4/𝑝2𝐶²₁𝜅²+1𝜆^{min{(𝑝−2)(2𝑟−1)/𝑝,0}}. (38) Plugging this estimate into (35) yields 𝔼1𝑚^𝑚_𝑖=1𝜉(𝑧_𝑖)−𝐿_𝐾𝑓_𝜌−𝑓_𝜆²_𝐾≤𝐶₂𝑚⁻¹𝜆^{min{(𝑝−2)(2𝑟−1)/𝑝,0}}1+^𝑚−1_𝑙=1𝛼_𝑙^{(𝑝−2)/𝑝}, (39) where 𝐶₂ is a constant only depending on 𝜅,𝑀 and ‖𝐿_𝐾^{−min{𝑟,1/2}}𝑓_𝜌‖_𝜌𝑋. Then combining (34) and (39) with (25), we complete the proof.

For 𝛼-mixing process we have the following proposition to get the bound of sample error in 𝓗_𝐾, and the proof can be directly obtained by the inequality (32).

Proposition 10. Under assumption of Proposition 9, one has 𝔼𝑓_𝐳,𝜆−𝑓_𝜆𝐾≤𝐶₃𝑚^−1/2𝜆⁻¹1+^𝑚−1_𝑙=1𝛼_𝑙^{(𝑝−2)/𝑝}, (40) where 𝐶₃=𝐶₂.

4. Error Bounds and Learning Rates

In this section we derive the learning rates, that is, the convergence rates of ‖𝑓_𝐳,𝜆−𝑓_𝜌‖_𝜌𝑋 and ‖𝑓_𝐳,𝜆−𝑓_𝜌‖_𝐾 as 𝑚→∞ by choosing the regularization parameter 𝜆 according to 𝑚. The following approximation error bound is needed to get the convergence rates.

Proposition 11. Supposing that 𝐿_𝐾^−𝑟𝑓_𝜌∈𝐿²_𝜌𝑋(𝑋) for some 𝑟>0, there holds 𝑓_𝜆−𝑓_𝜌𝜌𝑋≤𝜆^min{𝑟,1}𝐿_𝐾^{−min{𝑟,1}}𝑓_𝜌𝜌𝑋. (41) Moreover, when 𝑟≥1/2, that is, 𝑓_𝜌∈𝓗_𝐾, there holds 𝑓_𝜆−𝑓_𝜌𝐾≤𝜆^{min{𝑟−(1/2),1}}𝐿_𝐾^{−min{𝑟,3/2}}𝑓_𝜌𝜌𝑋. (42)

The first conclusion in Proposition 11 has been proved in [20], and the second one can be proved in the same way. To derive the learning rates, we need to balance the approximation error and sample error. For this purpose, the following simple facts are necessary: ^𝑚−1_𝑙=1𝑙^−𝑠≤1𝑚1−𝑠^1−𝑠1if0<𝑠<1,log𝑚if𝑠=1,𝑠−1if𝑠>1. (43)

Proof of Theorem 3. The estimate of learning rates in 𝐿²_𝜌𝑋(𝑋) norm is divided into two cases.
Case 1. For 0<𝑡<2, by (43) and 𝜙_𝑖≤𝑎𝑖^−𝑡, there is 1+4^𝑚−1_𝑖=1𝜙_𝑖^1/2≤1+4√𝑎^𝑚−1_𝑖=1𝑖^−𝑡/2≤8√1+𝑎𝑚2−𝑡^1−(𝑡/2). (44) Thus Proposition 7 yields that 𝔼𝑓_𝐳,𝜆−𝑓_𝜆𝜌𝑋√≤2𝐶1+16𝑎𝜆2−𝑡^−1/2𝑚^−𝑡/4+𝜆⁻¹𝑚^−3𝑡/8. (45) By Proposition 11 and Markov inequality, with confidence 1−𝜂, there holds 𝑓_𝐳,𝜆−𝑓_𝜌𝜌𝑋≤𝑂𝜆^min{𝑟,1}+𝜂⁻¹𝜆^−1/2𝑚^−𝑡/4+𝜆⁻¹𝑚^−3𝑡/8. (46)
For 0<𝑟<1/2, by taking 𝜆=𝑚^{−3𝑡/(8(𝑟+1))}, we can deduce the learning rate as 𝑂(𝑚^{−3𝑡𝑟/(8(𝑟+1))}). When 1/2≤𝑟<1, taking 𝜆=𝑚^{−𝑡/(2(2𝑟+1))}, the learning rate 𝑂(𝑚^{−𝑟𝑡/(2(2𝑟+1))}) can be derived. When 𝑟≥1, the desired convergence rate is obtained by taking 𝜆=𝑚^−𝑡/6.
Case 2. 𝑡≥2. With confidence 1−𝜂, there holds 𝑓_𝐳,𝜆−𝑓_𝜌𝜌𝑋=𝑂𝜆^min{𝑟,1}+𝜂⁻¹𝜆^−1/2𝑚^−1/2+𝜆⁻¹𝑚^−3/4log𝑚^3/4. (47) For 0<𝑟<1/2, taking 𝜆=𝑚^{−3/(4(𝑟+1))}, the learning rate 𝑂(𝑚^{−3𝑟/(4(𝑟+1))}(log𝑚)^3/4) can be derived, and for 1/2≤𝑟<1, by taking 𝜆=𝑚^{−1/(2𝑟+1)}, we can deduce the learning rate 𝑂(𝑚^{−𝑟/(2𝑟+1)}(log𝑚)^3/4). When 𝑟≥1, the desired convergence rate is obtained by taking 𝜆=𝑚^−1/3.
Next for bounding the generalization error in 𝓗_𝐾, Proposition 8 in connection with Proposition 11 tells us that with confidence 1−𝜂, 𝑓_𝐳−𝑓_𝜌𝐾≤𝜆^{min{𝑟−(1/2),1}}+𝜂⁻¹𝜆⁻¹𝑚^−1/21+4^𝑚−1_𝑖=1𝜙_𝑖^1/2. (48) The rest of the proof is analogous to the estimate of ‖𝑓_𝐳,𝜆−𝑓_𝜌‖_𝜌𝑋 mentioned previously.

Proof of Theorem 4. For 0<𝑡<1, by (43) and 𝛼_𝑙≤𝑏𝑙^−𝑡, there is 1+^𝑚−1_𝑙=1𝛼_𝑙^{(𝑝−2)/𝑝}≤1+𝑏^{(𝑝−2)/𝑝𝑚−1}_𝑙=1𝑙^{−((𝑝−2)/𝑝)𝑡}≤1+𝑝𝑏^{(𝑝−2)/𝑝}𝑡𝑚𝑝−𝑝−2^{1−((𝑝−2)/𝑝)𝑡},1+^𝑚−1_𝑙=1𝛼_𝑙≤1+𝑏^𝑚−1_𝑙=1𝑙^−𝑡≤𝑏1+𝑚1−𝑡^1−𝑡. (49) By Propositions 9 and 11 and Markov inequality, with confidence 1−𝜂, there holds 𝑓_𝐳,𝜆−𝑓_𝜌𝜌𝑋=𝑂𝜆^min{𝑟,1}+𝜂⁻¹𝜆^{min{(𝑝−2)(2𝑟−1)/2𝑝,0}−1/2}×𝑚^{−(𝑝−2)𝑡/2𝑝}+𝜆^−1/2𝑚^{−(𝑝−2)𝑡/2𝑝−𝑡/4}. (50)
For 0<𝑟<1/2, by taking 𝜆=𝑚^{−(𝑝−2)𝑡/2(2𝑟+𝑝−1)}, we can deduce the learning rate as 𝑂(𝑚^{−(𝑝−2)𝑡𝑟/2(2𝑟+𝑝−1)}). When 1/2≤𝑟<1, taking 𝜆=𝑚^{−(𝑝−2)𝑡/𝑝(2𝑟+1)}, the learning rate 𝑂(𝑚^{−(𝑝−2)𝑟𝑡/𝑝(2𝑟+1)}) can be derived. When 𝑟≥1, the desired convergence rate is obtained by taking 𝜆=𝑚^{−(𝑝−2)𝑡/3𝑝}.
The rest of the analysis is similar; we omit it here.

Acknowledgment

This paper is supported by the National Nature Science Foundation of China (no. 11071276).