Density Problem and Approximation Error in Learning Theory

Abstract

We study the density problem and approximation error of reproducing kernel Hilbert spaces for the purpose of learning theory. For a Mercer kernel 𝐾 on a compact metric space (𝑋, 𝑑), a characterization for the generated reproducing kernel Hilbert space (RKHS) ℋ_𝐾 to be dense in 𝐶(𝑋) is given. As a corollary, we show that the density is always true for convolution type kernels. Some estimates for the rate of convergence of interpolation schemes are presented for general Mercer kernels. These are then used to establish for convolution type kernels quantitative analysis for the approximation error in learning theory. Finally, we show by the example of Gaussian kernels with varying variances that the approximation error can be improved when we adaptively change the value of the parameter for the used kernel. This confirms the method of choosing varying parameters which is used often in many applications of learning theory.

1. Introduction

Learning theory investigates how to find function relations or data structures from random samples. For the regression problem, one usually has some experience and would expect that the (underlying) unknown function lies in some set of functions ℋ called the hypothesis space. Then one tries to find a good approximation in ℋ of the underlying function𝑓(under certain metric). The best approximation in ℋ is called the target function𝑓_ℋ. However,𝑓is unknown. What we have in hand is a set of random samples{(𝑥_𝑖,𝑦_𝑖)}^ℓ_𝑖=1. These samples are not given by𝑓exactly (𝑓(𝑥_𝑖)≠𝑦_𝑖). They are controlled by this underlying function𝑓with noise or some other uncertainties (𝑓(𝑥_𝑖)≈𝑦_𝑖). The most important model studied in learning theory [1] is to assume that the uncertainty is represented by a Borel probability measure𝜌on𝑋×𝑌, and the underlying function𝑓∶𝑋→𝑌is the regression function of𝜌given by 𝑓_𝜌(𝑥)=_𝑌𝑦𝑑𝜌𝑦∣𝑥,𝑥∈𝑋. (1) Here, 𝜌(𝑦∣𝑥)is the conditional probability measure at𝑥. Then, the samples{(𝑥_𝑖,𝑦_𝑖)}^ℓ_𝑖=1are independent and identically distributed drawers according to the probability measure𝜌. For the classification problem,𝑌={1,−1}and sign (𝑓_𝜌)is the optimal classifier.

Based on the samples, one can find a function from the hypothesis space ℋ that best fits the data𝐳∶={(𝑥_𝑖,𝑦_𝑖)}^ℓ_𝑖=1(with respect to certain loss functional). This function is called the empirical target function 𝑓_𝐳. When the number ℓ of samples is large enough, 𝑓_𝐳 is a good approximation of the target function 𝑓_ℋ with certain confidence. This problem has been extensively investigated and well developed in the literature of statistical learning theory. See, for example, [1–4].

What is less understood is the approximation of the underlying desired function𝑓by the target function𝑓_ℋ. For example, if one takes ℋ to be a polynomial space of some fixed degree, then𝑓can be approximated by functions from ℋ only when𝑓is a polynomial in ℋ.

In kernel machine learning such as support vector machines, one often uses reproducing kernel Hilbert spaces or their balls as hypothesis spaces. Here, we take (𝑋,𝑑(⋅,⋅)) to be a compact metric space and 𝑌=ℝ.

Definition 1. Let𝐾∶𝑋×𝑋→ℝbe continuous, symmetric, and positive semidefinite; that is, for any finite set of distinct points{𝑥₁,…,𝑥_ℓ}⊂𝑋, the matrix(𝐾(𝑥_𝑖,𝑥_𝑗))^ℓ_𝑖,𝑗=1is positive semidefinite. Such a kernel is called a Mercer kernel. It is called positive definite if the matrix(𝐾(𝑥_𝑖,𝑥_𝑗))^ℓ_𝑖,𝑗=1is always positive definite.

The reproducing kernel Hilbert space (RKHS) ℋ_𝐾 associated with a Mercer kernel𝐾is defined (see [5]) to be the completion of the linear span of the set of functions{𝐾_𝑥∶=𝐾(𝑥,⋅)∶𝑥∈𝑋}with the inner product⟨⋅,⋅⟩_ℋ𝐾=⟨⋅,⋅⟩_𝐾satisfying ^ℓ_𝑖=1𝑐_𝑖𝐾_𝑥𝑖²_𝐾=^ℓ_𝑖=1𝑐_𝑖𝐾_𝑥𝑖,^ℓ_𝑖=1𝑐_𝑖𝐾_𝑥𝑖𝐾=^ℓ_𝑖,𝑗=1𝑐_𝑖𝐾𝑥_𝑖,𝑥_𝑗𝑐_𝑗. (2) The reproducing kernel property is given by 𝐾_𝑥,𝑔_𝐾=𝑔(𝑥),∀𝑥∈𝑋,𝑔∈ℋ_𝐾. (3) This space can be embedded into𝐶(𝑋), the space of continuous functions on𝑋.

In kernel machine learning, one often takesℋ_𝐾or its balls as the hypothesis space. Then, one needs to know whether the desired function𝑓can be approximated by functions from the RKHS.

The first purpose of this paper is to study the density of the reproducing kernel Hilbert spaces in𝐶(𝑋)(or in𝐿²(𝑋)when𝑋is a subset of the Euclidean spaceℝ^𝑛). This will be done in Section 2 where some characterizations will be provided. Let us mention a simple example with detailed proof given in Section 6.

Example 2. Let𝑋=[0,1]and let 𝐾be a Mercer kernel given by 𝐾𝑥,𝑦=^+∞_𝑗=0𝑎_𝑗𝑥⋅𝑦^𝑗, (4) where𝑎_𝑗≥0for each𝑗 and∑^+∞_𝑗=0𝑎_𝑗<∞. Set𝐽∶={𝑗∈ℤ₊∶𝑎_𝑗>0}. Then, ℋ_𝐾is dense in𝐶(𝑋)if and only if 𝑎₀>0,_{𝑗∈𝐽⧵{0}}1𝑗=+∞. (5)

When the density holds, we want to study the convergence rate of the approximation by functions from balls of the RKHS as the radius tends to infinity. The quantity 𝐼𝑓,𝑅∶=inf_{𝑔𝐾≤𝑅}𝑓−𝑔 (6) is called the approximation error in learning theory. Some estimates have been presented by Smale and Zhou [6] for the𝐿²norm and many kernels. The second purpose of this paper is to investigate the convergence rate of the approximation error with the uniform norm as well as the𝐿²norm. Estimates will be given in Section 4, based on the analysis in Section 3 for interpolation schemes associated with general Mercer kernels. With this analysis, we can understand the approximation error with respect to marginal probability distribution induced by𝜌. Let us provide an example of Gaussian kernels to illustrate the idea. Notice that when the parameter𝜎of the kernel is allowed to change with𝑅, the rate of the approximation error may be improved. This confirms the method of adaptively choosing the parameter of the kernel, which is used in many applications (see e.g., [7]).

Example 3. Let 𝐾_𝜎𝑥,𝑦=exp−𝑥−𝑦²𝜎²,𝑥,𝑦∈𝑋=[0,1]^𝑛. (7) There exist positive constants𝐴and𝐵such that, for each𝑓∈𝐻^𝑠(ℝ^𝑛)and𝑅≥𝐴‖𝑓‖_𝐿2, there holds inf_{𝑔𝐾𝜎≤𝑅}𝑓−𝑔_𝐿2(𝑋)≤𝐵log𝑅^−𝑠/4 (8) when𝜎is fixed; while when𝜎may change with𝑅, there holds inf_{𝑔𝐾𝜎𝑅≤𝑅}𝑓−𝑔_𝐿2(𝑋)≤𝐵log𝑅^−𝑠. (9)

2. Density and Positive Definiteness

The density problem of reproducing kernel Hilbert spaces in𝐶(𝑋)was raised to the author by Poggio et al. See [8]. It can be stated as follows.

Given a Mercer kernel𝐾on a compact metric space(𝑋,𝑑(⋅,⋅)), when is the RKHSℋ_𝐾dense in𝐶(𝑋)?

By means of the dual space of𝐶(𝑋), we can give a general characterization. This is only a simple observation, but it does provide us useful information. For example, we will show that the density is always true for convolution type kernels. Also, for dot product type kernel, we can give a complete nice characterization for the density, which will be given in Section 6.

Recall the Riesz Representation Theorem asserting that the dual space of𝐶(𝑋)can be represented by the set of Borel measures on𝑋. For a Borel measure𝜇on𝑋, we define the integral operator𝐿_𝐾,𝜇associated with the kernel as 𝐿_𝐾,𝜇𝑓(𝑥)∶=_𝑋𝐾𝑥,𝑦𝑓𝑦𝑑𝜇𝑦,𝑥∈𝑋. (10) This is a compact operator on 𝐿²_𝜇(𝑋) if 𝜇 is a positive measure.

Theorem 4. Let 𝐾 be a Mercer kernel on a compact metric space (𝑋,𝑑). Then, the following statements are equivalent. (1)ℋ_𝐾is dense in𝐶(𝑋). (2)For any nontrivial positive Borel measure𝜇,ℋ_𝐾is dense in𝐿²_𝜇(𝑋). (3)For any nontrivial positive Borel measure𝜇,𝐿_𝐾,𝜇has no eigenvalue zero in𝐿²_𝜇(𝑋). (4)For any nontrivial Borel measure𝜇, as a function in𝐶(𝑋), _𝑋𝐾⋅,𝑦𝑑𝜇𝑦≠0. (11)

Proof. (1)⇒(2). This follows from the fact that𝐶(𝑋)is dense in𝐿²_𝜇(𝑋). See, for example, [9].
(2)⇒(3). Suppose that ℋ_𝐾is dense in𝐿²_𝜇(𝑋), but𝐿_𝐾,𝜇has an eigenvalue zero in𝐿²_𝜇(𝑋). Then, there exists a nontrivial function𝑓∈𝐿²_𝜇(𝑋)such that𝐿_𝐾,𝜇(𝑓)=0; that is, 𝐿_𝐾,𝜇𝑓(𝑥)=_𝑋𝐾𝑓𝑦𝑦=𝑥,𝑦𝑑𝜇_𝑋𝐾_𝑥𝑦𝑓𝑦𝑑𝜇𝑦=0. (12) The identity holds as functions in𝐿²_𝜇(𝑋). If the support of𝜇is𝑋, then this identity would imply that𝑓is orthogonal to each𝐾_𝑥with𝑥∈𝑋. When the support of𝜇is not𝑋, things are more complicated. Here, the support of𝜇, denoted as supp𝜇, is defined to be the smallest closed subset𝐹of𝑋satisfying𝜇(𝑋⧵𝐹)=0.
The property of the RKHS enables us to prove the general case. As the function𝐿_𝐾,𝜇(𝑓)is continuous, we know from (12) that, for each𝑥in supp𝜇, _supp𝜇𝐾_𝑥𝑦𝑓𝑦𝑑𝜇𝑦=_𝑋𝐾_𝑥𝑦𝑓𝑦𝑑𝜇𝑦=0. (13) This means for each𝑥in supp𝜇,𝑓⟂𝐾_𝑥in𝐿²_𝜇(supp 𝑋), where𝜇has been restricted onto supp𝜇. When we restrict𝐾onto supp𝜇×supp𝜇, the new kernel𝐾is again a Mercer kernel. Moreover, by (1),ℋ𝐾=ℋ_𝐾|_supp𝜇. It follows that span{𝐾_𝑥|_supp𝜇∶𝑥∈supp𝜇} is dense inℋ𝐾=ℋ_𝐾|_supp𝜇. The latter is dense in𝐿²_𝜇(supp𝜇). Therefore,𝑓is orthogonal to𝐿²_𝜇(supp𝜇); hence, as a function in𝐿²_𝜇(𝑋),𝑓is zero. This is a contradiction.
(3)⇒(4). Every nontrivial Borel measure𝜇can be uniquely decomposed as the difference of two mutually singular positive Borel measures:𝜇=𝜇⁺−𝜇⁻; that is, there exists a Borel set𝐴⊂𝑋such that𝜇⁺(𝐴)=𝜇⁺(𝑋)and𝜇⁻(𝐴)=0. With this decomposition, _𝑋𝐾𝑥,𝑦𝑑𝜇𝑦=_𝑋𝐾𝑥,𝑦𝜒_𝐴𝑦−𝜒_𝑋⧵𝐴𝑦𝑑𝜇=𝐿_𝐾,|𝜇|𝜒_𝐴−𝜒_𝑋⧵𝐴(𝑥). (14) Here, 𝜒_𝐴is the characteristic function of the set𝐴, and|𝜇|=𝜇⁺+𝜇⁻is the absolute value of𝜇. As|𝜇|is a nontrivial positive Borel measure and𝜒_𝐴−𝜒_𝑋⧵𝐴is a nontrivial function in𝐿²_|𝜇|(𝑋), statement (3) implies that, as a function in𝐿²_|𝜇|(𝑋),∫_𝑋𝐾(𝑥,𝑦)𝑑𝜇(𝑦)≠0. Since this function lies in𝐶(𝑋), it is nonzero as a function in𝐶(𝑋).
The last implication (4) ⇒ (1) follows directly from the Riesz Representation Theorem.

The proof of Theorem 4 also yields a characterization for the density of the RKHS in𝐿²_𝜇(𝑋).

Corollary 5. Let𝐾be a Mercer kernel on a compact metric space(𝑋,𝑑) and𝜇 a positive Borel measure on𝑋. Then, ℋ_𝐾is dense in𝐿²_𝜇(𝑋)if and only if𝐿_𝐾,𝜇has no eigenvalue zero in𝐿²_𝜇(𝑋).

The necessity has been verified in the proof of Theorem 4, while the sufficiency follows from the observation that an𝐿²_𝜇(𝑋)function𝑓lying in the orthogonal complement of span{𝐾_𝑥∶𝑥∈𝑋}gives an eigenfunction of𝐿_𝐾,𝜇with eigenvalue zero: 𝐾_𝑥,𝑓_{𝐿2𝜇(𝑋)}=_𝑋𝐾_𝑥𝑦𝑓𝑦𝑑𝜇𝑦=𝐿_𝐾,𝜇𝑓(𝑥)=0. (15)

Theorem 4 enables us to conclude that the density always holds for convolution type kernels𝐾(𝑥,𝑦)=𝑘(𝑥−𝑦)with𝑘∈𝐿²(ℝ^𝑛). The density for some convolution type kernels has been verified by Steinwart [10]. The author observed the density as a corollary of Theorem 4 when𝑘(𝜉)is strictly positive. Charlie Micchelli pointed out to the author that, for a convolution type kernel, the RKHS is always dense in𝐶(𝑋). So, the density problem is solved for these kernels.

Corollary 6. Let𝐾(𝑥,𝑦)=𝑘(𝑥−𝑦)be a nontrivial convolution type Mercer kernel onℝ^𝑛with𝑘∈𝐿²(ℝ^𝑛). Then, for any compact subset𝑋ofℝ^𝑛,ℋ_𝐾on𝑋is dense in𝐶(𝑋).

Proof. It is well known that𝐾is a Mercer kernel if and only if𝑘is continuous and𝑘(𝜉)≥0almost everywhere. We apply the equivalent statement (4) of Theorem 4 to prove our statement.
Let𝜇be a Borel measure on𝑋such that _𝑋𝐾𝑥,𝑦𝑑𝜇𝑦=0,∀𝑥∈𝑋. (16) Then, the inverse Fourier transform yields _𝑋ℝ𝑛𝑘(𝜉)𝑒^{𝑖𝜉⋅(𝑥−𝑦)}𝑦=𝑑𝜉𝑑𝜇_ℝ𝑛𝑘(𝜉)𝜇(𝜉)𝑒^{𝑖𝜉⋅𝑥}𝑑𝜉=0,∀𝑥∈𝑋. (17) Here, 𝜇(𝜉)=∫𝑒^{−𝑖𝜉⋅𝑦}𝑑𝜇(𝑦)is the Fourier transform of the Borel measure𝜇, which is an entire function.
Taking the integral on𝑋with respect to the measure𝜇, we have _ℝ𝑛𝑘(𝜉)𝜇(𝜉)_𝑋𝑒^{𝑖𝜉⋅𝑥}𝑑𝜇(𝑥)𝑑𝜉=_ℝ𝑛𝑘(𝜉)𝜇(𝜉)²𝑑𝜉=0. (18) For a nontrivial Borel measure𝜇supported on 𝑋, 𝜇(𝜉) vanishes only on a set of measure zero. Hence, 𝑘(𝜉)=0almost everywhere, which gives𝑘=0. Therefore, we must have𝜇=0. This proves the density by Theorem 4.

After the first version of the paper was finished, I learned that Micchelli et al. [11] proved the density for a class of convolution type kernels𝑘(𝑥−𝑦)with𝑘being the Fourier transform of a finite Borel measure. Note that a large family of convolution type reproducing kernels are given by radial basis functions; see, for example, [12].

Now we can state a trivial fact that the positive definiteness is a necessary condition for the density.

Corollary 7. Let𝐾be a Mercer kernel on a compact metric space(𝑋,𝑑). Ifℋ_𝐾is dense in𝐶(𝑋), then𝐾is positive definite.

Proof. Suppose to the contrary thatℋ_𝐾is dense in𝐶(𝑋), but there exists a finite set of distinct points{𝑥_𝑖}^ℓ_𝑖=1⊂𝑋such that the matrix𝐴_𝐱∶=(𝐾(𝑥_𝑖,𝑥_𝑗))^ℓ_𝑖,𝑗=1is not positive definite. By the Mercer kernel property,𝐴_𝐱is positive semidefinite. So it is singular, and we can find a nonzero vector𝑐∶=(𝑐₁,…,𝑐_ℓ)^𝑇∈ℝ^ℓsatisfying𝐴_𝐱𝑐=0. It follows that𝑐^𝑇𝐴_𝐱𝑐=0; that is, ^ℓ_𝑖=1𝑐_𝑖𝐾_𝑥𝑖²_𝐾=^ℓ_𝑖=1𝑐_𝑖𝐾_𝑥𝑖,^ℓ_𝑗=1𝑐_𝑗𝐾_𝑥𝑗𝐾=^ℓ_𝑖,𝑗=1𝑐_𝑖𝑐_𝑗𝐾𝑥_𝑖,𝑥_𝑗=0. (19) Thus, ^ℓ_𝑖=1𝑐_𝑖𝐾_𝑥𝑖=0. (20)
Now, we define a nontrivial Borel measure𝜇supported on{𝑥₁,…,𝑥_ℓ}as 𝜇𝑥_𝑖=𝑐_𝑖,𝑖=1,…,ℓ. (21) Then, for𝑥∈𝑋, _𝑋𝐾𝑥,𝑦𝑑𝜇𝑦=^ℓ_𝑖=1𝐾𝑥,𝑥_𝑖𝑐_𝑖=^ℓ_𝑖=1𝑐_𝑖𝐾_𝑥𝑖(𝑥)=0. (22) This is a contradiction to the equivalent statement (4) in Theorem 4 of the density.

Because of the necessity given in Corollary 7, one would expect that the positive definiteness is also sufficient for the density. Steve Smale convinced the author that this is not the case in general. This motivates us to present a constructive example of𝐶^∞kernel. Denote ‖𝑔‖_𝑊𝑚∞∶=∑^𝑚_𝑗=0‖𝑔^(𝑗)‖_𝐿∞ as the norm in the Sobolev space𝑊^𝑚_∞.

Example 8. Let𝑋=[0,1]. For every𝑚∈ℕand every𝑖∈{0,1,…,𝑚}, choose a real-valued𝐶^∞function𝜓_𝑖,𝑚(𝑥)on[0,1]such that 𝜓_𝑖,𝑚(𝑥)=𝑥^𝑖[]⧵1,∀𝑥∈0,1,1𝑚+1𝑚₁₀,𝜓_𝑖,𝑚(𝑥)𝑑𝑥=0. (23) Define𝐾on[0,1]×[0,1]by 𝐾𝑥,𝑦=^∞_𝑚=12^−𝑚∑^𝑚_𝑖=0𝜓_𝑖,𝑚(𝑥)𝜓_𝑖,𝑚𝑦∑^𝑚_𝑖=0𝜓_𝑖,𝑚²_𝑊𝑚∞,𝑥,𝑦∈[0,1]. (24) Then, 𝐾is a𝐶^∞Mercer kernel on[0,1]. It is positive definite, but the constant function1is not in the closure ofℋ_𝐾in 𝐶(𝑋). Hence, ℋ_𝐾is not dense in𝐶(𝑋).

Proof. The series in (24) converges in𝑊^𝑚_∞for any𝑚∈ℕ. Hence, 𝐾is𝐶^∞ and is a Mercer kernel on[0,1].
To prove the positive definiteness, we let{𝑥_𝑖}^ℓ_𝑖=1⊂[0,1]be a finite set of distinct points and(𝑐_𝑖)^ℓ_𝑖=1 a nonzero vector. Choose𝑚∈ℕsuch that 1𝑚≥ℓ−1,𝑚<min𝑥_𝑗∶𝑥_𝑗>0,𝑗∈{1,…,ℓ}. (25) Then, for each𝑗∈{1,…,ℓ}, either𝑥_𝑗=0or𝑥_𝑗>1/𝑚. Hence, 𝑥_𝑗∈[]⧵10,1,1𝑚+1𝑚,∀𝑗=1,…,ℓ. (26) By the construction of𝜓_𝑖,𝑚, there holds 𝜓_𝑖,𝑚𝑥_𝑗=𝑥^𝑖_𝑗,∀𝑖=0,1,…,𝑚,𝑗=1,…,ℓ. (27) Then, ^ℓ_𝑖,𝑗=1𝑐_𝑖𝑐_𝑗𝐾𝑥_𝑖,𝑥_𝑗≥2^−𝑚∑^𝑚_𝑖=0𝜓_𝑖,𝑚²_𝑊𝑚∞^𝑚_𝑖=0^ℓ_𝑗=1𝑐_𝑗𝜓_𝑖,𝑚𝑥_𝑗²≥2^−𝑚∑^𝑚_𝑖=0𝜓_𝑖,𝑚²_𝑊𝑚∞^ℓ−1_𝑖=0^ℓ_𝑗=1𝑐_𝑗𝜓_𝑖,𝑚𝑥_𝑗². (28) Now, the determinant of the matrix(𝑥^𝑖_𝑗)_{𝑖=0,1,…,ℓ−1,𝑗=1,…,ℓ}is a Vandermonde determinant and is nonzero. Since (𝑐_𝑗)^ℓ_𝑗=1 is a nonzero vector, we know that ∑^ℓ_𝑗=1𝑐_𝑗𝑥^𝑖_𝑗≠0 for some𝑖∈{0,1,…,ℓ−1}. It follows that ∑^ℓ_𝑖,𝑗=1𝑐_𝑖𝑐_𝑗𝐾(𝑥_𝑖,𝑥_𝑗)>0. Thus, 𝐾 is positive definite.
We now prove that𝟏, the constant function taking the value1everywhere, is not in the closure ofℋ_𝐾in𝐶(𝑋). In fact, the uniformly convergent series (24) and the vanishing property of𝜓_𝑖,𝑚imply that ¹₀𝐾𝑥,𝑦𝑑𝑦=¹₀𝐾_𝑥𝑦𝑑𝑦=0,∀𝑥∈𝑋. (29) Since span{𝐾_𝑥∶𝑥∈𝑋} is dense in ℋ_𝐾 and ℋ_𝐾 is embedded in 𝐶(𝑋), we know that ¹₀𝑓𝑦𝑑𝑦=0,∀𝑓∈ℋ_𝐾. (30) If 𝟏 could be uniformly approximated by a sequence {𝑓_𝑚} in ℋ_𝐾, then 1=¹₀𝟏𝑦𝑑𝑦=lim_𝑚→∞¹₀𝑓_𝑚𝑦𝑑𝑦=0, (31) which would be a contradiction. Therefore,ℋ_𝐾is not dense in𝐶(𝑋).

Combining the previous discussion, we know that the positive definiteness is a nice necessary condition for the density of the RKHS in𝐶(𝑋) but is not sufficient.

3. Interpolation Schemes for Reproducing Kernel Spaces

The study of approximation by reproducing kernel Hilbert spaces has a long history; see, for example, [13, 14]. Here, we want to investigate the rate of approximation as the RKHS norm of the approximant becomes large.

In the following sections, we consider the approximation error for the purpose of learning theory. The basic tool for constructing approximants is a set of nodal functions used in [6, 15, 16].

Definition 9. We say that{𝑢_𝑖(𝑥)∶=𝑢_𝑖,𝐱(𝑥)}^ℓ_𝑖=1is the set of nodal functions associated with the nodes𝐱∶={𝑥₁,…,𝑥_ℓ}⊂𝑋if𝑢_𝑖∈span{𝐾_𝑥𝑗}^ℓ_𝑗=1and 𝑢_𝑖𝑥_𝑗=𝛿_𝑖𝑗=1,if𝑗=𝑖,0,otherwise. (32)

The nodal functions have some nice minimization properties; see [6, 16].

In [15], we show that the nodal functions{𝑢_𝑖}^ℓ_𝑖=1associated with𝐱exist if and only if the Gramian matrix𝐴_𝐱∶=(𝐾(𝑥_𝑖,𝑥_𝑗))^ℓ_𝑖,𝑗=1is nonsingular. In this case, the nodal functions are uniquely given by 𝑢_𝑖(𝑥)=^ℓ_𝑗=1𝐴_𝐱⁻¹_𝑖,𝑗𝐾_𝑥𝑗(𝑥),𝑖=1,…,ℓ. (33)

Remark 10. When the RKHS has finite dimension 𝑚, then, for any ℓ≤𝑚 we can find nodal functions {𝑢_𝑗}^ℓ_𝑗=1 associated with some subset 𝐱={𝑥₁,…,𝑥_ℓ}⊂𝑋, while for ℓ>𝑚, no such nodal functions exist. When dimℋ_𝐾=∞, then, for any ℓ∈ℕ, we can find a subset 𝐱={𝑥₁,…,𝑥_ℓ}⊂𝑋 which possesses a set of nodal functions.

The nodal functions are used to construct an interpolation scheme: 𝐼_𝐱𝑓(𝑥)=^ℓ_𝑖=1𝑓𝑥_𝑖𝑢_𝑖,𝐱(𝑥),𝑥∈𝑋,𝑓∈𝐶(𝑋). (34) It satisfies 𝐼_𝐱(𝑓)(𝑥_𝑖)=𝑓(𝑥_𝑖) for𝑖=1,…,ℓ. Interpolation schemes have been applied to the approximation by radial basis functions in the vast literature; see, for example, [17–20].

The error𝐼_𝐱(𝑓)−𝑓for𝑓∈ℋ_𝐾will be estimated by means of a power function.

Definition 11. Let𝐾be a Mercer kernel on a compact metric space(𝑋,𝑑)and𝐱={𝑥₁,…,𝑥_ℓ}⊂𝑋. The power function𝜀_𝐾is defined on𝐱as 𝜀_𝐾(𝐱)∶=sup_𝑥∈𝑋inf_{𝑤∈ℝℓ}𝐾(𝑥,𝑥)−2^ℓ_𝑖=1𝑤_𝑖𝐾𝑥,𝑥_𝑖+^ℓ_𝑖,𝑗=1𝑤_𝑖𝐾𝑥_𝑖,𝑥_𝑗𝑤_𝑗^1/2. (35)

We know that𝜀_𝐾(𝐱)→0when𝑑_𝐱∶=max_𝑥∈𝑋min_{𝑖=1,…,ℓ}𝑑(𝑥,𝑥_𝑖)→0. If𝐾is Lipschitz𝑠on𝑋: 𝐾𝑥,𝑦−𝐾(𝑥,𝑡)≤𝐶𝑑𝑦,𝑡^𝑠, (36) then 𝜀_𝐾(𝐱)≤2𝐶𝑑^𝑠_𝐱. (37) Moreover, higher order regularity of𝐾implies faster convergence of𝜀_𝐾(𝐱). For details, see [16].

The error of the interpolation scheme for functions from RKHS can be estimated as follows.

Theorem 12. Let𝐾be a Mercer kernel and𝐴_𝐱 nonsingular for a finite set𝐱={𝑥₁,…,𝑥_ℓ}⊂𝑋. Define the interpolation scheme associated with𝐱as (34). Then, for𝑓∈ℋ_𝐾, there holds 𝐼_𝐱𝑓−𝑓_𝐶(𝑋)≤𝑓_𝐾𝜀_𝐾(𝐱),𝐼_𝐱𝑓_𝐾≤𝑓_𝐾. (38)(39)

Proof. Let𝑥∈𝑋. We apply the reproducing property (3) of the function𝑓in 𝐼_𝐱𝑓(𝑥)−𝑓(𝑥)=^ℓ_𝑖=1𝑓𝑥_𝑖𝑢_𝑖(𝑥)−𝑓(𝑥). (40) Then, 𝐼_𝐱𝑓(𝑥)−𝑓(𝑥)=^ℓ_𝑖=1𝑢_𝑖(𝑥)𝐾_𝑥𝑖,𝑓_𝐾−𝐾_𝑥,𝑓_𝐾=^ℓ_𝑖=1𝑢_𝑖(𝑥)𝐾_𝑥𝑖−𝐾_𝑥,𝑓_𝐾. (41) By the Schwartz inequality inℋ_𝐾, 𝐼_𝐱𝑓(𝑥)−𝑓(𝑥)≤^ℓ_𝑖=1𝑢_𝑖(𝑥)𝐾_𝑥𝑖−𝐾_𝑥𝐾𝑓_𝐾. (42) As⟨𝐾_𝑠,𝐾_𝑡⟩_𝐾=𝐾(𝑠,𝑡), we have ^ℓ_𝑖=1𝑢_𝑖(𝑥)𝐾_𝑥𝑖−𝐾_𝑥²_𝐾=𝐾(𝑥,𝑥)−2^ℓ_𝑖=1𝑢_𝑖(𝑥)𝐾𝑥,𝑥_𝑖+^ℓ_𝑖,𝑗=1𝑢_𝑖(𝑥)𝐾𝑥_𝑖,𝑥_𝑗𝑢_𝑗(𝑥). (43) However, the quadratic function 𝑄𝑤_𝑖^ℓ_𝑖=1∶=𝐾(𝑥,𝑥)−2^ℓ_𝑖=1𝑤_𝑖𝐾𝑥,𝑥_𝑖+^ℓ_𝑖,𝑗=1𝑤_𝑖𝐾𝑥_𝑖,𝑥_𝑗𝑤_𝑗 (44) overℝ^ℓtakes its minimum value at(𝑢_𝑖(𝑥))^ℓ_𝑖=1. Therefore, ^ℓ_𝑖=1𝑢_𝑖(𝑥)𝐾_𝑥𝑖−𝐾_𝑥𝐾≤𝜀_𝐾(𝐱). (45) It follows that 𝐼_𝐱𝑓(𝑥)−𝑓(𝑥)≤𝑓_𝐾𝜀_𝐾(𝐱). (46) This proves (38).
As𝐼_𝐱(𝑓)∈ℋ_𝐾 and𝐼_𝐱(𝑓)(𝑥_𝑖)=𝑓(𝑥_𝑖)for𝑖=1,…,ℓ, we know that 𝐼_𝐱𝑓𝑥_𝑖−𝑓𝑥_𝑖=⟨𝐾_𝑥𝑖,𝐼_𝐱𝑓−𝑓⟩_𝐾=0,𝑖=1,…,ℓ. (47) This means that𝐼_𝐱(𝑓)−𝑓is orthogonal to span{𝐾_𝑥𝑖}^ℓ_𝑖=1. Hence, 𝐼_𝐱(𝑓)is the orthogonal projection of𝑓onto span{𝐾_𝑥𝑖}^ℓ_𝑖=1. Thus, ‖𝐼_𝐱(𝑓)‖_𝐾≤‖𝑓‖_𝐾.

The regularity of the kernel in connection with Theorem 12 yields the rate of convergence of the interpolation scheme. As an example, from the estimate for𝜀_𝐾(𝐱)given in [16, Proposition 2], we have the following.

Corollary 13. Let𝑋=[0,1], 𝑠≤𝑁∈ℕ, and𝐾(𝑥,𝑦)be a𝐶^𝑠Mercer kernel such that𝐴_𝐱is nonsingular for𝐱={𝑗/𝑁}^𝑁−1_𝑗=0. Then, for𝑓∈ℋ_𝐾, there holds 𝐼_𝐱𝑓−𝑓_𝐶(𝑋)≤𝑓_𝐾(4𝑠)^𝑠1+𝑠2^𝑠()!𝜕𝑠−1𝜕𝑦^𝑠𝑠𝐾_∞𝑁^−𝑠. (48)

For convolution type kernels, the power function can be estimated in terms of the Fourier transform of the kernel function. This is of particular interest when the kernel function is analytic. Let us provide the details.

Assume that𝑘is a symmetric function in𝐿²(ℝ^𝑛)and𝑘(𝜉)>0almost everywhere onℝ^𝑛. Consider the Mercer kernel 𝐾𝑥,𝑦=𝑘𝑥−𝑦,𝑥,𝑦∈[0,1]^𝑛. (49) For𝑁∈ℕ, we define the following function to measure the regularity: 𝜆_𝑘(𝑁)1∶=𝑛1+2^𝑁𝑛−1×max_{1≤𝑗≤𝑛}(2𝜋)^−𝑛_{𝜉∈[−𝑁/2,𝑁/2]𝑛}𝑘(𝜉)𝜉_𝑗𝑁^𝑁+𝑑𝜉1+𝑁2^𝑁𝑛2(2𝜋)^−𝑛_{𝜉∉[−𝑁/2,𝑁/2]𝑛}𝑘(𝜉)𝑑𝜉. (50)

Remark 14. This function involves two parts. The first part is𝜉∈[−𝑁/2,𝑁/2]^𝑛, where(|𝜉_𝑗|/𝑁)^𝑁≤2^−𝑁; hence, it decays exponentially fast as𝑁becomes large. The second part is𝜉∉[−𝑁/2,𝑁/2]^𝑛, where𝜉is large. Then, the decay of𝑘(which is equivalent to the regularity of𝑘) yields the fast decay of the second part.
The power function𝜀_𝐾(𝐱)can be bounded by𝜆_𝑘(𝑁)on the regular points: 𝛼𝐱∶=𝑁_{𝛼∈{0,1,…,𝑁−1}}^𝑛. (51)

Proposition 15. For the convolution type kernel (49) and𝐱given by (51), one has𝜀_𝐾(𝐱)≤𝜆_𝑘(𝑁). (52) In particular, if 𝑘(𝜉)≤𝐶₀𝑒^{−𝜆|𝜉|},∀𝜉∈ℝ^𝑛 (53) for some constants𝐶₀>0and𝜆>4+2𝑛ln4, then there holds 𝜀_𝐾(𝐱)≤𝜆_𝑘(𝑁)≤4𝐶₀1max,4𝑒𝜆𝑒^{𝑛𝜆/2𝑁/2}. (54)

Proof. Choose{𝑤_𝛼∶=𝑤_𝛼,𝐱(𝑥)}_𝛼∈𝐱as the Lagrange interpolation polynomials on𝐱. It is a vector inℝ^𝑁𝑛for each𝑥∈𝑋. Then, 𝜀_𝐾(𝐱)≤sup_𝑥∈𝑋𝑄_𝑁(𝑥), where 𝑄_𝑁(𝑥)∶=𝑘(0)−2_𝛼∈𝐱𝑤_𝛼,𝐱(𝑥)𝑘(𝑥−𝛼)+_{𝛼,𝛽∈𝐱}𝑤_𝛼,𝐱(𝑥)𝑘𝛼−𝛽𝑤_𝛽,𝐱(𝑥). (55) In the proof of Theorem  2 in [16], we showed that𝑄_𝑁(𝑥)≤𝜆_𝑘(𝑁)for each𝑥∈[0,1]^𝑛. Therefore,𝜀_𝐾(𝐱)≤𝜆_𝑘(𝑁).
The estimate for𝜆_𝑘(𝑁)in the second part was verified in the proof of Theorem  3 in [16].

For the Gaussian kernels 𝐾𝑥,𝑦=exp−𝑥−𝑦²𝜎²,𝑥,𝑦∈[0,1]^𝑛, (56) it was proved in [16, Example 4] that, for𝑁≥80𝑛log2/𝜎², there holds 𝜀_𝐾(𝐱)≤𝜆_𝑘(𝑁)≤2√𝑒116𝑛^𝑁/2+4𝜎√𝜋2^−𝑛𝑁. (57)

4. Approximation Error in Learning Theory

Now, we can estimate the approximation error in learning theory by means of the interpolation scheme (34).

Consider the convolution type kernel (49) on 𝑋=[0,1]^𝑛. As in [6], we denote Λ_𝑘(𝑟)∶=inf_{𝜉∈[−𝑟𝜋,𝑟𝜋]𝑛}𝑘(𝜉)^−1/2,𝑟>0. (58) The approximation error (6) can be realized as follows.

Theorem 16. Let 𝑘∈𝐿²(ℝ^𝑛) be a symmetric function with 𝑘(𝜉)>0, and let the kernel on 𝑋=[0,1]^𝑛 be 𝐾(𝑥,𝑦)=𝑘(𝑥−𝑦). For 𝑓∈𝐿²(ℝ^𝑛) and 𝑀≤𝑁∈ℕ, we set 𝑓_𝑀∈𝐿²(ℝ^𝑛) by 𝑓_𝑀(𝜉)=𝑓(𝜉),𝑖𝑓𝜉∈[−𝑀𝜋,𝑀𝜋]^𝑛,0,𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒. (59) Then, with 𝐱={0,1/𝑁,…,(𝑁−1)/𝑁}^𝑛, one has(i)‖𝐼_𝐱(𝑓_𝑀)‖_𝐾≤‖𝑓‖_𝐿2Λ_𝑘(𝑁); (ii)‖𝑓_𝑀−𝐼_𝐱(𝑓_𝑀)‖_𝐶(𝑋)≤‖𝑓‖_𝐿2Λ_𝑘(𝑀)𝜀_𝐾(𝐱)≤‖𝑓‖_𝐿2Λ_𝑘(𝑀)𝜆_𝑘(𝑁); (iii)‖𝑓−𝑓_𝑀‖²_𝐿2(𝑋)≤(2𝜋)^−𝑛∫_{𝜉∉[−𝑀𝜋,𝑀𝜋]𝑛}|𝑓(𝜉)|²𝑑𝜉→0(𝑎𝑠𝑀→∞).

Proof. (i) For 𝑖,𝑗∈𝑋_𝑁∶={0,1,…,𝑁−1}^𝑛 and 𝑥_𝑖=𝑖/𝑁, expression (33) gives 𝑢_𝑖,𝑢_𝑗𝐾=_{𝑠,𝑡∈𝑋𝑁}𝐴_𝐱⁻¹_𝑖𝑠𝐴_𝐱⁻¹_𝑗𝑡𝐾_𝑥𝑠,𝐾_𝑥𝑡𝐾=_{𝑠,𝑡∈𝑋𝑁}𝐴_𝐱⁻¹_𝑖𝑠𝐴_𝐱⁻¹_𝑗𝑡𝐴_𝐱𝑡𝑠=𝐴_𝐱⁻¹_𝑖𝑗. (60) Then for 𝑔∈𝐶(𝑋) we have 𝐼_𝐱𝑔²_𝐾=_{𝑖∈𝑋𝑁}𝑔𝑥_𝑖𝑢_𝑖(𝑥)²_𝐾=_{𝑖,𝑗∈𝑋𝑁}𝑔𝑥_𝑖𝑔𝑥_𝑗𝑢_𝑖,𝑢_𝑗𝐾=𝑔_𝐱^𝑇𝐴_𝐱⁻¹𝑔_𝐱, (61) where 𝑔|_𝐱 is the vector (𝑔(𝑥_𝑖))_{𝑖∈𝑋𝑁}∈ℝ^𝑁𝑛. It follows that 𝐼_𝐱𝑔²_𝐾=𝑔_𝐱,𝐴_𝐱⁻¹𝑔_𝐱ℓ2≤𝐴_𝐱⁻¹₂𝑔_𝐱²_ℓ2=𝐴_𝐱⁻¹_{2𝑖∈𝑋𝑁}𝑔𝑥_𝑖², (62) where ‖𝐴_𝐱⁻¹‖₂ denotes the (operator) norm of the matrix 𝐴_𝐱⁻¹ in (ℝ^𝑁𝑛,ℓ²).
We apply the previous analysis to the function 𝑓_𝑀 satisfying _{𝑗∈𝑋𝑁}𝑓_𝑀𝑥_𝑗²=_{𝑗∈𝑋𝑁}(2𝜋)^−𝑛_{𝜉∈[−𝑁𝜋,𝑁𝜋]𝑛}𝑓_𝑀(𝜉)𝑒^{𝑖⋅(𝑗/𝑁)⋅𝜉}𝑑𝜉²=_{𝑗∈𝑋𝑁}(2𝜋)^−𝑛_{𝜉∈[−𝜋,𝜋]𝑛}𝑓_𝑀(𝑁𝜉)𝑒^{𝑖𝑗⋅𝜉}𝑁^𝑛𝑑𝜉²≤(2𝜋)^−𝑛_{𝜉∈[−𝜋,𝜋]𝑛}𝑓_𝑀(𝑁𝜉)𝑁^𝑛2𝑑𝜉≤𝑁^𝑛𝑓²_𝐿2. (63) Then, 𝐼_𝐱𝑓_𝑀²_𝐾≤𝐴_𝐱⁻¹₂𝑁^𝑛𝑓²_𝐿2. (64)
Now, we need to estimate the norm ‖𝐴_𝐱⁻¹‖₂. For convolution type kernels, such an estimate was given in [15, Theorem 2] by means of methods from the radial basis function literature, for example, [17, 21–24]. We have 𝐴_𝐱⁻¹₂≤𝑁^−𝑛Λ_𝑘(𝑁)². (65) Therefore, 𝐼_𝐱𝑓_𝑀𝐾≤𝑓_𝐿2Λ_𝑘(𝑁). (66) This proves the statement in (i).
(ii) Let 𝑥∈𝑋. Then 𝑓_𝑀(𝑥)−𝐼_𝐱𝑓_𝑀(𝑥)=(2𝜋)^−𝑛_{𝜉∈[−𝑀𝜋,𝑀𝜋]𝑛}𝑓(𝜉)×𝑒^{𝑖𝑥⋅𝜉}−_{𝑗∈𝑋𝑁}𝑢_𝑗,𝐱(𝑥)𝑒^{𝑖𝑥𝑗⋅𝜉}𝑑𝜉. (67) By the Schwartz inequality, 𝑓_𝑀(𝑥)−𝐼_𝐱𝑓_𝑀(𝑥≤())2𝜋^−𝑛_{𝜉∈[−𝑀𝜋,𝑀𝜋]𝑛}𝑓(𝜉)²𝑘(𝜉)𝑑𝜉^1/2×(2𝜋)^−𝑛_ℝ𝑛𝑘(𝜉)𝑒^{𝑖𝑥⋅𝜉}−_{𝑗∈𝑋𝑁}𝑢_𝑗,𝐱(𝑥)𝑒^{𝑖𝑥𝑗⋅𝜉2}𝑑𝜉^1/2. (68) The first term is bounded by ‖𝑓‖_𝐿2Λ_𝑘(𝑀). The second term is 𝑘(0)−2_{𝑗∈𝑋𝑁}𝑢_𝑗,𝐱(𝑥)𝑘𝑥−𝑥_𝑗+_{𝑖,𝑗∈𝑋𝑁}𝑢_𝑖,𝐱(𝑥)𝑘𝑥_𝑖−𝑥_𝑗𝑢_𝑗,𝐱(𝑥)^1/2 (69) which can be bounded by 𝜀_𝐾(𝐱), as shown in the proof of Theorem 12. Therefore, by (52), 𝑓_𝑀−𝐼_𝐱𝑓_{𝑀𝐶(𝑋)}≤𝑓_𝐿2Λ_𝑘(𝑀)𝜀_𝐾(𝐱)≤𝑓_𝐿2Λ_𝑘(𝑀)𝜆_𝑘(𝑁). (70)
(iii) By the Plancherel formula, 𝑓−𝑓_𝑀²_{𝐿2(ℝ𝑛)}=(2𝜋)^−𝑛_{𝜉∉[−𝑀𝜋,𝑀𝜋]𝑛}𝑓(𝜉)²𝑑𝜉. (71) This proves all the statements in Theorem 16.

Theorem 16 provides quantitative estimates for the approximation error: 𝑓−𝐼_𝐱𝑓_{𝑀𝐿2(𝑋)}≤(2𝜋)^−𝑛_{𝜉∉[−𝑀𝜋,𝑀𝜋]𝑛}𝑓(𝜉)²𝑑𝜉^1/2+𝑓_𝐿2Λ_𝑘(𝑀)𝜆_𝑘(𝑁) (72) with 𝐼_𝐱𝑓_𝑀𝐾≤𝑓_𝐿2Λ_𝑘(𝑁). (73) Choose 𝑁=𝑁(𝑀)≥𝑀 such that Λ_𝑘(𝑀)𝜆_𝑘(𝑁)→0 as 𝑀→+∞; we have ‖𝑓−𝐼_𝐱(𝑓_𝑀)‖_𝐿2(𝑋)→0 and the RKHS norm of 𝐼_𝐱(𝑓_𝑀) is controlled by the asymptotic behavior of Λ_𝑘(𝑁).

Denote by Λ_𝑘⁻¹ the inverse function of Λ_𝑘: Λ_𝑘⁻¹(𝑅)∶=max𝑟>0∶Λ_𝑘(𝑟)𝑘(𝜉)≤𝑅=max𝑟>0∶≥𝑅⁻²∀𝜉∈[−𝑟𝜋,𝑟𝜋]^𝑛. (74) Then, our estimate for the approximation error can be given as follows.

Corollary 17. Let 𝑋=[0,1]^𝑛 and 𝑓∈𝐻^𝑠(ℝ^𝑛). Then, for 𝑅>‖𝑓‖_𝐿2, inf_{‖𝑔‖𝐾≤𝑅}𝑓−𝑔_𝐿2(𝑋)≤inf_{0<𝑀≤𝑁𝑅}𝑓_𝐻𝑠(𝑀𝜋)^−𝑠+𝑓_𝐿2Λ_𝑘(𝑀)𝜆_𝑘𝑁_𝑅, (75) where 𝑁_𝑅∶=[Λ_𝑘⁻¹(𝑅/‖𝑓‖_𝐿2)]. If 𝑠>𝑛/2, then inf_{‖𝑔‖𝐾≤𝑅}𝑓−𝑔_𝐶(𝑋)≤inf_{0<𝑀≤𝑁𝑅}𝑓_𝐻𝑠𝑀𝑠−𝑛/2^{𝑛/2−𝑠}+𝑓_𝐿2Λ_𝑘(𝑀)𝜆_𝑘𝑁_𝑅. (76) In particular, when 𝐶₁exp−𝜆₁𝜉^𝑑≤𝑘(𝜉)≤𝐶₀exp−𝜆𝜉,∀𝜉∈ℝ^𝑛 (77) for some 𝐶₀,𝐶₁,𝑑,𝜆₁>0 and 𝜆>4+2𝑛log4, one has𝐼𝑓,𝑅=inf_{‖𝑔‖𝐾≤𝑅}𝑓−𝑔_𝐿2(𝑋)≤2^𝑑(𝑑+1)𝑛^𝑑/2𝜋^𝑑𝜆₁^{𝑠/𝑑(𝑑+1)}×𝜋^−𝑠𝑓_𝐻𝑠+2^𝑠+2𝐶₀𝐶₁𝑓_𝐿2×1log𝑅+2log𝐶₁−log𝑓_𝐿2^{−𝑠/𝑑(𝑑+1)} (78) provided that with the function 𝐺(𝑟)∶=(1/√𝑛𝜋)((2/𝜆₁)log(𝑟𝐶₁/‖𝑓‖_𝐿2))^1/𝑑, 𝑅 satisfies 𝐺(𝑅)≥16𝜆₁𝑛^𝑑/2𝜋^𝑑−logmax1/𝑒𝜆,4^𝑛/𝑒^𝜆/2𝑑+1,log𝐺(𝑅)𝐺(𝑅)≤−logmax1/𝑒𝜆,4^𝑛/𝑒^𝜆/2(𝑑+1)2^𝑑+4𝑠. (79)

Proof. The first part is a direct consequence of Theorem 16 when we choose 𝑁 to be 𝑁_𝑅, the integer part of Λ_𝑘⁻¹(𝑅/‖𝑓‖_𝐿2).
To see the second part, we note that (77) in connection with Proposition 15 implies with Λ∶=max{1/𝑒𝜆,4^𝑛/𝑒^𝜆/2}, 𝜆_𝑘(𝑁)≤4𝐶₀exp𝑁logΛ2,Λ_𝑘(𝑟)≤𝐶₁exp−𝜆₁√𝑛𝑟𝜋^𝑑−1/2=1𝐶₁𝜆exp¹2√𝑛𝜋^𝑑𝑟^𝑑. (80) Then, Λ_𝑘⁻¹(𝑅/‖𝑓‖_𝐿2)≥𝐺(𝑅).
For 𝑅≥(‖𝑓‖_𝐿2/𝐶₁)exp{(𝜆₁/2)(√𝑛𝜋)^𝑑}, we can choose 𝑀∈ℕ such that 12{𝐺(𝑅)}^1/(𝑑+1)≤𝑀≤{𝐺(𝑅)}^1/(𝑑+1). (81) Choose 𝑁∈ℕ such that 𝑀^𝑑+12≤𝑁≤𝑀^𝑑+1. (82) Then, 𝑀≤𝑁, and by Theorem 16, 𝐼_𝐱𝑓_𝑀𝐾≤𝑓_𝐿2𝐶₁𝜆exp¹2√𝑛𝜋^𝑑𝑀^𝑑(𝑑+1)Λ≤𝑅,_𝑘(𝑀)𝜆_𝑘(𝑁)≤4𝐶₀𝐶₁exp𝑁logΛ4×+𝑁logΛ4+𝜆₁√𝑛𝜋^𝑑𝑁^{−1/(𝑑+1)}. (83) When 𝑁^1/(𝑑+1)≥4𝜆₁√𝑛𝜋^𝑑,−logΛlog𝑁𝑁≤−logΛ(𝑑+1),4𝑠 (84) there holds Λ_𝑘(𝑀)𝜆_𝑘(𝑁)≤4𝐶₀𝐶₁exp𝑁logΛ4≤4𝐶₀𝐶₁𝑁^{−𝑠/(𝑑+1)}. (85) Hence, 𝑓−𝐼_𝐱𝑓_{𝑀𝐿2(𝑋)}≤𝑓_𝐻𝑠(𝑀𝜋)^−𝑠+𝑓_𝐿24𝐶₀𝐶₁2𝑀^𝑠≤𝑓_𝐻𝑠𝜋^−𝑠+2^𝑠𝑓_𝐿24𝐶₀𝐶₁×2^𝑠(𝐺(𝑅))^{−𝑠/(𝑑+1)}. (86) When 𝑅 satisfies (79), we know that 𝑁^1/(𝑑+1)≥𝑀2≥14𝐺(𝑅)^{1/𝑑1/(𝑑+1)}≥4𝜆₁√𝑛𝜋^𝑑,−logΛlog𝑁𝑁≤2log𝑀^𝑑+1𝑀^𝑑+1≤2^𝑑+2log𝐺(𝑅)𝐺(𝑅)≤−logΛ(𝑑+1).4𝑠 (87) Hence, (84) holds true. This proves our statements.

For the Gaussian kernels, we have the following.

Proposition 18. Let 𝐾𝑥,𝑦=exp−𝑥−𝑦²𝜎²,𝑥,𝑦∈𝑋=[0,1]^𝑛. (88) Denote 𝐶_𝜎,𝑛∶=𝜎²𝑛𝜋²/4min{log(4√𝑛),𝑛log2} and 𝐶_{𝜎,𝑛,𝑠}∶=(𝜎√𝑛𝜋)^𝑠/2(𝐶^𝑠/2_𝜎,𝑛+(𝜎√𝜋)^−𝑛/2(2√𝑒+4/𝜎√𝜋)). If 𝑓∈𝐻^𝑠(ℝ^𝑛), then one has𝐼𝑓,𝑅≤𝐶_{𝜎,𝑛,𝑠}𝑓_𝐻𝑠+𝑓_𝐿2×𝑛log𝑅+2log𝜎√𝜋−log𝑓_𝐿2^−𝑠/4 (89) and when 𝑠>𝑛/2, inf_{‖𝑔‖𝐾≤𝑅}𝑓−𝑔_𝐶(𝑋)≤𝐶_{𝜎,𝑛,𝑠−𝑛/2}𝑓_𝐻𝑠+𝑓_𝐿2×𝑛log𝑅+2log𝜎√𝜋−log𝑓_𝐿2^{𝑛/8−𝑠/4} (90) for 𝑅 satisfying 𝑅>𝑓_𝐿2𝜎√𝜋^−𝑛/2𝜎×exp²𝑛𝜋²max𝐶_𝜎,𝑛,80𝑛log2/𝜎²+1²8,132𝐶_𝜎,𝑛𝜎√log𝜋^𝑛/2𝑅𝑓_𝐿2^1/2≥𝑠22√log2𝜎√+1𝑛𝜋2𝜎√log𝜋^𝑛/2𝑅𝑓_𝐿2. (91)(92)

Proof. The Fourier transform of 𝑘(𝑥)=exp{−|𝑥|²/𝜎²} is 𝑘(𝜉)=𝜎√𝜋^𝑛−𝜎exp²𝜉²4. (93) Then, Λ_𝑘(𝑟)=𝜎√𝜋^−𝑛/2𝜎exp²𝑛𝑟²𝜋²8. (94)
For 𝑅≥𝑓_𝐿2Λ_𝑘max𝐶_𝜎,𝑛,80𝑛log2𝜎²+1, (95) we can take 𝑁∈ℕ with 𝑁≥max{𝐶_𝜎,𝑛,80𝑛log2/𝜎²} such that 12Λ_𝑘⁻¹𝑅𝑓_𝐿2≤𝑁≤Λ_𝑘⁻¹𝑅𝑓_𝐿2. (96) Here, Λ_𝑘⁻¹ is the inverse function of Λ_𝑘: Λ_𝑘⁻¹(𝑟)=2√2𝜎√𝜎√𝑛𝜋log𝜋^𝑛/2𝑟^1/2. (97) Then, ‖𝑓‖_𝐿2Λ_𝑘(𝑁)≤𝑅. Let 𝑀≤𝑁. By Theorem 16, ‖𝐼_𝐱(𝑓_𝑀)‖_𝐾≤𝑅.
By Corollary 17 and (57), 𝑓−𝐼_𝐱𝑓_{𝑀𝐿2(𝑋)}≤𝑓_𝐻𝑠(𝑀𝜋)^−𝑠+𝑓_𝐿2Λ_𝑘(𝑀)2√4𝑒+𝜎√𝜋exp−𝑁𝐶₀, (98) where 𝐶₀∶=min{log(4√𝑛),𝑛log2}. Choose 𝑀∈ℕ such that 12𝑁𝐶_𝜎,𝑛≤𝑀≤𝑁𝐶_𝜎,𝑛. (99) With this choice, 𝜎²𝑛𝑀²𝜋²/8≤𝐶₀𝑁/2. Therefore, 𝑓−𝐼_𝐱𝑓_{𝑀𝐿2(𝑋)}≤𝑓_𝐻𝑠4𝐶_𝜎,𝑛𝜋²𝑁^𝑠/2+𝑓_𝐿2𝜎√𝜋^−𝑛/2×2√4𝑒+𝜎√𝜋−𝐶exp⁰2𝑁≤𝐶^′_{𝜎,𝑛,𝑠}𝑓_𝐻𝑠+𝑓_𝐿2𝑅×max𝑓_𝐿2^−𝑠/2−𝐶,exp⁰4Λ_𝑘⁻¹𝑅𝑓_𝐿2, (100) where 𝐶^′_{𝜎,𝑛,𝑠}=𝐶^𝑠/2_𝜎,𝑛+𝜎√𝜋^−𝑛/22√4𝑒+𝜎√𝜋. (101)
When 𝐶₀4Λ_𝑘⁻¹𝑅𝑓_𝐿2≥𝑠2logΛ_𝑘⁻¹𝑅𝑓_𝐿2, (102) there holds 𝑓−𝐼_𝐱𝑓_{𝑀𝐿2(𝑋)}≤𝐶^′_{𝜎,𝑛,𝑠}𝑓_𝐻𝑠+𝑓_𝐿2×Λ_𝑘⁻¹𝑅𝑓_𝐿2^−𝑠/2. (103) This yields the first estimate.
When 𝑠>𝑛/2, the same method gives the error with the uniform norm.

5. Learning with Varying Kernels

Proposition 18 in the last section shows that, for a fixed Gaussian kernel, the approximation error 𝐼(𝑓,𝑅) behaves as 𝐼𝑓,𝑅≤𝐶log𝑅^−𝑠/4 (104) for functions 𝑓 in 𝐻^𝑠.

In this section, we consider the learning with varying kernels. Such a method is used in many applications where we have to choose suitable parameters for the reproducing kernel. For example, in [7] Gaussian kernels with different parameters in different directions are considered. Here, we study the case when the variance parameter keeps the same in all directions. Our analysis shows that the approximation error may be improved when the kernel changes with the RKHS norm 𝑅 of the empirical target function.

Proposition 19. Let 𝐾_𝜎𝑥,𝑦=exp−𝑥−𝑦²𝜎²,𝑥,𝑦∈𝑋=[0,1]^𝑛. (105) There exist positive constants 𝐴_𝑛,𝑠 and 𝐵_𝑛,𝑠, depending only on 𝑛 and 𝑠, such that for each 𝑓∈𝐻^𝑠(ℝ^𝑛) and 𝑅≥𝐴_𝑛,𝑠‖𝑓‖_𝐿2, one can find some 𝜎=𝜎_𝑅 satisfying inf_{‖𝑔‖𝐾𝜎𝑅≤𝑅}𝑓−𝑔_𝐿2(𝑋)≤𝐵_𝑛,𝑠log𝑅^−𝑠. (106)

Proof. Take 𝜎=80𝑛log2𝑁^1/2,𝑁14𝑛𝜋5min2+log𝑛2log2,𝑛^1/2𝑁≤𝑀≤12𝑛𝜋5min2+log𝑛2log2,𝑛^1/2, (107) where 𝑁 depends on 𝑅. Denote 𝐶_𝑛∶=𝑛𝜋²/4min{log(4√𝑛),𝑛log2}. As in the proof of Proposition 18, we have 𝑓−𝐼_𝐱𝑓_{𝑀𝐿2(𝑋)}≤(𝑁𝜋)^−𝑠𝑓_𝐻𝑠320𝑛𝐶_𝑛log2^𝑠/2+𝑁^𝑛/2𝑓_𝐿280𝑛log2^𝑛/4×2√4√𝑒+𝑁−𝐶80𝑛𝜋log2exp⁰2𝑁. (108)
When 𝑁 is large enough, with a constant 𝐶^′_𝑛,𝑠 depending on 𝑛 and 𝑠, this yields 𝑓−𝐼_𝐱𝑓_{𝑀𝐿2(𝑋)}≤𝐶^′_𝑛,𝑠𝑓_𝐻𝑠+𝑓_𝐿2𝑁^−𝑠. (109)
Finally, we determine 𝑁 by requiring 𝑓_𝐿2Λ_𝑘(𝑁)=𝑓_𝐿2𝑁^𝑛/480𝑛𝜋log2^−𝑛/4exp10𝑛²𝜋²𝑓𝑁log2≤𝑅≤_𝐿2Λ_𝑘(𝑁+1). (110) There is a constant 𝐴_𝑛,𝑠>0 depending only on 𝑛 and 𝑠 such that, for 𝑅≥𝐴_𝑛,𝑠‖𝑓‖_𝐿2, an integer 𝑁 satisfying all the previous requirements and (𝑁+1)^𝑛/4≤exp10𝑛²𝜋²(𝑁+1)log2 (111) exists. This makes all the estimates valid. It follows that 𝑅≤𝑓_𝐿2Λ_𝑘()≤𝑓𝑁+1_𝐿280𝑛𝜋log2^−𝑛/4exp20𝑛²𝜋²(𝑁+1)log2. (112) Hence, 1𝑁+1≥20𝑛²𝜋²𝑅log2log𝑓_𝐿280𝑛𝜋log2^𝑛/4. (113) Therefore, there holds ‖𝐼_𝐱(𝑓_𝑀)‖_𝐾𝜎≤𝑅 and 𝑓−𝐼_𝐱𝑓_{𝑀𝐿2(𝑋)}≤2^𝑠𝐶^′_𝑛,𝑠𝑓_𝐻𝑠+𝑓_𝐿220𝑛²𝜋²log2^𝑠×𝑛log𝑅+4log80𝑛𝜋log2−log𝑓_𝐿2^−𝑠. (114) This verifies our claim for the approximation error in 𝐿²(𝑋).

Let us mention the following problem concerning learning with Gaussian kernels with changing variances.

Problem 20. What is the optimal rate of convergence of sup_{‖𝑓‖𝐻𝑠=1}inf𝑓−𝑔_𝐿2(𝑋)∶𝑔_𝐾𝜎≤𝑅forsome𝜎>0 (115) as 𝑅 tends to infinity?

6. Dot Product Kernels

In this section, we illustrate our results by the family of dot product type kernels. These kernels take the form 𝐾𝑥,𝑦=^+∞_𝑗=0𝑎_𝑗𝑥⋅𝑦^𝑗,𝑥,𝑦∈ℝ^𝑛. (116) When ∑^+∞_𝑗=0|𝑎_𝑗|𝑅^2𝑗<∞ for some 𝑅>0, the kernel 𝐾 is a Mercer kernel on 𝑋∶={𝑥∈ℝ^𝑛∶|𝑥|≤𝑅} if and only if 𝑎_𝑗≥0 for each 𝑗≥0. See [25–28]. Here, we will characterize the density for this family as [29]. Denote 𝑥^𝛼∶=Π^𝑛_𝑖=1𝑥^𝛼𝑖_𝑖 and _𝛼^|𝛼|∶=(𝛼₁+⋯𝛼_𝑛)!/(Π^𝑛_𝑖=1𝛼_𝑗!) for 𝑥=(𝑥₁,…,𝑥_𝑛)∈ℝ^𝑛 and 𝛼=(𝛼₁,…,𝛼_𝑛)∈ℤ^𝑛₊.

Corollary 21. Let 𝑅>0, 𝑋∶=[0,𝑅]^𝑛, and the kernel 𝐾 be given by (116), where 𝑎_𝑗≥0 for each 𝑗∈ℤ₊ and ∑^+∞_𝑗=0𝑎_𝑗𝑅^2𝑗<∞. Set 𝐽∶={𝛼∈ℤ^𝑛₊∶𝑎_|𝛼|>0}. Then, ℋ_𝐾 is dense in 𝐶(𝑋) if and only if span{𝑥^𝛼∶𝛼∈𝐽} is dense in 𝐶(𝑋). Thus, the density depends only on the location of positive coefficients in (116). In particular, when 𝑛=1, ℋ_𝐾 is dense in 𝐶[0,𝑅] if and only if 𝑎₀>0,_{𝑗∈𝐽⧵{0}}1𝑗=+∞. (117)

Proof. Note that 𝐾_𝑥𝑦=𝐾𝑥,𝑦=𝑎₀+^+∞_𝑗=1𝑎_{𝑗|𝛼|=𝑗}|𝛼|𝛼𝑥^𝛼𝑦^𝛼=_𝛼∈𝐽𝑎_|𝛼||𝛼|𝛼𝑥^𝛼𝑦^𝛼. (118)
Sufficiency. Suppose that span{𝑥^𝛼∶𝛼∈𝐽} is dense in 𝐶(𝑋), but ℋ_𝐾 is not dense in 𝐶(𝑋). Then, by Theorem 4 there exists a nontrivial Borel measure 𝜇 on 𝑋 such that _𝑋𝐾𝑥,𝑦𝑑𝜇𝑦=0,∀𝑥∈𝑋. (119) Taking the integral with respect to 𝜇 and using (118), we have 0=_𝑋𝐾𝑥,𝑦𝑑𝜇(𝑥)𝑑𝜇𝑦=_𝛼∈𝐽𝑎_|𝑔𝑎||𝛼|𝛼^𝑋𝑥^𝛼𝑑𝜇(𝑥)². (120) Since 𝑎_|𝛼|>0 for each 𝛼∈𝐽, there holds _𝑋𝑥^𝛼𝑑𝜇(𝑥)=0,∀𝛼∈𝐽. (121) That is, 𝜇 annihilates each 𝑥^𝛼 for 𝛼∈𝐽. But span{𝑥^𝛼∶𝛼∈𝐽} is dense in 𝐶(𝑋); 𝜇 also annihilates all functions in 𝐶(𝑋), which is a contradiction.
Necessity. If span{𝑥^𝛼∶𝛼∈𝐽} is not dense in 𝐶(𝑋), then there exists a nontrivial Borel measure 𝜇 annihilating each 𝑥^𝛼; that is, ∫_𝑋𝑥^𝛼𝑑𝜇(𝑥)=0 for each 𝛼∈𝐽. Then, (118) tells us that, for each 𝑥∈𝑋, _𝑋𝐾𝑥,𝑦𝑑𝜇𝑦=_𝛼∈𝐽𝑎_|𝑔𝑎||𝛼|𝛼𝑥^𝛼_𝑋𝑦^𝛼𝑑𝜇𝑦=0. (122) This in connection with Theorem 4 implies that ℋ_𝐾 is not dense in 𝐶(𝑋). This proves the first statement of Corollary 21.
The second statement follows from the classical Muntz Theorem in approximation theory (see [30]): for a strictly increasing sequence of nonnegative numbers 𝜆₀<𝜆₁,…, span{𝑥^𝜆𝑗∶𝑗∈ℤ₊} is dense in 𝐶[0,𝑅] if and only if 𝜆₀=0 and ∑^+∞_𝑗=11/𝜆_𝑗=+∞.

The conclusion in Example 2 follows directly from Corollary 21.

By Corollary 21, we can provide more examples of dot product positive definite kernels whose corresponding RKHS is not dense. The following is such an example. However, compared with Example 8, it is not constructive, in the sense that no function outside the closure of ℋ_𝐾 is explicitly given.

Example 22. Let 𝑋=[0,1] and define 𝐾𝑥,𝑦=1+^+∞_𝑘=12^{−𝑘𝑘−1}_𝑚=0𝑥⋅𝑦^2𝑘+𝑚. (123) Then, 𝐾 is a positive definite Mercer kernel on 𝑋, but ℋ_𝐾 is not dense in 𝐶(𝑋).

Proof. Observe that the assumption in Corollary 21 holds for 𝐾, 𝑎₀=1>0 and 𝐽⧵{0}=^+∞_𝑘=12^𝑘,2^𝑘+1,…,2^𝑘+𝑘−1. (124) Since ∑_{𝑗∈𝐽⧵{0}}(1/𝑗)=∑^+∞_𝑘=1∑^𝑘−1_𝑖=0(1/(2^𝑘+𝑖))≤∑^+∞_𝑘=1(𝑘/2^𝑘)<+∞, Corollary 21 tells us that ℋ_𝐾 is not dense in 𝐶(𝑋).
What is left is to show that the Mercer kernel 𝐾 is positive definite. Suppose to the contrary that there exist a finite set of distinct points 𝐼⊂𝑋 and a nonzero vector 𝑐=(𝑐_𝑠)_𝑠∈𝐼 such that _{𝑠,𝑡∈𝐼}𝑐_𝑠𝑐_𝑡𝐾(𝑠,𝑡)=0. (125) Denote 𝐾𝑥,𝑦=^+∞_𝑘=12^{−𝑘𝑘−1}_𝑚=0𝑥⋅𝑦^2𝑘+𝑚. (126) Then, _{𝑠,𝑡∈𝐼}𝑐_𝑠𝑐_𝑡𝐾(𝑠,𝑡)=_𝑠∈𝐼𝑐_𝑠²+_{{0}𝑠,𝑡∈𝐼⧵}𝑐_𝑠𝑐_𝑡𝐾(𝑠,𝑡)=0. (127) Hence, ∑_𝑠∈𝐼𝑐_𝑠=0 which implies that 𝐼⧵{0}≠∅ and (𝑐_𝑠)_{𝑠∈𝐼⧵{0}} is a nonzero vector. Also, 0=_{{0}𝑠,𝑡∈𝐼⧵}𝑐_𝑠𝑐_𝑡𝐾()=𝑠,𝑡_+∞𝑘=02^{−𝑘𝑘−1}_{𝑚=0𝑠∈𝐼⧵{0}}𝑠^2𝑘+𝑚𝑐_𝑠². (128) It follows that _{𝑠∈𝐼⧵{0}}𝑠^2𝑘+𝑚𝑐_𝑠=0,∀𝑘∈ℕ,𝑚=0,1,…,𝑘−1. (129) Choose an integer 𝑘 which is not less than #(𝐼⧵{0}), the number of elements in the set 𝐼⧵{0}. Then, we know that the linear system _{𝑠∈𝐼⧵{0}}𝑠^2𝑘+𝑚𝑥_𝑠=0,𝑚=0,1,…,#(𝐼⧵{0}) (130) has a nonzero solution (𝑐_𝑠)_{𝑠∈𝐼⧵{0}}. Hence, the matrix (𝑠^2𝑘+𝑚)_{𝑠∈𝐼⧵{0},𝑚=0,1,…,#(𝐼⧵{0})} is singular. So, there exists a nonzero vector (𝑑_𝑚)^{#(𝐼⧵{0})}_𝑚=0 such that ^{#(𝐼⧵{0})}_𝑚=0𝑠^2𝑘+𝑚𝑑_𝑚=0,∀𝑠∈𝐼⧵{0}. (131) As each element 𝑠 in 𝐼⧵{0} is nonzero, we have ^{#(𝐼⧵{0})}_𝑚=0𝑠^𝑚𝑑_𝑚=0,∀𝑠∈𝐼⧵{0}. (132) However, the determinant of the matrix (𝑠^𝑚)_{𝑠∈𝐼⧵{0},𝑚=0,1,…,#(𝐼⧵{0})} is a Vandermonde determinant and is nonzero. This is a contradiction, as the linear system having this matrix as the coefficient matrix has a nonzero solution. Therefore, the Mercer kernel 𝐾 is positive definite.

An alternative simpler proof for the positive definiteness of the kernel in Example 22 can be given by means of the recent results in [25, 26].

After characterizing the density, we can then apply our analysis in Section 3 and provide some estimates for the convergence rate of the approximation error under the assumption that all the coefficients 𝑎_𝑗 in (116) are strictly positive. We will not provide details here, but only show the application of the interpolation scheme (34) to polynomials.

If 𝑓(𝑥)=∑_{|𝛼|≤𝑀}𝑐_𝛼𝛼^|𝛼|𝑥^𝛼, then 𝐼_𝐱𝑓(𝑥)−𝑓(𝑥)=_{|𝛼|≤𝑀}𝑐_𝛼|𝛼|𝛼^ℓ𝑗=1𝑢_𝑗(𝑥)𝑥^𝛼_𝑗−𝑥^𝛼. (133) It follows from the Schwartz inequality that 𝐼_𝐱𝑓(𝑥)−𝑓(𝑥)²≤_{|𝛼|≤𝑀}𝑐_𝛼^2|𝛼|_𝛼𝑎_|𝛼|×_{|𝛼|≤𝑀}𝑎_|𝛼||𝛼|𝛼^ℓ𝑗=1𝑢_𝑗(𝑥)𝑥^𝛼_𝑗−𝑥^𝛼2. (134) The first term can be bounded by _{|𝛼|≤𝑀}|𝛼|𝛼𝑐_𝛼²min_{𝑗=0,1,…,𝑀}𝑎_𝑗⁻¹ (135) while the second is bounded by _{𝛼∈ℤ𝑛+}𝑎_|𝛼||𝛼|𝛼^{ℓ𝑖,𝑗=1}𝑢_𝑖(𝑥)𝑢_𝑗(𝑥)𝑥^𝛼_𝑖𝑥^𝛼_𝑗−2^ℓ_𝑗=1𝑢_𝑗(𝑥)𝑥^𝛼_𝑗𝑥^𝛼+𝑥^𝛼⋅𝑥^𝛼=𝜀_𝐾(𝐱). (136) Thus, the approximation error can be given in terms of the regularity of the kernel 𝐾. The regularity of the approximated function yields the rate of approximation by polynomials 𝑓=𝑓_𝑀 while the asymptotic behavior of the coefficients 𝑎_𝑗 in (116) provides the control of the RKHS norm of 𝐼_𝐱(𝑓_𝑀).

Acknowledgments

The author would like to thank Charlie Micchelli for proving Corollary 6 in the general form, Allan Pinkus for clarifying Example 2, Tommy Poggio for raising the density problem, Steve Smale for suggestions on positive definiteness and approximation error in learning theory, and Grace Wahba for knowledge on earlier work of approximation by reproducing kernel Hilbert spaces. The work described in this paper is partially supported by a Grant from the Research Grants Council of Hong Kong (Project no. CityU 104710).