- About this Journal ·
- Abstracting and Indexing ·
- Advance Access ·
- Aims and Scope ·
- Annual Issues ·
- Article Processing Charges ·
- Articles in Press ·
- Author Guidelines ·
- Bibliographic Information ·
- Citations to this Journal ·
- Contact Information ·
- Editorial Board ·
- Editorial Workflow ·
- Free eTOC Alerts ·
- Publication Ethics ·
- Reviewers Acknowledgment ·
- Submit a Manuscript ·
- Subscription Information ·
- Table of Contents
Abstract and Applied Analysis
Volume 2013 (2013), Article ID 134727, 8 pages
Coefficient-Based Regression with Non-Identical Unbounded Sampling
School of Mathematics and Computational Science, Guangdong University of Business Studies, Guangzhou, Guangdong 510320, China
Received 18 January 2013; Accepted 15 April 2013
Academic Editor: Qiang Wu
Copyright © 2013 Jia Cai. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
We investigate a coefficient-based least squares regression problem with indefinite kernels from non-identical unbounded sampling processes. Here non-identical unbounded sampling means the samples are drawn independently but not identically from unbounded sampling processes. The kernel is not necessarily symmetric or positive semi-definite. This leads to additional difficulty in the error analysis. By introducing a suitable reproducing kernel Hilbert space (RKHS) and a suitable intermediate integral operator, elaborate analysis is presented by means of a novel technique for the sample error. This leads to satisfactory results.
1. Introduction and Preliminary
We study coefficient-based least squares regression with indefinite kernels from non-identical unbounded sampling processes. In our setting, functions are defined on a compact subset of and take values in . Let be a Borel probability measure on . A sample is drawn independently from different Borel probability measures (), . Let be the marginal distribution of on and the marginal distribution of on . We assume that the sequence converges exponentially fast in the dual of the Hölder space . Here the Hölder space () is defined as the space of all continuous functions on with the following norm finite : where
Definition 1. Let ; we say that the sequence converges exponentially fast in to a probability measure on or converges exponentially in short if there exist and 1 such that
By the definition of the dual space , the decay condition (3) can be expressed as The regression function is given by where is the conditional distribution of at and is unknown, so cannot be obtained directly. The aim of regression problem is to learn a good approximation of from sample . This is an ill-posed problem and regularization scheme is needed.
Classical learning algorithm is conducted by a scheme in a reproducing kernel Hilbert space (RKHS)  associated with a Mercer kernel , which is defined to be a continuous, symmetric, and positive semi-definite (p.s.d.) function. RKHS is defined to be the completion of the linear span of with the inner product . Define ; then the regularized regression problem is given by It has been well understood due to lots of the literature ([3–5] and the references therein). Here we consider the indefinite kernel scheme in a hypothesis space depending on the sample ; this space is defined by
And the regularized penalty term is imposed on the coefficients of function . Indefinite kernel means does not need to satisfy symmetry and p.s.d. condition except for continuity and boundedness. Define ; then is a Mercer kernel. For more introductions about learning with indefinite kernels, please see [6–8]. For all, if we define and , since is compact and is continous, and its adjoint are both compact operators. Hence , . The learning algorithm we are interested in this paper takes the following form:
We define the following coefficient-based regularizer:
Then we have . By using the integral operator technique from , in , Sun and Wu gave the capacity independent estimate for the convergence rate of , where . Shi investigated the error analysis in a data dependent hypothesis space for general kernels . Sun and Guo conducted error analysis for the Mercer kernels by uniform bounded non-i.i.d. sampling . In this paper, we study learning algorithm (8) by non-identical unbounded sampling processes with indefinite kernels.
Lemma 2. Let be a separable Hilbert space and a compact operator on ; then can be factored as where and is a partial isometry on with being orthogonal projection onto .
We immediately have the following proposition .
Proposition 3. Consider as a subspace of ; then and , where is a partial isometry on with being the orthogonal projection onto .
We use the RKHS to approximate , hence define
In order to estimate , we construct where . Then we can decompose the error term into the following three parts:
We will conduct the error analysis in several steps. The major contribution we make is on the sample error estimate; the main difficulty is the non-identical unbounded sampling of the samples; we overcome this difficulty by introducing a suitable intermediate operator.
2. Key Analysis and Main Results
Definition 4. We say that the Mercer kernel satisfies the kernel condition of order if, for some constant , and for all ,
Since sample is drawn from unbounded sampling processes, we will assume the following moment hypothesis condition .
Moment Hypothesis. There exist constants and such that
There is a large literature on error analysis for learning algorithm (6); see, for example, [4, 5, 14–17]. But most obtained results are presented under the standard assumption that almost surely ( is a constant). This excludes the case of Gaussian noise. The moment hypothesis condition is a natural generalization of the condition . Wang and Zhou considered error analysis for algorithm (6) under condition (15). Our main results are about learning rates of algorithm (8) under conditions (3), (14) and the approximation ability of in terms of .
Now we can state our general results on learning rates for algorithm (8).
Theorem 5. Assume moment hypothesis condition (15); satisfies condition (3) and satisfies condition (14); , for ; take with ; then where is a constant depending on and, but not on or , and will be given explicitly in Section 3.3.
Remark 6. If we take , then our rate is . The proof of Theorem 5 will be conducted in Section 3, where the error term is decomposed into three parts. In , the authors consider the coefficient-based regression with the Mercer kernels by uniform bounded non-i.i.d sampling; the best rate of order was obtained.
When the samples are drawn i.i.d from measure , we have the following result.
Here we get the same learning rate as the one in . But our rate is derived under a relaxation condition of the sampling output.
3. Error Analysis
In this section, we will state the error analysis in several steps.
3.1. Regularization Error Estimation
In this subsection, we address a bound for the regularization error . The error estimate for regularization error has been investigated in lots of the literature in learning theory ([4, 18] and the references therein); we will omit the proof and quote it directly.
Proposition 8. Assume for some and ; the following bound for approximation error holds: where , , and when , where .
3.2. Estimate for the Measure Error
This subsection is devoted to the analysis of the term caused by the difference of measures, which we called measure error. The ideas of proof are from . Before giving the result, let us state a crucial lemma first.
Lemma 9. Assume satisfies condition (14); then
Proof. For any , we see that where . Now we need to estimate and , respectively. For the term , it is easy to see that The estimation of is more involved: Since then Therefore Combining the estimation of and , we get When condition (14) is satisfied, it was proved in  that is included in with the inclusion bounded Then This completes the proof.
Proposition 10. Assume for some ; satisfies condition (14); the following bound for measure error holds: where and.
Proof. From (11), simple calculation shows that . Recalling (12), we can see that
Applying Lemma 9 to the case , we get
By the definition of and noticing (3), we can see
This in connection with Proposition 8 yields the conclusion.
3.3. Sample Error Estimation
In this subsection we will conduct the estimation of the term . At first, we give some notations. Let be the space of bounded continuous functions on with supremum norm . Define sampling operator by (). For , let and be operators from to defined as
It is easy to see that both and are bounded operators. Recall the definition of ; then Computing the gradient of the above equation, we immediately have  Hence . Employing the method as shown in , we can decompose the sample error into two parts:
Now we state our estimation for the sample error. The estimates are more involved since the sample is drawn by non-identical unbounded sampling processes. We overcome the difficulty by introducing a stepping integral operator which plays an intermediate role in the estimates, and the definition of it will be given later.
Proof. We will estimate and II, respectively:
where and is the norm on , for ; noticing (36), we can have
According to the definition of , for any , ; this implies that . Therefore
For the term , let
and . Then and
Applying the same method as shown in the proof of Lemma 4.1 in , we can see that when all the indices and are pairwise different, there holds ; therefore
This together with (45) yields
The term is more involved; recall that Hence
If we define and therefore If and are pairwise distinct, then . If or , By the Cauchy-Schwartz inequality, for any , Hence we only need to give a bound for . Simple calculation shows By the same method, we know that
Applying the conclusion as shown in  and together with the above bound, we can see that Hence
This yields then This together with (49) yields the conclusion.
Proof of Theorem 5. Theorem 11 ensures that
For , Proposition 10 tells that and Proposition 8 shows that since
Combining all the bounds together and noting that with , we can get the conclusion of Theorem 5 by taking .
Proof of Theorem 7. When the samples are drawn i.i.d. from measure , then . Hence
Let ; then
The conclusion follows by discussing the relationship between and .
The author would like to thank Professor Hongwei Sun for useful discussions which have helped to improve the presentation of the paper. The work described in this paper is supported partially by National Natural Science Foundation of China (Grant no. 11001247) and Doctor Grants of Guangdong University of Business Studies (Grant no. 11BS11001).
- S. Smale and D.-X. Zhou, “Online learning with Markov sampling,” Analysis and Applications, vol. 7, no. 1, pp. 87–113, 2009.
- N. Aronszajn, “Theory of reproducing kernels,” Transactions of the American Mathematical Society, vol. 68, pp. 337–404, 1950.
- F. Cucker and D.-X. Zhou, Learning Theory: An Approximation Theory Viewpoint, Cambridge University Press, Cambridge, UK, 2007.
- S. Smale and D.-X. Zhou, “Learning theory estimates via integral operators and their approximations,” Constructive Approximation, vol. 26, no. 2, pp. 153–172, 2007.
- Q. Wu, Y. Ying, and D.-X. Zhou, “Learning rates of least-square regularized regression,” Foundations of Computational Mathematics, vol. 6, no. 2, pp. 171–192, 2006.
- H. Sun and Q. Wu, “Indefinite kernel network with dependent sampling,” Analysis and Applications. Accepted.
- Q. Wu and D.-X. Zhou, “Learning with sample dependent hypothesis spaces,” Computers and Mathematics with Applications, vol. 56, no. 11, pp. 2896–2907, 2008.
- Q. Wu, “Regularization networks with indefinite kernels,” Journal of Approximation Theory, vol. 166, pp. 1–18, 2013.
- H. Sun and Q. Wu, “Least square regression with indefinite kernels and coefficient regularization,” Applied and Computational Harmonic Analysis, vol. 30, no. 1, pp. 96–109, 2011.
- L. Shi, “Learning theory estimate for coefficient-based regularized regression,” Applied and Computational Harmonic Analysis, vol. 34, no. 2, pp. 252–265, 2013.
- H. Sun and Q. Guo, “Coefficient regularized regression with non-iid sampling,” International Journal of Computer Mathematics, vol. 88, no. 15, pp. 3113–3124, 2011.
- J. B. Conway, A Course in Operator Theory, American Mathematical Society, 2000.
- C. Wang and D.-X. Zhou, “Optimal learning rates for least squares regularized regression with unbounded sampling,” Journal of Complexity, vol. 27, no. 1, pp. 55–67, 2011.
- A. Caponnetto and E. De Vito, “Optimal rates for the regularized least-squares algorithm,” Foundations of Computational Mathematics, vol. 7, no. 3, pp. 331–368, 2007.
- E. De Vito, A. Caponnetto, and L. Rosasco, “Model selection for regularized least-squares algorithm in learning theory,” Foundations of Computational Mathematics, vol. 5, no. 1, pp. 59–85, 2005.
- S. Mendelson and J. Neeman, “Regularization in kernel learning,” The Annals of Statistics, vol. 38, no. 1, pp. 526–565, 2010.
- I. Steinwart, D. Hush, and C. Scovel, “Optimal rates for regularized least-squares regression,” in Proceedings of the 22nd Annual Conference on Learning Theory, pp. 79–93, 2009.
- S. Smale and D.-X. Zhou, “Shannon sampling. II. Connections to learning theory,” Applied and Computational Harmonic Analysis, vol. 19, no. 3, pp. 285–302, 2005.
- D.-X. Zhou, “Capacity of reproducing kernel spaces in learning theory,” IEEE Transactions on Information Theory, vol. 49, no. 7, pp. 1743–1752, 2003.