Research Article | Open Access
A Note on Wavelet Estimation of the Derivatives of a Regression Function in a Random Design Setting
We investigate the estimation of the derivatives of a regression function in the nonparametric regression model with random design. New wavelet estimators are developed. Their performances are evaluated via the mean integrated squared error. Fast rates of convergence are obtained for a wide class of unknown functions.
We consider the nonparametric regression model with random design described as follows. Let be random variables defined on a probability space , where are i.i.d. random variables such that and , are i.i.d. random variables with common density , and is an unknown regression function. It is assumed that and are independent for any . We aim to estimate , that is, the th derivative of , for any integer , from .
In the literature, various estimation methods have been proposed and studied. The main ones are the kernel methods (see, e.g., [1–5]), the smoothing splines, and local polynomial methods (see, e.g., [6–9]). The object of this note is to introduce new efficient estimators based on wavelet methods. Contrary to the others, they have the benefit of enjoying local adaptivity against discontinuities thanks to the use of a multiresolution analysis. Reviews on wavelet methods can be found in, for example, Antoniadis , Härdle et al. , and Vidakovic . To the best of our knowledge, only Cai  and Petsa and Sapatinas  have proposed wavelet estimators for from (1) but defined with a deterministic equidistant design; that is, . The consideration of a random design complicates significantly the problem and no wavelet estimators exist in this case. This motivates our study.
In the first part, assuming that is known, we propose two wavelet estimators: the first one is linear nonadaptive and the second one nonlinear adaptive. Both use the approach of Prakasa Rao  initially developed in the context of the density estimation problem. Then we determine their rates of convergence by considering the mean integrated squared error (MISE) and assuming that belongs to Besov balls. In a second part, we develop a linear wavelet estimator in the case where is unknown. It is derived from the one introduced by Pensky and Vidakovic  considering the estimation of from (1). We evaluate its rate of convergence again under the MISE over Besov balls. The obtained rates of convergence are similar to those attained by wavelet estimators for the derivatives of a density (see, e.g., [15, 17, 18]).
The organization of this note is as follows. The next section describes some basics on wavelets and Besov balls. Our estimators and their rates of convergence are presented in Section 3. The proofs are carried out in Section 4.
This section is devoted to the presentation of the considered wavelet basis and the Besov balls.
2.1. Wavelet Basis
We set We consider the wavelet basis on introduced by Cohen et al. . Let and be the initial wavelet functions of the Daubechies wavelets family with (see, e.g., ). These functions have the distinction of being compactly supported and belong to the class for . For any , we set and, for ,
With appropriated treatments at the boundaries, there exists an integer such that, for any integer , forms an orthonormal basis of . For any integer and , we have the following wavelet expansion: where These quantities are called the wavelet coefficients of . See, for example, Cohen et al.  and Mallat .
2.2. Besov Balls
We consider the following wavelet sequential definition of the Besov balls. We say that with , , , and if there exists a constant such that and (6) satisfy with the usual modifications if or .
The interest of Besov balls is to contain various kinds of homogeneous and inhomogeneous functions . For particular choices of , , and , correspond to standard balls of function spaces, as the Hölder and Sobolev balls (see, e.g., [11, 22]).
In this section, we set the assumptions on the model, present our wavelet estimators, and determine their rates of convergence under the MISE over Besov balls.
We formulate the following assumptions.(K1)We have for any .(K2)There exists a constant such that (K3)There exists a constant such that (K4)There exists a constant such that
3.2. Wavelet Estimators: When Is Known
We consider the wavelet basis with to ensure that and belong to .
Linear Wavelet Estimator. We define the linear wavelet estimator by where and is an integer chosen a posteriori.
The definition of is motivated by the following unbiased property: using the independence between and , , and integrations by parts with (K1), we obtain which is the wavelet coefficient of associated with .
This approach was initially introduced by Prakasa Rao  for the estimation of the derivatives of a density. Its adaptation to (1) gives a suitable alternative to the wavelet methods developed by Cai  and Petsa and Sapatinas  in the case , specially in the treatment of the random design.
Note that, for the standard case , this estimator has been considered and studied in Chesneau .
Theorem 1 investigates the rate of convergence attained by under the MISE assuming that belongs to Besov balls.
Theorem 1. Suppose that (K1), (K2), and (K3) are satisfied and that with , , , and . Let be defined by (11) with such that
, and denotes the integer part of .
Then there exists a constant such that
The rate of convergence corresponds to the one obtained in the derivatives density estimation framework. See, for example, Prakasa Rao  and Chaubey et al. [17, 18]. For , Theorem 1 becomes [23, Theorem ], with .
In the rest of the study, the rate of convergence will be taken for benchmark. However, we do not claim that it is the optimal one in a minimax sense; the lower bounds are not determined. However, from some logical considerations, it is a serious candidate.
Hard Thresholding Wavelet Estimator. We define the hard thresholding wavelet estimator by , where is defined by (12), is the indicator function, is a large enough constant, is the integer satisfying and .
The construction of is an adaptation of the hard thresholding wavelet estimator introduced by Delyon and Juditsky  to the estimation of from (1). It used the modern version developed by Chaubey et al. . The advantage of over (11) is that is adaptive; thanks to the thresholding in (17), its performance does not depend on the knowledge of the smoothness of . The second thresholding in (17) enables us to relax some assumptions on the model, and, in particular, to only suppose on (its density can be unknown). Basics and important results on hard thresholding wavelet estimators can be found in, for example, Donoho and Johnstone [26, 27], Donoho et al. [28, 29], and Delyon and Juditsky .
Theorem 2 determines the rate of convergence attained by under the MISE assuming that belongs to Besov balls.
Theorem 2. Suppose that (K1), (K2), and (K3) are satisfied and that with , , or . Let be defined by (16). Then there exists a constant such that
The proof is based on a general result proved by [25, Theorem 6.1]. Let us observe that, for the case , is equal to the rate of convergence attained by up to a logarithmic factor (see Theorem 1). However, for the case , it is significantly better in terms of power.
3.3. Wavelet Estimators: When Is Unknown
In the case where is unknown, we propose the linear wavelet estimator defined by where , is an integer chosen a posteriori, refers to (K3), and is an estimator of constructed from the random variables . For instance, we can consider the linear wavelet estimator by where and is an integer chosen a posteriori.
The estimator is close to the “NES linear wavelet estimator” proposed by Pensky and Vidakovic  for . However, there are notable differences in the thresholding in (21), the partitioning of the variables, and the definition of , making the study of its performance under the MISE more simpler (see the proofs of Theorem 3 below).
Theorem 3 determines an upper bound of the MISE of and then exhibits its rate of convergence when belongs to Besov balls.
Theorem 3. Suppose that (K1), (K2), and (K3) are satisfied and that with , , , and . Let be defined by (20). Then there exists a constant such that
In addition, suppose that (K4) is satisfied, with , , , and ; consider with the estimator defined by (22) with such that and such that
Then there exists a constant such that
The first point of Theorem 3 is proved for any estimator of depending on . Taking , it corresponds to the upper bound of the MISE for established in the proof of Theorem 1. Note that the rate of convergence described in the second point is slower to the one attained by (see Theorem 1). The fact that the smoothness of influences the performance of and, a fortiori, seems natural. This phenomenon also appears in [16, Theorem 2.1], for .
Remark 4. If exists but is unknown, we can define as (20) with instead of in the threshold of (21). The impact of this modification is a logarithmic term in Theorem 3; that is, Moreover, choosing such that there exists a constant such that
Remark 5. Note that the assumption (K4) has been only used in the second point of Theorem 3.
Conclusion and Perspectives. We explore the estimation of from (1). Distinguishing the cases where is known or not, we propose wavelet methods and prove that they attain fast rates of convergence under the MISE assuming that .
Perspectives of this work are(i)to develop an adaptive wavelet estimator, as the hard thresholding one, for the estimation of in the case where is unknown;(ii)to relax assumptions on the model. Indeed, several techniques exist to relax (K3); that is, has potential zeros. See, for example, Kerkyacharian and Picard , Gaïffas , and Antoniadis et al. . However, their adaptations to the estimation of are more difficult than they appear at first glance;(iii)to consider dependent .
These aspects need further investigations that we leave for a future work.
In this section, denotes any constant that does not depend on , , and . Its value may change from one term to another and may depend on or .
Proof of Theorem 1. First of all, we expand the function on at the level given by (14):
where and .
Since forms an orthonormal basis of , we get Using the fact that is an unbiased estimator of (see (13)), are i.i.d., the inequalities: for any random variable , and , , and (K2) and (K3), we have Using , the change of variables , and the fact that is compactly supported, we obtain Therefore and, for satisfying (14), it holds that On the other hand, we have (see [11, Corollary 9.2]), which implies It follows from (32), (36), and (37) that Theorem 1 is proved.
Proof of Theorem 2. Observe that, for , any integer and any , (i)using arguments similar to (13), we obtain
(ii)using arguments similar to (33) and (34), we have
Applying [25, Theorem 6.1], (presented in Appendix) with , , , , and with , , either or , we prove the existence of a constant such that Theorem 2 is proved.
Proof of Theorem 3. As in the proof of Theorem 1, we first expand the function on at the level given by (26):
Since forms an orthonormal basis of , we get Using (see [11, Corollary 9.2]), we have Let be (12) with and (26). The elementary inequality , , yields where Proceeding as in (36), we get Let us now investigate the upper bound for .
The triangular inequality gives Moreover, we have It follows from the triangular inequality, the indicator function, (K3), , and the Markov inequality that Hence where Let us now consider . For any random variable , we have the equality where denotes the expectation of conditionally to and , the variance of conditionally to . Therefore where Let us now observe that, owing to the independence of , the random variables conditionally to are independent. This remark combines with the inequalities: for any random variable and , , the independence between and , (K2) and (K3), yields Thanks to the support compact of , we have . Therefore, using , On the other hand, by the Hölder inequality for conditional expectations and arguments similar to (33) and (34), we get Hence It follows from (55), (58), and (60) that Putting (46), (48), and (61) together, we get
Combining (44), (45), and (62), we obtain
A slight adaptation of , Proposition 1, gives the following result. Suppose that (K4) is satisfied and with , , , and . Let be defined by (22) with as (25). Then there exists a constant such that
Therefore, choosing as (26) and using (63), we have Theorem 3 is proved.
Appendixwhere , and is the integer satisfying Here, we suppose that there exist (i) functions with for any ,(ii)two sequences of real numbers and satisfying and Such that, for , (A1) any integer and any , (A2) there exist two constants, and , such that, for any integer and any ,
Let be (A.1) under (A1) and (A2). Suppose that with , or . Then there exists a constant such that
Conflict of Interests
The author declares that there is no conflict of interests regarding the publication of this paper.
The author is thankful to the reviewers for their comments which have helped in improving the presentation.
- T. Gasser and H.-G. Müller, “Estimating regression functions and their derivatives by the kernel method,” Scandinavian Journal of Statistics, Theory and Applications, vol. 11, no. 3, pp. 171–185, 1984.
- W. Härdle and T. Gasser, “On robust kernel estimation of derivatives of regression functions,” Scandinavian Journal of Statistics, vol. 12, no. 3, pp. 233–240, 1985.
- Y. P. Mack and H.-G. Müller, “Derivative estimation in nonparametric regression with random predictor variable,” Sankhya: The Indian Journal of Statistics, Series A, vol. 51, no. 1, pp. 59–72, 1989.
- D. Ruppert and M. P. Wand, “Multivariate locally weighted least squares regression,” The Annals of Statistics, vol. 22, no. 3, pp. 1346–1370, 1994.
- M. P. Wand and M. C. Jones, Kernel Smoothing, Chapman and Hall, London, UK, 1995.
- C. Stone, “Additive regression and other nonparametric models,” The Annals of Statistics, vol. 13, no. 2, pp. 689–705, 1985.
- G. Wahba and Y. H. Wang, “When is the optimal regularization parameter insensitive to the choice of the loss function?” Communications in Statistics: Theory and Methods, vol. 19, no. 5, pp. 1685–1700, 1990.
- S. Zhou and D. A. Wolfe, “On derivative estimation in spline regression,” Statistica Sinica, vol. 10, no. 1, pp. 93–108, 2000.
- R. Jarrow, D. Ruppert, and Y. Yu, “Estimating the interest rate term structure of corporate debt with a semiparametric penalized spline model,” Journal of the American Statistical Association, vol. 99, no. 465, pp. 57–66, 2004.
- A. Antoniadis, “Wavelets in statistics: a review,” Journal of the Italian Statistical Society Series B, vol. 6, no. 2, pp. 97–130, 1997.
- W. Härdle, G. Kerkyacharian, D. Picard, and A. Tsybakov, Wavelets, Approximation, and Statistical Applications, vol. 129 of Lecture Notes in Statistics, Springer, New York, NY, USA, 1998.
- B. Vidakovic, Statistical Modeling by Wavelets, John Wiley & Sons, New York, NY, USA, 1999.
- T. Cai, “On adaptive wavelet estimation of a derivative and other related linear inverse problems,” Journal of Statistical Planning and Inference, vol. 108, no. 1-2, pp. 329–349, 2002.
- A. Petsa and T. Sapatinas, “On the estimation of the function and its derivatives in nonparametric regression: a bayesian testimation approach,” Sankhya, Series A, vol. 73, no. 2, pp. 231–244, 2011.
- B. L. S. Prakasa Rao, “Nonparametric estimation of the derivatives of a density by the method of wavelets,” Bulletin of Informatics and Cybernetics, vol. 28, no. 1, pp. 91–100, 1996.
- M. Pensky and B. Vidakovic, “On non-equally spaced wavelet regression,” Annals of the Institute of Statistical Mathematics, vol. 53, no. 4, pp. 681–690, 2001.
- Y. P. Chaubey, H. Doosti, and B. L. S. P. Rao, “Wavelet based estimation of the derivatives of a density with associated variables,” International Journal of Pure and Applied Mathematics, vol. 27, no. 1, pp. 97–106, 2006.
- Y. P. Chaubey, H. Doosti, and B. L. S. P. Rao, “Wavelet based estimation of the derivatives of a density for a negatively associated process,” Journal of Statistical Theory and Practice, vol. 2, no. 3, pp. 453–463, 2008.
- A. Cohen, I. Daubechies, and P. Vial, “Wavelets on the interval and fast wavelet transforms,” Applied and Computational Harmonic Analysis, vol. 1, no. 1, pp. 54–81, 1993.
- I. Daubechies, Ten Lectures on Wavelets, Society for Industrial and Applied Mathematics, 1992.
- S. Mallat, A Wavelet Tour of Signal Processing, The Sparse Way, with Contributions from Gabriel Peyré, Elsevier, Academic Press, Amsterdam, The Netherlands, 3rd edition, 2009.
- Y. Meyer, Wavelets and Operators, Cambridge University Press, Cambridge, UK, 1992.
- C. Chesneau, “Regression with random design: a minimax study,” Statistics and Probability Letters, vol. 77, no. 1, pp. 40–53, 2007.
- B. Delyon and A. Juditsky, “On minimax wavelet estimators,” Applied and Computational Harmonic Analysis, vol. 3, no. 3, pp. 215–228, 1996.
- Y. P. Chaubey, C. Chesneau, and H. Doosti, “Adaptive wavelet estimation of a density from mixtures under multiplicative censoring,” http://hal.archives-ouvertes.fr/hal-00918069.
- D. L. Donoho and J. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika, vol. 81, no. 3, pp. 425–455, 1994.
- D. L. Donoho and I. M. Johnstone, “Adapting to unknown smoothness via wavelet shrinkage,” Journal of the American Statistical Association, vol. 90, no. 432, pp. 1200–1224, 1995.
- D. L. Donoho, I. M. Johnstone, G. Kerkyacharian, and D. Picard, “Wavelet shrinkage: asymptopia?” Journal of the Royal Statistical Society, Series B, vol. 57, no. 2, pp. 301–369, 1995.
- D. L. Donoho, I. M. Johnstone, G. Kerkyacharian, and D. Picard, “Density estimation by wavelet thresholding,” Annals of Statistics, vol. 24, no. 2, pp. 508–539, 1996.
- G. Kerkyacharian and D. Picard, “Regression in random design and warped wavelets,” Bernoulli, vol. 10, no. 6, pp. 1053–1105, 2004.
- S. Gaïffas, “Sharp estimation in sup norm with random design,” Statistics and Probability Letters, vol. 77, no. 8, pp. 782–794, 2007.
- A. Antoniadis, M. Pensky, and T. Sapatinas, “Nonparametric regression estimation based on spatially inhomogeneous data: minimax global convergence rates and adaptivity,” ESAIM: Probability and Statistics, vol. 18, pp. 1–41, 2014.
Copyright © 2014 Christophe Chesneau. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.