- About this Journal ·
- Abstracting and Indexing ·
- Advance Access ·
- Aims and Scope ·
- Article Processing Charges ·
- Articles in Press ·
- Author Guidelines ·
- Bibliographic Information ·
- Citations to this Journal ·
- Contact Information ·
- Editorial Board ·
- Editorial Workflow ·
- Free eTOC Alerts ·
- Publication Ethics ·
- Reviewers Acknowledgment ·
- Submit a Manuscript ·
- Subscription Information ·
- Table of Contents
Journal of Probability and Statistics
Volume 2012 (2012), Article ID 834107, 33 pages
General Bootstrap for Dual ϕ-Divergence Estimates
1Laboratoire de Mathématiques Appliquées, Université de Technologie de Compiègne, B.P. 529, 60205 Compiègne Cedex, France
2LSTA, Université Pierre et Marie Curie, 4 Place Jussieu, 75252 Paris Cedex 05, France
Received 30 May 2011; Revised 29 September 2011; Accepted 16 October 2011
Academic Editor: Rongling Wu
Copyright © 2012 Salim Bouzebda and Mohamed Cherfi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
A general notion of bootstrapped ϕ-divergence estimates constructed by exchangeably weighting sample is introduced. Asymptotic properties of these generalized bootstrapped ϕ-divergence estimates are obtained, by means of the empirical process theory, which are applied to construct the bootstrap confidence set with asymptotically correct coverage probability. Some of practical problems are discussed, including, in particular, the choice of escort parameter, and several examples of divergences are investigated. Simulation results are provided to illustrate the finite sample performance of the proposed estimators.
The -divergence modeling has proved to be a flexible tool and provided a powerful statistical modeling framework in a variety of applied and theoretical contexts (refer to [1–4] and the references therein). For good recent sources of references to the research literature in this area along with statistical applications, consult [2, 5]. Unfortunately, in general, the limiting distribution of the estimators, or their functionals, based on -divergences depends crucially on the unknown distribution, which is a serious problem in practice. To circumvent this matter, we will propose, in this work, a general bootstrap of -divergence-based estimators and study some of its properties by means of sophisticated empirical process techniques. A major application for an estimator is in the calculation of confidence intervals. By far the most favored confidence interval is the standard confidence interval based on a normal or a Student’s -distribution. Such standard intervals are useful tools, but they are based on an approximation that can be quite inaccurate in practice. Bootstrap procedures are an attractive alternative. One way to look at them is as procedures for handling data when one is not willing to make assumptions about the parameters of the populations from which one sampled. The most that one is willing to assume is that the data are a reasonable representation of the population from which they come. One then resamples from the data and draws inferences about the corresponding population and its parameters. The resulting confidence intervals have received the most theoretical study of any topic in the bootstrap analysis.
Our main findings, which are analogous to that of Cheng and Huang , are summarized as follows. The -divergence estimator and the bootstrap -divergence estimator are obtained by optimizing the objective function based on the independent and identically distributed (i.i.d) observations and the bootstrap sample , respectively, where are independent draws with replacement from the original sample. We will mention that can alternatively be expressed as where the bootstrap weights are given by
In this paper, we will consider the more general exchangeable bootstrap weighting scheme that includes Efron’s bootstrap [7, 8]. The general resampling scheme was first proposed in  and extensively studied by Bickel and Freedman , who suggested the name “weighted bootstrap”; for example, Bayesian Bootstrap when is equal in distribution to the vector of spacings of ordered uniform random variables, that is,
The interested reader may refer to . The case
was considered in [12, Remark 2.3] and [13, Remark 5]. The Bickel and Freedman result concerning the empirical process has been subsequently generalized for empirical processes based on observations in , as well as in very general sample spaces and for various set and function-indexed random objects (see, e.g., [14–18]). In this framework,  developed similar results for a variety of other statistical functions. This line of research was continued in the work of [20, 21]. There is a huge literature on the application of the bootstrap methodology to nonparametric kernel density and regression estimation, among other statistical procedures, and it is not the purpose of this paper to survey this extensive literature. This being said, it is worthwhile mentioning that the bootstrap as per Efron’s original formulation (see ) presents some drawbacks. Namely, some observations may be used more than once while others are not sampled at all. To overcome this difficulty, a more general formulation of the bootstrap has been devised: the weighted (or smooth) bootstrap, which has also been shown to be computationally more efficient in several applications. We may refer to [22–24]. Holmes and Reinert  provided new proofs for many known results about the convergence in law of the bootstrap distribution to the true distribution of smooth statistics employing the techniques based on Stein’s method for empirical processes. Note that other variations of Efron’s bootstrap are studied in  using the term “generalized bootstrap.” The practical usefulness of the more general scheme is well documented in the literature. For a survey of further results on weighted bootstrap, the reader is referred to .
The remainder of this paper is organized as follows. In the forthcoming section we recall the estimation procedure based on -divergences. The bootstrap of -divergence estimators is introduced, in detail, and their asymptotic properties are given in Section 3. In Section 4, we provide some examples explaining the computation of the -divergence estimators. In Section 5, we illustrate how to apply our results in the context of right censoring. Section 6 provides simulation results in order to illustrate the performance of the proposed estimators. To avoid interrupting the flow of the presentation, all mathematical developments are relegated to the appendix.
2. Dual Divergence-Based Estimates
The class of dual divergence estimators has been recently introduced by Keziou  and Broniatowski and Keziou . Recall that the -divergence between a bounded signed measure and a probability measure on , when is absolutely continuous with respect to , is defined by
where is a convex function from ][ to with . We will consider only -divergences for which the function is strictly convex and satisfies the domain of , is an interval with end points
The Kullback-Leibler, modified Kullback-Leibler, , modified , and Hellinger divergences are examples of -divergences; they are obtained, respectively, for , , , , and . The squared Le Cam distance (sometimes called the Vincze-Le Cam distance) and -error are obtained, respectively, for
We extend the definition of these divergences on the whole space of all bounded signed measures via the extension of the definition of the corresponding functions on the whole real space as follows: when is not well defined on or well defined but not convex on , we set for all . Notice that, for the -divergence, the corresponding function is defined on whole and strictly convex. All the above examples are particular cases of the so-called “power divergences,” introduced by Cressie and Read  (see also [4, Chapter 2], and also Rényi’s paper  is to be mentioned here), which are defined through the class of convex real-valued functions, for in ,, and . (For all , we define .) So, the -divergence is associated to , the to , the to , the to , and the Hellinger distance to . In the monograph by , the reader may find detailed ingredients of the modeling theory as well as surveys of the commonly used divergences.
Let be some identifiable parametric model with a compact subset of . Consider the problem of estimation of the unknown true value of the parameter on the basis of an i.i.d sample . We will assume that the observed data are from the probability space . Let be a function of class , strictly convex such that As it is mentioned in , if the function satisfies the following conditions: then the assumption (2.5) is satisfied whenever , where stands for the -divergence between and ; refer to [31, Lemma 3.2]. Also the real convex functions (2.4), associated with the class of power divergences, all satisfy the condition (2.5), including all standard divergences. Under assumption (2.5), using Fenchel duality technique, the divergence can be represented as resulting from an optimization procedure, this result was elegantly proved in [1, 3, 28]. Broniatowski and Keziou  called it the dual form of a divergence, due to its connection with convex analysis. According to , under the strict convexity and the differentiability of the function , it holds where the equality holds only for . Let and be fixed, and put and in (2.7), and then integrate with respect to , to obtain where and Furthermore, the supremum in this display (2.8) is unique and reached in , independently upon the value of . Naturally, a class of estimators of , called “dual -divergence estimators” (DDEs), is defined by where is the function defined in (2.9) and, for a measurable function ,
The class of estimators satisfies Formula (2.10) defines a family of -estimators indexed by the function specifying the divergence and by some instrumental value of the parameter . The -divergence estimators are motivated by the fact that a suitable choice of the divergence may lead to an estimate more robust than the maximum likelihood estimator (MLE) one; see . Toma and Broniatowski  studied the robustness of the DDEs through the influence function approach; they treated numerous examples of location-scale models and give sufficient conditions for the robustness of DDEs. We recall that the maximum likelihood estimate belongs to the class of estimates (2.10). Indeed, it is obtained when , that is, as the dual modified -divergence estimate. Observe that and , and hence
Keeping in mind definitions (2.10), we get
independently upon .
3. Asymptotic Properties
In this section, we will establish the consistency of bootstrapping under general conditions in the framework of dual divergence estimation. Define, for a measurable function ,
where ’s are the bootstrap weights defined on the probability space . In view of (2.10), the bootstrap estimator can be rewritten as The definition of , defined in (3.2), implies that The bootstrap weights ’s are assumed to belong to the class of exchangeable bootstrap weights introduced in . In the sequel, the transpose of a vector will be denoted by . We will assume the following conditions.(W.1)The vector is exchangeable for all ; that is, for any permutation of , the joint distribution of is the same as that of .(W.2) for all , and for all .(W.3), where(W.4) One has (W.5).
In Efron’s nonparametric bootstrap, the bootstrap sample is drawn from the nonparametric estimate of the true distribution, that is, empirical distribution. Thus, it is easy to show that and conditions (W.1)–(W.5) are satisfied. In general, conditions (W.3)–(W.5) are easily satisfied under some moment conditions on ; see [23, Lemma 3.1]. In addition to Efron’s nonparametric bootstrap, the sampling schemes that satisfy conditions (W.1)–(W.5) include Bayesian bootstrap, Multiplier bootstrap, Double bootstrap, and Urn bootstrap. This list is sufficiently long to indicate that conditions (W.1)–(W.5), are not unduly restrictive. Notice that the value of in (W.5) is independent of and depends on the resampling method, for example, for the nonparametric bootstrap and Bayesian bootstrap and for the double bootstrap. A more precise discussion of this general formulation of the bootstrap can be found in [23, 34, 35].
There exist two sources of randomness for the bootstrapped quantity, that is, : the first comes from the observed data and the second is due to the resampling done by the bootstrap, that is, random ’s. Therefore, in order to rigorously state our main theoretical results for the general bootstrap of -divergence estimates, we need to specify relevant probability spaces and define stochastic orders with respect to relevant probability measures. Following [6, 36], we will view as the th coordinate projection from the canonical probability space onto the th copy of . For the joint randomness involved, the product probability space is defined as
Throughout the paper, we assume that the bootstrap weights ’s are independent of the data ’s, thus Given a real-valued function defined on the above product probability space, for example, , we say that is of an order in -probability if, for any , as ,
and that is of an order in -probability if, for any , there exists a such that, as ,
where the superscript “” denotes the outer probability; see  for more details on outer probability measures. For more details on stochastic orders, the interested reader may refer to , in particular, Lemma 3 of the cited reference.
To establish the consistency of , the following conditions are assumed in our analysis.(A.1)One has for any open set containing .(A.2)One has The following theorem gives the consistency of the bootstrapped estimate.
Theorem 3.1. Assume that conditions (A.1) and (A.2) hold. Suppose that conditions (A.3)–(A.5) and (W.1)–(W.5) hold. Then is a consistent estimate of ; that is,
The proof of Theorem 3.1 is postponed until the appendix.
then we say that is a -Glivenko-Cantelli class of functions. If is a class of functions for which
where is a mean-zero -Brownian bridge process with (uniformly) continuous sample paths with respect to the semimetric , defined by
then we say that is a -Donsker class of functions. Here
and is a -Brownian bridge process on if it is a mean-zero Gaussian process with covariance function
Remark 3.2. (i) Condition (A.1) is the “well-separated” condition, compactness of the parameter space and the continuity of divergence imply that the optimum is well separated, provided the parametric model is identified; see [37, Theorem 5.7].
(ii) Condition (A.2) holds if the class is shown to be -Glivenko-Cantelli, by applying [34, Lemma ] and [6, Lemma ].
For any fixed , define the class of functions and as We will say a class of functions if possesses enough measurability for randomization with i.i.d multipliers to be possible, that is, can be randomized, in other words, we can replace by . It is known that , for example, if is countable, if are stochastically separable in , or if is image admissible Suslin; see [21, pages 853 and 854].
To state our result concerning the asymptotic normality, we will assume the following additional conditions.(A.3) The matrices are nonsingular.(A.4) The class and is -Donsker.(A.5) The class and is -Donsker.
Conditions (A.4) and (A.5) ensure that the “size” of the function classes and are reasonable so that the bootstrapped empirical processes
indexed, respectively, by and , have a limiting process conditional on the original observations; we refer, for instance, to [23, Theorem 2.2]. The main result to be proved here may now be stated precisely as follows.
Theorem 3.3. Assume that and fulfil (2.12) and (3.3), respectively. In addition suppose that Assume that conditions (A.3)–(A.5) and (W.1)–(W.5) hold. Then one has in -probability. Furthermore, in -probability. Consequently, where “≤” is taken componentwise and “” is given in (W.5), whose value depends on the used sampling scheme, and where and are given in condition (A.3). Thus, one has
The proof of Theorem 3.1 is captured in the forthcoming appendix.
Remark 3.4. Note that an appropriate choice of the bootstrap weights ’s implicates a smaller limit variance; that is, is smaller than 1. For instance, typical examples are i.i.d-weighted bootstraps and the multivariate hypergeometric bootstrap; refer to [23, Examples 3.1 and 3.4].
Following , we will illustrate how to apply our results to construct the confidence sets. A lower th quantile of bootstrap distribution is defined to be any fulfilling
where is an infimum over the given set only if there does not exist a in such that
Keep in mind the assumed regularity conditions on the criterion function, that is, in the present framework, we can, without loss of generality, suppose that
Making use of the distribution consistency result given in (3.27), we can approximate the th quantile of the distribution of
Therefore, we define the percentile-type bootstrap confidence set as In a similar manner, the th quantile of can be approximated by , where is the th quantile of the hybrid quantity , that is,
Thus, the hybrid-type bootstrap confidence set would be defined as follows: Note that and are not unique by the fact that we assume is a vector. Recall that, for any ,
According to the quantile convergence theorem, that is, [37, Lemma 21.1], we have, almost surely,
When applying quantile convergence theorem, we use the almost sure representation, that is, [37, Theorem 2.19], and argue along subsequences. Considering Slutsky’s Theorem which ensures that
we further have
The above arguments prove the consistency of the hybrid-type bootstrap confidence set, that is, (3.42), and can also be applied to the percentile-type bootstrap confidence set, that is, (3.41). For an in-depth study and more rigorous proof, we may refer to [37, Lemma 23.3]. The above discussion may be summarized as follows.
Corollary 3.5. Under the conditions in Theorem 3.3, one has, as ,
It is well known that the above bootstrap confidence sets can be obtained easily through routine bootstrap sampling.
Remark 3.6. Notice that the choice of weights depends on the problem at hand: accuracy of the estimation of the entire distribution of the statistic, accuracy of a confidence interval, accuracy in large deviation sense, and accuracy for a finite sample size; we may refer to  and the references therein for more details. Barbe and Bertail  indicate that the area where the weighted bootstrap clearly performs better than the classical bootstrap is in term of coverage accuracy.
3.1. On the Choice of the Escort Parameter
The very peculiar choice of the escort parameter defined through has the same limit properties as the MLE one. The DDE , in this case, has variance which indeed coincides with the MLE one; see, for instance, [28, Theorem 2.2, (1) (b)]. This result is of some relevance, since it leaves open the choice of the divergence, while keeping good asymptotic properties. For data generated from the distribution , Figure 1 shows that the global maximum of the empirical criterion is zero, independently of the value of the escort parameter (the sample mean , in Figure 1(a) and the median in Figure 1(b)) for all the considered divergences which is in agreement with the result of [39, Theorem 6], where it is showed that all differentiable divergences produce the same estimator of the parameter on any regular exponential family, in particular the normal models, which is the MLE one, provided that the conditions (2.6) and are satisfied.
Unlike the case of data without contamination, the choice of the escort parameter is crucial in the estimation method in the presence of outliers. We plot in Figure 2 the empirical criterion , where the data are generated from the distribution
where , , and stands for the Dirac measure at . Under contamination, when we take the empirical “mean,” , as the value of the escort parameter , Figure 2(a) shows how the global maximum of the empirical criterion shifts from zero to the contamination point. In Figure 2(b), the choice of the “median” as escort parameter value leads to the position of the global maximum remaining close to , for Hellinger , , and -divergence , while the criterion associated to the -divergence is still affected by the presence of outliers.
In practice, the consequence is that if the data are subject to contamination the escort parameter should be chosen as a robust estimator of , say . For more details about the performances of dual -divergence estimators for normal density models, we refer to .
Keep in mind the definitions (2.8) and (2.9). In what follows, for easy reference and completeness, we give some usual examples of divergences, discussed in [41, 42], of divergences and the associated estimates; we may refer also to  for more examples and details.(i)Our first example is the Kullback-Leibler divergence: The estimate of is given by and the estimate of the parameter , with escort parameter , is defined as follows: (ii)The second one is the -divergence: The estimate of is given by and the estimate of the parameter , with escort parameter , is defined by (iii)Another example is the Hellinger divergence: The estimate of is given by and the estimate of the parameter , with escort parameter , is defined by (iv)All the above examples are particular cases of the so-called “power divergences,” which are defined through the class of convex real-valued functions, for in , The estimate of is given by and the parameter estimate is defined by
Remark 4.1. The computation of the estimate requires calculus of the integral in the formula (2.9). This integral can be explicitly calculated for the most standard parametric models. Below, we give a closed-form expression for Normal, log-Normal, Exponential, Gamma, Weibull, and Pareto density models. Hence, the computation of can be performed by any standard nonlinear optimization code. Unfortunately, the explicit formula of , generally, cannot be derived, which also is the case for the ML method. In practical problems, to obtain the estimate , one can use the Newton-Raphson algorithm taking as initial point the escort parameter . This algorithm is a powerful technique for solving equations numerically, performs well since the objective functions are concave and the estimated parameter is unique for functions ; for instance, refer to [1, Remark 3.5].
4.1. Example of Normal Density
Consider the case of power divergences and the Normal model
Simple calculus gives, for in ,
In the particular case, , it follows that, for ,
which leads to the maximum likelihood estimate independently upon .
4.2. Example of Log-Normal Density
Consider the case of power divergences and the log-Normal model
Simple calculus gives, for in ,
4.3. Example of Exponential Density
Consider the case of power divergences and the Exponential model
We have, for in , Then, using this last equality, one finds
In more general case, we may consider the Gamma density combined with the power divergence. The Gamma model is defined by
where is the Gamma function
Simple calculus gives, for in ,
which implies that
4.4. Example of Weibull Density
Consider the case of power divergences and the Weibull density model, with the assumption that is known and is the parameter of interest to be estimated, and recall that
Routine algebra gives, for in ,
which implies that
4.5. Example of the Pareto Density
Consider the case of power divergences and the Pareto density
Simple calculus gives, for in , As before, using this last equality, one finds
which leads to the maximum likelihood estimate, given by
independently upon .
Remark 4.2. The choice of divergence, that is, the statistical criterion, depends crucially on the problem at hand. For example, the -divergence among various divergences in the nonstandard problem (e.g., boundary problem estimation) is more appropriate. The idea is to include the parameter domain into an enlarged space, say , in order to render the boundary value an interior point of the new parameter space, . Indeed, Kullback-Leibler, modified Kullback-Leibler, modified , and Hellinger divergences are infinite when takes negative values on nonnegligible (with respect to ) subset of the support of , since the corresponding is infinite on , when belongs to . This problem does not hold in the case of -divergence, in fact, the corresponding is finite on ; for more details refer to [41, 42, 44], and consult also [1, 45] for related matter. It is well known that when the underlying model is misspecified or when the data are contaminated the maximum likelihood or other classical parametric methods may be severely affected and lead to very poor results. Therefore, robust methods, which automatically circumvent the contamination effects and model misspecification, can be used to provide a compromise between efficient classical parametric methods and the semiparametric approach provided they are reasonably efficient at the model; this problem has been investigated in [46, 47]. In [41, 42], simulation results show that the choice of -divergence has good properties in terms of efficiency robustness. We mention that some progress has been made on automatic data-based selection of the tuning parameter , appearing in formula (1) of ; the interested reader is referred to [48, 49]. It is mentioned in , where semiparametric minimum distance estimators are considered, that the MLE or inversion-type estimators involve solving a nonlinear equation which depends on some initial value. The second difficulty is that the objective function is not convex in , in general, which give the situation of multiple roots. Thus, in general, “good” consistent initial estimates are necessary and the DDE should serve that purpose.
5. Random Right Censoring
Let be i.i.d survival times with continuous survival function , and let be independent censoring times with d.f. . In the censoring setup, we observe only the pair and , where is the indicator function of the event , which designs whether an observation has been censored or not. Let denote the observed data points, and let
be the distinct death times. Now define the death set and risk set as follows: for ,
Kaplan and Meier’s  estimator of , denoted here by , may be written as follows:
One may define a generally exchangeable weighted bootstrap scheme for the Kaplan-Meier estimator and related functionals as follows (cf. [38, page 1598]):
Let be -integrable, and put
Note that we have used the following identity. Let , , , , be real numbers
In the similar way, we define a more appropriate representation, which will be used in the sequel, as follows:
where, for ,
Here, are ordered -values and denotes the concomitant associated with . Hence, we may write For the right censoring situation, the bootstrap DDEs, is defined by replacing in (2.10) by , that is, The corresponding estimating equation for the unknown parameter is then given by where we recall that
and, for ,
Consider the lifetime distribution to be the one-parameter exponential with density , . Following , the Kaplan-Meier integral may be written as
Observe that this divergence leads to the AMLE, independently upon the value of .
For more details about dual -divergence estimators in right censoring, we refer to ; we leave this study open for future research. We mention that the bootstrapped estimators, in this framework, are obtained by replacing the weights by in the preceding formulas.
In this section, series of experiments were conducted in order to examine the performance of the proposed random weighted bootstrap procedure of the DDEs, defined in (3.2). We provide numerical illustrations regarding the mean-squared error (MSE) and the coverage probabilities. The computing program codes were implemented in R.
The values of are chosen to be , which corresponds, as indicated above, to the well-known standard divergences: -divergence, , the Hellinger distance, , and the -divergence, respectively. The samples of sizes considered in our simulations are 25, 50, 75, 100, 150, and 200, and the estimates, DDEs , are obtained from 500 independent runs. The value of escort parameter is taken to be the MLE, which, under the model, is a consistent estimate of , and the limit distribution of the DDE , in this case, has variance which indeed coincides with the MLE; for more details on this subject, we refer to [28, Theorem 2.2, (1) (b)], as it is mentioned in Section 3.1. The bootstrap weights are chosen to be
In Figure 3, we plot the densities of the different estimates, it shows that the proposed estimators perform reasonably well.
Tables 3 and 4 provide the MSE of various estimates under the Exponential model . As expected, the MLE produces most efficient estimators. A close look at the results of the simulations show that the DDEs perform well under the model. For large sample size , the estimator based on the Hellinger distance is equivalent to that of the MLE. Indeed in terms of empirical MSE, the DDE with produces the same MSE as the MLE, while the performance of the other estimators is comparable.
Tables 5, 6, 7, and 8, provide the empirical coverage probabilities of the corresponding 0.95 weighted bootstrap confidence intervals based on weighted bootstrap estimators. Notice that the empirical coverage probabilities as in any other inferential context, the greater the sample size, the better. From the results reported in these tables, we find that for large values of the sample size , the empirical coverage probabilities are all close to the nominal level. One can see that the DDE with has the best empirical coverage probability which is near the assigned nominal level.
6.1. Right Censoring Case
This subsection presents some simulations for right censoring case discussed in Section 5. A sample is generated from and an exponential censoring scheme is used; the censoring distribution is taken to be that the proportion of censoring is . To study the robustness properties of our estimators, of the observations are contaminated by . The DDE’s are calculated for samples of sizes 25, 50, 100, and 150, and the hole procedure is repeated 500 times. We can see from Table 9 that the DDEs perform well under the model in terms of MSE and are an attractive alternative to the AMLE.
Table 10 shows the variation in coverage of nominal asymptotic confidence intervals according to the sample size. Clearly, under coverage of the confidence intervals, the DDEs have poor coverage probabilities due to the censoring effect. However, for small- and moderate-sized samples, the DDEs associated to outperform the AMLE.
Remark 6.1. In order to extract methodological recommendations for the use of an appropriate divergence, it will be interesting to conduct extensive Monte Carlo experiments for several divergences or investigate theoretically the problem of the choice of the divergence which leads to an “optimal” (in some sense) estimate in terms of efficiency and robustness, which would go well beyond the scope of the present paper. Another challenging task is how to choose the bootstrap weights for a given divergence in order to obtain, for example, an efficient estimator.
This section is devoted to the proofs of our results. The previously defined notation continues to be used in the following.
Proof of Theorem 3.3. Keep in mind the following definitions: In view of the fact that , then a little calculation shows that Consequently, we have the following inequality: According to [23, Theorem 2.2] under condition (A.4), we have in -probability. In view of the CLT, we have . By applying a Taylor series expansion, we have where is between and . By condition (A.5) and [23, Theorem 2.2], we conclude that the right term in (A.4) is of order in -probability. By the fact that is assumed to be consistent, we have in -probability. An analogous argument yields that is of order , by the consistency of , we have in -probability. Finally, based on (3.3). In summary, (A.3) can be rewritten as follows: in -probability. On the other hand, by a Taylor series expansion, we can write Clearly it is straightforward to combine (A.7) with (A.6), to infer the following: in -probability, by considering again the consistency of and condition (A.3) and making use of (A.8) to complete the proof of (3.23). We next prove (3.24). Introduce By some algebra, we obtain Obviously, in -probability and . We also know that the order of is