Robust Wild Bootstrap for Stabilizing the Variance of Parameter Estimates in Heteroscedastic Regression Models in the Presence of Outliers

Abstract

Nowadays bootstrap techniques are used for data analysis in many other fields like engineering, physics, meteorology, medicine, biology, and chemistry. In this paper, the robustness of Wu (1986) and Liu (1988)'s Wild Bootstrap techniques is examined. The empirical evidences indicate that these techniques yield efficient estimates in the presence of heteroscedasticity problem. However, in the presence of outliers, these estimates are no longer efficient. To remedy this problem, we propose a Robust Wild Bootstrap for stabilizing the variance of the regression estimates where heteroscedasticity and outliers occur at the same time. The proposed method is based on the weighted residuals which incorporate the MM estimator, robust location and scale, and the bootstrap sampling scheme of Wu (1986) and Liu (1988). The results of this study show that the proposed method outperforms the existing ones in every respect.

1. Introduction

Bootstrap technique was first proposed by Efron [1]. It is a computer intensive method that can replace theoretical formulation with extensive use of computer. The attractive feature of the bootstrap technique is that it does not rely on the normality or any other distributional assumptions and is able to estimate standard error of any complicated estimator without any theoretical calculations. These interesting properties of the bootstrap method have to be traded off with computational cost and time. There are considerable papers that deal with bootstrap methods in the literatures (see [2–5]). The classical bootstrap methods are known to be a good general procedure for estimating a sampling distribution under the independent and identically distributed (i.i.d.) models. Let us consider a standard linear regression model:𝑌=𝑋𝛽+𝜀, (1.1) where 𝑌=(𝑦₁,𝑦₂,…,𝑦_𝑛)^𝑇,𝑋=(𝑥₁,𝑥₂,…,𝑥_𝑛)^𝑇, and 𝜀=(𝜀₁,𝜀₂,…,𝜀_𝑛)^𝑇. In this equation 𝛽 is a 𝑘×1 vector of unknown parameters, Y is an 𝑛×1 vector, X is an 𝑛×𝑘 data matrix of full rank 𝑘≤𝑛, and 𝜀 is an 𝑛×1 vector of unobservable random errors with E (𝜀) = 0 and 𝑉(𝜀)=𝜎²𝐼. In practice the i.i.d. set-up is often violated, as, for example, the homoscedastic assumption of Var(𝜀_𝑖)=𝜎²𝐼 is often violated. Wu [6] proposed a weighted bootstrap technique which gives better performance under both the homoscedastic and heteroscedastic models. However, a better alternative approximation is developed by Liu [7] following the suggestions of Liu [7] and Beran [8]. This type of weighted bootstraps is called the wild bootstrap in the literature. Several attempts have been made to use the Wu and Liu wild bootstrap techniques to remedy the problem of heteroscedasticity (see [6, 7, 9, 10]).

Salibian-Barrera and Zamar [11] pointed out that the problem of classical bootstrap is that the proportion of outliers in the bootstrap sample might be greater than that of the original data. Hence, the entire inferential procedure of bootstrap would be erroneous in the presence of outliers. As an alternative, robust bootstrap technique has been drawn a greater attention to the statisticians (see [11–15]). However, not much work is devoted to bootstrap technique when both outliers and heteroscedasticity are present in a data. Those wild bootstrap techniques can only rectify the problem of heteroscedasticity and not resistant to outliers. Moreover, these procedures are based on the OLS estimate which is very sensitive to outliers. We introduce the classical wild bootstrap in Section 2. In Section 3, we discuss the newly proposed robust wild bootstrap methods. A numerical example and a simulation study are presented in Sections 4 and 5, respectively. The conclusion of the study is given in Section 6.

2. Wild Bootstrap Techniques

In regression analysis, the most popular and widely used bootstrap technique is the fixed-x resampling or bootstrapping the residuals [2]. This bootstrapping procedure is based on the ordinary least squares (OLS) residuals summarized as follows.

Step 1. Fit a model 𝑦_𝑖=𝑓(𝑥_𝑖,𝛽_ols) by the OLS method to the original sample of observations to get 𝛽_ols and hence the fitted model is 𝑦_𝑖=𝑓(𝑥_𝑖,𝛽_ols).

Step 2. Compute the OLS residuals 𝜀_𝑖̂= 𝑦_𝑖−𝑦_𝑖 and each residual 𝜀_𝑖̂ has equal probability, 1/𝑛.

Step 3. Draw a random sample 𝜀^∗₁,𝜀^∗₂,...,𝜀^∗_𝑛 from 𝜀_𝑖̂ with simple random sampling with replacement and attached to 𝑦_𝑖 for obtaining fixed-𝑥 bootstrap values 𝑦_𝑖^∗𝑏 where𝑦_𝑖^∗𝑏=𝑓(𝑥_𝑖,𝛽_ols)+𝜀_𝑖^∗𝑏.

Step 4. Fit the OLS to the bootstrapped values 𝑦_𝑖^∗𝑏 on the fixed-𝑥 to obtain 𝛽^∗𝑏_ols.

Step 5. Repeat Steps 3 and 4 for B times to get 𝛽^∗𝑏1_ols,...,𝛽^∗𝑏𝐵_ols where B is the bootstrap replications.

We call this bootstrap scheme Boot_ols since it is based on the OLS method.

When heteroscedasticity is present in the data, the variances of the data are different and neither of these bootstrap schemes can yield efficient estimates of the parameters. Wu [6] showed that they are inconsistent and asymptotically biased under the heteroscedasticity. Wu [6] proposed a wild bootstrap (weighted bootstrap) that can be used to obtain the standard error which is asymptotically correct under heteroscedasticity of unknown form. Wu slightly modified Step 3 of the OLS bootstrap and kept the other steps unchanged. For each i, draw a value 𝑡^∗_𝑖, with replacement, from a distribution with zero mean and unit variance and attached to 𝑦_𝑖 for obtaining fixed-𝑥 bootstrap values 𝑦_𝑖^∗𝑏, where 𝑦_𝑖^∗𝑏=𝑓(𝑥_𝑖,𝛽_ols)+𝑡^∗_𝑖𝜀_𝑖̂/1−ℎ_𝑖𝑖 and ℎ_ii=𝑥^𝑇_𝑖(𝑋^𝑇𝑋)𝑥_𝑖 is the 𝑖th leverage. Note that the variance of 𝑡^∗_𝑖𝜀_𝑖̂ is not constant when the original errors are not homoscedastic. Therefore, this bootstrap scheme takes into consideration the nonconstancy of the error variances. As an alternative [6], 𝑡^∗_𝑖 can be chosen, with replacement, from 𝑎₁,𝑎₂,...,𝑎_𝑛, where𝑎_𝑖=𝜀_𝑖̂−𝜀_𝑖̂𝑛⁻¹∑^𝑛_𝑖=1𝜀_𝑖̂−𝜀̂² (2.1) with 𝜀̂=𝑛⁻¹∑^𝑛_𝑖=1𝜀_𝑖̂. For a regression model with intercept term, 𝜀_𝑖̂ approximately equals zero. This is nonparametric implementation of Wu’s bootstrap since the resampling is done from the empirical distribution function of the (normalized) residuals. We call this method Wu’s bootstrap and denote it by Boot_wu.

Following the idea of Wu [6], another wild bootstrap technique was proposed by Liu [7] in which 𝑡^∗_𝑖 is randomly selected from a population that has third central moment equal to one with zero mean and unit variance. Such kind of selection is used to correct the skewness term in the Edgeworth expansion of the sampling distribution of 𝐼^𝑇𝛽, where I is an n-vector of ones. Liu’s bootstrap can be conducted by drawing random numbers 𝑡^∗_𝑖 in the following two ways.(1)𝑡^∗_𝑖=𝑍_𝑖−𝐸(𝑍_𝑖),𝑖=1,2,...,𝑛, and 𝑍₁,𝑍₂,...,𝑍_𝑛 are independently and identically distributed having density 𝑔_𝑍(𝑥)=[𝛼^𝛽/(𝛽−1)!]𝑥^𝛽−1𝑒^−𝑎𝑥𝐼_(𝑥>0), where 𝛼=2 and 𝛽=4.(2)𝑡^∗_𝑖=𝐻_𝑖𝐷_𝑖−𝐸(𝐻_𝑖)𝐸(𝐷_𝑖),𝑖=1,2,...,𝑛, where 𝐻₁,𝐻₂,...,𝐻_𝑛 are independently and identically distributed normal distribution with mean (1/2)(17/6)+1/6 and variance 1/2. 𝐷₁,𝐷₂,...,𝐷_𝑛 are also independently and identically distributed normal distribution with mean (1/2)(17/6)−1/6 and variance 1/2. 𝐻_𝑖’s and 𝐷_𝑖’s are independent.

It is worth mentioning that selecting random numbers 𝑡^∗_𝑖 by procedure 1 or procedure 2 of Liu [7] will produce third central moment equal to one. Following Cribari-Neto and Zarkos [16], we consider the second procedure of drawing the random sample 𝑡^∗_𝑖. We call this bootstrap scheme as Boot_liu.

3. Proposed Robust Wild Bootstrap Techniques

We have discussed the classical wild bootstrap procedures which are based on the OLS residuals. It is now evident that the OLS suffers a huge setback in the presence of outliers since it has 0% breakdown [17]. Since the wild bootstrap samples are based on the OLS residuals, it is not resistant to outliers. Hence, in this article we propose to use the high-breakdown and high-efficiency robust MM estimator [18] to obtain the robust residuals. It is expected that for good data point, the residuals of the MM estimator are approximately the same as the OLS residuals. On the other hand, the residuals of the MM estimator would be larger for outlier observation. We assign weights to the MM residuals. The standardized residuals |𝜀^MM_𝑖̂|/𝜎_MM are computed, where 𝜎_MM is the square root of the mean squares error of the residuals of the MM estimates (see [19]). Following the idea of Furno [20], weights equal to one and 𝑐/(|𝜀^MM_𝑖̂|/𝜎_MM) are assigned to |𝜀^MM_𝑖̂|/𝜎_MM≤𝑐 and |𝜀^MM_𝑖̂|/𝜎_MM>𝑐, respectively, where 𝑐 is an arbitrary constant which is chosen between 2 and 3. We multiply the new weights with the residuals of the MM estimates and the resultants are denoted by 𝜀^WMM_𝑖̂. It is now expected that not only the residuals corresponding to the good data points but also the residuals corresponding to the bad data point of the MM residuals tend to be similar to the OLS residuals with no outliers. Based on the new weighted residuals 𝜀^WMM_𝑖̂, we propose to robustify Boot_ols, Boot_wu, and Boot_liu. We call the resulting robust bootstraps RBoot_ols, RBoot_wu, and RBoot_liu.

We propose to replace the OLS residuals by 𝜀^WMM_𝑖̂ in Step 3 of the Boot_ols. That is, the bootstrap sample 𝜀^∗₁,𝜀^∗₂,...,𝜀^∗_𝑛 is drawn from 𝜀^WMM_𝑖̂ with simple random sampling and the other steps remain unchanged. We call this bootstrap scheme Rboot_ols. Now we will discuss the formulation of robust wild bootstrap based on Wu’s procedures. The algorithm is summarized as follows.

Step 1. Fit a model 𝑦_𝑖=𝑥_𝑖𝛽+𝜀_𝑖 by the MM estimator to the original sample of observations to get the robust parameters 𝛽_MM and hence the fitted model is 𝑦_𝑖=𝑥_𝑖𝛽_MM.

Step 2. Compute the residuals of the MM estimate 𝜀^MM_𝑖̂=𝑦_𝑖−𝑦_𝑖. Then assign weight to each residual, 𝜀^MM_𝑖̂, such that the weight equals 1 if |𝜀^MM_𝑖̂|/𝜎_MM≤𝑐 and equals 𝑐/(|𝜀^MM_𝑖̂|/𝜎_MM) if |𝜀^MM_𝑖̂|/𝜎_MM>𝑐.

Step 3. The final weighted residuals of the MM estimates denoted by 𝜀^WMM_𝑖̂ are formulated by multiplying the weights obtained in Step 2 with the residuals of the MM estimates. That is, 𝜀^WMM_𝑖̂=1×𝜀^MM_𝑖̂ if the observation corresponds to good data point (no outliers) and 𝜀^WMM_𝑖̂=𝑐/(|𝜀^MM_𝑖̂|/𝜎_𝑀𝑀)×𝜀^MM_𝑖̂ if the observation corresponds to outliers.

Step 4. Construct a bootstrap sample (𝑦^∗_𝑖,𝑋), where 𝑦^∗_𝑖=𝑥_𝑖𝛽_MM+𝑡^∗_𝑖𝜀^WMM_𝑖̂1−ℎ_𝑖𝑖, (3.1) and 𝑡^∗ is a random sample following Wu [6] procedure.

Step 5. The OLS procedure is then applied to the bootstrap sample(𝑦^∗_𝑖,𝑋), and the resultant estimate is denoted by ^𝑅𝛽^∗=(𝑋^𝑇𝑋)⁻¹𝑋^𝑇𝑦^∗. Here, the robust estimates are very reliable since the bootstrap sample is constructed based on the robust weighted residuals, 𝜀^WMM_𝑖̂.

Step 6. Repeat Steps 4 and 5 for B times, where B is the bootstrap replications.

As discussed earlier, in the classical scheme of Wu’s bootstrap, the quantity 𝑡^∗_𝑖 is drawn from a population that has mean zero and variance equal to one or, 𝑡^∗_𝑖 can be drawn from normalized residuals𝑎₁,𝑎₂,...,𝑎_𝑛, that is, 𝑎_𝑖=𝜀_𝑖̂−𝜀_𝑖̂𝑛⁻¹∑^𝑛_𝑖=1𝜀_𝑖̂−𝜀̂². (3.2) However, following Maronna et al. [21], we suggest computing the robust normalized residuals based on median and normalized median absolute deviations (NMADs) instead of mean and standard deviation which are not robust. Thus,^𝑅𝑎_𝑖=𝜀^WMM_𝑖̂−median𝜀^WMM_𝑖̂NMAD_norm𝜀^WMM_𝑖̂, (3.3) where NMAD_norm=median{|𝜀^WMM_𝑖̂−median(𝜀^WMM_𝑖̂)|}/0.6745. We call this proposed robust nonparametric bootstrap as RBoot_wu.

In this paper we also want to robustify the wild bootstrap based on the Liu [7] algorithm. It is important to note that the only difference between the Wu and Liu implementation of wild bootstrap is the choice of the random sample 𝑡^∗_𝑖. In the proposed robust bootstrap based on the Liu wild bootstrap, we choose the random sample 𝑡^∗_𝑖 exactly the same manner as the classical Liu bootstrap. We call this bootstrap scheme as RBoot_liu.

4. Numerical Example

In this section, a numerical example is presented to assess the performance of the robust wild bootstrap methods. In order to compare the robustness of the classical and robust wild bootstrap in the presence of outliers, the Concrete Compressive Strength data is taken from Yeh [22]. Concrete is the most important material in civil engineering. The concrete compressive strength is a function of the eight output such as cement (Kg/m³), blast furnace slag (Kg/m³), fly ash (Kg/m³), water (Kg/m³), superplasticizer (Kg/m³), coarse aggregate (Kg/m³), fine aggregate (Kg/m³), and age of testing (days). The residuals versus fitted values are plotted in Figure 1 that show a funnel shape suggesting a heterogeneous error variances for the data (see [19]).

Figure 1 

Residuals versus Fitted values plot of Concrete Compressive Strength data.

We checked whether this data set contain any outliers or not by using Least trimmed of Squares (LTSs) residuals. It is found that 61 observations (about 6% of the sample of size 1030) appear to be outliers. The robust and non-robust (Classical) wild bootstrap methods were then applied to the data by considering two types of situations, namely, the data with outliers and data without outliers (omitted the outlying data points). The results are based on 500 bootstraps and are given in Table 1.

Table 1 

Wild bootstrap standard errors of the parameters for the Concrete Compressive Strength data.

The standard errors of the parameter estimates from robust and nonrobust wild bootstrap methods are exhibited in Table 1. The average standard errors of the parameter estimates are also shown. When there are no outliers, the standard errors of the classical wild bootstrap are reasonably closed to the standard errors of the robust wild bootstrap. It is interesting to note that the classical wild bootstrap methods provide larger standard errors compared to the wild bootstrap methods when outliers are present in the data.

We cannot make a final conclusion yet, just by observing the results of the real data, but a reasonable interpretation up to this stage is that the classical wild bootstrap is affected by outliers.

5. Simulation Study

In this section, the performances of the proposed robust wild bootstrap estimators are evaluated based on a simulation study. At first we generate some artificial data to see the performance of proposed bootstrap techniques. The final investigation of the performance of the proposed estimators is verified by the simulation approach on bootstrap samples.

5.1. Artificial Data

We follow the data generation technique of Cribari-Neto and Zarkos [16] and MacKinnon and White [23]. The design of this experiment involves a linear model with two covariates:𝑦_𝑖=𝛽₀+𝛽₁𝑥_1𝑖+𝛽₂𝑥_2𝑖+𝜎_𝑖𝜀_𝑖. (5.1) We consider the sample sizes n = 20, 60, 100. For n = 20 the covariate values 𝑥_1𝑖 were obtained from 𝑈(0,1) and the covariate values 𝑥_2𝑖 were obtained from N(0,1). These observations were replicated three and five times for creating the sample of size n = 60 and n = 100, respectively. The data generation was performed using 𝛽₀=𝛽₁=𝛽₂=1. For all i under the homoscedasticity, 𝜎_𝑖=1. However, the main interest here is to find the heteroscedastic model. In this respect, we create a heteroscedastic generating mechanism following Cribari-Neto [24]’s work, where 𝜎²_𝑖=exp3.2𝑥_1𝑖. (5.2) The degree of heteroscedasticity was measured by 𝜎℘=max²_𝑖min𝜎²_𝑖,𝑖=1,2,...,𝑛, (5.3) The degree of heteroscedasticity remains constant for different sample sizes since the covariate values are replicated for generating different sample sizes. In our study the degree of heterogeneity was approximately ℘=4. We focus on the situation where regression design would include outliers. To generate a certain percentages of outliers in Model (5.1), some i.i.d. normal errors 𝜀_𝑖’s were replaced by N(5, 10). Hence the contaminated heteroscedastic model becomes𝑦_𝑖=𝛽₀+𝛽₁𝑥_1𝑖+𝛽₂𝑥_2𝑖+𝜎_𝑖𝜀_i(cont.), (5.4) where 𝜀_𝑖(cont.)=𝛼𝑁(0,1)+(1−𝛼)𝑁(5,10) and 𝛼 is chosen according to level of percentage of outliers. In this study we choose the 5%, 10%, 15%, and 20% outliers in the model; that is, 𝛼 is 0.95, 0.80, 0.85, and 0.80, respectively. Now for each sample size, the OLS, the classical, and the proposed robust wild bootstrap were then applied to the data. The replications of the bootstrap were 500 in each model for the different sample sizes. It is noteworthy that the bootstrap is extremely computer intensive, and S-plus programming language was used for computing the bootstrap estimates.

The wild bootstrap standard errors of the estimates for different sample sizes and different percentage of contaminations are computed. The bootstrap standard errors of Boot_ols, Boot_wu, and Boot_liu are obtained by taking the square root to the main diagonal of the covariance matrix:(𝐵−1)^𝐵−1_𝑏=1𝛽^∗𝑏−𝛽^∗𝛽^∗𝑏−𝛽^∗𝑇, (5.5) where 𝛽^∗=(1/𝐵)∑^𝐵_𝑏=1𝛽^∗𝑏. On the other hand, the bootstrap standard errors of RBoot_ols, RBoot_wu, and RBoot_liu are obtained by taking the square root to the main diagonal of the covariance matrix as given in (5.5); the only essential difference is, however, we replace the usual bootstrap estimates by the robust bootstrap estimates.

The influences of outliers on the standard errors of the estimates are visible in Figures 2, 3, and 4. In these plots, the average standard errors of the parameters estimates are plotted at different levels of outliers for different bootstrap methods. The results presented in Figures 2–4 show that the performances of the wild bootstrap estimates are fairly close to the classical estimates at the 0% level of contamination. It emerges that the average standard errors of the RBoot_wu and RBoot_liu are closer to the average standard errors of the classical Boot_wu and Boot_liu, respectively, in “clean” data, regardless of the percentage of outliers. However, at the 5%, 10%, 15%, and 20% levels of contaminations, the classical standard errors of the bootstrap estimates become unduly large. On the contrary, it is interesting to see that not much influence is visible for the robust wild bootstrap techniques of RBoot_wu and RBoot_liu, at the different percentage levels of outliers. It is also observed that the performance of RBoot_liu is the best overall followed by RBoot_wu.

Figure 2 

The average effect of outliers on standard errors of parameters for sample size 𝑛=20.

Figure 3 

The average effect of outliers on standard errors of parameters for sample size 𝑛=60.

Figure 4 

The average effect of outliers on standard errors of parameters for sample size 𝑛=100.

5.2. Simulation Approach on Bootstrap Sample

In the previous section, we used artificial data sets for different sample sizes. Now we would like to investigate the performances of different bootstrap estimators where data sets are generated by Monte Carlo simulations. Let us consider a heteroscedastic model which is given by 𝑦_𝑖=𝛽₀+𝛽₁𝑥_1𝑖+𝛽₂𝑥_2𝑖+𝜎_𝑖𝜀_𝑖. (5.6) The covariate values of 𝑥_1𝑖 and 𝑥_2𝑖 are generated from𝑈(0,1) for sample sizes 20, 60, and 100. We have also considered 𝛽₀=𝛽₁=𝛽₂=1 as the true parameters in this model and the heteroscedasticity generating function was 𝜎²_𝑖=exp(0.4𝑥_1𝑖+0.4𝑥_2𝑖). In this study the level of heteroscedasticity is set as ℘=max(𝜎²_𝑖)/min(𝜎²_𝑖)=4.

In each simulation run and for the different sample size, 𝜀_𝑖’s were generated from N(0, 1) for the data with no outliers. However, for generating the 5% and 10% outliers, the 95% and 90% of𝜀_𝑖’s were generated from N(0, 1) and the 5% and 10% were generated from N(0, 20). It is worth mentioning that although such simulations are extremely computer intensive, the simulation for each sample size entails a total of 250000 replications with 500 replications and 500 bootstrap samples each. This simulation procedure was performed following the design of Cribari-Neto and Zarkos [16] and Furno [20].

The simulation results for the different bootstrap methods are presented in Tables 2–4. Table 2 shows the biasness measures of the non-robust and robust wild bootstrap techniques. It is observed that for the different sample sizes, the biasness of the Boot_ols, the Boot_liu, and the Boot_wu increases with the increase in the percentage of outliers. On the other hand, the RBoot_wu, and the RBoot_liu are slightly biased with the increase in the percentage of outliers. We can draw the same conclusion from the mean of the biasness of the estimates. The standard errors of the non-robust and robust wild bootstrap are presented in Table 3. It is observed that the standard errors of the classical bootstrap estimates increase with the increase in the percentage of outliers for different sample sizes. However, the robust bootstrap estimates are slightly affected by these outliers. By investigating the average standard errors of the estimates, it is also observed that the robust wild bootstrap techniques provide less standard error of the estimates in the presence of outliers. Finally, the robustness of different bootstrapping techniques are evaluated based on robustness measures defined in (5.5). Here the percentage robustness measure, that is, the ratio of the RMSEs of the estimators compared with the RMSEs of the OLS estimator for good data is presented in Table 4. From this table we see that the OLS and the classical bootstrap methods perform poorly. In the presence of outliers, the efficiency of the classical bootstrap estimates is very low. However, the efficiency of the robust bootstrap estimates is fairly closed to 100%.

Table 2 

Biasness measures of the non-robust and robust wild bootstrap.

Table 3 

Standard errors of the non-robust and robust wild bootstrap.

Table 4 

Robustness measure of RMSE of the non-robust and robust wild bootstrap.

6. Concluding Remarks

This paper examines the performance of classical wild bootstrap techniques which were proposed by Wu [6] and Liu [7] in the presence of heteroscedasticity and outliers. Both the artificial example and simulation study show that the classical bootstrap techniques perform poorly in the presence of outliers in the heteroscedastic model although they perform superbly for “clean” data. We attempt to robustify those classical bootstrap techniques to gain better efficiency in the presence of outliers. The numerical results show that the newly proposed robust wild bootstrap techniques, namely, the RBoot_wu and RBoot_liu outperform the classical wild bootstrap techniques when both outliers and heteroscedasticity are present in the data. RBoot_liu performs slightly better than RBoot_wu. Another advantage of using the RBoot_wu and the RBoot_liu is that no diagnosis for the data is required before the application of these methods.

Acknowledgment

The authors are grateful to the referees for valuable suggestions and comments that help them to improve the paper.