On the Performance of the Measure for Diagnosing Multiple High Leverage Collinearity-Reducing Observations
There is strong evidence indicating that the existing measures which are designed to detect a single high leverage collinearity-reducing observation are not effective in the presence of multiple high leverage collinearity-reducing observations. In this paper, we propose a cutoff point for a newly developed high leverage collinearity-influential measure and two existing measures ( and ) to identify high leverage collinearity-reducing observations, the high leverage points which hide multicollinearity in a data set. It is important to detect these observations as they are responsible for the misleading inferences about the fitting of the regression model. The merit of our proposed measure and cutoff point in detecting high leverage collinearity-reducing observations is investigated by using engineering data and Monte Carlo simulations.
High leverage points are the observations that fall far from the majority of explanatory variables in the data set (see [1–4]). It is now evident that high leverage point is another prime source of multicollinearity; a near-linear dependency of two or more explanatory variables . Hadi  pointed out that this source of multicollinearity is a special case of collinearity-influential observations; the observations which might induce or disrupt the multicollinearity pattern of a data. High leverage points that induce multicollinearity are referred as high leverage collinearity-enhancing observations while those that reduce multicollinearity in their presence are called high leverage collinearity-reducing observations [6–10]. Collinearity-influential observations are usually points with high leverages, though all high leverage points are not necessarily collinearity-influential observations .
It is very important to detect collinearity-influential observations because they are responsible for misleading conclusion about the fitting of a regression model, which gives wrong sign problem of regression coefficients and produces large variances to the regression estimates. Not many studies have been conducted in the literature on collinearity-influential measures and we will discuss these methods in Section 2. Nonetheless most of the existing methods are not successful in the detection of multiple high leverage collinearity-influential observations although their performances are considered good for the detection of a single observation. Moreover these measures do not have specific cutoff points to indicate the existence of collinearity-influential observations . These shortcomings motivated us to propose a new detection measure in such situation. Notably, the proposed measure is based on the Diagnostic Robust Generalized Potential (DRGP) method developed by Habshah et al.  and will be presented in Section 3. Section 4 exhibits the development of the collinearity-influential observations that can be classified as high leverage collinearity-enhancing or collinearity-reducing observations. Bagheri et al.  presented numerical examples and a simulation study to propose a novel high leverage collinearity-influential measure and a cutoff point for the detection of high leverage collinearity-enhancing observations. The authors also recommended cutoff points for collinearity-influential measures introduced by Hadi  and Sengupta and Bhimasankaram . It is also important to identify high leverage collinearity-reducing observations. However, these observations are more difficult to diagnose because they hide the effect of multicollinearity in the classical analysis. Following Hadi , Imon , and Habshah et al. , in Section 5, we propose a cutoff point for Bagheri‘s et al. , Hadi , and Sengupta and Bhimasankaram ’s measures to identify high leverage collinearity-reducing observations. A numerical example and simulation study are performed in Sections 6 and 7, respectively, to evaluate the performance of our proposed measure () and compare its performance with Hadi  and Sengupta and Bhimasankaram ’s measures ( and ). Conclusion of the study will be presented in Section 8.
2. Collinearity-Influential Measures
Let consider a multiple linear regression model as follows: where is an () vector of response or dependent variable, is an matrix of predictors (), is a () vector of unknown finite parameters to be estimated and is an () vector of random errors. We let denote the th column of the matrix; therefore, . Furthermore, multicollinearity is defined in terms of the linear dependence of the columns of .
Belsley et al.  proposed the singular-value decomposition of () matrix for diagnosing multicollinearity as follows: where is the matrix in which the columns that are associated with the nonzero eigenvalue of () is , (the matrix of eigenvectors of ) is , , , and is a diagonal matrix with nonnegative diagonal elements, , , which is called singular-values of . Condition number of matrix denoted as CN is another multicollinearity diagnostic measures which is obtained by first computing the Condition Index (CI) of the matrix and is defined as where are the singular values of the matrix. The CN corresponds to the largest values of . To make the condition indices comparable from one data set to another, the independent variables should first be scaled to have the same length. Scaling the independent variables prevents the eigen analysis to be dependent on the variables’ units of measurements. Belsley  stated that CN of matrix between 10 to 30 indicates moderate to strong multicollinearity, while a value of more than 30 reflects severe multicollinearity.
Hadi  noted that most collinearity-influential observations are points with high leverages, but not all high leverage points are collinearity-influential observations. He defined a measure for the influence of the th row of matrix on the condition index denoted as , where is computed by the eigenvalue of and when the th row of matrix has been deleted. Due to the lack of symmetry of Hadi’s measure, Sengupta and Bhimasankaram  proposed a collinearity-influential measure for each row of observations, defined as Unfortunately, they did not propose practical cutoff points for and and only mentioned the conditions for collinearity-enhancing and collinearity-reducing observations. To fill the gap, Bagheri et al.  suggested a cutoff point for and for detecting collinearity-enhancing observations as where is the Collinearity-Influential Measure cutoff point for the identification of collinearity-enhancing observations whereby can be or . for is an indicator that the th observation is a collinearity-enhancing observation.
3. Diagnostics Robust Generalised Potential for Identification of High Leverage Points
The th diagonal elements of the hat matrix, , is a traditionally used measure for detecting high leverage points and is defined as
Hoaglin and Welsch  suggested twice-the-mean-rule cutoff points for the hat matrix. Hadi  pointed out that the leverage diagnostics may not be successful to identify high leverage points and introduced a single-case-deleted measure, known as potential, and is defined as or where is the data matrix with the th row deleted. Imon  pointed that potentials may be very successful in the identification of a single high leverage point, but they fail to identify multiple high leverage points. To rectify this problem, Imon  proposed a group deletion version of potentials (GP), known as generalized potentials. Prior to defining the GP, Imon  partitioned the data into a set of “good” cases “remaining” in the analysis and a set of “bad” cases “deleted” from the analysis which were denoted as and . Nonetheless, Imon’s measure has drawbacks which are due to the inefficient procedure that he used for the determination of the initial deletion set . To overcome this shortcoming, Habshah et al.  proposed the diagnostic robust generalized potential (DRGP) where the suspected cases (bad cases) were identified by Robust Mahalanobis Distance (RMD), based on the Minimum Volume Ellipsoid (MVE). Rousseeuw  defined RMD based on MVE as follows: where and are robust locations and shape estimates of the MVE, respectively. In the second step of DRGP (MVE), the GPs are computed based on the set of and obtained from RMD (MVE). The low leverage points (if any) are put back into the estimation data set after inspecting the GP proposed by Imon  which are defined as follows: where . He suggested the cutoff point of as where can be taken as a constant value of 2 or 3.
The DRGP (MVE) have been proven to be very effective in the identification of multiple high leverage points.
4. The New Proposed High Leverage Collinearity-Influential Observations Measures
As already mentioned in the preceding section, the main reason of developing a new measure of high leverage collinearity-influential measure is due to the fact that the commonly used measures failed to detect multiple high leverage collinearity-influential observations. In addition, not many papers related to this measure have been published in the literatures. It is important mentioning that the collinearity-influential measure which were proposed by Hadi  and Sengupta and Bhimasankaram  are related to the Hadi’s single-case-deleted leverage measure . Since the robust generalized potentials that was developed by Habshah et al.  was very successful in the identification of multiple high leverage points compared to other widely used methods, Bagheri et al.  utilized a similar approach in developing multiple High Leverage Collinearity-Influential Measure (HLCIM). The proposed measure is formulated based on Sengupta and Bhimasankaram ’s measure with slight modification whereby almost similar approach of DRGP (MVE)  was adapted. Hence it is referred as HLCIM (DRGP) and denoted as . This new measure is defined as follows: where is the suspected group of multiple high leverage collinearity-influential observations diagnosed by DRGP(MVE), , is the number of elements in group, and is the remaining good observations. As such, following Habshah et al.  approach, three conditions should be considered in defining . Bagheri et al.  summarized the algorithm of HLCIM (DRGP) in three steps as follows.
Step 1. Calculate DRGP (MVE), , for . Form as a high leverage collinearity-influential suspected group whereby its members consist of observations which correspond to that exceed the . Obviously the rest of the observations belong to , the remaining group.
Step 2. Compute high leverage collinearity-influential values, , as follows.(i)If only a single member in the group, the size of is , and , calculate where indicates the condition number of the matrix without the th high leverage points. In this way, . (ii)If more than one member in the group, calculate where indicates the condition number of the matrix without the entire group minus the th high leverage points, where belongs to the suspected group.(iii)For any observation in the group, compute where refers to the condition number of the matrix without the entire group of high leverage points plus the th additional observation of the remaining group.
Step 3. If any values for does not exceed the cutoff points in (2.6), put back the th observation to the group. Otherwise, group is the high leverage collinearity-enhancing observations.
Bagheri et al.  only defined the cutoff point for to indicate high leverage collinearity-enhancing observations and they did not suggest cutoff point for collinearity-reducing observations. The authors considered to be high leverage collinearity-enhancing observations if is less than the cutoff points; that is median for < 0, where is a chosen value 3 and may be , or .
Since high leverage collinearity-reducing observations are also responsible for the misleading inferential statements, it is very crucial to detect their presence. In the following section, we propose a cutoff point for identifying high leverage collinearity-reducing observations.
It is important mentioning that not all which exceed the cutoff point are high leverage points. This is true for the situation when exceeds the cutoff point but belongs to the remaining group, . In this situation, the observation is considered as collinearity-influential observations since they are not high leverage points.
5. The New Proposed Cutoff Point for HLCIM (DRGP)
Hadi  and Sengupta and Bhimasankaram  mentioned that a large positive value of their collinearity-influential measures, and , respectively, indicates that the th observation is a collinearity-reducing observation. However, they did not suggest any cutoff points to indicate which observations are collinearity-enhancing and which are collinearity-reducing. Bagheri et al.  proposed a nonparametric cutoff point for high leverage collinearity-enhancing observations. Their work has inspired us to investigate high leverage collinearity-reducing observations among the observations that correspond to positive values of high leverage collinearity-influential measures. Figure 1 presents the normal distribution plot of . Based on this figure, any value that exceeds median can be utilized as a cutoff point for . Hence, we propose the following cutoff point: where is the Collinearity-Influential Measure cutoff point for Collinearity-Reducing Observations. can be , or . for is an indicator that the th observation is a collinearity-reducing observation.
6. A Numerical Example
A numerical example is presented to compare the performance of the newly proposed measure () with the existing measures and . An engineering data taken from Montgomery et al.  is used in this study. It represents the relationship between thrust of a jet-turbine engine () and six independent variables. The independent variables are primary speed of rotation (), secondary speed of rotation (), fuel flow rate (), pressure (), exhaust temperature (), and ambient temperature at time of test (). It is important mentioning that, the explanatory variables of this data are scaled before analysis in order to prevent the condition number to be dominated by large measurement units of some explanatory variables. Prior to analysis of this data, the explanatory variables have been scaled following Stewart’s  scaling method as There are other alternative scaling methods which can be found in Montgomery et al. , Stewart , and Hadi .
The matrix plot in Figure 2 and the collinearity diagnostics presented in Table 1 suggest that this data set has severe multicollinearity problem (CN 47.78). We would like to diagnose whether high leverage points are the cause of this problem. As such, it is necessary to detect the presence of high leverage points in this data set.
The index plot of DRGP (MVE) presented in Figure 3 suggests that observations 6 and 20 are high leverage points. By deleting these two observations from the data set, CN increases to 52.09. It seems that these two high leverages are collinearity-reducing observations.
The effect of these two high leverage points on collinearity pattern of the data is further investigated by applying , and with their respective new cutoff point introduced in (5.1) for detecting high leverage collinearity-reducing observations. Figure 4 illustrates the index plot of these measures. According to this plot, all these three measures have indicated that observations 6 and 20 as high leverage collinearity-reducing observations. Nevertheless, besides observations 6 and 20, they detect a few more observations as collinearity-reducing observations. It is interesting to note that none of the observations are detected as high leverage collinearity-enhancing observations or collinearity-enhancing observations.
It is worth mentioning that we do not have any information about the source of the two existing high leverage collinearity-reducing observations (cases 6 and 20). Therefore, we cannot control the magnitude and the number of added high leverages points to the data in order to study the effectiveness of our proposed measures. In this respect, we have modified this data set in two different patterns following . Habshah et al.  indicated that in the collinear data set, when high leverages exist in just one explanatory variable or in different positions of two explanatory variables; these leverages will be collinearity-reducing observations. Thus, the first pattern is when we replaced observations 5, 6, 19, and 20 of with a fixed large value of 50000. The second pattern is created by replacing the large value of 50000 to for observations 5, 6 and observations 19, 20 of .
The DRGP (MVE) index plot for Figure 5 reveals that observations 5, 6, 19, 20 are detected as high leverage points for modified jet turbine engine data set.
Figures 6 and 7 present the index plot of , and for the first and the second pattern of the modified jet turbine engine data set. The results of in these figures agree reasonably well with Bagheri’s et al.  findings that when high leverage points exist in just one explanatory variable (first pattern) or in different positions of two explanatory variables (second pattern) in collinear data sets, these observations are referred as collinearity-reducing observations. For both patterns, correctly identified that observations 5, 6, 19, and 20 are high leverage collinearity-reducing observations. However, for the first pattern, both and are not successful in detecting all of observations; 5, 6, 19, and 20 as high leverage collinearity-reducing observations. In the first pattern, they only correctly detected observations 19 and 20 as high leverage collinearity-reducing observations. However, none of the added high leverage collinearity-reducing observations can be detected by these two measures in the second pattern. It is important to note that for the first and the second patterns, the values of and for the observations 5 and 6, and observation 19, respectively are becoming negative. This indicates that for both patterns, and have wrongly indicated these observations as suspected high leverage collinearity-enhancing observations.
7. Monte Carlo Simulation Study
In this section, we report a Monte Carlo simulation study that is designed to assess the performance of our new proposed measure in detecting multiple high leverage collinearity-reducing observations and to compare its performance with two commonly used measures ( and ). Following Lawrence and Arthur , simulated data sets with three independent regressors were generated as follows: where the , ; are Uniform (0, 1). The value of which represents the correlation between the two explanatory variables are chosen to be equal to 0.95. This amount of correlation causes high multicollinearity between explanatory variables. Different percentage of high leverage points are considered in this study. The level of high leverage points varied from . Different sample sizes from , and 300 with replication of 10,000 times were considered. Following the idea of Habshah et al. , two different contamination patterns were created. In the first pattern, 100 () percent observations of one of the generated collinear explanatory variables were replaced by high leverages with unequal weights. In this pattern the explanatory variable and the observation which needed to be replaced by high leverage point were chosen randomly. The second pattern is created by replacing the first 100 () percent of one of the collinear explanatory variable and the last 100 () percent of another collinear explanatory variable with high leverages with unequal weights. The two independent variables are also randomly selected and the replacement of the high leverage point to the observations in different positions of explanatory variables was also performed randomly. Following Habshah et al.  and Bagheri et al. , the high leverage values with unequal weights in these two patterns were generated such that the values corresponding to the first high leverage point are kept fixed at 10 and those of the successive values are created by multiplying the observations index, by 10. The three diagnostic measures , and with the proposed cutoff point were introduced to (5.1) and were applied to each simulated data. The results based on the average values are presented in Table 2. The and HLCIO in Table 2 indicate, respectively, the percentage and the number of added high leverage collinearity-reducing observations to the simulated data sets. Furthermore, the number of high leverage points which is detected by DRGP (MVE) is denoted as HL. It is interesting to point out that the percentage of the high leverage point, detected by DRGP (MVE) denoted as HL in Table 2 is more than the percentage of the added high leverage collinearity-reducing observations to the simulated data sets, . However, by increasing the sample size and the percentage of added high leverage points to the simulated data, both percentages became exactly the same. The CN1 and the CN2 indicate the condition number of matrix without and with high leverage collinearity-reducing observations, respectively. Moreover, Cut and Cut()2 represent the number of high leverage collinearity-reducing observations and the number of collinearity-reducing observations which have been detected by cutoff .
Table 2 clearly shows the merit of our new proposed measure for high leverage collinearity-influential measure exhibited in (4.1). It can be observed that no other measures that were considered in this experiment performed satisfactorily except for our proposed measure. The simulated data sets have been created collinearly which produced large values of CN1, condition number of simulated data sets without high leverage points (CN1 30). The added multiple high leverage collinearity-influential observations reduces multicollinearity among the simulated explanatory variables; this reduction may result from the smaller values of CN2 compared to CN1. It is important mentioning that the reduction of the CN2 values for the second pattern was much more significant compared to CN2 for the first pattern. We can conclude that the influence of the added high leverage points to different positions of two explanatory variables for changing the multicollinearity pattern of simulated data, is more significant compared to the added high leverage points to only one explanatory variable.
The results of Table 2 for the first pattern of simulated data sets indicate that for small sample sizes () our proposed measure could not indicate the exact amount of high leverage collinearity-reducing observations. However, by increasing the sample size and the percentage of added high leverage points to the simulated data sets, the measure is capable of detecting the exact amount of added high leverage collinearity-reducing observations. It is evident by looking at the value of Cut()1 is exactly the same as HLCIO. On the other hand, the other two collinearity-influential measures, and , failed to indicate the exact amount of high leverage collinearity-reducing observations. It is worth noting that all of these three measures also detect some points as collinearity-reducing observations (see the Cut()2 in Table 2, where is , or ). Similar results will be obtained if pattern 1 can be drawn for the second pattern of the simulated data sets. Compared to the first contamination pattern, it is clearly seen that and almost completely failed to detect either high leverage collinearity-reducing observations or collinearity-reducing observations. Our proposed measure did a credible job where it is successfully detect high leverage collinearity-reducing observations for both contaminated patterns.
The presence of high leverage points and multicollinearity are inevitable in real data sets and they have an unduly effects on the parameter estimation of multiple linear regression models. These leverage points may be high leverage collinearity-enhancing or high leverage collinearity-reducing observations. It is crucial to detect these observations in order to reduce the destructive effects of multicollinearity on regression estimates which lead to misleading conclusion. It is easier to diagnose the presence of high leverage points which increase the collinearity among the explanatory variables compared to those which reduce collinearity. In this respect, it is very important to explore a sufficient measure with an accurate cutoff point for detecting high leverage collinearity-reducing observations. In this paper, we proposed a precise cutoff point for a novel existing measure to detect high leverage collinearity-reducing observations. By using an engineering data and a simulation study, we confirmed that the widely used measures failed to detect multiple high leverage collinearity-reducing observations. Furthermore, our proposed cutoff point successfully detects multiple high leverage collinearity-reducing observations.
D. C. Montgomery, E. A. Peck, and G. G. Viving, Introduction to Linear Regression Analysis, John Wiley & Sons, New York, NY, USA, 3rd edition, 2001.View at: MathSciNet
M. H. Kutner, C. J. Nachtsheim, and J. Neter, Applied Linear Regression Models, McGraw-Hill, New York, NY, USA, 2004.
S. Chatterjee and A. S. Hadi, Regression Analysis by Examples, John Wiley & Sons, New York, NY, USA, 4th edition, 2006.
M, A. Habshah, Bagheri, A. H. M. R, and Imon, “The application of robust multicollinearity diagnostic method based on robust coefficient determination to a non-collinear data,” Journal of Applied Sciences, vol. 10, no. 8, pp. 611–619, 2010.View at: Google Scholar
M. Habshah, A. Bagheri, and A. H. M. R. Imon, “High leverage collinearity-enhancing observations and its effect on multicollinearity pattern; Monte Carlo simulation study,” Sains Malaysiana, vol. 40, no. 12, pp. 1437–1447, 2011.View at: Google Scholar
A. Bagheri, H. Midi, and A. H. M. R. Imon, “The effect of collinearity-influential observations on collinear data set: A monte carlo simulation study,” Journal of Applied Sciences, vol. 10, no. 18, pp. 2086–2093, 2010.View at: Google Scholar
A. Bagheri and H. Midi, “On the performance of robust variance inflation factors,” International Journal of Agricultural and Statistical Sciences, vol. 7, no. 1, pp. 31–45, 2011.View at: Google Scholar
M. Habshah, M. R. Norazan, and A. H. M. R. Imon, “The performance of diagnostic-robust generalized potentials for the identification of multiple high leverage points in linear regression,” Journal of Applied Statistics, vol. 36, no. 5-6, pp. 507–520, 2009.View at: Publisher Site | Google Scholar | MathSciNet
A. S. Hadi, “A new measure of overall potential influence in linear regression,” Computational Statistics and Data Analysis, vol. 14, no. 1, pp. 1–27, 1992.View at: Google Scholar
D. A. Belsley, E. Kuh, and R. E. Welsch, Regression Diagnostics: Identifying Influential Data and Sources of Collinearity, John Wiley & Sons, New York, NY, USA, 1980.View at: MathSciNet
D. A. Belsley, Conditioning Diagnostics-Collinearity and Weak Data in Regression, John Wiley & Sons, New York, NY, USA, 1991.View at: MathSciNet
D. C. Hoaglin and R. E. Welsch, “The Hat Matrix in regression and ANOVA,” Journal of the American Statistical Association, vol. 32, no. 1, pp. 17–22, 1978.View at: Google Scholar
D. C. Montgomery, G. C. Runger, and N. F. Hubele, Engineering Statistics, John Wiley & Sons, New York, NY, USA, 5nd edition, 2011.
K. D. Lawrence and J. L. Arthur, Robust Regression; Analysis and Applications, Marcel Dekker, New York, NY, USA, 1990.