Research Article  Open Access
On the Performance of the Measure for Diagnosing Multiple High Leverage CollinearityReducing Observations
Abstract
There is strong evidence indicating that the existing measures which are designed to detect a single high leverage collinearityreducing observation are not effective in the presence of multiple high leverage collinearityreducing observations. In this paper, we propose a cutoff point for a newly developed high leverage collinearityinfluential measure and two existing measures ( and ) to identify high leverage collinearityreducing observations, the high leverage points which hide multicollinearity in a data set. It is important to detect these observations as they are responsible for the misleading inferences about the fitting of the regression model. The merit of our proposed measure and cutoff point in detecting high leverage collinearityreducing observations is investigated by using engineering data and Monte Carlo simulations.
1. Introduction
High leverage points are the observations that fall far from the majority of explanatory variables in the data set (see [1–4]). It is now evident that high leverage point is another prime source of multicollinearity; a nearlinear dependency of two or more explanatory variables [2]. Hadi [5] pointed out that this source of multicollinearity is a special case of collinearityinfluential observations; the observations which might induce or disrupt the multicollinearity pattern of a data. High leverage points that induce multicollinearity are referred as high leverage collinearityenhancing observations while those that reduce multicollinearity in their presence are called high leverage collinearityreducing observations [6–10]. Collinearityinfluential observations are usually points with high leverages, though all high leverage points are not necessarily collinearityinfluential observations [5].
It is very important to detect collinearityinfluential observations because they are responsible for misleading conclusion about the fitting of a regression model, which gives wrong sign problem of regression coefficients and produces large variances to the regression estimates. Not many studies have been conducted in the literature on collinearityinfluential measures and we will discuss these methods in Section 2. Nonetheless most of the existing methods are not successful in the detection of multiple high leverage collinearityinfluential observations although their performances are considered good for the detection of a single observation. Moreover these measures do not have specific cutoff points to indicate the existence of collinearityinfluential observations [10]. These shortcomings motivated us to propose a new detection measure in such situation. Notably, the proposed measure is based on the Diagnostic Robust Generalized Potential (DRGP) method developed by Habshah et al. [11] and will be presented in Section 3. Section 4 exhibits the development of the collinearityinfluential observations that can be classified as high leverage collinearityenhancing or collinearityreducing observations. Bagheri et al. [10] presented numerical examples and a simulation study to propose a novel high leverage collinearityinfluential measure and a cutoff point for the detection of high leverage collinearityenhancing observations. The authors also recommended cutoff points for collinearityinfluential measures introduced by Hadi [5] and Sengupta and Bhimasankaram [12]. It is also important to identify high leverage collinearityreducing observations. However, these observations are more difficult to diagnose because they hide the effect of multicollinearity in the classical analysis. Following Hadi [13], Imon [14], and Habshah et al. [11], in Section 5, we propose a cutoff point for Bagheri‘s et al. [10], Hadi [5], and Sengupta and Bhimasankaram [12]’s measures to identify high leverage collinearityreducing observations. A numerical example and simulation study are performed in Sections 6 and 7, respectively, to evaluate the performance of our proposed measure () and compare its performance with Hadi [5] and Sengupta and Bhimasankaram [12]’s measures ( and ). Conclusion of the study will be presented in Section 8.
2. CollinearityInfluential Measures
Let consider a multiple linear regression model as follows: where is an () vector of response or dependent variable, is an matrix of predictors (), is a () vector of unknown finite parameters to be estimated and is an () vector of random errors. We let denote the th column of the matrix; therefore, . Furthermore, multicollinearity is defined in terms of the linear dependence of the columns of .
Belsley et al. [15] proposed the singularvalue decomposition of () matrix for diagnosing multicollinearity as follows: where is the matrix in which the columns that are associated with the nonzero eigenvalue of () is , (the matrix of eigenvectors of ) is , , , and is a diagonal matrix with nonnegative diagonal elements, , , which is called singularvalues of . Condition number of matrix denoted as CN is another multicollinearity diagnostic measures which is obtained by first computing the Condition Index (CI) of the matrix and is defined as where are the singular values of the matrix. The CN corresponds to the largest values of . To make the condition indices comparable from one data set to another, the independent variables should first be scaled to have the same length. Scaling the independent variables prevents the eigen analysis to be dependent on the variables’ units of measurements. Belsley [16] stated that CN of matrix between 10 to 30 indicates moderate to strong multicollinearity, while a value of more than 30 reflects severe multicollinearity.
Hadi [5] noted that most collinearityinfluential observations are points with high leverages, but not all high leverage points are collinearityinfluential observations. He defined a measure for the influence of the th row of matrix on the condition index denoted as , where is computed by the eigenvalue of and when the th row of matrix has been deleted. Due to the lack of symmetry of Hadi’s measure, Sengupta and Bhimasankaram [12] proposed a collinearityinfluential measure for each row of observations, defined as Unfortunately, they did not propose practical cutoff points for and and only mentioned the conditions for collinearityenhancing and collinearityreducing observations. To fill the gap, Bagheri et al. [10] suggested a cutoff point for and for detecting collinearityenhancing observations as where is the CollinearityInfluential Measure cutoff point for the identification of collinearityenhancing observations whereby can be or . for is an indicator that the th observation is a collinearityenhancing observation.
3. Diagnostics Robust Generalised Potential for Identification of High Leverage Points
The th diagonal elements of the hat matrix, , is a traditionally used measure for detecting high leverage points and is defined as
Hoaglin and Welsch [17] suggested twicethemeanrule cutoff points for the hat matrix. Hadi [13] pointed out that the leverage diagnostics may not be successful to identify high leverage points and introduced a singlecasedeleted measure, known as potential, and is defined as or where is the data matrix with the th row deleted. Imon [14] pointed that potentials may be very successful in the identification of a single high leverage point, but they fail to identify multiple high leverage points. To rectify this problem, Imon [14] proposed a group deletion version of potentials (GP), known as generalized potentials. Prior to defining the GP, Imon [14] partitioned the data into a set of “good” cases “remaining” in the analysis and a set of “bad” cases “deleted” from the analysis which were denoted as and . Nonetheless, Imon’s measure has drawbacks which are due to the inefficient procedure that he used for the determination of the initial deletion set . To overcome this shortcoming, Habshah et al. [11] proposed the diagnostic robust generalized potential (DRGP) where the suspected cases (bad cases) were identified by Robust Mahalanobis Distance (RMD), based on the Minimum Volume Ellipsoid (MVE). Rousseeuw [18] defined RMD based on MVE as follows: where and are robust locations and shape estimates of the MVE, respectively. In the second step of DRGP (MVE), the GPs are computed based on the set of and obtained from RMD (MVE). The low leverage points (if any) are put back into the estimation data set after inspecting the GP proposed by Imon [14] which are defined as follows: where . He suggested the cutoff point of as where can be taken as a constant value of 2 or 3.
The DRGP (MVE) have been proven to be very effective in the identification of multiple high leverage points.
4. The New Proposed High Leverage CollinearityInfluential Observations Measures
As already mentioned in the preceding section, the main reason of developing a new measure of high leverage collinearityinfluential measure is due to the fact that the commonly used measures failed to detect multiple high leverage collinearityinfluential observations. In addition, not many papers related to this measure have been published in the literatures. It is important mentioning that the collinearityinfluential measure which were proposed by Hadi [5] and Sengupta and Bhimasankaram [12] are related to the Hadi’s singlecasedeleted leverage measure [13]. Since the robust generalized potentials that was developed by Habshah et al. [11] was very successful in the identification of multiple high leverage points compared to other widely used methods, Bagheri et al. [10] utilized a similar approach in developing multiple High Leverage CollinearityInfluential Measure (HLCIM). The proposed measure is formulated based on Sengupta and Bhimasankaram [12]’s measure with slight modification whereby almost similar approach of DRGP (MVE) [11] was adapted. Hence it is referred as HLCIM (DRGP) and denoted as . This new measure is defined as follows: where is the suspected group of multiple high leverage collinearityinfluential observations diagnosed by DRGP(MVE), , is the number of elements in group, and is the remaining good observations. As such, following Habshah et al. [11] approach, three conditions should be considered in defining . Bagheri et al. [10] summarized the algorithm of HLCIM (DRGP) in three steps as follows.
Step 1. Calculate DRGP (MVE), , for . Form as a high leverage collinearityinfluential suspected group whereby its members consist of observations which correspond to that exceed the . Obviously the rest of the observations belong to , the remaining group.
Step 2. Compute high leverage collinearityinfluential values, , as follows.(i)If only a single member in the group, the size of is , and , calculate where indicates the condition number of the matrix without the th high leverage points. In this way, . (ii)If more than one member in the group, calculate where indicates the condition number of the matrix without the entire group minus the th high leverage points, where belongs to the suspected group.(iii)For any observation in the group, compute where refers to the condition number of the matrix without the entire group of high leverage points plus the th additional observation of the remaining group.
Step 3. If any values for does not exceed the cutoff points in (2.6), put back the th observation to the group. Otherwise, group is the high leverage collinearityenhancing observations.
Bagheri et al. [10] only defined the cutoff point for to indicate high leverage collinearityenhancing observations and they did not suggest cutoff point for collinearityreducing observations. The authors considered to be high leverage collinearityenhancing observations if is less than the cutoff points; that is median for < 0, where is a chosen value 3 and may be , or .
Since high leverage collinearityreducing observations are also responsible for the misleading inferential statements, it is very crucial to detect their presence. In the following section, we propose a cutoff point for identifying high leverage collinearityreducing observations.
It is important mentioning that not all which exceed the cutoff point are high leverage points. This is true for the situation when exceeds the cutoff point but belongs to the remaining group, . In this situation, the observation is considered as collinearityinfluential observations since they are not high leverage points.
5. The New Proposed Cutoff Point for HLCIM (DRGP)
Hadi [5] and Sengupta and Bhimasankaram [12] mentioned that a large positive value of their collinearityinfluential measures, and , respectively, indicates that the th observation is a collinearityreducing observation. However, they did not suggest any cutoff points to indicate which observations are collinearityenhancing and which are collinearityreducing. Bagheri et al. [10] proposed a nonparametric cutoff point for high leverage collinearityenhancing observations. Their work has inspired us to investigate high leverage collinearityreducing observations among the observations that correspond to positive values of high leverage collinearityinfluential measures. Figure 1 presents the normal distribution plot of . Based on this figure, any value that exceeds median can be utilized as a cutoff point for . Hence, we propose the following cutoff point: where is the CollinearityInfluential Measure cutoff point for CollinearityReducing Observations. can be , or . for is an indicator that the th observation is a collinearityreducing observation.
6. A Numerical Example
A numerical example is presented to compare the performance of the newly proposed measure () with the existing measures and . An engineering data taken from Montgomery et al. [19] is used in this study. It represents the relationship between thrust of a jetturbine engine () and six independent variables. The independent variables are primary speed of rotation (), secondary speed of rotation (), fuel flow rate (), pressure (), exhaust temperature (), and ambient temperature at time of test (). It is important mentioning that, the explanatory variables of this data are scaled before analysis in order to prevent the condition number to be dominated by large measurement units of some explanatory variables. Prior to analysis of this data, the explanatory variables have been scaled following Stewart’s [20] scaling method as There are other alternative scaling methods which can be found in Montgomery et al. [1], Stewart [20], and Hadi [5].
The matrix plot in Figure 2 and the collinearity diagnostics presented in Table 1 suggest that this data set has severe multicollinearity problem (CN 47.78). We would like to diagnose whether high leverage points are the cause of this problem. As such, it is necessary to detect the presence of high leverage points in this data set.

The index plot of DRGP (MVE) presented in Figure 3 suggests that observations 6 and 20 are high leverage points. By deleting these two observations from the data set, CN increases to 52.09. It seems that these two high leverages are collinearityreducing observations.
The effect of these two high leverage points on collinearity pattern of the data is further investigated by applying , and with their respective new cutoff point introduced in (5.1) for detecting high leverage collinearityreducing observations. Figure 4 illustrates the index plot of these measures. According to this plot, all these three measures have indicated that observations 6 and 20 as high leverage collinearityreducing observations. Nevertheless, besides observations 6 and 20, they detect a few more observations as collinearityreducing observations. It is interesting to note that none of the observations are detected as high leverage collinearityenhancing observations or collinearityenhancing observations.
(a)
(b)
(c)
It is worth mentioning that we do not have any information about the source of the two existing high leverage collinearityreducing observations (cases 6 and 20). Therefore, we cannot control the magnitude and the number of added high leverages points to the data in order to study the effectiveness of our proposed measures. In this respect, we have modified this data set in two different patterns following [7]. Habshah et al. [7] indicated that in the collinear data set, when high leverages exist in just one explanatory variable or in different positions of two explanatory variables; these leverages will be collinearityreducing observations. Thus, the first pattern is when we replaced observations 5, 6, 19, and 20 of with a fixed large value of 50000. The second pattern is created by replacing the large value of 50000 to for observations 5, 6 and observations 19, 20 of .
The DRGP (MVE) index plot for Figure 5 reveals that observations 5, 6, 19, 20 are detected as high leverage points for modified jet turbine engine data set.
(a)
(b)
Figures 6 and 7 present the index plot of , and for the first and the second pattern of the modified jet turbine engine data set. The results of in these figures agree reasonably well with Bagheri’s et al. [10] findings that when high leverage points exist in just one explanatory variable (first pattern) or in different positions of two explanatory variables (second pattern) in collinear data sets, these observations are referred as collinearityreducing observations. For both patterns, correctly identified that observations 5, 6, 19, and 20 are high leverage collinearityreducing observations. However, for the first pattern, both and are not successful in detecting all of observations; 5, 6, 19, and 20 as high leverage collinearityreducing observations. In the first pattern, they only correctly detected observations 19 and 20 as high leverage collinearityreducing observations. However, none of the added high leverage collinearityreducing observations can be detected by these two measures in the second pattern. It is important to note that for the first and the second patterns, the values of and for the observations 5 and 6, and observation 19, respectively are becoming negative. This indicates that for both patterns, and have wrongly indicated these observations as suspected high leverage collinearityenhancing observations.
(a)
(b)
(c)
(a)
(b)
(c)
7. Monte Carlo Simulation Study
In this section, we report a Monte Carlo simulation study that is designed to assess the performance of our new proposed measure in detecting multiple high leverage collinearityreducing observations and to compare its performance with two commonly used measures ( and ). Following Lawrence and Arthur [21], simulated data sets with three independent regressors were generated as follows: where the , ; are Uniform (0, 1). The value of which represents the correlation between the two explanatory variables are chosen to be equal to 0.95. This amount of correlation causes high multicollinearity between explanatory variables. Different percentage of high leverage points are considered in this study. The level of high leverage points varied from . Different sample sizes from , and 300 with replication of 10,000 times were considered. Following the idea of Habshah et al. [7], two different contamination patterns were created. In the first pattern, 100 () percent observations of one of the generated collinear explanatory variables were replaced by high leverages with unequal weights. In this pattern the explanatory variable and the observation which needed to be replaced by high leverage point were chosen randomly. The second pattern is created by replacing the first 100 () percent of one of the collinear explanatory variable and the last 100 () percent of another collinear explanatory variable with high leverages with unequal weights. The two independent variables are also randomly selected and the replacement of the high leverage point to the observations in different positions of explanatory variables was also performed randomly. Following Habshah et al. [11] and Bagheri et al. [10], the high leverage values with unequal weights in these two patterns were generated such that the values corresponding to the first high leverage point are kept fixed at 10 and those of the successive values are created by multiplying the observations index, by 10. The three diagnostic measures , and with the proposed cutoff point were introduced to (5.1) and were applied to each simulated data. The results based on the average values are presented in Table 2. The and HLCIO in Table 2 indicate, respectively, the percentage and the number of added high leverage collinearityreducing observations to the simulated data sets. Furthermore, the number of high leverage points which is detected by DRGP (MVE) is denoted as HL. It is interesting to point out that the percentage of the high leverage point, detected by DRGP (MVE) denoted as HL in Table 2 is more than the percentage of the added high leverage collinearityreducing observations to the simulated data sets, . However, by increasing the sample size and the percentage of added high leverage points to the simulated data, both percentages became exactly the same. The CN1 and the CN2 indicate the condition number of matrix without and with high leverage collinearityreducing observations, respectively. Moreover, Cut and Cut()_{2} represent the number of high leverage collinearityreducing observations and the number of collinearityreducing observations which have been detected by cutoff .
(a)  
 
(b)  
 
(c)  

Table 2 clearly shows the merit of our new proposed measure for high leverage collinearityinfluential measure exhibited in (4.1). It can be observed that no other measures that were considered in this experiment performed satisfactorily except for our proposed measure. The simulated data sets have been created collinearly which produced large values of CN_{1}, condition number of simulated data sets without high leverage points (CN_{1} 30). The added multiple high leverage collinearityinfluential observations reduces multicollinearity among the simulated explanatory variables; this reduction may result from the smaller values of CN_{2} compared to CN_{1}. It is important mentioning that the reduction of the CN_{2} values for the second pattern was much more significant compared to CN_{2} for the first pattern. We can conclude that the influence of the added high leverage points to different positions of two explanatory variables for changing the multicollinearity pattern of simulated data, is more significant compared to the added high leverage points to only one explanatory variable.
The results of Table 2 for the first pattern of simulated data sets indicate that for small sample sizes () our proposed measure could not indicate the exact amount of high leverage collinearityreducing observations. However, by increasing the sample size and the percentage of added high leverage points to the simulated data sets, the measure is capable of detecting the exact amount of added high leverage collinearityreducing observations. It is evident by looking at the value of Cut()_{1} is exactly the same as HLCIO. On the other hand, the other two collinearityinfluential measures, and , failed to indicate the exact amount of high leverage collinearityreducing observations. It is worth noting that all of these three measures also detect some points as collinearityreducing observations (see the Cut()_{2} in Table 2, where is , or ). Similar results will be obtained if pattern 1 can be drawn for the second pattern of the simulated data sets. Compared to the first contamination pattern, it is clearly seen that and almost completely failed to detect either high leverage collinearityreducing observations or collinearityreducing observations. Our proposed measure did a credible job where it is successfully detect high leverage collinearityreducing observations for both contaminated patterns.
8. Conclusion
The presence of high leverage points and multicollinearity are inevitable in real data sets and they have an unduly effects on the parameter estimation of multiple linear regression models. These leverage points may be high leverage collinearityenhancing or high leverage collinearityreducing observations. It is crucial to detect these observations in order to reduce the destructive effects of multicollinearity on regression estimates which lead to misleading conclusion. It is easier to diagnose the presence of high leverage points which increase the collinearity among the explanatory variables compared to those which reduce collinearity. In this respect, it is very important to explore a sufficient measure with an accurate cutoff point for detecting high leverage collinearityreducing observations. In this paper, we proposed a precise cutoff point for a novel existing measure to detect high leverage collinearityreducing observations. By using an engineering data and a simulation study, we confirmed that the widely used measures failed to detect multiple high leverage collinearityreducing observations. Furthermore, our proposed cutoff point successfully detects multiple high leverage collinearityreducing observations.
References
 D. C. Montgomery, E. A. Peck, and G. G. Viving, Introduction to Linear Regression Analysis, John Wiley & Sons, New York, NY, USA, 3rd edition, 2001. View at: MathSciNet
 Md. Kamruzzaman and A. H. M. R. Imon, “High leverage point: another source of multicollinearity,” Pakistan Journal of Statistics, vol. 18, no. 3, pp. 435–447, 2002. View at: Google Scholar  Zentralblatt MATH  MathSciNet
 M. H. Kutner, C. J. Nachtsheim, and J. Neter, Applied Linear Regression Models, McGrawHill, New York, NY, USA, 2004.
 S. Chatterjee and A. S. Hadi, Regression Analysis by Examples, John Wiley & Sons, New York, NY, USA, 4th edition, 2006.
 A. S. Hadi, “Diagnosing collinearityinfluential observations,” Computational Statistics & Data Analysis, vol. 7, no. 2, pp. 143–159, 1989. View at: Publisher Site  Google Scholar  Zentralblatt MATH  MathSciNet
 M, A. Habshah, Bagheri, A. H. M. R, and Imon, “The application of robust multicollinearity diagnostic method based on robust coefficient determination to a noncollinear data,” Journal of Applied Sciences, vol. 10, no. 8, pp. 611–619, 2010. View at: Google Scholar
 M. Habshah, A. Bagheri, and A. H. M. R. Imon, “High leverage collinearityenhancing observations and its effect on multicollinearity pattern; Monte Carlo simulation study,” Sains Malaysiana, vol. 40, no. 12, pp. 1437–1447, 2011. View at: Google Scholar
 A. Bagheri, H. Midi, and A. H. M. R. Imon, “The effect of collinearityinfluential observations on collinear data set: A monte carlo simulation study,” Journal of Applied Sciences, vol. 10, no. 18, pp. 2086–2093, 2010. View at: Google Scholar
 A. Bagheri and H. Midi, “On the performance of robust variance inflation factors,” International Journal of Agricultural and Statistical Sciences, vol. 7, no. 1, pp. 31–45, 2011. View at: Google Scholar
 A. Bagheri, M. Habshah, and R. H. M. R. Imon, “A novel collinearityinfluential observation diagnostic measure based on a group deletion approach,” Communications in Statistics, vol. 41, no. 8, pp. 1379–1396, 2012. View at: Publisher Site  Google Scholar  MathSciNet
 M. Habshah, M. R. Norazan, and A. H. M. R. Imon, “The performance of diagnosticrobust generalized potentials for the identification of multiple high leverage points in linear regression,” Journal of Applied Statistics, vol. 36, no. 56, pp. 507–520, 2009. View at: Publisher Site  Google Scholar  MathSciNet
 D. Sengupta and P. Bhimasankaram, “On the roles of observations in collinearity in the linear model,” Journal of the American Statistical Association, vol. 92, no. 439, pp. 1024–1032, 1997. View at: Publisher Site  Google Scholar  MathSciNet
 A. S. Hadi, “A new measure of overall potential influence in linear regression,” Computational Statistics and Data Analysis, vol. 14, no. 1, pp. 1–27, 1992. View at: Google Scholar
 A. H. M. R. Imon, “Identifying multiple high leverage points in linear regression,” Journal of Statistical Studies, vol. 3, pp. 207–218, 2002, Special Volume in Honour of Professor Mir Masoom Ali. View at: Google Scholar  Zentralblatt MATH  MathSciNet
 D. A. Belsley, E. Kuh, and R. E. Welsch, Regression Diagnostics: Identifying Influential Data and Sources of Collinearity, John Wiley & Sons, New York, NY, USA, 1980. View at: MathSciNet
 D. A. Belsley, Conditioning DiagnosticsCollinearity and Weak Data in Regression, John Wiley & Sons, New York, NY, USA, 1991. View at: MathSciNet
 D. C. Hoaglin and R. E. Welsch, “The Hat Matrix in regression and ANOVA,” Journal of the American Statistical Association, vol. 32, no. 1, pp. 17–22, 1978. View at: Google Scholar
 P. Rousseeuw, “Multivariate estimation with high breakdown point,” in Mathematical Statistics and Applications, pp. 283–297, Reidel, Dordrecht, The Netherlands, 1985. View at: Google Scholar  MathSciNet
 D. C. Montgomery, G. C. Runger, and N. F. Hubele, Engineering Statistics, John Wiley & Sons, New York, NY, USA, 5nd edition, 2011.
 G. W. Stewart, “Collinearity and least squares regression,” Statistical Science, vol. 2, no. 1, pp. 68–100, 1987. View at: Google Scholar  Zentralblatt MATH  MathSciNet
 K. D. Lawrence and J. L. Arthur, Robust Regression; Analysis and Applications, Marcel Dekker, New York, NY, USA, 1990.
Copyright
Copyright © 2012 Arezoo Bagheri and Habshah Midi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.