A New Ridge-Type Estimator for the Linear Regression Model: Simulations and Applications

Kibria, B. M. Golam; Lukman, Adewale F.

doi:https://doi.org/10.1155/2020/9758378

Scientifica

On this page

Abstract Introduction Data Availability Conflicts of Interest Acknowledgments References Copyright Related Articles

Research Article | Open Access

Volume 2020 | Article ID 9758378 | https://doi.org/10.1155/2020/9758378

A New Ridge-Type Estimator for the Linear Regression Model: Simulations and Applications

B. M. Golam Kibria¹and Adewale F. Lukman^2,3

Academic Editor: Osman Kucuk

Received20 Jan 2020

Accepted28 Feb 2020

Published28 Apr 2020

Abstract

The ridge regression-type (Hoerl and Kennard, 1970) and Liu-type (Liu, 1993) estimators are consistently attractive shrinkage methods to reduce the effects of multicollinearity for both linear and nonlinear regression models. This paper proposes a new estimator to solve the multicollinearity problem for the linear regression model. Theory and simulation results show that, under some conditions, it performs better than both Liu and ridge regression estimators in the smaller MSE sense. Two real-life (chemical and economic) data are analyzed to illustrate the findings of the paper.

1. Introduction

To describe the problem, we consider the following linear regression model:where y is an vector of the response variable, is a known full rank matrix of predictor or explanatory variables, is an vector of unknown regression parameters, is an vector of errors such that , and , is an identity matrix. The ordinary least squares estimator (OLS) of β in (1) is defined aswhere is the design matrix.

The OLS estimator dominates for a long time until it was proven inefficient when there is multicollinearity among the predictor variables. Multicollinearity is the existence of near-to-strong or strong-linear relationship among the predictor variables. Different authors have developed several estimators as an alternative to the OLS estimator. These include Stein estimator [1], principal component estimator [2], ridge regression estimator [3], contraction estimator [4], modified ridge regression (MRR) estimator [5], and Liu estimator [6]. Also, some authors have developed two-parameter estimators to combat the problem of multicollinearity. The authors include Akdeniz and Kaçiranlar [7]; Özkale and Kaçiranlar [8]; Sakallıoğlu and Kaçıranlar [9]; Yang and Chang [10]; and very recently Roozbeh [11]; Akdeniz and Roozbeh [12]; and Lukman et al. [13, 14], among others.

The objective of this paper is to propose a new one-parameter ridge-type estimator for the regression parameter when the predictor variables of the model are linear or near-to-linearly related. Since we want to compare the performance of the proposed estimator with ridge regression and Liu estimator, we will give a short description of each of them as follows.

1.1. Ridge Regression Estimator

Hoerl and Kennard [3] originally proposed the ridge regression estimator. It is one of the most popular methods to solve the multicollinearity problem of the linear regression model. The ridge regression estimator is obtained by minimizing the following objective function:with respect to β, will yield the normal equationswhere k is the nonnegative constant. The solution to (4) gives the ridge estimator which is defined aswhere , and k is the biasing parameter. Hoerl et al. [15] defined the harmonic-mean version of the biasing parameter for the ridge regression estimator as follows:where is the estimated mean squared error form OLS regression using equation (1) and is ith coefficient of and is defined under equation (17). There are a high number of techniques suggested by various authors to estimate the biasing parameters. To mention a few, McDonald and Galarneau [16]; Lawless and Wang [17]; Wichern and Churchill [18]; Kibria [19]; Sakallıoğlu and Kaçıranlar [9]; Lukman and Ayinde [20]; and recently, Saleh et al. [21], among others.

1.2. Liu Estimator

The Liu estimator of is obtained by augmenting to (1) and then applying the OLS estimator to estimate the parameter. The Liu estimator is obtained to bewhere . The biasing parameter d for the Liu estimator is defined as follows:where is the ith eigenvalue of the matrix and which is defined under equation (17). If is negative, Özkale and Kaçiranlar [8] adopt the following alternative biasing parameter:where is the ith component of .

For more on the Liu [6] estimator, we refer our readers to Akdeniz and Kaçiranlar [7]; Liu [22]; Alheety and Kibria [23]; Liu [24]; Li and Yang [25]; Kan et al. [26]; and very recently, Farghali [27], among others.

In this article, we propose a new one-parameter estimator in the class of ridge and Liu estimators, which will carry most of the characteristics from both ridge and Liu estimators.

1.3. The New One-Parameter Estimator

The proposed estimator is obtained by minimizing the following objective function:with respect to β, will yield the normal equationswhere k is the nonnegative constant. The solution to (11) gives the new estimator aswhere , and . The new proposed estimator will be called the Kibria–Lukman (KL) estimator and denoted by .

1.3.1. Properties of the New Estimator

The proposed estimator is a biased estimator unless k = 0.and the mean square error matrix (MSEM) is defined as

To compare the performance of the four estimators (OLS, RR, Liu, and KL), we rewrite (1) in the canonical form which giveswhere and . Here, Q is an orthogonal matrix such that Z’Z = QX’XQ = Λ = diag (λ₁, λ₂, …, λ_p). The OLS estimator of is

The ridge estimator (RE) of iswhere and k is the biasing parameter.where .

The Liu estimator of iswhere .where .

The proposed one-parameter estimator of iswhere and .

The following notations and lemmas are needful to prove the statistical property of :

Lemma 1. Let n × n matrices M > 0 and N > 0 (or N ≥ 0); then, M > N if and only if λ₁ (NM⁻¹) < 1, where λ₁ (NM⁻¹) is the largest eigenvalue of matrix NM⁻¹ [28].

Lemma 2. Let M be an n × n positive definite matrix, that is, M > 0 and be some vector; then, if and only if [29].

Lemma 3. Let , i = 1, 2, be two linear estimators of . Suppose that , where denotes the covariance matrix of and . Consequently,if and only if , where [30].

The other parts of this article are as follows. The theoretical comparison among the estimators and estimation of the biasing parameters are given in Section 2. A simulation study has been constructed in Section 3. We conducted two numerical examples in Section 4. This paper ends up with concluding remarks in Section 5.

2. Comparison among the Estimators

2.1. Comparison between and

The difference between and is

We have the following theorem.

Theorem 1. If k > 0, estimator is superior to estimator using the MSEM criterion, that is, if and only if

Proof. The difference between (15) and (19) iswhere will be positive definite (pd) if and only if We observed that, for k > 0, .
Consequently, is pd.

2.2. Comparison between and

The difference between and is

Theorem 2. When estimator is superior to in the MSEM sense if and only ifwhere

Proof. Using the dispersion matrix difference,It is obvious that, for k > 0, G > 0 and H > 0. According to Lemma 1, it is clear that G-H > 0 if and only if , where is the maximum eigenvalue of the matrix Consequently, is pd.

2.3. Comparison between and

The difference between and is

We have the following theorem.

Theorem 3. If k > 0 and 0 < d < 1, estimator is superior to estimator using the MSEM criterion, that is, if and only ifwhere .

Proof. Using the difference between the dispersion matrix,where and We observed that is pd if and only if or . Obviously for k > 0 and 0 < d < 1, Consequently, is pd.

2.4. Determination of Parameter k

There is a need to estimate the parameter of the new estimator for practical use. The ridge biasing parameter and the Liu shrinkage parameter were determined by both Hoerl and Kennard [3] and Liu [6], respectively. Different authors have developed other estimators of these ridge parameters. To mention a few, these include Hoerl et al. [15]; Kibria [19]; Kibria and Banik [31]; and Lukman and Ayinde [20], among others. The optimal value of k is the one that minimizes

Differentiating with respect to k gives and setting , we obtain

The optimal value of k in (39) depends on the unknown parameter and . These two estimators are replaced with their unbiased estimate. Consequently, we have

Following Hoerl et al. [15], the harmonic-mean version of (40) is defined as

According to Özkale and Kaçiranlar [8], the minimum version of (41) is defined as

3. Simulation Study

Since theoretical comparisons among the estimators, ridge regression, Liu and KL in Section 2 give the conditional dominance among the estimators, a simulation study has been conducted using the R 3.4.1 programming languages to see a better picture about the performance of the estimators.

3.1. Simulation Technique

The design of the simulation study depends on factors that are expected to affect the properties of the estimator under investigation and the criteria being used to judge the results. Since the degree of collinearity among the explanatory variable is of central importance, following Gibbons [32] and Kibria [19], we generated the explanatory variables using the following equation:where are independent standard normal pseudo-random numbers and represents the correlation between any two explanatory variables. We consider and 7 in the simulation. These variables are standardized so that and are in correlation forms. The n observations for the dependent variable y are determined by the following equation:where are i.i.d N (0, σ²), and without loss of any generality, we will assume zero intercept for the model in (44). The values of β are chosen such that = 1 [33]. Since our main objective is to compare the performance of the proposed estimator with ridge regression and Liu estimators, we consider k = d = 0.1, 0.2, …, 1. We have restricted k between 0 and 1 as Wichern and Churchill [18] have found that the ridge regression estimator is better than the OLS when k is between 0 and 1. Kan et al. [26] also suggested a smaller value of k (less than 1) is better. Simulation studies are repeated 1,000 times for the sample sizes n = 30 and 100 and σ² = 1, 25, and 100. For each replicate, we compute the mean square error (MSE) of the estimators by using the following equation:where would be any of the estimators (OLS, ridge, Liu, or KL). Smaller MSE of the estimators will be considered the best one.

The simulated results for n = 30, , and ρ = 0.70, 0.80 and ρ = 0.90, 0.99 are presented in Tables 1 and 2, respectively, and for n = 100, , and ρ = 0.7, 0.80 and ρ = 0.90, 0.99 are presented in Tables 3 and 4, respectively. The corresponding simulated results for n = 30, 100 and are presented in Tables 5–8. For a better visualization, we have plotted MSE vs. d for n = 30, σ = 10, and ρ = 0.70, 0.90, and 0.99 in Figures 1–3, respectively. We also plotted MSE vs σ for n = 30, d = .50, and ρ = 0.90 and 0.99, which is presented in Figures 4 and 5, respectively. Finally, to see the effect of sample size on MSE, we plotted MSE vs. sample size for d = 0.5 and ρ = 0.90 and presented in Figure 6.

(a)

(b)

(c)

(d)

(a)

(b)

(c)

(d)

(a)

(b)

(c)

(d)

(a)

(b)

(c)

(d)

(a)

(b)

(c)

(d)

(a)

(b)

(c)

(d)

3.2. Simulation Results and Discussion

From Tables 1–8 and Figures 1–6, it appears that, as the values of σ increase, the MSE values also increase (Figure 3), while the sample size increases as the MSE values decrease (Figure 4). Ridge, Liu, and proposed KL estimators uniformly dominate the ordinary least squares (OLS) estimator. In general, from these tables, an increase in the levels of multicollinearity and the number of explanatory variables increase the estimated MSE values of the estimators. The figures consistently show that the OLS estimator performs worst when there is multicollinearity. From Figures 1–6 and simulation Tables 1–8, it clearly indicated that, for or less, the proposed estimator uniformly dominates the ridge regression estimator, while Liu performed much better than both proposed and ridge estimators for small d, say 0.3 or less. When , the ridge regression performs the best for higher k, while the proposed estimator performs the best for say k (say 0.3 or less). When d = k = 0.5 and , both ridge and KL estimators outperform the Liu estimator. None of the estimators uniformly dominates each other. However, it appears that our proposed estimator, KL, performs better in the wider space of d = k in the parameter space. If we review all Tables 1–8, we observed that the conclusions about the performance of all estimators remain the same for both and .

4. Numerical Examples

To illustrate our theoretical results, we consider two datasets: (i) famous Portland cement data originally adopted by Woods et al. [34] and (ii) French economy data from Chatterjee and Hadi [35], and they are analyzed in the following sections, respectively.

4.1. Example 1: Portland Data

These data are widely known as the Portland cement dataset. It was originally adopted by Woods et al. [34]. It has also been analyzed by the following authors: Kaciranlar et al. [36]; Li and Yang [25]; and recently by Lukman et al. [13]. The regression model for these data is defined aswhere = heat evolved after 180 days of curing measured in calories per gram of cement, = tricalcium aluminate, = tricalcium silicate, = tetracalcium aluminoferrite, and = -dicalcium silicate. The correlation matrix of the predictor variables is given in Table 9.

The variance inflation factors are VIF₁ = 38.50, VIF₂ = 254.42, VIF₃ = 46.87, and VIF₄ = 282.51. Eigenvalues of are , and , and the condition number of is approximately 424. The VIFs, the eigenvalues, and the condition number all indicate the presence of severe multicollinearity. The estimated parameters and MSE are presented in Table 10. It appears from Table 11 that the proposed estimator performed the best in the sense of smaller MSE.

4.2. Example 2: French Economy Data

The French economy data in Chatterjee and Hadi [37] are considered in this example. It has been analyzed by Malinvard [38] and Liu [6], among others. The variables are imports, domestic production, stock formation, and domestic consumption. All are measured in milliards of French francs for the years 1949 through 1966.

The regression model for these data is defined aswhere = IMPORT, = domestic production, = stock formation, and = domestic consumption. The correlation matrix of the predicted variable is given in Table 12.

The variance inflation factors are and . The eigenvalues of the matrix are λ₁ = 161779, λ₂ = 158, and λ₃ = 49.61, and the condition number is 32612. If we review the above correlation matrix, VIFs, and condition number, it can be said that there is presence of severe multicollinearity existing in the predictor variables.

The biasing parameter for the new estimator is defined in (41) and (42). The biasing parameter for the ridge and Liu estimator is provided in (6), (8), and (9), respectively.

We analyzed the data using the biasing parameters for each of the estimators and presented the results in Tables 10 and 11. It can be seen from Tables 10 and 11 that the proposed estimator performed the best in the sense of smaller MSE.

5. Summary and Concluding Remarks

In this paper, we introduced a new biased estimator to overcome the multicollinearity problem for the multiple linear regression model and provided the estimation technique of the biasing parameter. A simulation study has been conducted to compare the performance of the proposed estimator and Liu [6] and ridge regression estimators [3]. Simulation results evidently show that the proposed estimator performed better than both Liu and ridge under some condition on the shrinkage parameter. Two sets of real-life data are analyzed to illustrate the benefits of using the new estimator in the context of a linear regression model. The proposed estimator is recommended for researchers in this area. Its application can be extended to other regression models, for example, logistic regression, Poisson, ZIP, and related models, and those possibilities are under current investigation [37, 39, 40].

Data Availability

Data will be made available on request.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

We are dedicating this article to those who lost their lives because of COVID-19.

References

C. Stein, “Inadmissibility of the usual estimator for mean of multivariate normal distribution,” in Proceedings of the Third Berkley Symposium on Mathematical and Statistics Probability, J. Neyman, Ed., vol. 1, pp. 197–206, Springer, Berlin, Germany, 1956.
View at: Google Scholar
W. F. Massy, “Principal components regression in exploratory statistical research,” Journal of the American Statistical Association, vol. 60, no. 309, pp. 234–256, 1965.
View at: Publisher Site | Google Scholar
A. E. Hoerl and R. W. Kennard, “Ridge regression: biased estimation for nonorthogonal problems,” Technometrics, vol. 12, no. 1, pp. 55–67, 1970.
View at: Publisher Site | Google Scholar
L. S. Mayer and T. A. Willke, “On biased estimation in linear models,” Technometrics, vol. 15, no. 3, pp. 497–508, 1973.
View at: Publisher Site | Google Scholar
B. F. Swindel, “Good ridge estimators based on prior information,” Communications in Statistics-Theory and Methods, vol. 5, no. 11, pp. 1065–1075, 1976.
View at: Publisher Site | Google Scholar
K. Liu, “A new class of biased estimate in linear regression,” Communication in Statistics- Theory and Methods, vol. 22, pp. 393–402, 1993.
View at: Google Scholar
F. Akdeniz and S. Kaçiranlar, “On the almost unbiased generalized liu estimator and unbiased estimation of the bias and mse,” Communications in Statistics-Theory and Methods, vol. 24, no. 7, pp. 1789–1797, 1995.
View at: Publisher Site | Google Scholar
M. R. Özkale and S. Kaçiranlar, “The restricted and unrestricted two-parameter estimators,” Communications in Statistics-Theory and Methods, vol. 36, no. 15, pp. 2707–2725, 2007.
View at: Publisher Site | Google Scholar
S. Sakallıoğlu and S. Kaçıranlar, “A new biased estimator based on ridge estimation,” Statistical Papers, vol. 49, no. 4, pp. 669–689, 2008.
View at: Google Scholar
H. Yang and X. Chang, “A new two-parameter estimator in linear regression,” Communications in Statistics-Theory and Methods, vol. 39, no. 6, pp. 923–934, 2010.
View at: Publisher Site | Google Scholar
M. Roozbeh, “Optimal QR-based estimation in partially linear regression models with correlated errors using GCV criterion,” Computational Statistics & Data Analysis, vol. 117, pp. 45–61, 2018.
View at: Publisher Site | Google Scholar
F. Akdeniz and M. Roozbeh, “Generalized difference-based weighted mixed almost unbiased ridge estimator in partially linear models,” Statistical Papers, vol. 60, no. 5, pp. 1717–1739, 2019.
View at: Publisher Site | Google Scholar
A. F. Lukman, K. Ayinde, S. Binuomote, and O. A. Clement, “Modified ridge-type estimator to combat multicollinearity: application to chemical data,” Journal of Chemometrics, vol. 33, no. 5, p. e3125, 2019.
View at: Publisher Site | Google Scholar
A. F. Lukman, K. Ayinde, S. K. Sek, and E. Adewuyi, “A modified new two-parameter estimator in a linear regression model,” Modelling and Simulation in Engineering, vol. 2019, Article ID 6342702, 10 pages, 2019.
View at: Publisher Site | Google Scholar
A. E. Hoerl, R. W. Kannard, and K. F. Baldwin, “Ridge regression:some simulations,” Communications in Statistics, vol. 4, no. 2, pp. 105–123, 1975.
View at: Publisher Site | Google Scholar
G. C. McDonald and D. I. Galarneau, “A monte carlo evaluation of some ridge-type estimators,” Journal of the American Statistical Association, vol. 70, no. 350, pp. 407–416, 1975.
View at: Publisher Site | Google Scholar
J. F. Lawless and P. Wang, “A simulation study of ridge and other regression estimators,” Communications in Statistics-Theory and Methods, vol. 5, no. 4, pp. 307–323, 1976.
View at: Publisher Site | Google Scholar
D. W. Wichern and G. A. Churchill, “A comparison of ridge estimators,” Technometrics, vol. 20, no. 3, pp. 301–311, 1978.
View at: Publisher Site | Google Scholar
B. M. G. Kibria, “Performance of some new ridge regression estimators,” Communications in Statistics-Simulation and Computation, vol. 32, no. 1, pp. 419–435, 2003.
View at: Publisher Site | Google Scholar
A. F. Lukman and K. Ayinde, “Review and classifications of the ridge parameter estimation techniques,” Hacettepe Journal of Mathematics and Statistics, vol. 46, no. 5, pp. 953–967, 2017.
View at: Google Scholar
A. K. M. E. Saleh, M. Arashi, and B. M. G. Kibria, Theory of Ridge Regression Estimation with Applications, Wiley, Hoboken, NJ, USA, 2019.
K. Liu, “Using Liu-type estimator to combat collinearity,” Communications in Statistics-Theory and Methods, vol. 32, no. 5, pp. 1009–1020, 2003.
View at: Publisher Site | Google Scholar
K. Alheety and B. M. G. Kibria, “On the Liu and almost unbiased Liu estimators in the presence of multicollinearity with heteroscedastic or correlated errors,” Surveys in Mathematics and its Applications, vol. 4, pp. 155–167, 2009.
View at: Google Scholar
X.-Q. Liu, “Improved Liu estimator in a linear regression model,” Journal of Statistical Planning and Inference, vol. 141, no. 1, pp. 189–196, 2011.
View at: Publisher Site | Google Scholar
Y. Li and H. Yang, “A new Liu-type estimator in linear regression model,” Statistical Papers, vol. 53, no. 2, pp. 427–437, 2012.
View at: Publisher Site | Google Scholar
B. Kan, Ö. Alpu, and B. Yazıcı, “Robust ridge and robust Liu estimator for regression based on the LTS estimator,” Journal of Applied Statistics, vol. 40, no. 3, pp. 644–655, 2013.
View at: Publisher Site | Google Scholar
R. A. Farghali, “Generalized Liu-type estimator for linear regression,” International Journal of Research and Reviews in Applied Sciences, vol. 38, no. 1, pp. 52–63, 2019.
View at: Google Scholar
S. G. Wang, M. X. Wu, and Z. Z. Jia, Matrix Inequalities, Chinese Science Press, Beijing, China, 2nd edition, 2006.
R. W. Farebrother, “Further results on the mean square error of ridge regression,” Journal of the Royal Statistical Society: Series B (Methodological), vol. 38, no. 3, pp. 248–250, 1976.
View at: Publisher Site | Google Scholar
G. Trenkler and H. Toutenburg, “Mean squared error matrix comparisons between biased estimators-an overview of recent results,” Statistical Papers, vol. 31, no. 1, pp. 165–179, 1990.
View at: Publisher Site | Google Scholar
B. M. G. Kibria and S. Banik, “Some ridge regression estimators and their performances,” Journal of Modern Applied Statistical Methods, vol. 15, no. 1, pp. 206–238, 2016.
View at: Publisher Site | Google Scholar
D. G. Gibbons, “A simulation study of some ridge estimators,” Journal of the American Statistical Association, vol. 76, no. 373, pp. 131–139, 1981.
View at: Publisher Site | Google Scholar
J. P. Newhouse and S. D. Oman, “An evaluation of ridge estimators. A report prepared for United States air force project RAND,” 1971.
View at: Google Scholar
H. Woods, H. H. Steinour, and H. R. Starke, “Effect of composition of Portland cement on heat evolved during hardening,” Industrial & Engineering Chemistry, vol. 24, no. 11, pp. 1207–1214, 1932.
View at: Publisher Site | Google Scholar
S. Chatterjee and A. S. Hadi, Regression Analysis by Example, Wiley, Hoboken, NJ, USA, 1977.
S. Kaciranlar, S. Sakallioglu, F. Akdeniz, G. P. H. Styan, and H. J. Werner, “A new biased estimator in linear regression and a detailed analysis of the widely-analysed dataset on portland cement,” Sankhyā: The Indian Journal of Statistics, Series B, vol. 61, pp. 443–459, 1999.
View at: Google Scholar
S. Chatterjee and A. S. Haadi, Regression Analysis by Example, Wiley, Hoboken, NJ, USA, 2006.
E. Malinvard, Statistical Methods of Econometrics, North-Holland Publishing Company, Amsterdam, Netherlands, 3rd edition, 1980.
D. N. Gujarati, Basic Econometrics, McGraw-Hill, New York, NY, USA, 1995.
A. F. Lukman, K. Ayinde, and A. S. Ajiboye, “Monte Carlo study of some classification-based ridge parameter estimators,” Journal of Modern Applied Statistical Methods, vol. 16, no. 1, pp. 428–451, 2017.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2020 B. M. Golam Kibria and Adewale F. Lukman. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

3772

Downloads

1829

Citations