Robust Quadratic Regression and Its Application to Energy-Growth Consumption Problem

Abstract

We propose a robust quadratic regression model to handle the statistics inaccuracy. Unlike the traditional robust statistic approaches that mainly focus on eliminating the effect of outliers, the proposed model employs the recently developed robust optimization methodology and tries to minimize the worst-case residual errors. First, we give a solvable equivalent semidefinite programming for the robust least square model with ball uncertainty set. Then the result is generalized to robust models under 𝑙₁- and 𝑙_∞-norm critera with general ellipsoid uncertainty sets. In addition, we establish a robust regression model for per capital GDP and energy consumption in the energy-growth problem under the conservation hypothesis. Finally, numerical experiments are carried out to verify the effectiveness of the proposed models and demonstrate the effect of the uncertainty perturbation on the robust models.

1. Introduction

Traditional regression analysis is a useful tool to model the linear or nonlinear relationship between the observed data. In the simplest linear regression model, there is only one explanatory variable 𝑥 (the regressors) and one dependent variable 𝑦 (the regressand) that is assumed to be an affine function of 𝑥; it is further extended to a polynomial regression model where 𝑦 is an 𝑛th order polynomial of 𝑥. In this case, the corresponding multivariate regression model contains more than one explanatory variable. To make the regression models work well, there are several specific assumptions on the model and the observed data.

Consider the following standard multivariate linear regression model: 𝑦_𝑖=^𝑛_𝑗=1𝑥_𝑖𝑗+𝜀_𝑖,𝑖=1,…,𝑚, (1) where (𝑥_𝑖,𝑦_𝑖)^𝑚_𝑖=1 are given observed data and 𝜀 is a random error vector. Assuming that (𝜀_𝑖)^𝑚_𝑖=1 have zero mean and constant variance, they are independent of each other. Besides the assumption on the random errors, there is another important weak exogeneity assumption that the explanatory variables are known deterministic values. Under this assumption, one can arbitrarily transform their values and construct any complex function relationship between the regressors and the regressand. For example, in this case the polynomial regression is merely a linear regression with regressors {𝑥¹_𝑖,𝑥²_𝑖,…,𝑥^𝑝_𝑖}.

Although this weak exogeneity assumption makes the linear regression model very powerful to fit the given data or predict the regressand for given known regressors, it may lead to overfitting or inconsistent estimations [1]. Actually this assumption may be quite unreasonable in some case. For instance, in the process of collecting data, there is often unavoidable observation noise that makes the observed data quite inaccurate. Furthermore, in statistics the incomplete sampling approach sometimes can only give an approximation of the real values.

Researches on regression models with imprecise data have been reported. One way to handle the noisy observation is the measure error model or the errors-in-variable model, where it is assumed that there exist some unknown latent (or true) variables that follow the true functional relationship, and the actual observations are affected by certain random noise [2]. Based on different assumptions about random noise, there are a variety of regression models, such as the method of moments [3] that is based on the third-(or higher-) order joint cumulants of observable variables and the Deming regression [4] assuming that the ratio of the noise variance is known. A brief historical overview of linear regression with errors in variables can be found in [5].

In addition to the errors-in-variable models, motivated by the robust optimization theory under uncertainty, studies on the robust regression models are reported. In such case, the perturbations are deterministic and unknown but bounded. Ghaoui and Lebret [6] study the robust linear regression with bounded uncertainty sets under least square criterion. They utilize the second-order cone programming (SOCP) and semi-definite programming to minimize the worst-case residual errors. Shivaswamy et al. [7] propose SOCP formulations for robust linear classification and regression models when the first two moments of the uncertain data are computable. Ben-Tal et al. [8] provide an excellent framework of robust linear classification and regression. Based on general assumption on the uncertainty sets, they provide explicit equivalent formulations for robust least squares, 𝑙₁, 𝑙_∞, and Huber penalty regressions. For more results regarding robust classification and regression that are similar to this work, we refer the readers to [9–11]. Also according to [8], the traditional robust statistic approaches (see [12]), which try to reject the outliers in data, are different from the point of view in this paper as the authors here intend to minimize the maximal (worst-case) residual errors. However in order to overcome the confliction, a two-step approach can be easily implemented. First, the outliers are identified, and related data is removed. Then, our proposed method is applied in order to safely eliminate the effect generated from imprecise data. We employ this approach in the real energy-growth regression problem in Section 3.

Besides the regression models, there are a wide variety of forecasting models, such as support vector machines, decision tree, neural network, and Bayes classifier. For example, [13] utilizes the support vector machine based on trend-based segmentation method for financial time series forecasting. The proposed models have been tested by using various stocks from America stock market with different trends [14]. proposes a new adaptive local linear prediction method to reduce the parameters uncertainties in the prediction of a chaotic time series. Real hydrological time series are used to validate the effectiveness of the proposed methods. More related literatures can be found in [15] (chaotic time series analysis), [16] (fractal time series), and [17] (knowledge-based Green's kernel for support vector regression). Compared with these models, we focus on the handling of the statistics inaccuracy. The regression model is an appropriate basis to develop effective and tractable robust models.

In this paper, we try to extend the robust linear regression model to general multivariate quadratic regression and provide equivalent tractable formulations. Different from the simple extension from the classical linear model to classical polynomial (even general nonlinear) models under the weak exogeneity assumption, the perturbation of explanatory variables in the quadratic terms will affect the model in a complex nonlinear manner. Although [8, 12] have discussed the robust polynomial interpolation problem, only an upper bound and the corresponding suboptimal coefficients are given. They further conjecture that the proposed problem cannot be solved exactly in polynomial time. Our proposed robust multivariate quadratic regression model in this paper also needs to solve a complex biquadratic min-max optimization problem. However, under certain assumption on the uncertainty sets, we can obtain a series of equivalent semidefinite programming formulations for robust quadratic regression under different residual error criteria.

In particular, we first extend the traditional quadratic regression model by introducing the separable ball (2-norm) uncertainty set and formulate the optimal robust regression problem as a min-max problem that tries to minimize the maximal residual error. By utilizing the S-lemma [18] and Schur complement lemma, we provide an equivalent semi-definite programming formulation for the robust least square quadratic regression model with ball uncertainty set. This result is then generalized to models with general ellipsoid uncertainty sets and under the 𝑙₁-, 𝑙_∞-norm criteria. Furthermore the robust quadratic regression models are applied to the economic growth and energy consumption regression problem. We take the per capital GDP as the explanatory variable and the per capital energy consumption as the dependent variable. Under the conservation hypothesis, we establish a corresponding robust model. Finally we test the proposed model on different history data sets and compare our models with the classical regression models.

The paper proceeds as follows. In Section 2, we present a general robust quadratic regression model, give a solvable equivalent semi-definite programming for the robust least square quadratic regression model with ball uncertainty set, and further generalize the result. In Section 3, the proposed models are applied to the energy-growth problem. Numerical experiments are carried out in Section 4, and Section 5 concludes this paper and gives future research directions.

2. Robust Quadratic Regression Models

2.1. General Robust Models

Consider the standard multivariate quadratic regression model: 𝑦=𝑥^𝑇𝑄𝑥+2𝛼^𝑇𝑥+𝛽, (2) where 𝑥∈𝑅^𝑛 denotes the 𝑛-dimension explanatory data, 𝑦∈𝑅 denotes the dependent data, and 𝑄∈𝑅^𝑛×𝑛, 𝛼∈𝑅^𝑛, and 𝛽∈𝑅 are unknown coefficients that will be determined based on certain minimal criteria.

Given a set of data 𝐷=[𝑋;𝑌^𝑇]∈𝑅^{(𝑛+1)×𝑚}, where 𝑋=[𝑥₁,…,𝑥_𝑚]∈𝑅^𝑛×𝑚 and 𝑌=[𝑦₁;…;𝑦_𝑚]∈𝑅^𝑚, we utilize the 𝑝-norm to measure the prediction error 𝑒_𝑝=𝑦𝛼,𝛽,𝑄¹−𝑥^𝑇₁𝑄𝑥₁−2𝛼^𝑇𝑥₁⋮𝑦−𝛽_𝑚−𝑥^𝑇_𝑚𝑄𝑥_𝑚−2𝛼^𝑇𝑥_𝑚−𝛽_𝑝. (3)

In traditional regression models, we assume that the explanatory data are precise and reliable. Based on this weak exogeneity assumption, the quadratic regression can be expressed as the following linear regression: min_{𝛼,𝛽,𝑄}𝑦₁−𝑄∘𝑋₁−2𝛼^𝑇𝑥₁⋮𝑦−𝛽_𝑚−𝑄∘𝑋_𝑚−2𝛼^𝑇𝑥_𝑚−𝛽_𝑝, (4) where 𝑋_𝑖=𝑥_𝑖𝑥^𝑇_𝑖 are the problem data and the linear operator ∘ for matrix 𝐴 and 𝐵∈𝑅^𝑠×𝑙 is defined as 𝐴∘𝐵=∑^𝑠_𝑖=1∑^𝑙_𝑗=1𝐴_𝑖,𝑗𝐵_𝑖,𝑗. Therefore, we can easily solve the above linear regression model for 𝑝=1,2 (the least square regression) and +∞.

To relax the weak exogeneity assumption, we assume that the real data are contained in the following uncertainty set: 𝑈=𝑋;𝑌^𝑇∶𝑥_𝑖=𝑥_𝑖+Δ𝑥_𝑖,𝑦_𝑖=𝑦_𝑖+Δ𝑦_𝑖,𝑖=1,…,𝑚,Δ𝑦_𝑖;Δ𝑥_{𝑖𝑖=1,…,𝑚2}≤𝛿. (5)

To minimize the worst-case residual error, we establish the following robust quadratic regression model: min_{𝛼,𝛽,𝑄}max_{[𝑋;𝑌𝑇]∈𝑈}𝑦₁−𝑥^𝑇₁𝑄𝑥₁−2𝛼^𝑇𝑥₁⋮𝑦−𝛽_𝑚−𝑥^𝑇_𝑚𝑄𝑥_𝑚−2𝛼^𝑇𝑥_𝑚−𝛽_𝑝. (6)

From the computational perspective, although the robust linear regression problem (where the coefficients 𝑄 are set to zero) with a large variety of uncertainty sets can be efficiently solved, the robust quadratic regression problems are much more difficult. Actually for general uncertainty sets and least square criteria, even the inner maximization problem, which includes convex biquadratic polynomial as the objective function and general convex set as feasible set, is in general not solvable in polynomial run time. Next we will introduce some meaningful uncertainty sets and provide the corresponding tractable equivalences.

2.2. Separable Ball Uncertainty Sets Model

In this subsection, we consider the following separable ball uncertainty set: 𝑈_𝑠=𝑈₁×𝑈₂×⋯×𝑈_𝑚, (7) where 𝑈_𝑖=𝑥_𝑖;𝑦_𝑖∈𝑅^𝑛+1∶𝑥_𝑖=𝑥_𝑖+Δ𝑥_𝑖,𝑦_𝑖=𝑦_𝑖+Δ𝑦_𝑖,Δ𝑦_𝑖;Δ𝑥_𝑖2≤𝛿_𝑖, (8) and 𝛿_𝑖≥0. Thus the inner problem (IP) is of the following form (here we first consider square of the original objective function): (IP)max^𝑚_𝑖=1(𝑦_𝑖−𝑥^𝑇_𝑖𝑄𝑥_𝑖−2𝛼^𝑇𝑥_𝑖−𝛽)²s.t.𝑥_𝑖;𝑦_𝑖∈𝑈_𝑖,𝑖=1,…,𝑚. (9)

Note that for the inner problem, the separable uncertainty set and the summation form of the objective function allow us to decompose it into 𝑚 small scale subproblems with quadratic objective function and ball constraints. The quadratic objective function and constraints motivate us to use the following S-lemma to obtain an equivalent solvable reformulation.

Lemma 1 (inhomogeneous version of S-lemma [8]). Let 𝐴,𝐵 be symmetric matrices of the same size, and let the quadratic form 𝑥^𝑇𝐴𝑥+2𝑎^𝑇𝑥+𝛽 be strictly positive at some point. Then the implication 𝑥^𝑇𝐴𝑥+2𝑎^𝑇𝑥+𝛽≥0⟹𝑥^𝑇𝐵𝑥+2𝑏^𝑇𝑥+𝛽≥0 (10) holds true if and only if ∃𝜆≥0∶𝐵−𝜆𝐴(𝑏−𝜆𝑎)^𝑇𝑏−𝜆𝑎𝛽−𝜆𝛼⪰0. (11)

We can obtain the following equivalent semidefinite programming for the separable robust least square quadratic regression model.

Proposition 2. The robust least square quadratic regression model with separable uncertainty set 𝑈_𝑠 is equivalent to the following semidefinite programming: min_{𝛼,𝛽,𝑄,𝑣,𝑢,𝑟,𝑡,𝜏}𝑣𝑢s.t.^𝑖−𝑡_𝑖𝛿²_𝑖𝑡_𝑖𝑡_𝑖𝐼_𝑛×𝑛+𝐹⪰0,𝑖∈𝑀₊,𝑢_𝑖−𝜏_𝑖𝛿²_𝑖𝜏_𝑖𝜏_𝑖𝐼_𝑛×𝑛−𝐹⪰0,𝑖∈𝑀₊,𝑟_𝑖+𝑢_𝑖≥0,𝑟_𝑖−𝑢_𝑖≥0,𝑖∈𝑀₀,𝑟_𝑖=𝛽+𝑥^𝑇_𝑖𝑄𝑥_𝑖+2𝛼^𝑇𝑥_𝑖−𝑦_𝑖,𝑖=1,…,𝑚,𝑣𝑢^𝑇𝑢𝑣𝐼_𝑚×𝑚𝑡⪰0,_𝑖≥0,𝜏_𝑖≥0,𝑢_𝑖,𝑟_𝑖∈𝑅,𝑖=1,…,𝑚,𝑣,𝛽∈𝑅,𝛼∈𝑅^𝑛,𝑄∈𝑅^𝑛×𝑛, (12) where 𝑟𝐹=^𝑖−12(𝑄^𝑇𝑥_𝑖+𝛼)^𝑇−1200_1×𝑛𝑄^𝑇𝑥_𝑖+𝛼0_𝑛×1𝑄,𝑀₊=𝑖∶𝛿_𝑖{},𝑀>01,…,𝑚₀=𝑖∶𝛿_𝑖=0{1,…,𝑚}. (13)

Proof. First consider the inner maximization subproblem. It is obvious that IP_𝑖max𝑦_𝑖−𝑥^𝑇_𝑖𝑄𝑥_𝑖−2𝛼^𝑇𝑥_𝑖−𝛽²s.t.𝑥_𝑖;𝑦_𝑖∈𝑈_𝑖,⟺min𝑢²_𝑖s.t.𝑢_𝑖≥𝑦_𝑖−𝑥^𝑇_𝑖𝑄𝑥_𝑖−2𝛼^𝑇𝑥_𝑖−𝛽,∀𝑥_𝑖;𝑦_𝑖∈𝑈_𝑖. (14)
If 𝛿_𝑖=0, we have that 𝑥^𝑇_𝑖𝑄𝑥_𝑖+2𝛼^𝑇𝑥_𝑖+𝛽−𝑦_𝑖+𝑢_𝑖∀𝑥≥0,_𝑖;𝑦_𝑖∈𝑈_𝑖⟺𝑟_𝑖+𝑢_𝑖𝑥≥0,_𝑇𝑖𝑄𝑥_𝑖+2𝛼^𝑇𝑥_𝑖+𝛽−𝑦_𝑖−𝑢_𝑖∀𝑥≥0,_𝑖;𝑦_𝑖∈𝑈_𝑖⟺𝑟_𝑖−𝑢_𝑖≥0, (15) where 𝑟_𝑖=𝑥^𝑇_𝑖𝑄𝑥_𝑖+2𝛼^𝑇𝑥_𝑖+𝛽−𝑦_𝑖.
If 𝛿_𝑖>0, we can utilize the S-lemma as follows: 𝑥^𝑇_𝑖𝑄𝑥_𝑖+2𝛼^𝑇𝑥_𝑖+𝛽−𝑦_𝑖+𝑢_𝑖≥0,∀𝑥_𝑖;𝑦_𝑖∈𝑈_𝑖,⟺Δ𝑥^𝑇_𝑖𝑄Δ𝑥_𝑖+2𝑄^𝑇𝑥_𝑖+𝛼^𝑇Δ𝑥_𝑖−Δ𝑦_𝑖+𝑟_𝑖+𝑢_𝑖∀≥0,Δ𝑦_𝑖;Δ𝑥_𝑖2≤𝛿_𝑖,⟺1Δ𝑦_𝑖Δ𝑥_𝑖^𝑇𝑢_𝑖+𝑟_𝑖−12𝑄^𝑇𝑥_𝑖+𝛼^𝑇−1200_1×𝑛𝑄^𝑇𝑥_𝑖+𝛼0_𝑛×1𝑄×1Δ𝑦_𝑖Δ𝑥_𝑖≥0,∀Δ𝑦_𝑖;Δ𝑥_𝑖2≤𝛿_𝑖,⟺∃𝑡_𝑖𝑢≥0,s.t.^𝑖+𝑟_𝑖−𝑡_𝑖𝛿²_𝑖−12𝑄^𝑇𝑥_𝑖+𝛼^𝑇−12𝑡_𝑖0_1×𝑛𝑄^𝑇𝑥_𝑖+𝛼0_𝑛×1𝑄+𝑡_𝑖𝐼_𝑛×𝑛⪰0. (16) Note that in the last step, if 𝛿²_𝑖>0, then there exists (Δ𝑦_𝑖;Δ𝑥_𝑖)=(0;0) such that quadratic form 𝛿²_𝑖−‖(Δ𝑦_𝑖;Δ𝑥_𝑖)‖²₂ is strictly positive; thus the condition of S-lemma holds truly. Similarly we have that 𝑥^𝑇_𝑖𝑄𝑥_𝑖+2𝛼^𝑇𝑥_𝑖+𝛽−𝑦_𝑖−𝑢_𝑖≤0,∀𝑥_𝑖;𝑦_𝑖∈𝑈_𝑖,⟺Δ𝑥^𝑇_𝑖𝑄Δ𝑥_𝑖+2𝑄^𝑇𝑥_𝑖+𝛼^𝑇Δ𝑥_𝑖−Δ𝑦_𝑖+𝑟_𝑖−𝑢_𝑖∀≤0,Δ𝑦_𝑖;Δ𝑥_𝑖2≤𝛿_𝑖,⟺∃𝜏_𝑖𝑢≥0,s.t.^𝑖−𝑟_𝑖−𝜏_𝑖𝛿²_𝑖12−𝑄^𝑇𝑥_𝑖+𝛼^𝑇12𝜏_𝑖0_1×𝑛−𝑄^𝑇𝑥_𝑖−𝛼0_𝑛×1𝜏_𝑖𝐼_𝑛×𝑛−𝑄⪰0. (17) Thus the inner maximization problem is equivalent to the following semi-definite programming: (IP)min∑_{𝑢𝑖,𝑟𝑖,𝑡𝑖,𝜏𝑖}^𝑚_𝑖=1𝑢²_𝑖𝑢s.t.^𝑖+𝑟_𝑖−𝑡_𝑖𝛿²_𝑖−12𝑄^𝑇𝑥_𝑖+𝛼^𝑇−12𝑡_𝑖0_1×𝑛𝑄^𝑇𝑥_𝑖+𝛼0_𝑛×1𝑄+𝑡_𝑖𝐼_𝑛×𝑛⪰0,𝑖∈𝑀₊,𝑢_𝑖−𝑟_𝑖−𝜏_𝑖𝛿²_𝑖12−𝑄^𝑇𝑥_𝑖+𝛼^𝑇12𝜏_𝑖0_1×𝑛−𝑄^𝑇𝑥_𝑖−𝛼0_𝑛×1𝜏_𝑖𝐼_𝑛×𝑛−𝑄⪰0,𝑖∈𝑀₊,𝑟_𝑖+𝑢_𝑖≥0,𝑟_𝑖−𝑢_𝑖≥0𝑖∈𝑀₀,𝑟_𝑖=𝛽+𝑥^𝑇_𝑖𝑄𝑥_𝑖+2𝛼^𝑇𝑥_𝑖−𝑦_𝑖𝑡𝑖=1,…,𝑚,_𝑖≥0,𝜏_𝑖≥0,𝑢_𝑖,𝑟_𝑖∈𝑅𝑖=1,…,𝑚. (18) Note that based on the Schur complement lemma, the second-order cone constraint 𝑣≥∑^𝑚_𝑖=1𝑢²_𝑖 can also be formalized as the following semi-definite constraint: 𝑣𝑢^𝑇𝑢𝑣𝐼_𝑚×𝑚⪰0. (19) Thus we complete the proof by embedding the equivalent semi-definite programming into the outer problem.

Due to the advance of interior algorithms for conic programming, the above semidefinite programming can be efficiently solved in polynomial run time. There are several efficient and free software packages for solving the semidefinite programming, such as the SDPT3 [19]. Next we make several extensions based on the separable robust least square quadratic regression model.

2.3. Ellipsoid Uncertainty Set and More Norm Criterion

The above result on standard ball uncertainty set can be further extended to that on the following general ellipsoid uncertainty set: 𝑈^′_𝑠=𝑈^′₁×𝑈^′₂×⋯×𝑈^′_𝑚,𝑈^′_𝑖=𝑥_𝑖;𝑦_𝑖∈𝑅^𝑛+1∶𝑥_𝑖=𝑥_𝑖+Δ𝑥_𝑖,𝑦_𝑖=𝑦_𝑖+Δ𝑦_𝑖,𝑃_𝑖Δ𝑦_𝑖;Δ𝑥_𝑖2≤𝛿_𝑖, (20) where 𝑃_𝑖∈𝑅^{𝑘×(𝑛+1)}. Linear transformation operator 𝑃_𝑖 allows us to impose more restrictions on the uncertainty set. For example, if we choose the diagonal matrix 𝑃_𝑖=Diag{𝜎₁,…,𝜎_𝑛+1}, we can put different weights on deviation of components of (𝑥_𝑖;𝑦_𝑖); general matrix can further restrict the correlated deviation of different components.

To obtain the corresponding reformulation, we only need to modify the first two constraints based on the S-lemma as follows: 𝑢_𝑖−𝑡_𝑖𝛿²_𝑖𝑡_𝑖𝑃^𝑇_𝑖𝑃_𝑖+𝐹⪰0𝑖∈𝑀₊,𝑢_𝑖−𝜏_𝑖𝛿²_𝑖𝜏_𝑖𝑃^𝑇_𝑖𝑃_𝑖−𝐹⪰0𝑖∈𝑀₊. (21)

We further consider the robust quadratic regression models with 𝑙_∞-norm and 𝑙₁-norm criterion. Note that for 𝑙_∞-norm criteria, the inner maximization problem is of the following form: max_{1≤𝑖≤𝑚}max_{𝑥𝑖;𝑦𝑖∈𝑈𝑖}𝑦_𝑖−𝑥^𝑇_𝑖𝑄𝑥_𝑖−2𝛼^𝑇𝑥_𝑖,𝑦−𝛽⟺min𝑢∶𝑢≥_𝑖−𝑥^𝑇_𝑖𝑄𝑥_𝑖−2𝛼^𝑇𝑥_𝑖,∀𝑥−𝛽_𝑖;𝑦_𝑖∈𝑈_𝑖,1≤𝑖≤𝑚. (22) And for 𝑙₁-norm criteria, we have the following equivalent reformulation: max_{1≤𝑖≤𝑚}^𝑚_𝑖=1𝑦_𝑖−𝑥^𝑇_𝑖𝑄𝑥_𝑖−2𝛼^𝑇𝑥_𝑖∶𝑥−𝛽_𝑖;𝑦_𝑖∈𝑈_𝑖,1≤𝑖≤𝑚,⟺min^𝑚_𝑖=1𝑢_𝑖∶𝑢_𝑖≥𝑦_𝑖−𝑥^𝑇_𝑖𝑄𝑥_𝑖−2𝛼^𝑇𝑥_𝑖,∀𝑥−𝛽_𝑖;𝑦_𝑖∈𝑈_𝑖,1≤𝑖≤𝑚. (23)

Using the similar approach as in Proposition 2, both can be further reformulated as semi-definite programming.

Proposition 3. The separable robust quadratic regression model under 𝑙_∞-norm and 𝑙₁-norm criteria are equivalent to the following semidefinite programming, respectively: 𝑙_∞-normmin_{𝛼,𝛽,𝑄,𝑢,𝑟,𝑡,𝜏}𝑢s.t.𝑢−𝑡_𝑖𝛿²_𝑖𝑡_𝑖𝑡_𝑖𝐼_𝑛×𝑛+𝐹⪰0,𝑖∈𝑀₊,𝑢−𝜏_𝑖𝛿²_𝑖𝜏_𝑖𝜏_𝑖𝐼_𝑛×𝑛−𝐹⪰0,𝑖∈𝑀₊,𝑟_𝑖+𝑢≥0,𝑟_𝑖−𝑢≥0,𝑖∈𝑀₀,𝑟_𝑖=𝛽+𝑥^𝑇_𝑖𝑄𝑥_𝑖+2𝛼^𝑇𝑥_𝑖−𝑦_𝑖𝑡,𝑖=1,…,𝑚,_𝑖≥0,𝜏_𝑖≥0,𝑟_𝑖∈𝑅,𝑖=1,…,𝑚,𝑢,𝛽∈𝑅,𝛼∈𝑅^𝑛,𝑄∈𝑅^𝑛×𝑛,𝑙₁-normmin^𝑚_{𝛼,𝛽,𝑄,𝑢,𝑟,𝑡,𝜏𝑖=1}𝑢_𝑖𝑢s.t.^𝑖−𝑡_𝑖𝛿²_𝑖𝑡_𝑖𝑡_𝑖𝐼_𝑛×𝑛+𝐹⪰0,𝑖∈𝑀₊,𝑢_𝑖−𝜏_𝑖𝛿²_𝑖𝜏_𝑖𝜏_𝑖𝐼_𝑛×𝑛−𝐹⪰0,𝑖∈𝑀₊,𝑟_𝑖+𝑢_𝑖≥0,𝑟_𝑖−𝑢_𝑖≥0,𝑖∈𝑀₀,𝑟_𝑖=𝛽+𝑥^𝑇_𝑖𝑄𝑥_𝑖+2𝛼^𝑇𝑥_𝑖−𝑦_𝑖,𝑡𝑖=1,…,𝑚,_𝑖≥0,𝜏_𝑖≥0,𝑢_𝑖,𝑟_𝑖∈𝑅𝑖=1,…,𝑚,𝛽∈𝑅,𝛼∈𝑅^𝑛,𝑄∈𝑅^𝑛×𝑛, (24) where 𝑟𝐹=^𝑖−12𝑄^𝑇𝑥_𝑖+𝛼^𝑇−1200_1×𝑛𝑄^𝑇𝑥_𝑖+𝛼0_𝑛×1𝑄. (25)

3. Robust Energy-Growth Regression Models

Studies have been reported on the causal relationship between economic growth and energy consumption. In this section, we try to apply the proposed robust quadratic regression model to the energy-growth problem.

The seminal paper of J. Kraft and A. Kraft [20] first studies the casual relationship for USA. In a recent survey, Ilhan [21] categorizes the casual relationships into four types: no causality, unidirectional causality running from economic growth to energy consumption, the reverse case, and the bidirectional causality. Note that the resulted relationships depend on the selected data and analysis approaches. Sometimes the results obtained from different approaches conflict with each other when even using the data from the same country. For example, using the Toda-Yamamoto causality test method, Bowden and Payne [22] show that energy consumption plays an important role in economic growth in USA based on history data from 1949 to 2006 while using the same method Soytas and Sari [23] find that no causality exists between them based on USA data from 1960 to 2006. On the other hand, based on the same USA's data from 1947 to 1990, Cheng [24] and Stern [25] conclude different causalities by utilizing different analyzing approaches.

Unlike the previous energy-growth studies, we attempt to provide a long-run stationary regression model between the per capital GDP (G) and per capital energy consumption (EC). The underlying assumption of our model is similar to the traditional “conservation hypothesis” that means that an increase in real GDP will cause an increase in energy consumption [21]. The “per capital” perspective provides us with a new insight on the causality and new regression models. Figures 1 and 2 demonstrate the relationship between per capital energy consumption and per capital GDP in USA and Germany respectively. From the subfigures on the left hand side, we can see that in both countries there is a gradual increase in economy while the per capital energy consumption may decrease after reaching a certain level; the subfigures on the right hand side inspire us to establish a nonlinear regression model to characterize the relationship.

(a)

(b)

Figure 1 

Germany data from 1960 to 2006.

(a)

(b)

Figure 2 

USA data from 1870 to 2006.

To eliminate effect of the imprecise statistics data, we employ the proposed robust quadratic regression model and put different weights on the residual errors at different time points. Specifically we establish the following weighted robust quadratic regression model: min_{𝛼,𝛽,𝑞}max_{(𝐺𝑡,𝐸𝐶𝑡)∈𝑈𝜖𝑡}^𝑇_𝑡=1𝑤_𝑡(EC_𝑡−𝑞𝐺²_𝑡−2𝛼𝐺_𝑡−𝛽)^𝑝1/𝑝, (26) where the weight factor 𝑤_𝑡∈[0,1] represents the relative importance of the predicted residual error in the 𝑡th year. We could set 𝑤_𝑡=0 for the abnormal data point and set 𝑤_𝑡 as an increase function of 𝑡 to emphasize the importance of recent data. The uncertainty set is defined as 𝑈^𝜖_𝑡=𝐺_𝑡,EC_𝑡∶𝐺_𝑡−𝐺_𝑡,EC_𝑡−EC_𝑡2≤𝛿_𝑡, (27) where 𝛿_𝑡=𝜀𝐺²_𝑡+EC²_𝑡. Parameter 𝜀 controls the relative amplitude of the fluctuation in observed data.

The weighted robust quadratic regression model can be summarized as follows.(1)Solve the classical quadratic regression model using the nominal values (𝐺_𝑡,EC_𝑡)^𝑇_𝑡=1. (2)Based on the quadratic regression, remove the data with the first 𝑘 largest residual errors and set weights value 𝑤_𝑡. (3)Solve the equivalent semi-definite programming problem and return the final weighted robust quadratic regression model.

4. Numerical Experiments

In this section, we verify the effectiveness of the proposed robust quadratic regression models on several data sets. The equivalent semi-definite programming problem is solved by the SDPT3 solver [19]. Numerical experiments are implemented using MATLAB 7.7.0 and run on Intel(R) Core(TM)2 CPU E7400.

First we test the proposed robust least square quadratic regression (LS-RQR) model with Germany data from 1960 to 2006. As previously discussed, after the preliminary quadratic regression analysis, we will remove the data with the first 𝑘 largest residual errors, where 𝑘=3%×datasize. Then for the rest of data, we establish the classical least square quadratic regression (LS-CQR) and LS-RQR models, respectively.

Table 1 lists the computation results for LS-CQR and LS-RQR with a series of 𝜖 values. The listed Err value represents the mean square error from the nominal value, and 𝑇 represents the run time for solving the optimization problem. It is seen that the resulted robust model exhibits smaller absolute values of 𝑄, 𝛼, and 𝛽 with the increase of 𝜖 value; that is, the regression curve is more flat as the model parameters are less precise. It is obvious that one drawback of the robust model is that the mean square error will increase as uncertainty increases. Figure 3 plots the regression curves for different models and also supports our analysis of the effect of increasing data uncertainty on robust regression.

Table 1 

LS-CQR and LS-RQR models with different 𝜖.

Figure 3 

LS-CQR and LS-RQR models on Germany data.

To demonstrate the effectiveness of the robust models, we test the worst-case performance of the resulted models when 𝜖 varies from 0 to 0.1. Specifically, for each 𝜖 value, we randomly generate 500 groups of data from the defined uncertainty set 𝑈^𝜖_𝑡 and then calculate the maximal residual error at each data point. Figure 4 plots the worst-case error of LS-CQR model and LS-RQR models with 𝜖=0.01,0.03, and 0.05. It is seen that the error of LS-CQR model increases rapidly, and LS-RQR with 𝜖=0.05 has the most flat error curve. Figure 4 also indicates that it is critical to accurately estimate the variability of the data and set proper value for 𝜖. In our case, we recommend LS-RQR with 𝜖=0.03 that is almost always better than the traditional LS-CQR model.

Figure 4 

Mean square error of LS-CQR and LS-RQR models when 𝜖 varies.

Next we test the proposed RQR models under 𝑙₁ (L1-RQR) and 𝑙_∞-(LI-RQR) norm criteria on the same data set. Figure 5 plots the corresponding regression curves for the same uncertainty set 𝜖=0.02. For the same 𝜖 value, LI-RQR model can be considered as the most robust one, and L1-RQR and L2-RQR models are similar. It is noticeable that it contradicts with the traditional robust regression terms. For example, [26] refers to the 𝑙₁-norm regression as the robust regression model in the sense that the corresponding model is insensitive to the large residual errors(corresponding to the outliers). However, after removing the possible abnormal data points, here we try to make our regression analysis insensitive to the worst-case residual errors at each data point.

Figure 5 

RQR models under 𝑙1-,𝑙2- and 𝑙∞-norm criteria.

Finally we apply the proposed RQR model on more data sets, including USA data from 1870 to 2006, Switzerland data from 1965 to 2006, and Belgium data from 1960 to 2006. Figures 6, 7, and 8 give the resulted regression models and the worst-case residual errors for different 𝜖 values. It is seen that the proposed RQR models still almost always outperform the CQR model, especially for large uncertainty sets. Based on the robust quadratic regression models, these three countries reach the highest per capital energy consumption points at per capital GDP value around 23,000 while the peak values vary from 5.7 to 8.5 Ton.

(a)

(b)

Figure 6 

USA data from 1870 to 2006.

(a)

(b)

Figure 7 

Switzerland data from 1965 to 2006.

(a)

(b)

Figure 8 

Belgium data from 1960 to 2006.

5. Conclusions and Future Works

In this paper, we studied the multivariate quadratic regression model with imprecise statistic data. Unlike the traditional robust statistic approaches that focus on the detection of the outliers and the elimination of the effects, we employed the recently developed robust optimization framework and uncertainty set theory.

In particular, we first extended the existing robust linear regression results to the robust least square quadratic regression model with the separable ball uncertainty set. The specific form of the uncertainty set allowed us to use the well-known S-lemma and give the tractable equivalent semidefinite programming. We further generalized the result to robust models under 𝑙₁- and 𝑙_∞-norm criteria with general ellipsoid uncertainty sets. Next the proposed robust models were applied to the energy-growth problem. Under the classical conservation hypothesis, we employed the traditional quadratic regression model to remove the abnormal data and established a robust quadratic regression model for the per capital GDP and per capital energy consumption. Finally the proposed models were tested on the history data of Germany, USA, Switzerland, and Belgium. From the numerical experiments, we found that (1) the amplitude of the uncertainty perturbation 𝛿 plays a critical role on the robust models; (2) with the increase of 𝛿, the robust model has a more flat curve; (3) for the same 𝛿 value, compared with 𝑙₁- and 𝑙₂-norm models, 𝑙_∞-norm model is the most robust one; (4) as expected, the robust approach provides a serial robust regression models that can reduce the worst-case residual errors when the observed data contain noise.

For further research, robust polynomial (nonlinear) regression models are interesting in their own right. Although we may always reduce them to the linear regression model with polynomially (or nonlinearly) transformed uncertainty data set, it is still worth studying whether the resulted regression models are solvable for quadratic regression with coupled uncertainty sets.

Acknowledgment

This work was supported by Geological Survey Project of China (nos. 1212010881801, 1212011120995).