The Oracle Inequalities on Simultaneous Lasso and Dantzig Selector in High-Dimensional Nonparametric Regression

Abstract

During the last few years, a great deal of attention has been focused on Lasso and Dantzig selector in high-dimensional linear regression when the number of variables can be much larger than the sample size. Under a sparsity scenario, the authors (see, e.g., Bickel et al., 2009, Bunea et al., 2007, Candes and Tao, 2007, Candès and Tao, 2007, Donoho et al., 2006, Koltchinskii, 2009, Koltchinskii, 2009, Meinshausen and Yu, 2009, Rosenbaum and Tsybakov, 2010, Tsybakov, 2006, van de Geer, 2008, and Zhang and Huang, 2008) discussed the relations between Lasso and Dantzig selector and derived sparsity oracle inequalities for the prediction risk and bounds on the 𝐿_𝑝 estimation loss. In this paper, we point out that some of the authors overemphasize the role of some sparsity conditions, and the assumptions based on this sparsity condition may cause bad results. We give better assumptions and the methods that avoid using the sparsity condition. As a comparison with the results by Bickel et al., 2009, more precise oracle inequalities for the prediction risk and bounds on the 𝐿_𝑝 estimation loss are derived when the number of variables can be much larger than the sample size.

1. Introduction

During the last few years, a great deal of attention has been focused on the 𝐿₁ penalized least squares (Lasso) estimator of parameters in high-dimensional linear regression when the number of variables can be much larger than the sample size (e.g., see [1–12]). Quite recently, Candes and Tao [13] have proposed the Dantzig estimate for such linear models, and other authors [1, 6, 14–22] have discussed the Dantzig estimate and established the properties under a sparsity scenario, that is, when the number of nonzero components of the true vector of parameters is small.

Lasso estimators have also been studied in the nonparametric regression setup (see [23–26]). In particular, Bunea et al. [23, 24] obtain sparsity oracle inequalities for the prediction loss in this context and point out the implications for minimax estimation in classical nonparametric regression settings as well as for the problem of aggregation of estimators. Modified versions of Lasso estimators (nonquadratic terms and/or penalties slightly different from 𝐿₁) for nonparametric regression with random design are suggested and studied under prediction loss in Koltchinskii [27] and van de Geer [28]. Sparsity oracle inequalities for the Dantzig selector with random design are obtained by Koltchinskii [29]. In linear fixed design regression, Meinshausen and Yu [7] establish a bound on the 𝐿₂ loss for the coefficients of Lasso that are quite different from the bound on the same loss for the Dantzig selector proven in Candes and Tao [13]. Bickel et al. [15] show that, under a sparsity scenario, the Lasso and the Dantzig selector exhibit similar behavior, both for linear regression and for nonparametric regression models, for 𝐿₂ prediction loss, and for 𝐿_𝑝 loss in the coefficients for 1≤𝑝≤2. In the nonparametric regression model, they prove sparsity oracle inequalities for the Lasso and the Dantzig selector. Moreover, the Lasso and the Dantzig selector are approximately equivalent in terms of the prediction loss. They develop geometrical assumptions that are considerably weaker than those of Candes and Tao [13] for the Dantzig selector and Bunea et al. [23] for the Lasso.

We give the assumptions equivalent with assumptions by Bickel et al. [15] and derive oracle inequalities that are more precise than Bickel et al.’s [15] for the prediction risk in the general nonparametric regression model and bounds that are more precise than Bickel et al.’s [15] on the 𝐿_𝑝 estimation loss in the linear model when the number of variables can be much larger than the sample size. We begin, in the next section, by defining the Lasso and Dantzig procedures and the notation. In Section 3, we present our key three assumptions and discuss the relations between the assumptions and assumptions by Bickel et al. [15]. In Section 4, we give some equivalent results and sparsity oracle inequalities for the Lasso and Dantzig estimators in the general nonparametric regression model and improve corresponding results by Bickel et al. [15]. The concluding remarks are given in Section 5.

2. Definitions and Notations

Unless stated otherwise, all of our notations, definitions, and terminologies follow Bickel et al. [15]. Let (𝑍₁,𝑌₁),…,(𝑍_𝑛,𝑌_𝑛) be a sample of independent random pairs with 𝑌_𝑖=𝑓𝑍_𝑖+𝑊_𝑖,𝑖=1,…,𝑛, (1) where 𝑓∶ℤ→ℝ is an unknown regression function to be estimated, ℤ is a Borel subset of ℝ^𝑑, the 𝑍_𝑖’s are fixed elements in ℤ, and the regression errors 𝑊_𝑖 are Gaussian. Let 𝐹_𝑀={𝑓₁,…,𝑓_𝑀} be a finite dictionary of functions 𝑓_𝑗∶ℤ→ℝ, 𝑗=1,…,𝑀. We assume throughout that 𝑀≥2.

Consider the matrix 𝑋=(𝑓_𝑗(𝑍_𝑖)), 𝑖=1,…,𝑛, 𝑗=1,…,𝑀 and the vectors 𝐲=(𝑌₁,…,𝑌_𝑛)^𝑇, 𝐟=(𝑓(𝑍₁),…,𝑓(𝑍_𝑛))^𝑇, and 𝐰=(𝑊₁,…,𝑊_𝑛)^𝑇. With the notation 𝐲=𝐟+𝐰, (2) we will write |𝑥|_𝑝 for the 𝐿_𝑝 norm of 𝑥∈ℝ^𝑀, 1≤𝑝≤∞. The notation ‖⋅‖_𝑛 stands for the empirical norm 𝑔_𝑛=1𝑛𝑔²₂ (3) for any 𝑔∶ℤ→ℝ. We suppose that ‖𝑓_𝑗‖_𝑛≠0, 𝑗=1,…,𝑀. Set 𝑓_max=max_{1≤𝑗≤𝑀}𝑓_𝑗𝑛,𝑓_min=min_{1≤𝑗≤𝑀}𝑓_𝑗𝑛. (4)

For any 𝛽=(𝛽₁,…,𝛽_𝑀)^𝑇∈ℝ^𝑀 and 𝑍∈ℤ, define 𝑓_𝛽(𝑍)=∑^𝑀_𝑗=1𝛽_𝑗𝑓_𝑗(𝑍) and 𝐟_𝛽=(𝑓_𝛽(𝑍₁),…,𝑓_𝛽(𝑍_𝑛))^𝑇=𝑋𝛽. The estimates we consider are all of the form 𝑓𝛽(⋅), where 𝛽 is data determined. Since we consider mainly sparse vectors 𝛽, it will be convenient to define the following. Let 𝑀𝛽=^𝑀_𝑗=1𝐼_{{𝛽𝐽≠0}}=𝐽𝛽 (5) denote the number of nonzero coordinates of 𝛽, where 𝐼_{⋅} denotes the indicator function, 𝐽(𝛽)={𝑗∈{1,…,𝑀}∶𝛽_𝑗≠0}, and |𝐽| denotes the cardinality of 𝐽. For a vector 𝛿∈ℝ^𝑀 and a subset 𝐽⊂{1,…,𝑀}, we denote by 𝛿_𝐽 the vector in ℝ^𝑀 that has the same coordinates as 𝛿 on 𝐽 and zero coordinates on the complement 𝐽^𝑐 of 𝐽.

Define the Lasso solution 𝛽_𝐿=(𝛽_1,𝐿,…,𝛽_𝑀,𝐿)^𝑇 by 𝛽_𝐿=argmin_{𝛽∈ℝ𝑀}1𝑛𝑦−𝑋𝛽²₂+2𝑟^𝑀_𝑗=1𝑓_𝑗𝑛𝛽_𝑗, (6) where 𝑟>0 is some tuning constant, and introduce the corresponding Lasso estimator 𝑓_𝐿(𝑍)𝛽=𝑓_𝐿(𝑍)=^𝑀_𝑗=1𝛽_𝑗,𝐿𝑓_𝑗(𝑍). (7) The Dantzig selector is defined by 𝛽_𝐷=argmin𝛽₁∶1𝑛𝐷^−1/2𝑋^𝑇𝑦−𝑋𝛽_∞≤𝑟, (8) where 𝐷 is the diagonal matrix 𝐷=diag𝑓₁²_𝑛,𝑓₂²_𝑛,…,𝑓_𝑀²_𝑛. (9) The Dantzig estimator is defined by 𝑓_𝐷(𝑍)𝛽=𝑓_𝐷(𝑍)=^𝑀_𝑗=1𝛽_𝑗,𝐷𝑓_𝑗(𝑍), (10) where 𝛽_𝐷=(𝛽_1,𝐷,…,𝛽_𝑀,𝐷)^𝑇 is the Dantzig selector.

We refer to Bickel et al. [15] for detailed discussion of the Dantzig constraint and the constraint that the Lasso selector satisfies.

Finally, for any 𝑛≥1, 𝑀≥2, we consider the Gram matrix Ψ_𝑛=1𝑛𝑋^𝑇1𝑋=𝑛^𝑛_𝑖=1𝑓_𝑗𝑍_𝑖𝑓_𝑘𝑍_𝑖,1≤𝑗,𝑘≤𝑀, (11) and let 𝜙_max denote the maximal eigenvalue of Ψ_𝑛.

3. Discussion of the Assumptions

Under the sparsity scenario, we are typically interested in the case where 𝑀>𝑛 and even 𝑀≫𝑛. Here, sparsity specifies that the high-dimensional vector 𝛽 has coefficients that are mostly 0. Clearly, the matrix Ψ_𝑛 is degenerate, and ordinary least squares do not work in this case, since the require positive definiteness of Ψ_𝑛. That is, min_{𝛿∈ℝ𝑀,𝛿≠0}|𝑋𝛿|₂√𝑛|𝛿|₂>0. (12) It turns out that the Lasso and Dantzig selector require much weaker assumptions. The idea by Bickel et al. [15] is that the minimum in (12) be replaced by the minimum over a restricted set of vectors, and the norm |𝛿|₂ in the denominator of the condition be replaced by the 𝐿₂ norm of only a part of 𝛿. This is feasible. Because for the linear regression model, the residuals 𝛿=𝛽_𝐿−𝛽 and 𝛿=𝛽_𝐷−𝛽 satisfy 𝛿_𝐽^𝑐₀₁≤𝑐₀𝛿_𝐽01 (13) with 𝑐₀=1 by Candes and Tao [13] and 𝑐₀=3 by Bickel et al. [15], respectively, where 𝑐₀>0 and 𝐽₀=𝐽(𝛽) is the set of nonzero coefficients of the true parameter 𝛽 of the model; therefore, for any 𝛿 satisfying (13), we have 𝛿^𝑇Ψ_𝑛𝛿|𝛿|²₂≥𝛿^𝑇Ψ𝛿|𝛿|²₂−𝜀_𝑛1+𝑐₀²𝐽₀, (14) where Ψ is a positive definite matrix and 𝜀_𝑛=max|(Ψ_𝑛−Ψ)_𝑖𝑗|. Thus, we have a kind of “restricted” positive definiteness if 𝜀_𝑛|𝐽₀| is small enough. This results in the following restricted eigenvalue (RE) assumption.

Assumption RE(𝑠,𝑐₀) (Bickel et al. [15]). For some integer 𝑠 such that 1≤𝑠≤𝑀 and a positive number 𝑐₀, the following condition holds: 𝜅𝑠,𝑐₀≜min_{𝐽0⊆{},𝐽1,2,…,𝑀0≤𝑠}min_{𝛿𝛿≠0,𝑐𝐽01≤𝑐0𝛿𝐽01}|𝑋𝛿|₂√𝑛𝛿_𝐽02>0. (15) The purpose of giving this assumption may be in order to facilitate the use of |𝛿_𝐽^𝑐₀|₁≤𝑐₀|𝛿_𝐽0|₁ since they frequently use it in the proofs of their theorems and so do Candes and Tao [13].

Note that the role of |𝛿_𝐽^𝑐₀|₁≤𝑐₀|𝛿_𝐽0|₁ is only to restrict set of vectors; that is, {𝛿∈ℝ^𝑀∶𝛿≠0} restricts to {𝛿∈ℝ^𝑀∶𝛿≠0,|𝛿_𝐽^𝑐₀|₁≤𝑐₀|𝛿_𝐽0|₁}. Therefore, it is not necessary that the norm |𝛿|₂ in the denominator of (12) be replaced by the 𝐿₂ norm of only a part of 𝛿. We give the following assumptions.

Assumption RE𝜏₁(𝑠,𝑐₀). For some integer 𝑠 such that 1≤𝑠≤𝑀 and a positive number 𝑐₀, the following condition holds: 𝜏₁𝑠,𝑐₀≜min_{𝐽0⊆{},𝐽1,2,…,𝑀0≤𝑠}min_{𝛿𝛿≠0,𝑐𝐽01≤𝑐0𝛿𝐽01}|𝑋𝛿|₂√𝑛𝛿_𝐽01>0. (16)

Assumption RE𝜏₂(𝑠,𝑐₀). For some integer 𝑠 such that 1≤𝑠≤𝑀 and a positive number 𝑐₀, the following condition holds: 𝜏₂𝑠,𝑐₀≜min_{𝐽0⊆{},𝐽1,2,…,𝑀0≤𝑠}min_{𝛿𝛿≠0,𝑐𝐽01≤𝑐0𝛿𝐽01}|𝑋𝛿|₂√𝑛|𝛿|₁>0. (17)

Assumption RE𝜏₃(𝑠,𝑐₀). For some integer 𝑠 such that 1≤𝑠≤𝑀 and a positive number 𝑐₀, the following condition holds: 𝜏₃𝑠,𝑐₀≜min_{𝐽0⊆{},𝐽1,2,…,𝑀0≤𝑠}min_{𝛿𝛿≠0,𝑐𝐽01≤𝑐0𝛿𝐽01}|𝑋𝛿|₂√𝑛|𝛿|₂>0. (18)

Note that 𝜏₂(𝑠,𝑐₀)≤𝜏₁(𝑠,𝑐₀)≤𝜅(𝑠,𝑐₀) and 𝜏₂(𝑠,𝑐₀)≤𝜏₃(𝑠,𝑐₀)≤𝜅(𝑠,𝑐₀) since |𝛿_𝐽0|₂≤|𝛿_𝐽0|₁≤|𝛿|₁ and |𝛿_𝐽0|₂≤|𝛿|₂≤|𝛿|₁. Moreover, it is easy to see that for fixed 𝑛, the four assumptions are equivalent, and those assumptions 1–5 by Bickel et al. [15] are all sufficient conditions for assumptions RE𝜏₁(𝑠,𝑐₀), RE𝜏₂(𝑠,𝑐₀), and RE𝜏₃(𝑠,𝑐₀).

In Section 4, we will see that RE𝜏₁(𝑠,𝑐₀) and RE𝜏₂(𝑠,𝑐₀) are all better than 𝜅(𝑠,𝑐₀) since they use |𝛿_𝐽^𝑐₀|₁≤𝑐₀|𝛿_𝐽0|₁ and |𝛿_𝐽0|₁≤|𝐽₀|^1/2|𝛿_𝐽0|₂ as little as possible. Therefore, the inequalities given are more precise.

4. Comparisons with the Results by Bickel et al.

In the following, we give a bound of the prediction losses ‖𝑓_𝐿−𝑓‖²_𝑛 with respect to ‖𝑓_𝐷−𝑓‖²_𝑛 when the number of nonzero components of the Lasso or the Dantzig selector is small as compared to the sample size.

Theorem 1. Let 𝑊_𝑖 be independent 𝑁(0,𝜎²) random variables with 𝜎²>0. Fix 𝑛≥1, 𝑀≥2. Let assumption RE(𝑠,𝑐₀) or RE𝜏₁(𝑠,𝑐₀) be satisfied with 1≤𝑠≤𝑀, where 𝑐₀>0, and let ‖𝑓_𝑗‖_𝑛=1, 𝑗=1,…,𝑀. Consider the Lasso estimator 𝑓_𝐿 defined by (6)-(7) with 𝑟=𝐴𝜎log𝑀𝑛, (19) where 𝐴>2√2, and consider the Dantzig estimator 𝑓_𝐷 defined by (10) with the same 𝑟. If 𝑀(𝛽_𝐷)≤𝑠, then, with probability at least 1−𝑀^1−𝐴2/8, one has 𝑓_𝐿−𝑓²_𝑛≤𝑓_𝐷−𝑓²_𝑛+9𝐴²𝜎²𝑛log𝑀𝜏²₁𝑠,𝑐₀≤𝑓_𝐷−𝑓²_𝑛+9𝐴²𝑀𝛽_𝐷𝜎²𝑛log𝑀𝜅²𝑠,𝑐₀. (20)

Proof. Set 𝛿=𝛽_𝐿−𝛽_𝐷. We apply (B.1) by Bickel et al. [15] with 𝛽=𝛽_𝐷, which yields that, with probability at least 1−𝑀^1−𝐴2/8, 𝑓_𝐿−𝑓²_𝑛≤𝑓_𝐷−𝑓²_𝑛+4𝑟𝛿_𝐽01−𝑟|𝛿|₁, (21) where 𝐽₀=𝐽(𝛽_𝐷). From (B.16) by Bickel et al. [15], we have 𝑓_𝐿−𝑓²_𝑛≤𝑓_𝐷−𝑓²_𝑛+3𝑟|𝛿|₁−1𝑛|𝑋𝛿|²₂. (22) Then, 𝑓_𝐿−𝑓²_𝑛≤𝑓_𝐷−𝑓²_𝑛+3𝑟𝛿_𝐽01−1||4𝑛𝑋𝛿²₂.≤𝑓_𝐷−𝑓²_𝑛+3𝑟𝛿_𝐽01√𝑛|𝑋𝛿|₂²≤𝑓_𝐷−𝑓²_𝑛+9𝑟²𝜏²₁𝑠,𝑐₀≤𝑓_𝐷−𝑓²_𝑛+9𝑟²𝑀𝛽_𝐷𝜅²𝑠,𝑐₀. (23)

Corollary 2. Let the conditions of Theorem 1 hold, but with RE(𝑠,5) in place of RE(𝑠,𝑐₀). If 𝑀(𝛽_𝐷)≤𝑠, then, with probability at least 1−𝑀^1−𝐴2/8, one has 𝑓_𝐿−𝑓²_𝑛≤𝑓_𝐷−𝑓²_𝑛+9𝐴²𝑀𝛽_𝐷𝜎²𝑛log𝑀𝜅²().𝑠,5 (24)

This corollary greatly improves Theorem 5.2 by Bickel et al. [15]. The right-hand side of the inequality of Theorem 5.2 is 10𝑓_𝐷−𝑓²_𝑛+81𝐴²𝑀𝛽_𝐷𝜎²𝑛log𝑀𝜅²().𝑠,5 (25)

A general discussion of sparsity oracle inequalities can be found in Tsybakov [30]. Here, we prove a sparsity oracle inequality for the prediction loss of the Lasso estimators. Such inequalities have been recently obtained for the Lasso-type estimators in a number of settings, see [15, 23, 24, 27, 28].

Theorem 3. Let 𝑊_𝑖 be independent 𝑁(0,𝜎²) random variables with 𝜎²>0. Fix integers 𝑛≥1, 𝑀≥2, 1≤𝑠≤𝑀. Let assumption RE(𝑠,𝑐₀) or RE𝜏₁(𝑠,𝑐₀) be satisfied, where 𝑐₀>0. Consider the Lasso estimator 𝑓_𝐿 defined by (6)-(7) with 𝑟=𝐴𝜎log𝑀𝑛 (26) for some 𝐴>2√2. Then, with probability at least 1−𝑀^1−𝐴2/8, one has 𝑓_𝐿−𝑓²_𝑛≤𝑓_𝛽−𝑓²_𝑛+9𝑓²_max𝐴²𝜎²𝜏²₁𝑠,𝑐₀log𝑀𝑛≤𝑓_𝛽−𝑓²_𝑛+9𝑓²_max𝐴²𝜎²𝜅²𝑠,𝑐₀𝑀𝛽log𝑀𝑛. (27)

Proof. Fix an arbitrary 𝛽∈ℝ^𝑀 with 𝑀(𝛽)≤𝑠. Set 𝛿=𝐷^1/2(𝛽_𝐿−𝛽), 𝐽₀=𝐽(𝛽), where 𝐷^1/2=diag{||𝑓₁||_𝑛,…,||𝑓_𝑀||_𝑛}. On the event 𝐴 in p1723 by Bickel et al. [15], we get, from the first line in (B.1) by Bickel et al. [15], that 𝑓_𝐿−𝑓²_𝑛≤𝑓_𝛽−𝑓²_𝑛+4𝑟𝛿_𝐽01−𝑟|𝛿|₁. (28) Since 𝑓−𝑋𝛽_𝐿²₂=𝑓−𝑋𝛽²₂−2𝛿^𝑇𝐷^−1/2𝑋^𝑇𝑓−𝑋𝛽_𝐿−𝑋𝐷^−1/2𝛿²₂, (29) then 𝑓_𝐿−𝑓²_𝑛≤𝑓_𝛽−𝑓²_𝑛+2𝑛|𝛿|₁𝐷^−1/2𝑋^𝑇(𝑓−𝑋𝛽_𝐿)_∞−1𝑛𝑋𝐷^−1/2𝛿²₂. (30)
From (8) and (B.5) by Bickel et al. [15], we have 1𝑛𝐷^−1/2𝑋^𝑇(𝑓−𝑋𝛽_𝐿)_∞≤3𝑟2,1𝑛𝑋𝐷^−1/2𝛿²₂=1𝑛𝛿^𝑇𝐷^−1/2𝑋^𝑇𝑋𝐷^−1/21𝛿≥𝑛𝑓²_max|𝑋𝛿|²₂. (31) Thus, 𝑓_𝐿−𝑓²_𝑛≤𝑓_𝛽−𝑓²_𝑛+3𝑟|𝛿|₁−1𝑛𝑋𝐷^−1/2𝛿²₂≤𝑓_𝛽−𝑓²_𝑛+3𝑟|𝛿|₁−1𝑛𝑓²_max|𝑋𝛿|²₂. (32) From (28) and (32), we have 𝑓_𝐿−𝑓²_𝑛≤𝑓_𝛽−𝑓²_𝑛+3𝑟𝛿_𝐽01−14𝑛𝑓²_max|𝑋𝛿|²₂≤𝑓_𝛽−𝑓²_𝑛+3𝑟𝛿_𝐽01√𝑛𝑓_max|𝑋𝛿|₂²≤𝑓_𝛽−𝑓²_𝑛+9𝑟²𝑓²_max𝜏²₁𝑠,𝑐₀≤𝑓_𝛽−𝑓²_𝑛+9𝑟²𝑓²_max𝜅²𝑠,𝑐₀𝑀𝛽. (33)

Corollary 4. Let the conditions of Theorem 3 hold, but with RE(𝑠,3+4/𝜀) in place of RE(𝑠,𝑐₀). Then, with probability at least 1−𝑀^1−𝐴2/8, one has 𝑓_𝐿−𝑓²_𝑛≤𝑓_𝛽−𝑓²_𝑛+9𝑓²_max𝐴²𝜎²𝜅²()𝑀𝛽𝑠,3+4/𝜀log𝑀𝑛. (34)

This corollary greatly improves Theorem 6.1 by Bickel et al. [15]. The right-hand side of the inequality of Theorem 6.1 is (1+𝜀)inf_{𝛽∈ℝ𝑀,𝑀𝛽≤𝑠}𝑓_𝛽−𝑓²_𝑛+𝐶(𝜀)𝑓²_max𝐴²𝜎²𝜅²()𝑀𝛽𝑠,3+4/𝜀log𝑀𝑛, (35) where (1+𝜀)𝐶(𝜀)=4(𝜀+2)²/𝜀>9.

In the following, we assume that the vector of observations 𝐲=(𝑌₁,…,𝑌_𝑛)^𝑇 is of the form 𝐲=𝑋𝛽^∗+𝐰, (36) where 𝑋 is an 𝑛×𝑀 deterministic matrix, 𝛽^∗∈ℝ^𝑀, and 𝐰=(𝑊₁,…,𝑊_𝑛)^𝑇. We consider dimension 𝑀 that can be of order 𝑛 and even much larger. Then, 𝛽^∗ is, in general, not uniquely defined. For 𝑀>𝑛, if (36) is satisfied for 𝛽^∗=𝛽₀, then there exists an affine space 𝑈={𝛽^∗∶𝑋𝛽^∗=𝑋𝛽₀} of vectors satisfying (36). The Lasso estimator of 𝛽^∗ in (36) is defined by 𝛽_𝐿=argmin_{𝛽∈ℝ𝑀}1𝑛𝐲−𝑋𝛽²₂+2𝑟𝛽₁. (37) The correspondence between the notation here and that of the previous is 𝑓_𝛽²_𝑛=1𝑛𝑋𝛽²₂,𝑓_𝛽−𝑓²_𝑛=1𝑛𝑋(𝛽−𝛽^∗)²₂,𝑓_𝐿−𝑓²_𝑛=1𝑛𝑋(𝛽_𝐿−𝛽^∗)²₂. (38)

Theorem 5. Let 𝑊_𝑖 be independent 𝑁(0,𝜎²) random variables with 𝜎²>0. Let all the diagonal elements of the matrix 𝑋^𝑇𝑋/𝑛 be equal to 1, and let 𝑀(𝛽^∗)≤𝑠, where 1≤𝑠≤𝑀, 𝑛≥1, 𝑀≥2. Let assumption RE(𝑠,𝑐₀) or RE𝜏₁(𝑠,𝑐₀) be satisfied, where 𝑐₀>0. Consider the Lasso estimator 𝛽_𝐿 defined by (37) with 𝑟=𝐴𝜎log𝑀𝑛 (39) and 𝐴>2√2. Then, with probability at least 1−𝑀^1−𝐴2/8, one has 𝛽_𝐿−𝛽^∗₁≤4𝐴𝜏²₁𝑠,𝑐₀𝜎log𝑀𝑛≤4𝐴𝜅²𝑠,𝑐₀𝜎𝑠log𝑀𝑛,𝑋(𝛽_𝐿−𝛽^∗)²₂≤144𝐴²25𝜏²₁𝑠,𝑐₀𝜎²log𝑀≤144𝐴²25𝜅²𝑠,𝑐₀𝜎²𝑀𝛽𝑠log𝑀,_𝐿≤576𝜙_max25𝜏²₁𝑠,𝑐₀≤576𝜙_max25𝜅²𝑠,𝑐₀𝑠. (40)(41)(42)

Proof. Set 𝛿=𝛽_𝐿−𝛽^∗and 𝐽₀=𝐽(𝛽^∗). Using (B.1) and (B.2) by Bickel et al. [15], where we put 𝛽=𝛽^∗, 𝑟_𝑛,𝑗≡𝑟, and ‖𝑓_𝛽−𝑓‖_𝑛=0, we get that, on the event 𝐴 (i.e., with probability at least 1−𝑀^1−𝐴2/8), 1𝑛|𝑋𝛿|²₂≤4𝑟𝛿_𝐽01−𝑟|𝛿|₁. (43) From (B.16) by Bickel et al. [15], we have 2𝑛|𝑋𝛿|²₂≤3𝑟|𝛿|₁. (44) Then, 1𝑛|𝑋𝛿|²₂≤125𝑟𝛿_𝐽01. (45) By assumption RE(𝑠,𝑐₀) or RE𝜏₁(𝑠,𝑐₀), we obtain that, on 𝐴, 𝜏²₁𝑠,𝑐₀𝛿_𝐽0²₁≤1𝑛|𝑋𝛿|²₂≤125𝑟𝛿_𝐽01. (46) Thus, 𝛿_𝐽02≤𝛿_𝐽01≤125𝜏²₁𝑠,𝑐₀𝑟≤125𝜅²𝑠,𝑐₀𝑟√1𝑠,𝑛|𝑋𝛿|²₂≤14425𝜏²₁𝑠,𝑐₀𝑟²≤14425𝜅²𝑠,𝑐₀𝑟²𝑠. (47) From (43), we have 𝑟|𝛿|₁≤4𝑟𝛿_𝐽01−1𝑛|𝑋𝛿|²₂≤2𝑟𝛿_𝐽01√𝑛|𝑋𝛿|₂²≤4𝑟²𝜏²₁𝑠,𝑐₀≤4𝑠𝑟²𝜅²𝑠,𝑐₀. (48) Thus, |𝛿|₁≤4𝑟𝜏²₁𝑠,𝑐₀≤4𝑠𝑟𝜅²𝑠,𝑐₀. (49) The inequalities (49) and (47) coincide with (40) and (41), respectively. Next, (42) follows immediately from (B.3) in Bickel et al. [15] and (41).

Corollary 6. Let the conditions of Theorem 5 hold, but with RE(𝑠,3) in place of RE(𝑠,𝑐₀). Then, with probability at least 1−𝑀^1−𝐴2/8, one has 𝛽_𝐿−𝛽^∗₁≤4𝐴𝜅²(𝑠,3)𝜎𝑠log𝑀𝑛,𝑋(𝛽_𝐿−𝛽^∗)²₂≤144𝐴²25𝜅²()𝜎𝑠,3²𝑀𝛽𝑠log𝑀,_𝐿≤576𝜙_max25𝜅²𝑠,𝑐₀𝑠. (50)(51)(52)

This corollary improves Theorem 7.2 by Bickel et al. [15]. The right-hand sides of the inequalities of Theorem 7.2 are 16𝐴𝜅²(𝑠,3)𝜎𝑠log𝑀𝑛,16𝐴²𝜅²()𝜎𝑠,3²𝑠log𝑀,64𝜙_max𝜅²(𝑠,3)𝑠, (53) respectively. That is, they are 4, 25/9, and 25/9 times as large as (50)–(52), respectively.

5. Conclusions

We point out that |𝛿_𝐽^𝑐₀|₁≤𝑐₀|𝛿_𝐽0|₁ with 𝑐₀=1 by Candes and Tao [13] and 𝑐₀=3 by Bickel et al. [15] are only the sufficient condition of Lasso and Dantzig selector. Their role should not be overemphasized. That is, |𝛿_𝐽^𝑐₀|₁≤𝑐₀|𝛿_𝐽0|₁ should not be deliberately used in any case for solving inequality. We should use |𝛿_𝐽^𝑐₀|₁≤𝑐₀|𝛿_𝐽0|₁ as little as possible when proving inequalities.

In fact, the corresponding results have been enlarged due to the use of |𝛿_𝐽^𝑐₀|₁≤𝑐₀|𝛿_𝐽0|₁ when solving the problems of Lasso and Dantzig selector. When proving sparsity oracle inequalities for the prediction loss and bounds on the 𝐿_𝑝 estimation loss, using again |𝛿_𝐽^𝑐₀|₁≤𝑐₀|𝛿_𝐽0|₁ must be to enlarge the inequalities again and to result in reduced accuracy.

We have seen that RE𝜏₁(𝑠,𝑐₀) and RE𝜏₂(𝑠,𝑐₀) are all much better than 𝜅(𝑠,𝑐₀) since in RE𝜏₁(𝑠,𝑐₀) and RE𝜏₂(𝑠,𝑐₀) the use of |𝛿_𝐽^𝑐₀|₁≤𝑐₀|𝛿_𝐽0|₁ and |𝛿_𝐽0|₁≤|𝐽₀|^1/2|𝛿_𝐽0|₂ is less. Therefore, the inequalities given are more precise.