Properties and Iterative Methods for the 𝑄-Lasso

Abstract

We introduce the 𝑄-lasso which generalizes the well-known lasso of Tibshirani (1996) with 𝑄 a closed convex subset of a Euclidean m-space for some integer 𝑚≥1. This set 𝑄 can be interpreted as the set of errors within given tolerance level when linear measurements are taken to recover a signal/image via the lasso. Solutions of the 𝑄-lasso depend on a tuning parameter 𝛾. In this paper, we obtain basic properties of the solutions as a function of 𝛾. Because of ill posedness, we also apply 𝑙₁-𝑙₂ regularization to the 𝑄-lasso. In addition, we discuss iterative methods for solving the 𝑄-lasso which include the proximal-gradient algorithm and the projection-gradient algorithm.

1. Introduction

The lasso of Tibshirani [1] is the minimization problem: min_{𝑥∈ℝ𝑛}12‖𝐴𝑥−𝑏‖²₂+𝛾‖𝑥‖₁, (1) where 𝐴 is an 𝑚×𝑛 (real) matrix, 𝑏∈ℝ^𝑚, and 𝛾>0 is a tuning parameter. It is equivalent to the basis pursuit (BP) of Chen et al. [2]: min_{𝑥∈ℝ𝑛}‖𝑥‖₁subjectto𝐴𝑥=𝑏. (2) It is well known that both lasso and BP model a number of applied problems arising from machine learning, signal/image processing, and statistics, due to the fact that they promote the sparsity of a signal 𝑥∈ℝ^𝑛. Sparsity is popular phenomenon that occurs in practical problems since a solution may have a sparse representation in terms of an appropriate basis and therefore has been paid much attention.

Observe that both the lasso (1) and BP (2) can be viewed as the ℓ₁ regularization applied to the inverse linear system in ℝ^𝑛: 𝐴𝑥=𝑏. (3) In sparse recovery, the system (3) is underdetermined (i.e., 𝑚<𝑛 and often 𝑚≪𝑛 indeed). The theory of compressed sensing of Donoho [3] and Candès et al. [4, 5] makes a breakthrough that under certain conditions the underdetermined system (3) can determine a unique 𝑘-sparse solution. (Recall that a signal 𝑥∈ℝ^𝑛 is said to be 𝑘-sparse if the number of nonzero entries of 𝑥 is no bigger than 𝑘.)

However, due to errors of measurements, the system (3) is actually inexact: 𝐴𝑥≈𝑏. It turns out that the BP (2) is reformulated as min_{𝑥∈ℝ𝑛}‖𝑥‖₁subjectto‖𝐴𝑥−𝑏‖≤𝜀, (4) where 𝜀>0 is the tolerance level of errors and ‖⋅‖ is a norm on ℝ^𝑛 (often it is the ℓ_𝑝 norm ‖⋅‖_𝑝 for 𝑝=1,2,∞; a solution to (4) when the tolerance is measured by the ℓ_∞ norm ‖⋅‖_∞ is known as the Dantzig selector by Candès and Tao [6]; see also [7]).

Note that if we let 𝑄∶=𝐵_𝜀(𝑏) be the closed ball in ℝ^𝑚 around 𝑏 and with radius of 𝜀, then (4) is rewritten as min_{𝑥∈ℝ𝑛}‖𝑥‖₁subjectto𝐴𝑥∈𝑄. (5)

Let now 𝑄 be a nonempty closed convex subset of ℝ^𝑚 and let 𝑃_𝑄 be the projection from ℝ^𝑚 onto 𝑄. Then noticing the condition 𝐴𝑥∈𝑄 being equivalent to the condition 𝐴𝑥−𝑃_𝑄(𝐴𝑥)=0, we see that the problem (5) is solved via min_{𝑥∈ℝ𝑛}‖𝑥‖₁subjectto𝐼−𝑃_𝑄𝐴𝑥=0. (6) Applying the Lagrangian method, we arrive at the following equivalent minimization: min_{𝑥∈ℝ𝑛}12𝐼−𝑃_𝑄𝐴𝑥²₂+𝛾‖𝑥‖₁, (7) where 𝛾>0 is a Lagrangian multiplier (also interpreted as a regularization parameter).

Alternatively, we may view (7) as the ℓ₁ regularization of the inclusion 𝐴𝑥∈𝑄equivalently,theequation𝐼−𝑃_𝑄𝐴𝑥=0 (8) which extends the linear system (3) in an obvious way. We refer to the problem (7) as the 𝑄-lasso since it is the ℓ₁ regularization of inclusion (8) as lasso (1) is the ℓ₁ regularization of the linear system (3). Throughout the rest of this paper, we always assume that (8) is consistent (i.e., solvable).

𝑄-lasso (7) is also connected with the so-called split feasibility problem (SFP) of Censor and Elfving [8] (see also [9]) which is stated as finding a point 𝑥 with the property 𝑥∈𝐶,𝐴𝑥∈𝑄, (9) where 𝐶 and 𝑄 are closed convex subsets of ℝ^𝑛 and ℝ^𝑚, respectively. An equivalent minimization formulation of the SFP (9) is given as min_𝑥∈𝐶12𝐴𝑥−𝑃_𝑄𝐴𝑥²₂. (10) Its ℓ₁ regularization is given as the minimization min_𝑥∈𝐶12𝐴𝑥−𝑃_𝑄𝐴𝑥²₂+𝛾‖𝑥‖₁, (11) where 𝛾>0 is a regularization parameter. Problem (7) is a special case of (11) when the set of constraints, 𝐶, is taken to be the entire space ℝ^𝑛.

The purpose of this paper is to study the behavior, in terms of 𝛾>0, of solutions to the regularized problem (7). (We leave the more general problem (11) to further work, due to the fact that the involvement of another closed convex set 𝐶 brings some technical difficulties which are not easy to overcome.) We discuss iterative methods for solving the 𝑄-lasso, including the proximal-gradient method and the projection-gradient method, the latter being derived via a duality technique. Due to ill posedness, we also apply the ℓ₁-ℓ₂ regularization to the 𝑄-lasso.

2. Preliminaries

Let 𝑛≥1 be an integer and let ℝ^𝑛 be the Euclidean 𝑛-space. If 𝑝≥1, we use ‖⋅‖_𝑝 to denote the ℓ_𝑝 norm on ℝ^𝑛. Namely, for 𝑥=(𝑥_𝑗)^𝑡∈ℝ^𝑛, ‖𝑥‖_𝑝=^𝑛_𝑗=1𝑥_𝑗^𝑝1/𝑝,‖𝑥‖1≤𝑝<∞_∞=max_{1≤𝑗≤𝑛}𝑥_𝑗. (12) Let 𝐾 be a closed convex subset of ℝ^𝑛. Recall that the projection from ℝ^𝑛 to 𝐾 is defined as the operator 𝑃_𝐾(𝑥)=argmin_𝑢∈𝐾‖𝑥−𝑢‖²₂,𝑥∈ℝ^𝑛. (13) The projection 𝑃_𝐾 is characterized as follows: given𝑥∈ℝ^𝑛and𝑧∈𝐾∶𝑧=𝑃_𝐾𝑥⟺𝑥−𝑧,𝑦−𝑧≤0,𝑦∈𝐾. (14) Projections are nonexpansive. Namely, we have the following.

Proposition 1. One has that 𝑃_𝐾 is firmly nonexpansive in the sense that 𝑥−𝑦,𝑃_𝐾𝑥−𝑃_𝐾𝑦≥𝑃_𝐾𝑥−𝑃_𝐾𝑦²,𝑥,𝑦∈ℝ^𝑛. (15) In particular, 𝑃_𝐾 is nonexpansive; that is, ‖𝑃_𝐾𝑥−𝑃_𝐾𝑦‖≤‖𝑥−𝑦‖ for all 𝑥,𝑦∈ℝ^𝑛.

Recall that function 𝑓∶ℝ^𝑛→ℝ is convex if 𝑓(1−𝜆)𝑥+𝜆𝑦≤(1−𝜆)𝑓(𝑥)+𝜆𝑓𝑦 (16) for all 𝜆∈(0,1) and 𝑥,𝑦∈ℝ^𝑛. (Note that we only consider finite-valued functions.)

The subdifferential of a convex function 𝑓 is defined as the operator 𝜕𝑓 given by 𝜕𝑓(𝑥)=𝜉∈ℝ^𝑛∶𝑓𝑦≥𝑓(𝑥)+𝜉,𝑦−𝑥,𝑦∈ℝ^𝑛. (17) The inequality in (17) is referred to as the subdifferential inequality of 𝑓 at 𝑥. We say that 𝑓 is subdifferentiable at 𝑥 if 𝜕𝑓(𝑥) is nonempty. It is well known that, for an everywhere finite-valued convex function 𝑓 on ℝ^𝑛, 𝑓 is everywhere subdifferentiable.

Examples. (i) If 𝑓(𝑥)=|𝑥| for 𝑥∈ℝ, then 𝜕𝑓(0)=[−1,1]; (ii) of 𝑓(𝑥)=‖𝑥‖₁ for 𝑥∈ℝ^𝑛, then 𝜕𝑓(𝑥) is given componentwise by 𝜕𝑓(𝑥)_𝑗=sgn𝑥_𝑗,if𝑥_𝑗𝜉≠0,_𝑗∈[−1,1],if𝑥_𝑗=0,1≤𝑗≤𝑛. (18) Here sgn is the sign function; that is, for 𝑎∈ℝ, sgn(𝑎)=1,if𝑎>0,0,if𝑎=0,−1,if𝑎<0. (19)

Consider the unconstrained minimization problem min_{𝑥∈ℝ𝑛}𝑓(𝑥). (20) The following are well known.

Proposition 2. Let 𝑓 be everywhere finite-valued on ℝ^𝑛.(i) If 𝑓 is strictly convex, then (20) admits at most one solution.(ii) If 𝑓 is convex and satisfies the coercivity condition‖𝑥‖⟶∞⟹𝑓(𝑥)⟶∞, (21) then there exists at least one solution to (20). Therefore, if 𝑓 is both strictly convex and coercive, there exists one and only one solution to (20).

Proposition 3. Let 𝑓 be everywhere finite-valued convex on ℝ^𝑛 and 𝑧∈ℝ^𝑛. Suppose 𝑓 is bounded below (i.e., inf{𝑓(𝑥)∶𝑥∈ℝ^𝑛}>−∞). Then 𝑧 is a solution to minimization (20) if and only if it satisfies the first-order optimality condition: 0∈𝜕𝑓(𝑧). (22)

3. Properties of the 𝑄-Lasso

We study some basic properties of the 𝑄-lasso which is repeated below min_{𝑥∈ℝ𝑛}𝜑_𝛾(𝑥)1∶=2𝐴𝑥−𝑃_𝑄𝐴𝑥²₂+𝛾‖𝑥‖₁, (23) where 𝛾>0 is a regularization parameter. We also consider the following minimization (we call it 𝑄-least squares problem): min_{𝑥∈ℝ𝑛}12𝐴𝑥−𝑃_𝑄𝐴𝑥²₂. (24) Denote by 𝑆 and 𝑆_𝛾 the solution sets of (24) and (23), respectively. Since 𝜑_𝛾 is continuous, convex, and coercive (i.e., 𝜑_𝛾(𝑥)→∞ as ‖𝑥‖₂→∞), 𝑆_𝛾 is closed, convex, and nonempty. Notice also that since we assume the consistency of (8), we have 𝑆≠∅; moreover, the solution sets of (8) and (24) coincide.

Observe that the assumption that 𝑆≠∅ actually implies that 𝑆_𝛾 is uniformly bounded in 𝛾>0, as shown by the lemma below.

Lemma 4. Assume that (24) is consistent (i.e., 𝑆≠∅). Then, for 𝛾>0 and 𝑥_𝛾∈𝑆_𝛾, one has ‖𝑥_𝛾‖₁≤inf_𝑥∈𝑆‖𝑥‖₁.

Proof. Let 𝑥_𝛾∈𝑆_𝛾. In the relation 12𝐼−𝑃_𝑄𝐴𝑥_𝛾²₂+𝛾𝑥_𝛾1≤12𝐼−𝑃_𝑄𝐴𝑥²₂+𝛾‖𝑥‖₁,𝑥∈ℝ^𝑛, (25) taking 𝑥∈𝑆 yields (for 𝑃_𝑄𝑥∈𝑄) 12𝐼−𝑃_𝑄𝐴𝑥_𝛾²₂+𝛾𝑥_𝛾1≤𝛾‖𝑥‖₁,𝑥∈𝑆. (26) It follows that 𝑥_𝛾1≤‖𝑥‖₁,𝑥∈𝑆. (27) This proves the conclusion of the lemma.

Proposition 5. One has the following.(i) The functions𝜌𝛾∶=𝑥_𝛾1𝛾1,𝜂∶=2(𝐼−𝑃_𝑄)𝐴𝑥_𝛾²₂ (28) are well defined for 𝛾>0. That is, they do not depend upon particular choice of 𝑥_𝛾∈𝑆_𝛾.(ii) The function 𝜌(𝛾) is decreasing in 𝛾>0.(iii)The function 𝜂(𝛾) is increasing in 𝛾>0.(iv)(𝐼−𝑃_𝑄)𝐴𝑥_𝛾 is continuous in 𝛾>0.

Proof. For 𝑥_𝛾∈𝑆_𝛾, we have the optimality condition: 0∈𝜕𝜑_𝛾𝑥_𝛾=𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾+𝛾𝜕𝑥_𝛾1. (29) Here 𝐴^𝑡 is the transpose of 𝐴 and 𝜕 stands for the subdifferential in the sense of convex analysis. Equivalently, −1𝛾𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾∈𝜕𝑥_𝛾1. (30) It follows by the subdifferential inequality that 𝛾‖𝑥‖₁≥𝛾𝑥_𝛾1−𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝑥−𝑥_𝛾,∀𝑥∈ℝ^𝑛. (31) In particular, for 𝑥_𝛾∈𝑆_𝛾, 𝛾𝑥_𝛾1≥𝛾𝑥_𝛾1−𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝑥_𝛾−𝑥_𝛾. (32) Interchange 𝑥_𝛾 and 𝑥_𝛾 to get 𝛾𝑥_𝛾1≥𝛾𝑥_𝛾1−𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝑥_𝛾−𝑥_𝛾. (33) Adding up (32) and (33) yields 0≥𝐴𝑥_𝛾−𝐴𝑥_𝛾,𝐴𝑥_𝛾−𝐴𝑥_𝛾=𝐴𝑥_𝛾−𝐴𝑥_𝛾²₂. (34) Consequently, 𝐴𝑥_𝛾=𝐴𝑥_𝛾. Moreover, (32) and (33) imply that ‖𝑥_𝛾‖₁≥‖𝑥_𝛾‖₁ and ‖𝑥_𝛾‖₁≥‖𝑥_𝛾‖₁, respectively. Hence ‖𝑥_𝛾‖₁=‖𝑥_𝛾‖₁, and it follows that the functions 𝜌𝛾∶=𝑥_𝛾1𝛾1,𝜂∶=2𝐼−𝑃_𝑄𝐴𝑥_𝛾²₂𝑥_𝛾∈𝑆_𝛾 (35) are well defined for 𝛾>0.
Now substituting 𝑥_𝛽∈𝑆_𝛽 for 𝑥 in (31), we get 𝛾𝑥_𝛽1≥𝛾𝑥_𝛾1−𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝑥_𝛽−𝑥_𝛾. (36) Interchange 𝛾 and 𝛽 and 𝑥_𝛾 and 𝑥_𝛽 to find 𝛽𝑥_𝛾1≥𝛽𝑥_𝛽1−𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛽,𝑥_𝛾−𝑥_𝛽. (37) Adding up (36) and (37) and using the fact that (𝐼−𝑃_𝑄) is firmly nonexpansive, we deduce that 𝛾−𝛽𝑥_𝛽1−𝑥_𝛾1≥𝐼−𝑃_𝑄𝐴𝑥_𝛾−𝐼−𝑃_𝑄𝐴𝑥_𝛽,𝐴𝑥_𝛾−𝐴𝑥_𝛾≥𝐼−𝑃_𝑄𝐴𝑥_𝛾−𝐼−𝑃_𝑄𝐴𝑥_𝛽²₂. (38) We therefore find that if 𝛾>𝛽, then ‖𝑥_𝛽‖₁≥‖𝑥_𝛾‖₁. This proves that 𝜌(𝛾) is nonincreasing in 𝛾>0. From (38) it also follows that (𝐼−𝑃_𝑄)𝐴𝑥_𝛾 is continuous for 𝛾>0, which implies the continuity of 𝜂(𝛾) for 𝛾>0.
To see that 𝜂(𝛾) is increasing, we use the inequality (as 𝑥_𝛾∈𝑆_𝛾) 12(𝐼−𝑃_𝑄)𝐴𝑥_𝛾²₂+𝛾𝑥_𝛾1≤12(𝐼−𝑃_𝑄)𝐴𝑥_𝛽²₂+𝛾𝑥_𝛽1 (39) which implies that 𝜂𝛾≤𝜂𝛽+𝛾𝑥_𝛽1−𝑥_𝛾1. (40) Now if 𝛽>𝛾>0, then, as ‖𝑥_𝛽‖₁≤‖𝑥_𝛾‖₁, we immediately get that 𝜂(𝛾)≤𝜂(𝛽) and the increase of 𝜂 is proven.

Proposition 6. One has the following.(i)lim_𝛾→0𝜂(𝛾)=inf_𝑥∈ℝ^𝑛(1/2)‖(𝐼−𝑃_𝑄)𝐴𝑥‖²₂.(ii)lim_𝛾→0𝜌(𝛾)=min_𝑥∈𝑆‖𝑥‖₁.

Proof. (i) Taking the limit as 𝛾→0 in the inequality (and using the boundedness of (𝑥_𝛾)) 12𝐼−𝑃_𝑄𝐴𝑥_𝛾²₂+𝛾𝑥_𝛾1≤12𝐼−𝑃_𝑄𝐴𝑥²₂+𝛾‖𝑥‖₁,∀𝑥∈ℝ^𝑛, (41) yields lim_𝛾→0𝜂𝛾≤12(𝐼−𝑃_𝑄)𝐴𝑥²₂,∀𝑥∈ℝ^𝑛. (42) The result in (i) then follows.
As for (ii), we have, by (27), ‖𝑥_𝛾‖₁≤‖𝑥‖₁ for any 𝑥∈𝑆. In particular, ‖𝑥_𝛾‖₁≤‖𝑥^†‖₁, where 𝑥^† is an ℓ₁ minimum-norm element of 𝑆; that is, ‖𝑥^†‖₁=min_𝑥∈𝑆‖𝑥‖₁.
Assume 𝛾_𝑘→0 is such that 𝑥_𝛾𝑘→𝑥. Then for any 𝑥, 12𝐼−𝑃_𝑄𝐴𝑥²₂=lim_𝑘→∞12𝐼−𝑃_𝑄𝐴𝑥_𝛾𝑘²₂=lim_𝑘→∞12𝐼−𝑃_𝑄𝐴𝑥_𝛾𝑘²₂+𝛾_𝑘𝑥_𝛾𝑘1≤lim_𝑘→∞12𝐼−𝑃_𝑄𝐴𝑥²₂+𝛾_𝑘‖𝑥‖₁=12(𝐼−𝑃_𝑄)𝐴𝑥²₂. (43) It follows that 𝑥 solves the minimization problem: min_𝑥(1/2)‖(𝐼−𝑃_𝑄)𝐴𝑥‖²₂; that is, 𝑥∈𝑆. Consequently, lim_𝛾→0𝜌𝛾=lim_𝑘→∞𝜌𝛾_𝑘=lim_𝑘→∞𝑥_𝛾𝑘1=‖𝑥‖₁≤𝑥^†₁=min_𝑥∈𝑆‖𝑥‖₁. (44) This suffices to ensure that the conclusion of (ii) holds.

It is a challenging problem how to select the tuning (i.e., regularizing) parameter 𝛾 in lasso (1) and 𝑄-lasso (7). There is no general rule to universally select 𝛾 which should instead be selected in a case-to-case manner. The following result however points out that 𝛾 cannot be large.

Proposition 7. Let 𝑄 be a nonempty closed convex subset of ℝ^𝑚 and assume that 𝑄-lasso (7) is consistent (i.e., solvable). If 𝛾>max{‖𝐴^𝑡𝑃_𝑄𝐴𝑥‖_∞∶‖𝑥‖₁≤min_𝑣∈𝑆‖𝑣‖₁} (note that this condition is reduced to 𝛾>‖𝐴^𝑡𝑏‖_∞ for lasso (1) for which 𝑄={𝑏}), then 𝑥_𝛾=0. (Here 𝑆 is, as before, the solution set of the 𝑄-least squares problem (24).)

Proof. Let 𝑥_𝛾∈𝑆_𝛾. The optimality condition −𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾∈𝛾𝜕𝑥_𝛾1 (45) implies that −𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾𝑗=𝛾⋅sgn𝑥_𝛾𝑗,if𝑥_𝛾𝑗𝐴≠0,_𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾𝑗≤𝛾,if𝑥_𝛾𝑗=0. (46) Taking 𝑥=2𝑥_𝛾 in subdifferential inequality (31) yields 𝛾𝑥_𝛾1≥−𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝑥_𝛾=−_{𝑥𝛾𝑗≠0}𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾𝑗𝑥_𝛾𝑗=_{𝑥𝛾𝑗≠0}𝛾⋅sgn𝑥_𝛾𝑗𝑥_𝛾𝑗=𝛾_{𝑥𝛾𝑗≠0}𝑥_𝛾𝑗=𝛾‖𝑥‖₁. (47) It follows that 𝛾𝑥_𝛾1=−𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝑥_𝛾=−𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝐴𝑥_𝛾,=−𝐴𝑥_𝛾²₂+𝑃_𝑄𝐴𝑥_𝛾,𝐴𝑥_𝛾≤𝑃_𝑄𝐴𝑥_𝛾,𝐴𝑥_𝛾=𝐴^𝑡𝑃_𝑄𝐴𝑥_𝛾,𝑥_𝛾≤𝐴^𝑡𝑃_𝑄𝐴𝑥_𝛾∞𝑥_𝛾1. (48)(49)
Now by Lemma 4, we have ‖𝑥_𝛾‖₁≤min_𝑣∈𝑆‖𝑣‖₁. Hence, from (49) it follows that if 𝑥_𝛾≠0, we must have 𝛾≤max{‖𝐴^𝑡𝑃_𝑄𝐴𝑥‖_∞∶‖𝑥‖₁≤min_𝑣∈𝑆‖𝑣‖₁}. This completes the proof.

Notice that (48) shows that 𝜌(𝜆)=‖𝑥_𝛾‖₁ can be determined by 𝐴𝑥_𝛾. Hence, we arrive at the following characterization of solutions of 𝑄-lasso (23).

Proposition 8. Let 𝑄 be a nonempty closed convex subset of ℝ^𝑚 and let 𝛾>0 and 𝑥_𝛾∈𝑆_𝛾. Then 𝑥∈ℝ^𝑛 is a solution of the 𝑄-lasso (23) if and only if 𝐴𝑥=𝐴𝑥_𝛾 and ‖𝑥‖≤‖𝑥_𝛾‖. It turns out that 𝑆_𝛾=𝑥_𝛾+𝑁(𝐴)∩𝐵_𝜌(𝛾), (50) where 𝑁(𝐴)={𝑥∈ℝ^𝑛∶𝐴𝑥=0} is the null space of 𝐴 and where 𝐵_𝑟 denotes the closed ball centered at the origin and with radius of 𝑟>0. This shows that if one can find one solution to 𝑄-lasso (23), then all solutions are found by (50).

Proof. If 𝐴𝑥=𝐴𝑥_𝛾, then from the relations 𝜑_𝛾𝑥_𝛾=12𝐼−𝑃_𝑄𝐴𝑥_𝛾²₂+𝛾𝑥_𝛾1≤12𝐼−𝑃_𝑄𝐴𝑥²₂+𝛾‖𝑥‖₁=12𝐼−𝑃_𝑄𝐴𝑥_𝛾²₂+𝛾‖𝑥‖₁, (51) we obtain ‖𝑥_𝛾‖₁≤‖𝑥‖₁. This together with the assumption of ‖𝑥‖₁≤‖𝑥_𝛾‖₁ yields that ‖𝑥‖₁=‖𝑥_𝛾‖₁ which in turn implies that 𝜑_𝛾(𝑥)=𝜑_𝛾(𝑥_𝛾) and hence 𝑥∈𝑆_𝛾.

4. Iterative Methods

In this section we discuss the proximal iterative methods for solving 𝑄-lasso (7). The basics are Moreau's concept of proximal operators and their fundamental properties which are briefly mentioned below. (For the sake of our purpose, we however confine ourselves to the finite-dimensional setting.)

4.1. Proximal Operators

Let Γ₀(ℝ^𝑛) be the space of convex functions in ℝ^𝑛 that are proper, lower semicontinuous and convex.

Definition 9 (see [10, 11]). The proximal operator of 𝜑∈Γ₀(ℝ^𝑛) is defined by prox_𝜑(𝑥)∶=argmin_{𝑣∈ℝ𝑛}𝜑(𝑣)+12‖𝑣−𝑥‖²,𝑥∈ℝ^𝑛. (52) The proximal operator of 𝜑 of order 𝜆>0 is defined as the proximal operator of 𝜆𝜑; that is, prox_𝜆𝜑(𝑥)∶=argmin_{𝑣∈ℝ𝑛}𝜑(𝑣)+1‖‖2𝜆𝑣−𝑥²,𝑥∈ℝ^𝑛. (53)
For fundamental properties of proximal operators, the reader is referred to [12, 13] for details. Here we only mention the fact that the proximal operator prox_𝜆𝜑 can have a closed-form expression in some important cases as shown in the examples below [12].(a)If we take 𝜑 to be any norm ‖⋅‖ of ℝ^𝑛, then prox_{𝜆‖⋅‖}(𝑥)=𝜆1−‖𝑥‖𝑥,if‖𝑥‖>𝜆.0,if‖𝑥‖≤𝜆. (54) In particular, if we take 𝜑 to be the absolute value function of the real line ℝ, we get prox_𝜆|⋅|(𝑥)=sgn(𝑥)max{|𝑥|−𝜆,0} (55) which is also known as the scalar soft-thresholding operator.(b)Let {𝑒_𝑘}^𝑛_𝑘=1 be an orthonormal basis of ℝ^𝑛 and let {𝜔_𝑘}^𝑛_𝑘=1 be real positive numbers. Define 𝜑 by 𝜑(𝑥)=^𝑛_𝑘=1𝜔_𝑘𝑥,𝑒_𝑘. (56) Then prox_𝜑(𝑥)=∑^𝑛_𝑘=1𝛼_𝑘𝑒_𝑘, where 𝛼_𝑘=sgn𝑥,𝑒_𝑘max𝑥,𝑒_𝑘−𝜔_𝑘,0. (57) In particular, if 𝜑(𝑥)=‖𝑥‖₁ for 𝑥∈ℝ^𝑛, then prox_{𝜆‖⋅‖1}(𝑥)=prox_𝜆|⋅|𝑥₁,…,prox_𝜆|⋅|𝑥_𝑛=𝛼₁,…,𝛼_𝑛, (58) where 𝛼_𝑘=sgn(𝑥_𝑘)max{|𝑥_𝑘|−𝜆,0} for 𝑘=1,…,𝑛.

4.2. Proximal-Gradient Algorithm

The proximal operators can be used to minimize the sum of two convex functions: min_{𝑥∈ℝ𝑛}𝑓(𝑥)+𝑔(𝑥), (59) where 𝑓,𝑔∈Γ₀(ℝ^𝑛). It is often the case where one of them is differentiable. The following is an equivalent fixed point formulation of (59).

Proposition 10 (see [12, 14]). Let 𝑓,𝑔∈Γ₀(ℝ^𝑛). Let 𝑥^∗∈ℝ^𝑛 and 𝜆>0. Assume 𝑓 is finite valued and differentiable on ℝ^𝑛. Then 𝑥^∗ is a solution to (59) if and only if 𝑥^∗ solves the fixed point equation: 𝑥^∗=prox_𝜆𝑔∘𝐼−𝜆∇𝑓𝑥^∗. (60)

Fixed point equation (60) immediately yields the following fixed point algorithm which is also known as the proximal-gradient algorithm for solving (59).

Initialize 𝑥₀∈ℝ^𝑛 and iterate 𝑥_𝑘+1=prox_𝜆𝑘𝑔∘𝐼−𝜆_𝑘∇𝑓𝑥_𝑘, (61) where {𝜆_𝑘} is a sequence of positive real numbers.

Theorem 11 (see [12, 14]). Let 𝑓,𝑔∈Γ₀(ℝ^𝑛) and assume (59) is consistent. Assume in addition the following.(i)∇𝑓 is Lipschitz continuous on ℝ^𝑛: ∇𝑓(𝑥)−∇𝑓𝑦≤𝐿𝑥−𝑦,𝑥,𝑦∈ℝ^𝑛. (62)(ii)0<liminf_𝑛→∞𝜆_𝑛≤limsup_𝑛→∞𝜆_𝑛<2/𝐿. Then the sequence (𝑥_𝑘) generated by the proximal-gradient algorithm (61) converges to a solution of (59).

4.3. The Relaxed Proximal-Gradient Algorithm

The relaxed proximal-gradient algorithm generates a sequence (𝑥_𝑘) by the following iteration process.

Initialize 𝑥₀∈ℝ^𝑛 and iterate 𝑥_𝑘+1=1−𝛼_𝑘𝑥_𝑘+𝛼_𝑘prox_𝜆𝑘𝑔∘𝐼−𝜆_𝑘∇𝑓𝑥_𝑘, (63) where {𝛼_𝑘} is the sequence of relaxation parameters and {𝜆_𝑘} is a sequence of positive real numbers.

Theorem 12 (see [14]). Let 𝑓,𝑔∈Γ₀(ℝ^𝑛) and assume (59) is consistent. Assume in addition the following.(i)∇𝑓 is Lipschitz continuous on ℝ^𝑛: ∇𝑓(𝑥)−∇𝑓𝑦≤𝐿𝑥−𝑦,𝑥,𝑦∈ℝ^𝑛. (64)(ii)0<liminf_𝑛→∞𝜆_𝑛≤limsup_𝑛→∞𝜆_𝑛<2/𝐿. (iii)0<liminf_𝑛→∞𝛼_𝑛≤limsup_𝑛→∞𝛼_𝑛<4/(2+𝐿⋅limsup_𝑛→∞𝜆_𝑛). Then the sequence (𝑥_𝑘) generated by proximal-gradient algorithm (61) converges to a solution of (59).

If we take 𝜆_𝑛≡𝜆∈(0,2/𝐿), then the relaxation parameters 𝛼_𝑘 can be chosen from a larger pool; they are allowed to be close to zero. More precisely, we have the following theorem.

Theorem 13 (see [14]). Let 𝑓,𝑔∈Γ₀(ℝ^𝑛) and assume (59) is consistent. Define the sequence (𝑥_𝑘) by the following relaxed proximal algorithm: 𝑥_𝑘+1=1−𝛼_𝑛𝑥_𝑘+𝛼_𝑘prox_𝜆𝑔𝑥_𝑘−𝜆∇𝑓𝑥_𝑘. (65) Suppose that (a)∇𝑓 satisfies the Lipschitz continuity condition (i) in Theorem 12;(b)0<𝜆<2/𝐿 and 0≤𝛼_𝑘≤(2+𝜆𝐿)/4 for all 𝑘;(c)∑^∞_𝑛=1𝛼_𝑛((4/(2+𝜆𝐿))−𝛼_𝑘)=∞. Then (𝑥_𝑛) converges to a solution of (59).

4.4. Proximal-Gradient Algorithms Applied to Lasso

For 𝑄-lasso (7), we take 𝑓(𝑥)=(1/2)‖(𝐼−𝑃_𝑄)𝐴𝑥‖²₂ and 𝑔(𝑥)=𝛾‖𝑥‖₁. Noticing that ∇𝑓(𝑥)=𝐴^𝑡(𝐼−𝑃_𝑄)𝐴𝑥 which is Lipschitz continuous with constant 𝐿=‖𝐴‖²₂ for 𝐼−𝑃_𝑄 is nonexpansive, we find that proximal-gradient algorithm (61) is reduced to the following algorithm for solving 𝑄-lasso (7): 𝑥_𝑘+1=prox_{𝜆𝑘𝛾‖⋅‖1}𝐼−𝜆_𝑘𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝑘. (66)

The convergence theorem of general proximal-gradient algorithm (61) reads the following for 𝑄-lasso (7).

Theorem 14. Assume 0<liminf_𝑘→∞𝜆_𝑘≤limsup_𝑘→∞𝜆_𝑘<2/‖𝐴‖²₂. Then the sequence (𝑥_𝑘) generated by proximal-gradient algorithm (66) converges to a solution of lasso (7).

Remark 15. Relaxed proximal-gradient algorithms (63) and (65) also apply to 𝑄-lasso (7). We however do not elaborate on them in detail.

Remark 16. Proximal-gradient algorithm (61) can be reduced to a projection-gradient algorithm in the case where the convex function 𝑔 is homogeneous (i.e., 𝑔(𝑡𝑥)=𝑡𝑔(𝑥) for 𝑡≥0 and 𝑥∈ℝ^𝑛) because the homogeneity of 𝑔 implies that the proximal operator of 𝑔 is actually a projection; more precisely, we have prox_𝜆𝑔=𝑃_𝜆𝐾,𝜆>0, (67) where 𝐾=𝜕𝑔(0). As a result, proximal-gradient algorithm (61) is reduced to the following projection-gradient algorithm: 𝑥_𝑘+1=𝐼−𝑃_𝜆𝑘𝐾𝐼−𝜆_𝑘∇𝑓𝑥_𝑘. (68)
Now we apply projection-gradient algorithm (68) to 𝑄-lasso (7). In this case, we have 𝑓(𝑥)=(1/2)‖(𝐼−𝑃_𝑄)𝐴𝑥‖²₂ and 𝑔(𝑥)=𝛾‖𝑥‖₁ (homogeneous). Thus, ∇𝑓(𝑥)=𝐴^𝑡(𝐼−𝑃_𝑄)𝐴𝑥 and the convex set 𝐾=𝜕𝑔(0) is given as 𝐾=𝛾𝜕(‖𝑧‖₁)|_𝑧=0=𝛾[−1,1]^𝑛. We find that, for each positive number 𝜆>0, 𝑃_𝜆𝐾 is the projection of the Euclidean space ℝ^𝑛 to the ℓ_∞ ball with radius of 𝜆𝛾; that is, {𝑥∈ℝ^𝑛∶‖𝑥‖_∞≤𝜆𝛾}. It turns out that proximal-projection algorithm (66) is rewritten as a projection algorithm below: 𝑥_𝑘+1=𝐼−𝑃_{𝜆𝑘𝛾[−1,1]}^𝑛𝐼−𝜆_𝑘𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝑘. (69)

5. An ℓ₁-ℓ₂ Regularization for the 𝑄-Lasso

𝑄-lasso (7) may be ill posed and therefore needs to be regularized. Inspired by the elastic net [15] which regularizes lasso (1), we introduce an ℓ₁-ℓ₂ regularization for the 𝑄-Lasso as the minimization min_{𝑥∈ℝ𝑛}12(𝐼−𝑃_𝑄)𝐴𝑥²₂+𝛾‖𝑥‖₁1+𝛿2‖𝑥‖²₂=∶𝜑_𝛾,𝛿(𝑥), (70) where 𝛾>0 and 𝛿>0 are regularization parameters. This is indeed the traditional Tikhonov regularization applied to 𝑄-lasso (7).

Let 𝑥_𝛾,𝛿 be the unique solution of (70) and set 𝜑_𝛾(𝑥)1∶=2𝐼−𝑃_𝑄𝐴𝑥²₂+𝛾‖𝑥‖₁,𝜓_𝛿(𝑥)1∶=2(𝐼−𝑃_𝑄)𝐴𝑥²₂1+𝛿2‖𝑥‖²₂ (71) which are the limits of 𝜑_𝛾,𝛿(𝑥) as 𝛿→0 and 𝛾→0, respectively. Let 𝑆_𝛾=argmin_{𝑥∈ℝ𝑛}𝜑_𝛾(𝑥),𝑥_𝛿=argmin_{𝑥∈ℝ𝑛}𝜓_𝛿(𝑥). (72)

Proposition 17. Assume the 𝑄-least-squares problem min_{𝑥∈ℝ𝑛}12𝐼−𝑃_𝑄𝐴𝑥²₂ (73) is consistent (i.e., solvable) and let 𝑆 be its nonempty set of solutions.(i) As  𝛿→0   (for each fixed  𝛾>0),  𝑥_𝛾,𝛿→𝑥^†_𝛾  which is the (ℓ₂) minimum-norm solution to  𝑄-lasso (7). Moreover, as  𝛾→0, every cluster point of  𝑥^†_𝛾  is a   (ℓ₁)   minimum-norm solution of  𝑄-least squares problem (73), that is, a point in the set  argmin_𝑥∈𝑆‖𝑥‖₁.(ii)As  𝛾→0   (for each fixed  𝛿>0),  𝑥_𝛾,𝛿→𝑥_𝛿  which is the unique solution to the  ℓ₂  regularized problem:min_{𝑥∈ℝ𝑛}12(𝐼−𝑃_𝑄)𝐴𝑥²₂1+𝛿2‖𝑥‖²₂. (74) Moreover, as  𝛿→0,  𝑥_𝛿→𝑥  which is the  ℓ₂  minimal norm solution of   (73); that is,  𝑥=argmin_𝑥∈𝑆‖𝑥‖₂.

Proof. We have that 𝑥_𝛾,𝛿 satisfies the optimality condition: 0∈𝜕𝜑_𝛾,𝛿𝑥_𝛾,𝛿, (75) where the subdifferential of 𝜑_𝛾,𝛿 is given by 𝜕𝜑_𝛾,𝛿(𝑥)=𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥+𝛿𝑥+𝛾𝜕‖𝑥‖₁. (76) It turns out that the above optimality condition is reduced to −1𝛾𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝛿+𝛿𝑥_𝛾,𝛿∈𝜕𝑥_𝛾,𝛿1. (77) Using the subdifferential inequality, we obtain 𝛾‖𝑥‖₁≥𝛾𝑥_𝛾,𝛿1−𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝛿+𝛿𝑥_𝛾,𝛿,𝑥−𝑥_𝛾,𝛿 (78) for 𝑥∈ℝ^𝑛. Replacing 𝑥 with 𝑥_{𝛾′,𝛿′} for 𝛾^′>0 and 𝛿^′>0 yields 𝛾𝑥_{𝛾′,𝛿′1}≥𝛾𝑥_𝛾,𝛿1−𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝛿+𝛿𝑥_𝛾,𝛿,𝑥_{𝛾′,𝛿′}−𝑥_𝛾,𝛿. (79) Interchange 𝛾 and 𝛾^′ and 𝛿 and 𝛿^′ to get 𝛾^′𝑥_𝛾,𝛿1≥𝛾^′𝑥_{𝛾′,𝛿′1}−𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_{𝛾′,𝛿′}+𝛿^′𝑥_{𝛾′,𝛿′},𝑥_𝛾,𝛿−𝑥_{𝛾′,𝛿′}. (80) Adding up (79) and (80) results in 𝛾^′−𝛾𝑥_𝛾,𝛿1−𝑥_{𝛾′,𝛿′1}≥𝐼−𝑃_𝑄𝐴𝑥_{𝛾′,𝛿′}−𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝛿,𝐴^𝑡𝑥_{𝛾′,𝛿′}−𝐴^𝑡𝑥_𝛾,𝛿+𝛿^′𝑥_{𝛾′,𝛿′}−𝛿𝑥_𝛾,𝛿,𝑥_{𝛾′,𝛿′}−𝑥_𝛾,𝛿≥𝐼−𝑃_𝑄𝐴𝑥_𝛾,𝛿−𝐼−𝑃_𝑄𝐴𝑥_{𝛾′,𝛿′}²₂+𝛿^′−𝛿𝑥_{𝛾′,𝛿′},𝑥_{𝛾′,𝛿′}−𝑥_𝛾,𝛿+𝛿𝑥_{𝛾′,𝛿′}−𝑥_𝛾,𝛿²₂. (81) Since ℓ₁-ℓ₂ regularization (70) is the Tikhonov regularization of 𝑄-lasso (7), we get 𝑥_𝛾,𝛿2≤𝑥_𝛾2≤𝑐𝑥_𝛾1≤𝑐‖𝑥‖₁,𝑥_𝛾∈𝑆_𝛾,𝑥∈𝑆. (82) Here 𝑐>0 is a constant. It follows that {𝑥_𝛾,𝛿} is bounded.(i)For fixed 𝛾>0, we can use the theory of Tikhonov regularization to conclude that 𝑥_𝛾,𝛿 is continuous in 𝛿>0 and converges, as 𝛿→0, to 𝑥^†_𝛾 which is the (ℓ₂) minimum-norm solution to 𝑄-lasso (7), that is, the unique element 𝑥^†_𝛾∶=argmin_{𝑥∈𝑆𝛾}‖𝑥‖₂. By Proposition 6, we also find that every cluster point of 𝑥^†_𝛾, as 𝛾→0, lies in the set argmin_𝑥∈𝑆‖𝑥‖₁.(ii)Fix 𝛿>0 and use Proposition 6 to see that 𝑥_𝛾,𝛿→𝑥_𝛿 as 𝛾→0. Now the standard property of Tikhonov's regularization ensures that 𝑥_𝛿→argmin_𝑥∈𝑆‖𝑥‖₂ as 𝛿→0.

ℓ₁-ℓ₂ regularization (70) can be solved by proximal-gradient algorithm (61). Take 𝑓(𝑥)=(1/2)‖(𝐼−𝑃_𝑄)𝐴𝑥‖²₂+(1/2)𝛿‖𝑥‖²₂ and 𝑔(𝑥)=𝛾‖𝑥‖₁; then algorithm (61) is reduced to 𝑥_𝑘+1=prox_{𝜆𝑘𝛾‖⋅‖1}𝑥_𝑘−𝜆_𝑘𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝑘+𝛿𝑥_𝑘. (83) The convergence of this algorithm is given as follows.

Theorem 18. Assume 0<liminf_𝑘→∞𝜆_𝑘≤limsup_𝑘→∞𝜆_𝑘<2‖𝐴‖²₂.+𝛿 (84) Then the sequence (𝑥_𝑘) generated by algorithm (83) converges to the solution of ℓ₁-ℓ₂ regularization (70).

We can also take 𝑓(𝑥)=(1/2)‖(𝐼−𝑃_𝑄)𝐴𝑥‖²₂ and 𝑔(𝑥)=𝛾‖𝑥‖₁+(1/2)𝛿‖𝑥‖²₂. Then prox_𝜇𝑔(𝑥)=prox_{𝜈‖⋅‖1}((1/(1+𝜇𝛿))𝑥) with 𝜈=𝜇𝛾/(1+𝜇𝛿), and the proximal algorithm (61) is reduced to 𝑥_𝑘+1=prox_{𝜈𝑘‖⋅‖1}11+𝛿𝛾_𝑘𝑥_𝑘−𝜆_𝑘𝐴^𝑡𝐼−𝑃_𝑄𝐴𝑥_𝑘. (85) Here 𝜈_𝑘=𝛾𝜆_𝑘/(1+𝛿𝛾_𝑘). Convergence of this algorithm is given below.

Theorem 19. Assume 0<liminf_𝑘→∞𝜆_𝑘≤limsup_𝑘→∞𝜆_𝑘<2‖𝐴‖²₂. (86) Then the sequence (𝑥_𝑘) generated by the algorithm (85) converges to the solution of ℓ₁-ℓ₂ regularization (70).

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

The authors were grateful to the anonymous referees for their helpful comments and suggestions which improved the presentation of this paper. This work was funded by the Deanship of Scientific Research (DSR), King Abdulaziz University, under Grant no. 2-363-1433-HiCi. The authors, therefore, acknowledge the technical and financial support of KAU.