On the Convergence Analysis of the Alternating Direction Method of Multipliers with Three Blocks

Abstract

We consider a class of linearly constrained separable convex programming problems whose objective functions are the sum of three convex functions without coupled variables. For those problems, Han and Yuan (2012) have shown that the sequence generated by the alternating direction method of multipliers (ADMM) with three blocks converges globally to their KKT points under some technical conditions. In this paper, a new proof of this result is found under new conditions which are much weaker than Han and Yuan’s assumptions. Moreover, in order to accelerate the ADMM with three blocks, we also propose a relaxed ADMM involving an additional computation of optimal step size and establish its global convergence under mild conditions.

1. Introduction

In various fields of applied mathematics and engineering, many problems can be equivalently formulated as a separable convex optimization problem with two blocks; that is, given two closed convex functions 𝑓_𝑖∶ℜ^𝑛𝑖→ℜ∪{+∞},𝑖=1,2, to find a solution pair (𝑥^∗₁,𝑥^∗₂) of the following problem: min𝑓₁𝑥₁+𝑓₂𝑥₂s.t.𝐴₁𝑥₁+𝐴₂𝑥₂=𝑏, (1) where 𝐴_𝑖 is a matrix in ℜ^{𝑝×𝑛𝑖},  𝑖=1,2, and 𝑏 is a vector in ℜ^𝑝. The classical alternating direction method of multipliers (ADMM) [1, 2] applied to problem (1) yields the following scheme: 𝑥₁^𝑘+1=argmin_{𝑥1∈ℜ𝑛1}𝑓₁𝑥₁−𝐴^𝑇₁𝜆^𝑘,𝑥₁+𝛽2𝐴₁𝑥₁+𝐴₂𝑥^𝑘₂−𝑏²,𝑥₂^𝑘+1=argmin_{𝑥2∈ℜ𝑛2}𝑓₂𝑥₂−𝐴^𝑇₂𝜆^𝑘,𝑥₂+𝛽2𝐴₁𝑥₁^𝑘+1+𝐴₂𝑥₂−𝑏²,𝜆^𝑘+1=𝜆^𝑘−𝛽𝐴₁𝑥₁^𝑘+1+𝐴₂𝑥₂^𝑘+1−𝑏, (2) where 𝜆^𝑘 is a Lagrangian multiplier and 𝛽>0 is a penalty parameter. Possibly due to its simplicity and effectiveness, the ADMM with two blocks has received continuous attention both in theoretical and application domains. We refer to [3–8] for theoretical results on ADMM with two blocks and [9–13] for its efficient applications in high-dimensional statistics, compressive sensing, finance, image processing, and engineering, to name just a few.

In this paper, we concentrate on the linearly constrained convex programming problem with three blocks: min𝑓₁𝑥₁+𝑓₂𝑥₂+𝑓₃𝑥₃s.t.𝐴₁𝑥₁+𝐴₂𝑥₂+𝐴₃𝑥₃=𝑏, (3) where 𝑓₃∶ℜ^𝑛3→ℜ∪{+∞} is a closed convex function and 𝐴₃ is a matrix in ℜ^𝑝×𝑛3. For solving (3), a nature idea is to extend the ADMM with two blocks to the ADMM with three blocks in which the next iteration (𝑥₂^𝑘+1,𝑥₃^𝑘+1,𝜆^𝑘+1) is updated by 𝑥₂^𝑘+1,𝑥₃^𝑘+1,𝜆^𝑘+1∶=𝑥^𝑘₂,𝑥^𝑘₃,𝜆^𝑘, (4) where 𝑥^𝑘₁=argmin_{𝑥1∈ℜ𝑛1}𝑓₁𝑥₁−𝐴^𝑇₁𝜆^𝑘,𝑥₁+𝛽2𝐴₁𝑥₁+𝐴₂𝑥^𝑘₂+𝐴₃𝑥^𝑘₃−𝑏²,𝑥^𝑘₂=argmin_{𝑥2∈ℜ𝑛2}𝑓₂𝑥₂−𝐴^𝑇₂𝜆^𝑘,𝑥₂+𝛽2𝐴₁𝑥^𝑘₁+𝐴₂𝑥₂+𝐴₃𝑥^𝑘−𝑏²,𝑥^𝑘₃=argmin_{𝑥3∈ℜ𝑛3}𝑓₃𝑥₃−𝐴^𝑇₃𝜆^𝑘,𝑥₃+𝛽2𝐴₁𝑥^𝑘₁+𝐴₂𝑥^𝑘₂+𝐴₃𝑥₃−𝑏²,𝜆^𝑘=𝜆^𝑘−𝛽𝐴₁𝑥^𝑘₁+𝐴₂𝑥^𝑘₂+𝐴₃𝑥^𝑘₃−𝑏. (5) Similar to the ADMM with two blocks, the ADMM with three blocks has found numerous applications in a broad spectrum of areas, such as doubly nonnegative cone programming [14], high-dimensional statistics [15, 16], imaging science [17], and engineering [18]. Even though its numerical efficiency is clearly seen from those applications, the theoretical treatment of ADMM with three blocks is challenging and the convergence of the ADMM is still open given only the convex assumptions of the objective function. To alleviate this difficulty, the authors of [19, 20] proposed prediction-correction type methods to solve the general separable convex programming; however, numerical results show that the direct ADMM outperforms its variants substantially. Therefore, it is of great significance to investigate the theoretical performance of the ADMM with three blocks even only to provide sufficient conditions to guarantee the convergence. To the best of our knowledge, there exist only two works aiming to attack the convergence problem of the direct ADMM with three blocks. By using an error bound analysis method, Hong and Luo [21] proved the linear convergence of the ADMM with 𝑚 blocks for sufficiently small 𝛽 subject to some technical conditions. However, the sufficiently small requirement on 𝛽 makes the algorithm difficult to implement. In [22], Han and Yuan employed a contractive analysis method to establish the convergence of ADMM under the strongly convex assumptions of 𝑓_𝑖 and the parameter 𝛽 less than a threshold depending on all the strongly convex moduli. In this paper, we firstly prove the convergence of ADMM with three blocks under two conditions weaker than those of [22]. In our conditions, the threshold on the parameter 𝛽 only relies on the strongly convex moduli of 𝑓₂ and 𝑓₃, and furthermore 𝑓₁ is not necessarily strongly convex in one of our conditions. Also, the restricted range of 𝛽 in this paper is shown to be at least three times as big as that of [22].

In order to accelerate the ADMM with three blocks, we also propose a relaxed ADMM with three blocks which involves an additional computation of optimal step size. Specifically, with the triple (𝑥^𝑘₂,𝑥^𝑘₃,𝜆^𝑘), we first generate a predictor (𝑥^𝑘₂,𝑥^𝑘₃,𝜆^𝑘) according to (5) and then obtain (𝑥₂^𝑘+1,𝑥₃^𝑘+1,𝜆^𝑘+1) in the next iteration by 𝑥₂^𝑘+1=𝑥^𝑘₂−𝛾𝛼^∗_𝑘𝑥^𝑘₂−𝑥^𝑘₂,𝑥₃^𝑘+1=𝑥^𝑘₃−𝛾𝛼^∗_𝑘𝑥^𝑘₃−𝑥^𝑘₃,𝜆^𝑘+1=𝜆^𝑘−𝛾𝛼^∗_𝑘𝜆^𝑘−𝜆^𝑘, (6)

where 𝛾∈(0,2) and 𝛼^∗_𝑘 is special step size defined in (43). The convergence of the relaxed ADMM is also established under mild conditions. We should mention that it is possible to modify the analyses given in this paper to be problems with more than three blocks of separability. But this is not the focus of this paper.

The remaining parts of this paper are organized as follows. In Section 2, we list some preliminaries on the strongly convex function, subdifferential, and the ADMM with three blocks. In Section 3, we first show the contractive property of the distance between the sequence generated by ADMM with three blocks and the solution set and then prove the convergence of ADMM under certain conditions. In Section 4, we extend the direct ADMM with three blocks to the relaxed ADMM with an optimal step size and establish its convergence under suitable conditions. We conclude our paper in Section 5.

Notation. For any positive integer 𝑚, let 𝐼_𝑚 be the 𝑚×𝑚 identity matrix. We use ‖⋅‖ and ‖⋅‖₂ to denote the vector Euclidean norm and the spectral norm of matrices (defined as the maximum singular value of matrices). For any symmetric matrix 𝑆∈ℜ^𝑛×𝑛, we write ‖𝑥‖²_𝑆=𝑥^𝑇𝑆𝑥 for any 𝑥∈ℜ^𝑛. 𝐺 and 𝑀 are two (𝑛₂+𝑛₃+𝑝)×(𝑛₂+𝑛₃+𝑝) matrices defined by 𝐺∶=𝛽𝐴^𝑇₂𝐴₂000𝛽𝐴^𝑇₃𝐴₃0𝐼00𝛽,𝑀∶=2𝛽𝐴^𝑇₂𝐴₂000𝛽𝐴^𝑇₃𝐴₃0𝐼00𝛽, (7) respectively. For given 𝑥₁∈ℜ^𝑛1,𝑥₂∈ℜ^𝑛2,𝑥₃∈ℜ^𝑛3, and 𝜆∈ℜ^𝑝, we frequently use 𝑢 and 𝑣 to denote the joint vectors of 𝑥₂,𝑥₃,𝜆 and 𝑥₁,𝑥₂,𝑥₃,𝜆, respectively; that is, 𝑢=𝑥^𝑇₂,𝑥^𝑇₃,𝜆^𝑇𝑇,𝑣=𝑥^𝑇₁,𝑥^𝑇₂,𝑥^𝑇₃,𝜆^𝑇𝑇, (8) while 𝑢 and 𝑣̃ are the joint vectors corresponding to 𝑥₂,𝑥₃,𝜆 and 𝑥₁,𝑥₂,𝑥₃,𝜆.

2. Preliminaries

Throughout this paper, we assume 𝑓_𝑖,𝑖=1,2,3, are strongly convex functions with modulus 𝜇_𝑖≥0; that is 𝑓_𝑖(1−𝛼)𝑧+𝛼𝑧^′≤(1−𝛼)𝑓_𝑖(𝑧)+𝛼𝑓_𝑖𝑧^′−12𝜇_𝑖𝛼(1−𝛼)𝑧−𝑧^′2,∀𝑧,𝑧^′∈ℜ^𝑛𝑖, (9) for each 𝑖. Note that 𝑓_𝑖 is a strongly convex function with modulus 0 being equivalent to the convexity of 𝑓_𝑖. Let 𝑥 be a point of dom(𝑓_𝑖); the subdifferential of 𝑓_𝑖 at 𝑥 is defined by 𝜕𝑓_𝑖(𝑥)∶=𝑥^∗∣𝑓(𝑧)≥𝑓(𝑥)+𝑥^∗,𝑧−𝑥,∀𝑧. (10) From Proposition  6 in [23], we know that, for each 𝑖, 𝜕𝑓_𝑖 is strongly monotone with modulus 𝜇_𝑖 which means 𝑧₁−𝑧₂,𝑥₁−𝑥₂≥𝜇_𝑖𝑧₁−𝑧₂²≥0,∀𝑥₁,𝑥₂,𝑧₁∈𝜕𝑓_𝑖𝑥₁,𝑧₂∈𝜕𝑓_𝑖𝑥₂. (11)

The next lemma introduced in [22] plays a key role in the convergence analysis of the ADMM and the relaxed ADMM with three blocks.

Lemma 1. Let (𝑥^∗₁,𝑥^∗₂,𝑥^∗₃,𝜆^∗) be any KKT point of problem (3). Let 𝑣^𝑘̃ be generated by (5) from given 𝑢^𝑘. Then, one has 𝑢^𝑘−𝑢^∗,𝐺𝑢^𝑘−𝑢^𝑘≥³_𝑖=1𝜇_𝑖‖𝑥^𝑘_𝑖−𝑥^∗_𝑖‖²+𝜆^𝑘−𝜆^𝑘,³_𝑖=2𝐴_𝑖𝑥^𝑘_𝑖−𝑥^𝑘_𝑖+𝛽𝐴₃𝑥^𝑘₃−𝑥^∗₃,𝐴₂𝑥^𝑘₂−𝑥^𝑘₂. (12)

3. The ADMM with Three Blocks

In this section, we first investigate the contractive property of the distance between the sequence generated by ADMM with three blocks and the solution set under the condition that 0<𝛽≤min{𝜇₂/‖𝐴₂‖²₂,𝜇₃/‖𝐴₃‖²₂}.

Lemma 2. Let 𝑣^∗=(𝑥^∗₁,𝑥^∗₂,𝑥^∗₃,𝜆^∗) be a KKT point of problem (3) and let the sequence {𝑣^𝑘=(𝑥^𝑘₁,𝑥^𝑘₂,𝑥^𝑘₃,𝜆^𝑘)} be generated by the ADMM with three blocks. Then, it holds that 𝑢^𝑘+1−𝑢^∗2_𝑀≤𝑢^𝑘−𝑢^∗2_𝑀−𝛽𝐴₃(𝑥₃^𝑘+1−𝑥^𝑘₃)²−𝛽𝐴₁𝑥₁^𝑘+1+𝐴₂𝑥^𝑘₂+𝐴₃𝑥₃^𝑘+1−𝑏²−2𝜇₁𝑥₁^𝑘+1−𝑥^∗₁²−2𝑥₂^𝑘+1−𝑥^∗₂²_{𝜇2𝐼𝑛2−𝛽𝐴𝑇2𝐴2}−2𝑥₃^𝑘+1−𝑥^∗₃²_{𝜇3𝐼𝑛3−𝛽𝐴𝑇3𝐴3}. (13)

Proof. Since 𝑥^𝑗₃ minimizes 𝑓₃(⋅)−⟨𝐴^𝑇₃𝜆^𝑗,⋅⟩, we deduce from the first order optimality condition that 𝐴^𝑇₃𝜆^𝑗∈𝜕𝑓₃𝑥^𝑗₃,𝑗=0,1,…,𝑘. (14) By (14) and the monotonicity of 𝜕𝑓₃(⋅) (11), it is easily seen that 𝑥^𝑘₃−𝑥₃^𝑘+1,𝐴^𝑇₃𝜆^𝑘−𝐴^𝑇₃𝜆^𝑘+1≥0. (15) Then for each 𝑘, 𝑢^𝑘+1−𝑢^∗,𝐺𝑢^𝑘−𝑢^𝑘+1≥³_𝑖=1𝜇_𝑖𝑥_𝑖^𝑘+1−𝑥^∗_𝑖²+𝜆^𝑘−𝜆^𝑘+1,𝐴₂𝑥^𝑘_𝑖−𝑥₂^𝑘+1+𝛽𝐴₃𝑥₃^𝑘+1−𝑥^∗₃,𝐴₂𝑥₂^𝑘+1−𝑥^𝑘₂≥²_𝑖=1𝜇_𝑖𝑥_𝑖^𝑘+1−𝑥^∗_𝑖²+𝑥₃^𝑘+1−𝑥^∗₃²_{𝜇3𝐼𝑛3−𝛽𝐴𝑇3𝐴3}+𝜆^𝑘−𝜆^𝑘+1,𝐴₂𝑥^𝑘₂−𝑥₂^𝑘+1−𝛽4𝐴₂𝑥₂^𝑘+1−𝑥^𝑘₂², (16) where the last “≥” follows from the elementary inequality ⟨𝑥,𝑦⟩≥−‖𝑥‖²−14𝑦². (17) Since 𝐴₃𝑥₃^𝑘+1−𝑥^𝑘₃²≤2𝐴₃(𝑥₃^𝑘+1−𝑥^∗₃)²+2𝐴₃𝑥^𝑘₃−𝑥^∗₃², (18) by direct computations, we further obtain that 𝑢^𝑘−𝑢^∗2_𝐺≥𝑢^𝑘+1−𝑢^∗2_𝐺+𝑢^𝑘+1−𝑢^𝑘2_𝐺+2𝜇₁𝑥₁^𝑘+1−𝑥^∗₁²+2𝑥₂^𝑘+1−𝑥^∗₂²_{𝜇2𝐼𝑛2−𝐴𝛽/2𝑇2𝐴2}+2𝑥₃^𝑘+1−𝑥^∗₃²_{𝜇3𝐼𝑛3−𝛽𝐴𝑇3𝐴3}+2𝜆^𝑘−𝜆^𝑘+1,𝐴₂𝑥^𝑘₂−𝑥₂^𝑘+1−𝛽𝐴₂𝑥^𝑘₂−𝑥^∗₂², (19) which, together with 𝐺=𝑀−^{𝛽𝐴𝑇2𝐴2}₀₀, implies 𝑢^𝑘−𝑢^∗2_𝑀≥𝑢^𝑘+1−𝑢^𝑘2_𝐺+𝑢^𝑘+1−𝑢^∗2_𝑀+2𝜇₁𝑥₁^𝑘+1−𝑥^∗₁²+2𝑥₂^𝑘+1−𝑥^∗₂²_{𝜇2𝐼𝑛2−𝛽𝐴𝑇2𝐴2}+2𝑥₃^𝑘+1−𝑥^∗₃²_{𝜇3𝐼𝑛3−𝛽𝐴𝑇3𝐴3}+2𝜆^𝑘−𝜆^𝑘+1,𝐴₂𝑥^𝑘₂−𝑥₂^𝑘+1. (20) Note that 𝑥^𝑘₂−𝑥₂^𝑘+12_{𝛽𝐴𝑇2𝐴2}+2𝜆^𝑘−𝜆^𝑘+1,𝐴₂𝑥^𝑘₂−𝑥₂^𝑘+1+1𝛽𝜆^𝑘−𝜆^𝑘+12=𝛽𝐴₁𝑥₁^𝑘+1+𝐴₂𝑥^𝑘₂+𝐴₃𝑥₃^𝑘+1−𝑏². (21) We complete the proof of this lemma.

With the above preparation, we are ready to prove the convergence of the ADMM with three blocks for solving (3) given the following conditions.

Theorem 3. Let {𝑣^𝑘=(𝑥^𝑘₁,𝑥^𝑘₂,𝑥^𝑘₃,𝜆^𝑘)} be the sequence generated by the ADMM with three blocks. Then {𝑣^𝑘} converges to a KKT point of problem (3) if either of the following conditions holds:(i)𝜇₁>0 and 0<𝛽≤min{𝜇₂/‖𝐴₂‖²₂,𝜇₃/‖𝐴₃‖²₂}; (ii)𝐴₁ is of full column rank, 0<𝛽<𝜇₂‖𝐴₂‖²₂, and 𝛽≤𝜇₃‖𝐴₃‖²₂.

Proof. By the inequality (13), it follows that the sequence {𝐴₂𝑥^𝑘₂,𝐴₃𝑥^𝑘₃,𝜆^𝑘} is bounded. Recall that 𝐴₁𝑥₁^𝑘+1=𝜆^𝑘−𝜆^𝑘+1𝛽−𝐴₂𝑥₂^𝑘+1−𝐴₃𝑥₃^𝑘+1+𝑏. (22) Hence {𝐴₁𝑥^𝑘₁} is also bounded. Moreover, from (13) we see immediately that +∞>^∞_𝑘=1𝛽𝐴₃(𝑥₃^𝑘+1−𝑥^𝑘₃)²+𝛽𝐴₁𝑥₁^𝑘+1+𝐴₂𝑥^𝑘₂+𝐴₃𝑥₃^𝑘+1−𝑏²+^∞_𝑘=12𝜇₁𝑥₁^𝑘+1−𝑥^∗₁²+2𝑥₂^𝑘+1−𝑥^∗₂²_{𝜇2𝐼𝑛2−𝛽𝐴𝑇2𝐴2}+2𝑥₃^𝑘+1−𝑥^∗₃²_{𝜇3𝐼𝑛3−𝛽𝐴𝑇3𝐴3}. (23) According to the condition that 0<𝛽≤min{𝜇₂/‖𝐴₂‖²₂,𝜇₃/‖𝐴₃‖²₂}, we know ^∞_𝑘=1‖𝐴₃𝑥₃^𝑘+1−𝑥^𝑘₃‖²<∞,_∞𝑘=1‖𝐴₁𝑥₁^𝑘+1+𝐴₂𝑥^𝑘₂+𝐴₃𝑥₃^𝑘+1−𝑏‖²<+∞,_∞𝑘=1𝜇₁‖𝑥₁^𝑘+1−𝑥^∗₁‖²<+∞,_∞𝑘=1𝑥₂^𝑘+1−𝑥^∗₂²_{𝜇2𝐼𝑛2−𝛽𝐴𝑇2𝐴2}<+∞,_∞𝑘=1𝑥₃^𝑘+1−𝑥^∗₃²_{𝜇3𝐼𝑛3−𝛽𝐴𝑇3𝐴3}<+∞. (24) It therefore holds that lim_𝑘→∞𝐴₃𝑥₃^𝑘+1−𝑥^𝑘₃²=0,lim_𝑘→∞𝐴₁𝑥₁^𝑘+1+𝐴₂𝑥^𝑘₂+𝐴₃𝑥₃^𝑘+1−𝑏²=0,lim_𝑘→∞𝜇₁𝑥₁^𝑘+1−𝑥^∗₁²=0,lim_𝑘→∞𝑥₂^𝑘+1−𝑥^∗₂²_{𝜇2𝐼𝑛2−𝛽𝐴𝑇2𝐴2}=0,lim_𝑘→∞𝑥₃^𝑘+1−𝑥^∗₃²_{𝜇3𝐼𝑛3−𝛽𝐴𝑇3𝐴3}=0. (25)(26) Therefore, the sequence {𝜇₁‖𝑥^𝑘₁‖², ‖𝑥^𝑘₂‖²_{𝜇2𝐼𝑛2−𝛽𝐴}^𝑇_2𝐴2, ‖𝑥^𝑘₃‖²_{𝜇3𝐼𝑛3−𝛽𝐴}^𝑇_2𝐴2} is bounded, which, together with the boundedness of {𝐴₁𝑥^𝑘₁,𝐴₂𝑥^𝑘₂,𝐴₃𝑥^𝑘₃,𝜆^𝑘}, implies that {𝑥^𝑘₂,𝑥^𝑘₃,𝜆^𝑘} is bounded, and {𝑥^𝑘₁} is bounded given the condition 𝜇₁>0 or 𝐴₁ is of full column rank. Moreover, since 𝑥₃^𝑘+1−𝑥^𝑘₃²=𝐴₃𝑥₃^𝑘+1−𝐴₃𝑥^𝑘₃²+𝑥₃^𝑘+1−𝑥^𝑘₃²_{𝜇3𝐼𝑛3−𝐴𝑇3𝐴3}, (27) by the first equality in (25) and the third equality in (26), it holds that lim_𝑘→∞𝑥₃^𝑘+1−𝑥^𝑘₃=0. (28) We proceed to prove the convergence of ADMM by considering the following two cases.
Case  1  (𝜇₁>0 and 𝛽≤min(𝜇₂/‖𝐴₂‖²₂,𝜇₃/‖𝐴₃‖²₂)). In this case, the sequence {𝑥^𝑘₁} converges to 𝑥^∗₁ and then lim_𝑘→∞𝐴₂𝑥₂^𝑘+1−𝐴₂𝑥^𝑘₂=0,lim_𝑘→∞𝜆^𝑘+1−𝜆^𝑘=0. (29) By the second equality in (26), we deduce from (29) that lim_𝑘→∞𝑥₂^𝑘+1−𝑥^𝑘₂=0. (30) Since {𝑥^𝑘₂,𝑥^𝑘₃,𝜆^𝑘} is bounded, there exist a triple (𝑥^∞₂,𝑥^∞₃,𝜆^∞) and a subsequence {𝑛_𝑘} such that lim_𝑘→∞𝑥^𝑛𝑘₂=𝑥^∞₂,lim_𝑘→∞𝑥^𝑛𝑘₂=𝑥^∞₂,lim_𝑘→∞𝜆^𝑛𝑘=𝜆^∞, (31)which by combining (25), (29) with given conditions, implies lim_𝑘→∞𝑥^𝑛𝑘₂⁺¹=𝑥^∞₂,lim_𝑘→∞𝑥^𝑛𝑘₂⁺¹=𝑥^∞₂,lim_𝑘→∞𝜆^𝑛𝑘+1=𝜆^∞. (32) Note that 0∈𝜕𝑓₁𝑥₁^𝑘+1−𝐴^𝑇₁𝜆^𝑘+1+𝐴^𝑇₁𝐴₂𝑥^𝑘₂−𝑥₂^𝑘+1+𝐴^𝑇₁𝐴₃𝑥^𝑘₃−𝑥₃^𝑘+1,0∈𝜕𝑓₂𝑥₂^𝑘+1−𝐴^𝑇₂𝜆^𝑘+1+𝐴^𝑇₂𝐴₃𝑥^𝑘₃−𝑥₃^𝑘+1,0∈𝜕𝑓₃𝑥₃^𝑘+1−𝐴^𝑇₃𝜆^𝑘+1,𝜆^𝑘+1=𝜆^𝑘−𝛽𝐴₁𝑥₁^𝑘+1+𝐴₂𝑥₂^𝑘+1+𝐴₃𝑥₃^𝑘+1. (33) Then, by taking the limits on both sides of (33), using (25) and (29), and invoking the upper semicontinuous of 𝜕𝑓₁(⋅),  𝜕𝑓₂(⋅), and 𝜕𝑓₃(⋅) [24], one can immediately write 0∈𝜕𝑓₁𝑥^∗−𝐴^𝑇₁𝜆^∞,0∈𝜕𝑓₂𝑥^∞₂−𝐴^𝑇₂𝜆^∞,0∈𝜕𝑓₃𝑥^∞₃−𝐴^𝑇₃𝜆^∞,𝐴₁𝑥^∗+𝐴₂𝑥^∞₂+𝐴₃𝑥^∞₃=𝑏, (34) which indicates (𝑥^∗₁,𝑥^∞₂,𝑥^∞₃,𝜆^∞) is a KKT point of problem (3). Hence, the inequality (13) is also valid if (𝑥^∗₁,𝑥^∗₂,𝑥^∗₃,𝜆^∗) is replaced by (𝑥^∗₁,𝑥^∞₂,𝑥^∞₃,𝜆^∞). Then it holds that 2𝛽𝐴₂𝑥₂^𝑘+1−𝐴₂𝑥^∞₂²+𝛽𝐴₃𝑥₃^𝑘+1−𝐴₃𝑥^∞₃²+1𝛽𝜆^𝑘+1−𝜆^∞2≤2𝛽𝐴₂𝑥^𝑘₂−𝐴₂𝑥^∞₂²+𝛽𝐴₃𝑥^𝑘₃−𝐴₃𝑥^∞₃²+1𝛽𝜆^𝑘−𝜆^∞2, (35) which yields lim_𝑘→∞𝑥^𝑘₂−𝑥^∞₂²_{𝐴𝑇2𝐴2}=0,lim_𝑘→∞𝑥^𝑘₃−𝑥^∞₃²_{𝐴𝑇3𝐴3}=0,lim_𝑘→∞𝜆^𝑘=𝜆^∞. (36)(37) By adding the last two equalities in (26) to (36), we know lim_𝑘→∞𝑥^𝑘₂=𝑥^∞₂,lim_𝑘→∞𝑥^𝑘₃=𝑥^∞₃. (38) Therefore, we have shown that the whole sequence {(𝑥^𝑘₁,𝑥^𝑘₂,𝑥^𝑘₃,𝜆^𝑘)} converges to (𝑥^∗₁,𝑥^∞₂,𝑥^∞₃,𝜆^∞) under condition (i) in Theorem 3.
Case  2  (𝐴₁ is of full column rank, 0<𝛽<𝜇₂/‖𝐴₂‖²₂, and 𝛽≤𝜇₃/‖𝐴₃‖²₂). In this case, the sequence {𝑥^𝑘₂} converges to 𝑥^∗₂ and {𝑥^𝑘₁} is bounded. From the second equality in (25) and (28), we have lim_𝑘→∞‖𝐴₁𝑥₁^𝑘+1−𝐴₁𝑥^𝑘₁‖=0,lim_𝑘→∞‖𝜆^𝑘−𝜆^𝑘+1‖=0. (39) Since 𝐴₁ is of full column rank, it therefore holds that lim_𝑘→∞‖𝑥₁^𝑘+1−𝑥^𝑘₁‖=0. (40) Let (𝑥^∞₁,𝑥^∞₃,𝜆^∞) be a cluster point of the sequence {𝑥^𝑘₁,𝑥^𝑘₃,𝜆^𝑘}. Following a similar proof in Case 1, we are able to show (𝑥^∞₁,𝑥^∗₂,𝑥^∞₃,𝜆^∞) is a KKT point of problem (3) and the whole sequence {(𝑥^𝑘₁,𝑥^𝑘₂,𝑥^𝑘₃,𝜆^𝑘)} converges to this point.

Remark 4 (see [22]). the authors proved the convergence of the ADMM under the conditions that 𝑓₁,𝑓₂, and 𝑓₃ are strongly convex and 0<𝛽<min_1≤𝑖≤3{𝜇_𝑖/3‖𝐴_𝑖‖²₂}. Our result improves the upper bound min_1≤𝑖≤3{𝜇_𝑖/3‖𝐴_𝑖‖²₂} by min{𝜇₂/‖𝐴₂‖²₂,𝜇₃/‖𝐴₃‖²₂}. Moreover, in our condition (ii), the strongly convexity assumption is only imposed on 𝑓₂ and 𝑓₃ while 𝑓₁ is not necessarily strongly convex with positive modulus.

4. The Relaxed ADMM with Three Blocks

For the ADMM with two blocks, Ye and Yuan [25] developed a variant of alternating direction method with an optimal step size. Numerical results demonstrated that an additional computation on the optimal step size would improve the efficiency of the new variant of ADMM. In this section, by adopting the essential idea of Ye and Yuan [25], we propose a relaxed ADMM with three blocks to accelerate the ADMM via an optimal step size. For notational simplicity, we write Φ𝑢^𝑘,𝑢^𝑘∶=3𝛽4𝐴₂(𝑥^𝑘₂−𝑥^𝑘₂)²+𝛽𝐴₃(𝑥₃−𝑥^𝑘₃)²+1𝛽𝜆^𝑘−𝜆^𝑘2+𝜆^𝑘−𝜆^𝑘,𝐴₂𝑥^𝑘₂−𝑥^𝑘₂+𝐴₃𝑥^𝑘₃−𝑥^𝑘₃. (41) With 𝑢^𝑘=(𝑥^𝑘₂,𝑥^𝑘₃,𝜆^𝑘), the new iterate of extended ADMM is produced by 𝑢^𝑘+1=𝑢^𝑘−𝛾𝛼^∗𝑢^𝑘−𝑢^𝑘,𝛾∈(0,2), (42) where 𝑢^𝑘 is the solution of (5) and 𝛼^∗ is defined by 𝛼^∗Φ𝑢∶=^𝑘,𝑢^𝑘‖𝑢^𝑘−𝑢^𝑘‖²_𝐺. (43)

Lemma 5. Let the sequence {𝑢^𝑘} be generated by the relaxed ADMM with three blocks. Then, if 0<𝛽≤𝜇₃/‖𝐴₃‖²₂, the following statements are valid:(i)Φ(𝑢^𝑘,𝑢^𝑘)≥(1/6)‖𝑢^𝑘−𝑢^𝑘+1‖²_𝐺 and thus 𝛼^∗≥1/6; (ii)‖𝑢^𝑘+1−𝑢^∗‖²_𝐺≤‖𝑢^𝑘−𝑢^∗‖²_𝐺−(1/36)𝛾(2−𝛾)‖𝑢^𝑘−𝑢^𝑘‖²_𝐺 − (1/3)𝛾𝜇₁‖𝑥^𝑘₁−𝑥^∗₁‖² − (1/3)𝛾𝜇₂‖𝑥^𝑘₂−𝑥^∗₂‖²−(1/3)𝛾‖𝑥^𝑘₃−𝑥^∗₃‖²_{𝜇3𝐼−𝛽𝐴}^𝑇_3𝐴3.

Proof. By direct computations to Φ(𝑢^𝑘,𝑢^𝑘), we know that Φ𝑢^𝑘,𝑢^𝑘=3𝛽4𝐴₂𝑥^𝑘₂−𝑥^𝑘₂²+𝛽𝐴₃(𝑥₃−𝑥^𝑘₃)²+1𝛽𝜆^𝑘−𝜆^𝑘2+𝜆^𝑘−𝜆^𝑘,𝐴₂𝑥^𝑘₂−𝑥^𝑘₂+𝐴₃𝑥^𝑘₃−𝑥^𝑘₃≥3𝛽4𝐴₂(𝑥^𝑘₂−𝑥^𝑘₂)²+𝛽𝐴₃(𝑥₃−𝑥^𝑘₃)²+1𝛽𝜆^𝑘−𝜆^𝑘2−𝛽2𝐴₂𝑥^𝑘₂−𝑥^𝑘₂²−1𝜆2𝛽^𝑘−𝜆^𝑘2−3𝛽4𝐴₃(𝑥^𝑘₃−𝑥^𝑘₃)²−1𝜆3𝛽^𝑘−𝜆^𝑘2=𝛽4𝐴₂(𝑥^𝑘₂−𝑥^𝑘₂)²+𝛽4𝐴₃(𝑥^𝑘₃−𝑥^𝑘₃)²+1𝜆6𝛽^𝑘−𝜆^𝑘2, (44) where the second inequality follows Cauchy inequality. It therefore holds that Φ𝑢^𝑘,𝑢^𝑘≥16𝑢^𝑘−𝑢^𝑘2_𝐺, (45) which completes the proof of the first part. By Lemma 1 and the elementary inequality (17), it can be easily verified that 𝑢^𝑘−𝑢^∗,𝐺𝑢^𝑘−𝑢^𝑘≥Φ𝑢^𝑘,𝑢^𝑘+𝜇₁𝑥^𝑘₁−𝑥^∗₁²+𝜇₂𝑥^𝑘₂−𝑥^∗₂²+𝑥^𝑘₃−𝑥^∗₃²_{𝜇3𝐼𝑛3−𝛽𝐴𝑇3𝐴3} (46) and then 𝑢^𝑘+1−𝑢^∗2_𝐺=𝑢^𝑘−𝑢^∗−𝛾𝛼^∗𝑢^𝑘−𝑢^𝑘2_𝐺≤𝑢^𝑘−𝑢^∗2_𝐺−𝛾2−𝛾𝛼^∗2×𝑢^𝑘−𝑢^𝑘2_𝐺−2𝛾𝛼^∗𝜇₁𝑥^𝑘₁−𝑥^∗₁²−2𝜇₂𝛾𝛼^∗𝑥^𝑘₂−𝑥^∗₂²−2𝛾𝛼^∗𝑥^𝑘₃−𝑥^∗₃²_{𝜇3𝐼𝑛3−𝛽𝐴𝑇3𝐴3}. (47) This, together with the fact that 𝛼^∗≥1/6, completes the proof.

Based on the above inequality, we are able to prove the following convergence result of the relaxed ADMM with three blocks. Since the proof is in line with that of Theorem 3, we omit it.

Theorem 6. Let {𝑣^𝑘=(𝑥^𝑘₁,𝑥^𝑘₂,𝑥^𝑘₃,𝜆^𝑘)} be the sequence generated by the relaxed ADMM. Then {𝑣^𝑘} converges to a KKT point of problem (3) under the conditions that 0<𝛽≤𝜇₃/‖𝐴₃‖²₂ and 𝐴₁,𝐴₂, and 𝐴₃ are of full column rank.

5. Conclusion Remarks

In this paper, we take a step to investigate the ADMM for separable convex programming problems with three blocks. Based on the contractive analysis of the distance between the sequence and the solution set, we establish theoretical results to guarantee the global convergence of ADMM with three blocks under weaker conditions than those employed in [22]. By adopting the essential idea of [25], we also present a relaxed ADMM with an optimal step size to accelerate the ADMM and prove its convergence under mild assumptions.

Acknowledgment

The first author is supported by the Natural Science Foundation of Jiangsu Province and the National Natural Science Foundation of China under Project no. 71271112. The second author is supported by university natural science research fund of jiangsu province under grant no. 13KJD110002.