Abstract
The augmented Lagrangian method (ALM) is one of the most successful first-order methods for convex programming with linear equality constraints. To solve the two-block separable convex minimization problem, we always use the parallel splitting ALM method. In this paper, we will show that no matter how small the step size and the penalty parameter are, the convergence of the parallel splitting ALM is not guaranteed. We propose a new convergent parallel splitting ALM (PSALM), which is the regularizing ALM’s minimization subproblem by some simple proximal terms. In application this new PSALM is used to solve video background extraction problems and our numerical results indicate that this new PSALM is efficient.
1. Introduction
Many problems arising from machine learning, such as compressive sensing [1, 2], the video background extraction problem [3–5], batch images alignment [6, 7], and transform invariant low-rank textures [8, 9] can be formulated as separable convex programming with linear constraints. In this paper, we consider the following two-block separable convex programming:where , , the matrix on ; , the -dimensional vector on ; is the subset of . Here, are the proper lower semicontinuous convex functions, i.e.,
The solution set of problem (1), denoted by , is assumed to be nonempty. Note that theoretical results to problem (1) can easily be extended to matrix variables.
Let be a penalty parameter, and the augmented Lagrangian functions of problem (1) can be written aswhere is the Lagrangian multiplier. It is well-known that the augmented Lagrangian method (ALM) [10] is one of the most successful first-order methods for convex programming with linear constraints. Applying it to problem (1), we can obtain the following procedure:where is the step size. It has been proved that the sequence generated by the iterative scheme (4) is Fejér monotone and has the global convergence. In many practical problems, the proximal mapping of , defined ashas closed-form expression. For example, consider the video background extraction problem given bywhere
are the singular values of the matrix L, is the singular value decomposition of L, and are elements of the matrix S. Then, by [11], the proximal mapping (5) is rewritten as
Similarly, by [12], the proximal mapping (5) is given aswhere is defined byand is the sign function. However, the proximal mapping of does not admit closed-form expression so that the iterative scheme (4) for problem (5) has to resort some inner solver to compute its minimization problem in each iteration, which inevitably creates a chain of problems, such as an efficient solver to solve the minimization problem and stopping criterion to ensure the global convergence of inexact version of the iterative scheme (5). Therefore, we should not ignore the separable structure of the objective function of problem (1). That is, we’d better split the minimization subproblem in (5) into some smaller scale subproblems to fully utilize the individual property of .
The alternating direction method of multipliers (ADMM) can decompose a complicated problem into some small-scale subproblems and solve these subproblems in a sequential manner or a parallel manner. Focusing on the way of splitting the augmented Lagrangian function in the spirit of the well-known alternating direction method (ADM), Tao and Yuan [4] have proposed a variant of the alternating splitting augmented Lagrangian method (ASALM) with convergent property, which can solve three-block separable convex programming. He et al. [13] propose a splitting method for solving a separable convex minimization problem with linear constraints, where the objective function is expressed as the sum of m individual functions without coupled variables. Aybat and Iyenga [14] have designed a variant of ADMM with increasing penalty for stable principal component pursuit (ADMIP) to solve the SPCP problem. Its preliminary computational tests show that ADMIP works very well in practice and outperforms ASALM which is a state-of-the-art ADMM algorithm for the SPCP problem with a constant penalty parameter.
In this paper, following the abovementioned procedure, we split the minimization problem of ALM in the Jacobian manner and get the following iterative scheme:
The iterative scheme (11) takes advantages of the separable structure and parallel architectures. Unfortunately, it suffers from the weakness that its convergence cannot be guaranteed under the convex assumption of and . To verify this, we shall apply the iterative scheme (11) to solve problem (1) given in Section 2 and show that the generated sequence diverges even . So, in Section 2, we propose a new parallel splitting ALM for problem (1), which is not only convergent but also keeps the good properties of the original ALM. Our numerical results in Section 3 indicate that the new PSALM is efficient.
The remainder of this paper is organized as follows. In Section 2, an example is given showing that the iterative scheme (11) maybe diverge, and a new parallel splitting ALM for problem (1) is proposed, which has the global convergence in both ergodic and nonergodic senses. For illustration of our main results, two numerical examples are given in Section 3 and a brief conclusion is drawn in Section 4.
2. Algorithm and Convergence Results
In this section, we will show that the iterative scheme (11) with any small positive α and β can be diverge so that a new parallel splitting ALM for problem (1) is proposed and the corresponding convergence results are established.
In what follows, will stand for the n-dimensional Euclidean space, and set
By the characterization of optimality condition for convex optimization, problem (1) can be readily characterized as the following mixed variational inequalities denoted by MVI :where
Because is a linear mapping with the skew-symmetric coefficient matrix, it satisfies the following property:
The solution set of MVI is denoted by , which is nonempty under the assumption .
Based on the prototype algorithm proposed by He and Yuan in [15], we can introduce the following prototype algorithm to solve MVI :
A prototype algorithm for MVI is denoted by ProAlo
[Prediction] For given , find and Q satisfyingwhere the matrix Q has the property which is positive definite
[Correction] Determine a nonsingular matrix M, a scalar , and generate the new iterate via
Similar to [15], we have the following convergence condition, which ensures that the ProAlo is globally convergent and has worst-case convergence rate in ergodic and nonergodic senses.
Convergence Condition
The matrices are positive definite.
Now we give an example to show that the iterative scheme (11) maybe diverge for problem (1).
Example 1. Consider the linear equation [16]Obviously, the linear equation (18) is a special case of problem (1) with the specifications: . The augmented Lagrangian function of problem (18) is given byThen, applying the iterative scheme (11) to solve problem (18) gives the following procedure:whereThree eigenvalues of the matrix P areFor any , we have , let is the spectral radius of P. Hence, the iterative scheme (20) with is divergent, which also indicates that the iterative scheme (11) diverge for problem (1) under the traditional assumption. In the following, we shall design a new parallel splitting ALM for problem (1), which maximally inherits the structure of the iterative scheme (11).
Algorithm 1. new PSALM Step 1. Given an initial point ; let parameters , , and the tolerance . Set . Step 2. Compute the new iterate via Step 3. If , then stop; otherwise, replace by k, and go to Step 2.
Remark 1. The new PSALM iterative scheme (23) not only preserves the main advantage of the ALM iterative scheme (11) in treating the objective functions and individually, but also numerically accelerates (11) with some values of the step size α, e.g., , which can be achieved when .
In the following, we further make the following assumption about problem (1).
Assumption 1. The matrices in problem (1) are both full column ranks.
To establish the convergence results of new PSALM, we first cast it to a special case of the ProAlo. To accomplish this, let us define an auxiliary sequence as follows:This together with the updated formula of λ in (23) givesThus,where the matrix M is defined as
Lemma 1. Let be the sequence generated by the new PSALM and be defined as in (24), and it holds thatwhere the matrix Q is defined by
Proof. By the first-order optimality condition of the two minimization problems in (23), for any , we haveThe definition of the variable in (24) givesAdding the above three inequalities, using the notation of and , we can get the compact form (28) and this completes the proof.
Remark 2. If , then by (24), we have . So, . Substituting it into the right-hand side of (28), we haveSo, , and the stopping criterion of the PSALM is reasonable.
With inequalities (26) and (28) in hand, to establish the convergence results of the PSALM, we only need to ensure the matrices defined as in (27) and (29) satisfy the convergence condition.
Lemma 2. Let the matrices be defined as in (27) and (29). If and Assumption 1 holds, then(i)The matrix is positive definite(ii)The matrix is symmetric and positive definite(iii)The matrix is symmetric and positive definite
Proof. (i)From the definition of the matrix Q, we have which is positive definite from and Assumption 1.(ii)From the definition of the matrices , we have which is symmetric and positive definite by Assumption 1.(iii)Similarly, from the definition of , we havewhereFrom Assumption 1, we only need to prove the matrix is positive definite. In fact, it can be written aswhere denotes the matrix Kronecker product. Thus, we only need to prove the 3 order matrixis positive definite. Its three eigenvalues are , . Then, solving the following three inequalitieswe get . Therefore, the matrix G is positive definite for any . In addition, from its expression, the matrix is obviously symmetric. This completes the proof.
Lemma 2 indicates that the matrices defined as in (27) and (29) satisfy convergence condition, and according to the result in [15], we get the following convergence results of new PSALM.
Theorem 1 (global convergence). Let be the sequence generated by new PSALM. Then, it converges to some , which belongs to .
Theorem 2 (ergodic convergence rate). Let be the sequence generated by new PSALM, be the corresponding sequence defined as in (24). SetThen, for any integer , we have
Theorem 3 (nonergodic convergence rate). Let be the sequence generated by new PSALM. Then, for any and integer , we havewhere is a constant.
Note that the right-hand side of (41) is independent of the distance between the initial iterate and the solution set . Therefore, (41) is not a reasonable criterion to measure the nonergodic convergence rate. In the following, we give a refined result to measure the nonergodic convergence rate of the sequence generated by new PSALM.
Lemma 4. Let be the sequence generated by new PSALM. Then, for any , we have
Proof. The proof is similar to that of Lemma 3.1 in [15] and is omitted for brevity of this paper.
Theorem 4 (refined ergodic convergence rate). Let be the sequence generated by the PSALM, and setThen, for any integer , there exists a constant such that
Proof. Choose . Then, for any , we have . From the notion of and (15), we haveSet in (43), and we obtainCombining the above two inequalities givesSumming the abovementioned inequality from to yieldsDividing the both sides of the above inequality by t, we obtainThen, it follows from the convexity of and thatSince (51) holds for any λ, we setand obtainSetand we thus obtainSince , we haveCombining the above two inequalities giveswhich completes the proof.
The following example is to test the influence of the parameters on this new PSALM.
Example 2 (go on with Example 1). For this new PSALM, we set , and the resulting iterative scheme isFurthermore, we set the initial point , and the stopping criterion is set asor the number of iterations exceeds 10000.
The left subplot of Figure 1 graphically depicts the numerical results when . It reveals that when is fixed, the larger α is, the better of the new PSALM’s performance is. The right subplot of Figure 1 graphically depicts the numerical results when . It reveals that when ν takes some values near 1, the new PSALM always performs best. This is in conformity with the intuition because the large value of this parameter makes the proximal term taking a too heavy weight in the objective function , leads to a very small increment of the variable , and finally decreases the convergence speed of the new PSALM. On the contrary, the small value of this parameter also makes the step size to be very small. Therefore, how to choose a suitable combination of is a challenging issue.
Remark 3. The step size α is at the denominator of the right-hand side terms of (45). Therefore, the bigger α is, the smaller the right-hand side terms of (45) are, and the new PSALM may need less iteration to generate an ε-approximate solution of problem (1).
3. Numerical Results
In this section, we demonstrate the potential efficiency of this new PSALM by solving the video background extraction problem (6).
3.1. Simulation Study
Firstly, we generate a low-rank matrix and a sparse matrix . Secondly, we get an observation matrix be an observation matrix. Thirdly, we want to recover the low-rank matrix and the sparse matrix by problem (6) with D as the input matrix and as the unknown matrices. In the following, let represent the low-rank ratio of and the ratios of the number of nonzeros entries of (i.e., ), respectively. We use the following Matlab script to generate matrix D:(i)(ii)(iii)
We set and tested
We empirically set in problem (6), and in the PSALM. The initial iterate , and use the stopping criterionor the number of iterations exceeds 1000. Let be the numerical solution of problem (6) obtained by the PSALM, and we measure the quality of recovery by the relative error to , which is defined by
In Table 1, we report the recovery results on simulated matrices with size . Note that for the recovered sparse matrix , to eliminate the roundabout error, we have reset the component whose the absolute value is less than 0.01 to be 0, i.e., . From this table, it can be seen that (i) the rank of the low-rank matrix can be exactly recovered, while the sparse matrix is recovered approximately; (ii) though is not recovered completely, the relative error RelErr indicates that the precision of the recovered is high.
In Figure 2, we present the convergence result for problem (6) with . From this figure, it can be observed that when the number of iterations , the relative error RelErr achieves at a stable state. It decreases almost linearly with k before , while it almost does not decrease any more after .
3.2. An Application Example
In this section, we apply the new PSALM(NEWPSALM) to solve a concrete problem of model (6): the video background extraction problem [17], which aims to subtract the background from (the low-rank matrix L) a video clip (stacking as the matrix D) and meanwhile detect the moving objects (the sparse matrix S). We set in problem (6). We have downloaded the video clip: Highway from the Internet. For comparison, we also applied the iteration method in [4] (denoted by TYVASALM) and the iteration method in [13] (denoted by HTYIMA) to solve this problem.
Figure 3 presents the extracted backgrounds and foregrounds images by TYVASALM, HTYIMA, and NEWPSALM. Clearly, all methods can extract satisfactory results
(a)
(b)
(c)
Table 2 lists the detail experimental results, where the number of iterations (Iteration), CPU time in seconds (CPU), the rank of the recovered final low-rank matrix (Rank), and the number of nonzeros of the recovered final sparse matrix (Sparse) are reported. From the results given in Table 2, we see that NEWPSALM is faster than the other two methods in terms of both iteration numbers and CPU time. In particular, NEWPSALM outperforms all the other two methods in recovering low rank and sparse components. These results clearly illustrate the efficiency and robustness of NEWPSALM.
4. Conclusion
In this paper, we have proposed a new parallel splitting augmented Lagrangian method(NEWPSALM) for two-block separable convex programming and have established its various convergence results, including global convergence, ergodic, and nonerogdic convergence rate. Numerical results indicate that it is efficient for the video background extraction problem.
Practical experiments indicate that the ALM and its various variants with the step size can often accelerate the convergence speed, while in the new PSALM, to ensure the step size , the parameter ν must larger than 1, such as only if . However, as we have mentioned in Example 2, large ν often decreases the convergence speed of the new PSALM. Therefore, the ALM type methods with less restrictions imposed on deserve further researching, and we leave this problem as a future research work.
Data Availability
The data used to support the findings of this study are available from http://www.SceneBackgroundModeling.net.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (71532015, 71528007, and 71771179) and Foundation of First Class Discipline of Zhejiang-A (Zhejiang University of Finance and Economics-Statistics).