- About this Journal ·
- Abstracting and Indexing ·
- Aims and Scope ·
- Annual Issues ·
- Article Processing Charges ·
- Articles in Press ·
- Author Guidelines ·
- Bibliographic Information ·
- Citations to this Journal ·
- Contact Information ·
- Editorial Board ·
- Editorial Workflow ·
- Free eTOC Alerts ·
- Publication Ethics ·
- Reviewers Acknowledgment ·
- Submit a Manuscript ·
- Subscription Information ·
- Table of Contents
Abstract and Applied Analysis
Volume 2013 (2013), Article ID 912846, 6 pages
Convergence Analysis of the Relaxed Proximal Point Algorithm
1School of Economics and Management, Southeast University, Nanjing 210096, China
2Department of Mathematics, Nanjing University, Nanjing 210093, China
Received 3 May 2013; Accepted 9 June 2013
Academic Editor: Abdellah Bnouhachem
Copyright © 2013 Min Li and Yanfei You. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Recently, a worst-case convergence rate was established for the Douglas-Rachford alternating direction method of multipliers (ADMM) in an ergodic sense. The relaxed proximal point algorithm (PPA) is a generalization of the original PPA which includes the Douglas-Rachford ADMM as a special case. In this paper, we provide a simple proof for the same convergence rate of the relaxed PPA in both ergodic and nonergodic senses.
The finite-dimensional variational inequality (VI), denoted by , is to find a vector such that where is a nonempty closed convex set in and is a monotone mapping from into itself. The solution set, denoted by is assumed to be nonempty. We refer to [1–4] for the pivotal roles of VIs in various fields such as economics, transportation, and engineering.
As is well known, proximal point algorithm (PPA), which was presented originally in  and mainly developed in [6, 7], is a well-developed approach to solving . Let be the current approximation of a solution of (1); then PPA generates the new iterate by solving the following auxiliary VI: where is a positive constant. Compared to the monotone VI (1), (2) is easier to handle since it is a strongly monotone VI. In this paper, we focus on the relaxed proximal point algorithm (PPA) proposed by Gol’shtein and Tret’yakov in , which combines the PPA step (3a) with a relaxation step (3b) as follows:
where is a relaxation factor and is a symmetric positive semidefinite matrix. In particular, is called an under-relaxation factor when or an over-relaxation factor when , and the relaxed PPA reduces to the original PPA (2) when and . For convenience, we still use the notation to represent the nonnegative number in our analysis.
The Douglas-Rachford alternating direction methods of multipliers (ADMM) scheme proposed by Glowinski and Marrocco in  (see also ) is a commonplace tool to solve the convex minimization problem with linear constraints and a separable objective function as follows: where , , , , and are closed convex sets and : and : are convex smooth functions. The iterative scheme of ADMM for solving (4) at the -th iteration runs as
where and is a positive constant. As shown in , ADMM can be regarded as an application of the relaxed PPA with (i.e., the original PPA (2)) and Without further assumption on , the matrix defined previously can be guaranteed as a symmetric and positive semidefinite matrix. Recently, He and Yuan in  have shown a worst-case convergence rate of the ADMM scheme (5a), (5b), and (5c) in an ergodic sense. You et al. in  have proved the same convergence rate of the Lagrangian PPA-based contraction methods with nonsymmetric linear proximal term in an ergodic sense. The purpose of this paper is to establish the convergence rate of the relaxed PPA (3a) and (3b) in both ergodic and nonergodic senses.
In this section, we review some preliminaries which are useful for further discussions. More specially, we recall a useful characterization on , the variational reformulation of (4), the relationship of the ADMM in [9, 10], and the relaxed PPA in  for solving this variational reformulation.
Theorem 1. The solution set of is convex, and it can be characterized as
In the following, we will give a variational reformulation of (4). It is easy to see that the model (4) can be characterized by a variational inequality problem: find such that where Note that the mapping is monotone since and are convex. As shown in , the ADMM scheme (5a), (5b), and (5c) is identical with the following iterative scheme in a cyclical sense: Based on the definition (6) of the matrix , we can rewrite (11a), (11b), (11c), and (12) as a special case of the relaxed PPA with immediately.
3. The Contraction of the Relaxed Proximal Point Algorithm
In this section, we prove the contraction of the relaxed PPA. First, we give an important lemma.
Proof. First, using (3a), we have Since (see (3b)), we have Thus, it suffices to show that By setting , , , and in the identity we derive that On the other hand, using (3b), we have Combining the last two equations, we obtain (17). The assertion (14) follows immediately. The proof is completed.
Proof. Setting in (14), we get On the other hand, since is monotone and , we have It follows from the previous two inequalities that The proof is completed.
4. Ergodic Worst-Case Convergence Rate
In this section, we will establish an ergodic worst-case convergence rate for the relaxed PPA in the sense that after iterations of such an algorithm, we can find such that with and .
Proof. From (14), we have Since is monotone, from the previous inequality, we have Summing the inequality (29) over , we obtain Since , is a convex combination of and thus . Using the notation of , we derive The assertion (27) follows from the previous inequality immediately.
It follows from Theorem 4 that the sequence is bounded. According to (21), the sequence is also bounded. Therefore, there exists a constant such that Recall that is the average of . Thus, we have . For any , we get Thus, for any given , after at most iterations, we have which means that is an approximate solution of with an accuracy of . That is, a worst-case convergence rate of the relaxed PPA in an ergodic sense is established.
Note that this convergence rate is in an ergodic sense and is a convex combination of the previous vectors with equal weights. One may ask if we can establish the same convergence rate in a nonergodic sense directly for the sequence generated by the relaxed PPA (3a) and (3b), and this is the main purpose of the next section.
5. Nonergodic Worst-Case Convergence Rate
This section shows that the relaxed PPA has a worst-case convergence rate in a nonergodic sense. First, we establish two important inequalities in the following lemmas.
Proof. Setting in (3a), we have Note that (3a) is also true for , and thus we have Setting in the previous inequality, we obtain Adding (36) and (38) and using the monotonicity of , we get (35) immediately.
Proof. First, adding the term to the both sides of (35), we get Reordering in the previous inequality to , we get Substituting the term (see (3b)) into the left-hand side of the last inequality, we obtain (39). The proof is completed.
Next, we prove that is monotonically nonincreasing.
Note that is convex and closed (see Theorem 1). Let . Then, for any given , Theorem 9 shows that the relaxed PPA (3a) and (3b) needs at most iterations to ensure that . Recall that is a solution of if . In other words, if , we have since is a positive semidefinite matrix. And thus from (3a), it follows that which means that is a solution of according to (1). A worst-case convergence rate in a nonergodic sense for the relaxed PPA (3a) and (3b) is thus established from Theorem 9.
6. Concluding Remarks
This paper established the worst-case convergence rate in both ergodic and nonergodic senses for the relaxed PPA. Recall that ADMM is a primal application of the relaxed PPA with . And thus ADMM also has the same worst-case convergence rate in both ergodic and nonergodic senses.
This work was supported by the National Natural Science Foundation of China (Grant no. 11001053), Program for New Century Excellent Talents in University (Grant no. NCET-12-0111), and Natural Science Foundation of Jiangsu Province, China (Grant no. BK2012662).
- S. Dafermos and A. Nagurney, “Supply and demand equilibration algorithms for a class of market equilibrium problems,” Transportation Science, vol. 23, no. 2, pp. 118–124, 1989.
- M. C. Ferris and J. S. Pang, “Engineering and economic applications of complementarity problems,” SIAM Review, vol. 39, no. 4, pp. 669–713, 1997.
- D. Kinderlehrer and G. Stampacchia, An Introduction to Variational Inequalities and their Applications, vol. 88, Academic Press, New York, NY, USA, 1980.
- A. Nagurney, Network Economics: A Variational Inequality Approach, vol. 1, Kluwer Academic, Dordrecht, The Netherlands, 1993.
- B. Martinet, “Régularisation D'inéquations Variationnelles par Approximations Successives,” Revue Française dInformatique et de Recherche Opérationelle, vol. 4, pp. 154–158, 1970.
- R. T. Rockafellar, “Monotone operators and the proximal point algorithm,” SIAM Journal on Control and Optimization, vol. 14, no. 5, pp. 877–898, 1976.
- R. T. Rockafellar, “Augmented Lagrangians and applications of the proximal point algorithm in convex programming,” Mathematics of Operations Research, vol. 1, no. 2, pp. 97–116, 1976.
- E. G. Gol'shtein and N. V. Tret'yakov, “Modified Lagrangians in convex programming and their generalizations,” Mathematical Programming Study, no. 10, pp. 86–97, 1979.
- R. Glowinski and A. Marrocco, “Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité, d'une classe de problèmes de Dirichlet non linéaires,” RAIRO, vol. 9, no. R-2, pp. 41–76, 1975.
- D. Gabay and B. Mercier, “A dual algorithm for the solution of nonlinear variational problems via finite-element approximations,” Computers & Mathematics with Applications, vol. 2, no. 1, pp. 17–40, 1976.
- X. J. Cai, G. Y. Gu, B. S. He, and X. M. Yuan, A Proximal Point Algorithm Revisit on the Alternating Direction Method of Multipliers, Science China Mathematics.
- B. He and X. Yuan, “On the convergence rate of the Douglas-Rachford alternating direction method,” SIAM Journal on Numerical Analysis, vol. 50, no. 2, pp. 700–709, 2012.
- Y. F. You, X. L. Fu, and B. S. He, Lagrangian PPA-Based Contraction Methods for Linearly Constrained Convex Optimization, manuscript, 2013.
- F. Facchinei and J. S. Pang, Finite-Dimensional Variational Inequalities and Complementarity Problems, Springer Series in Operations Research, Springer, New York, NY, USA, 2003.
- Yu. Nesterov, “Gradient methods for minimizing composite functions,” Mathematical Programming, 2012.