Advance in Nonlinear Analysis: Algorithm, Convergence and ApplicationsView this Special Issue
Research Article | Open Access
Yuanying Qiu, Dandan Cui, Wei Xue, Gaohang Yu, "A Self-Adjusting Spectral Conjugate Gradient Method for Large-Scale Unconstrained Optimization", Abstract and Applied Analysis, vol. 2013, Article ID 814912, 8 pages, 2013. https://doi.org/10.1155/2013/814912
A Self-Adjusting Spectral Conjugate Gradient Method for Large-Scale Unconstrained Optimization
This paper presents a hybrid spectral conjugate gradient method for large-scale unconstrained optimization, which possesses a self-adjusting property. Under the standard Wolfe conditions, its global convergence result is established. Preliminary numerical results are reported on a set of large-scale problems in CUTEr to show the convergence and efficiency of the proposed method.
Consider the following unconstrained optimization problem: where is a nonlinear smooth function and its gradient is available. Conjugate gradient methods are very efficient for solving (1), especially when the dimension is large, and have the following iterative form: where is a steplength obtained by a line search, and is the search direction defined by where is a scalar and denotes the gradient of at point .
There are at least six formulas for , which are given below: where and denotes the Euclidean norm. In the above six methods, HS, PR, and LS methods are especially efficient in real computations, but one may not globally converge for general functions. FR, CD, and DY methods are globally convergent, but they perform much worse. To combine the good numerical performance of HS method and the nice global convergence property of DY method, Dai and Yuan  proposed an efficient hybrid formula for which is defined as the following form:
Their studies suggested that the HSDY method (5) has the same advantage of avoiding the propensity of short steps as the HS method . They also proved that the HSDY method with the standard wolfe line search produces a descent search direction at each iteration and converges globally. Descent condition may be crucial for the convergence analysis of conjugate gradient methods with inexact line searches [2, 3]. Further, there are some modified conjugate gradient methods [4–7] which possess the sufficiently descent property without any line search condition. Recently, Yu  proposed a spectral version of HSDY method: where with , . The numerical experiments show that this simple preconditioning technique benefits to its performance.
In this paper, based on a new conjugate condition , we propose a new hybrid spectral conjugate gradient method with defined by where
A full description of DS-HSDY method is formally given as follows.
Algorithm 1 (DS-HSDY conjugate gradient method).
Data. Choose constants , , and . Given an initial point , set . Let .
Step 1. If , then stop.
Step 2. Determine satisfying the standard Wolfe condition: Then update .
Step 3. Compute , and . Then update such as Set and go to Step 1.
The rest of the paper is organized as follows. In the next section, we show that the DS-HSDY method possesses a self-adjusting property. In Section 3, we establish its global convergence result under the standard Wolfe line search conditions. Section 4 gives some numerical results on a set of large-scale unconstrained test problems in CUTEr to illustrate the convergence and efficiency of the proposed method. Finally we have a Conclusion section.
2. Self-Adjusting Property
In this section, we prove that the DS-HSDY method possesses a self-adjusting property. To begin with, we assume that otherwise, a stationary point has been found, and define the two following important quantities: The quantity shows the size of , where is a quantity showing the descent degree of . In fact, if , is a descent direction. Furthermore, if for some constant , then we have the sufficient descent condition On the other hand, it follows from (12) that Hence Combining with (17) yields Dividing both sides of (18) by and using (7), we obtain It follows from (19) and the definitions of and that
Additionally, we assume that there exist positive constants and such that then we have the following result.
Proof. Summing (20) over the iterates and noting that , we get
Since , it follows from (25) that
Equations (21), (26), and yield
Furthermore, we have
Thus (24) holds with .
Noting that and , it is easy to derive that (22) and (23) hold with and , respectively. Hence the proof is complete.
Theorem 3. Consider the method (2), (8), and (12), where is a descent direction. If (21) holds, then for any , there exist constants , , and such that, for any k, the relations hold for at least values of .
Proof. The proof is similar to the Theorem 2 in , so we omit it here.
3. Global Convergence
Throughout the paper, we assume that the following assumptions hold.
Assumption 1. (1) is bounded below in the level set ;
(2) in a neighborhood of , is differentiable and its gradient is Lipschitz continuous; namely, there exists a constant such that Under Assumption 1 on , we could get a useful lemma.
Lemma 4. Suppose that is a starting point for which Assumption 1 holds. Consider any method in the form (2), where is a descent direction and satisfies the weak Wolfe conditions; then one has that For DS-HSDY method, one has the following global convergence result.
Theorem 5. Suppose that is a starting point for which Assumption 1 hold. Consider DS-HSDY method; if for all , then one has that Further, the method converges in the sense that
Proof. Since , it is obvious that . Assume that . By (10) and the definition of the , we have , then . In addition, from (8), we have
Let , then we have . By (12) with replaced by , and multiplying it by , we have
From this and the formula for , we get
At the same time, if we define
it follows from (39) that
Then we have by (10), with replaced by , that
Furthermore, we have
The above relation, (40), (41), and the fact that imply that . Thus by induction, (32) holds.
We now prove (33) by contradiction and assume that there exists some constant such that Since , we have that Dividing both sides of (44) by and using (36) and (40), we obtain In addition, since and , we have that , or equivalently which with (37) yields By (45) and (47), we obtain Using (48) recursively and noting that , Then we get from this and (43) that which indicates This contradicts the Zoutendijk condition (31). Hence we complete the proof.
4. Numerical Result
In this section, we compare the performance of DS-HSDY method to method , HSDY method , and S-HSDY method . The test problems are taken from CUTEr (http://hsl.rl.ac.uk/cuter-www/problems.html) with the standard initial points. All codes are written in double precision Fortran and complied with f77 (default compiler settings) on a PC (AMD Athlon XP 2500 + CPU 1.84 GHz). Our line search subroutine computes such that the Wolfe conditions (10) and (11) hold with and . We use the condition or as the stopping criterion. The numerical results are presented in Tables 1, 2, 3, and 4 with the form NI/Nfg/T, where we report the dimension of the problem (), the number of iteration (NI), the number of function evaluations (Nfg), and the CPU time () in 0.01 seconds.
Figure 1 shows the performance of these test methods relative to the CPU time, which were evaluated using the profiles of Dolan and Moré . That is, for each method, we plot the fraction of problems for which the method is within a factor of the best time. The top curve is the method that solved the most problems in a time that was within a factor of the best time. Clearly, the left side of the figure gives the percentage of the test problems for which a method is the fastest. As we can see from Figure 1, DS-HSDY method has the best performance which performs better than S-HSDY method, HSDY method, and the well-known method.
In this paper, we proposed an efficient hybrid spectral conjugate gradient method with self-adjusting property. Under some suitable assumptions, we established the global convergence result for the DS-HSDY method. Numerical results indicated that the proposed method is efficient for large-scale unconstrained optimization problems.
This work was partly supported by the National Natural Science Foundation of China (no. 61262026), the JGZX program of Jiangxi Province (20112BCB23027), and the science and technology program of Jiangxi Education Committee (LDJH12088). The authors would also like to thank the editor and an anonymous referees for their comments and suggestions on the first version of the paper, which led to significant improvements of the presentation.
- Y. H. Dai and Y. Yuan, “An efficient hybrid conjugate gradient method for unconstrained optimization,” Annals of Operations Research, vol. 103, pp. 33–47, 2001.
- Y. H. Dai and Y. Yuan, “A nonlinear conjugate gradient method with a strong global convergence property,” SIAM Journal on Optimization, vol. 10, no. 1, pp. 177–182, 1999.
- Z. X. Wei, G. Y. Li, and L. Q. Qi, “Global convergence of the Polak-Ribière-Polyak conjugate gradient method with an Armijo-type inexact line search for nonconvex unconstrained optimization problems,” Mathematics of Computation, vol. 77, no. 264, pp. 2173–2193, 2008.
- W. W. Hager and H. Zhang, “A new conjugate gradient method with guaranteed descent and an efficient line search,” SIAM Journal on Optimization, vol. 16, no. 1, pp. 170–192, 2005.
- L. Zhang, W. Zhou, and D.-H. Li, “A descent modified Polak-Ribière-Polyak conjugate gradient method and its global convergence,” IMA Journal of Numerical Analysis, vol. 26, no. 4, pp. 629–640, 2006.
- G. Yuan, “Modified nonlinear conjugate gradient methods with sufficient descent property for large-scale optimization problems,” Optimization Letters, vol. 3, no. 1, pp. 11–21, 2009.
- G. Yu, L. Guan, and W. Chen, “Spectral conjugate gradient methods with sufficient descent property for large-scale unconstrained optimization,” Optimization Methods and Software, vol. 23, no. 2, pp. 275–293, 2008.
- G. Yu, Nonlinear self-scaling conjugate gradient methods for large-scale optimization problems [Ph.D. thesis], Sun Yat-Sen University, Guangzhou, China, 2007.
- G. Li, C. Tang, and Z. Wei, “New conjugacy condition and related new conjugate gradient methods for unconstrained optimization,” Journal of Computational and Applied Mathematics, vol. 202, no. 2, pp. 523–539, 2007.
- Y.-H. Dai, “New properties of a nonlinear conjugate gradient method,” Numerische Mathematik, vol. 89, no. 1, pp. 83–98, 2001.
- J. C. Gilbert and J. Nocedal, “Global convergence properties of conjugate gradient methods for optimization,” SIAM Journal on Optimization, vol. 2, no. 1, pp. 21–42, 1992.
- E. D. Dolan and J. J. Moré, “Benchmarking optimization software with performance profiles,” Mathematical Programming, vol. 91, no. 2, pp. 201–213, 2002.
Copyright © 2013 Yuanying Qiu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.