Advances in Numerical Analysis

Advances in Numerical Analysis / 2013 / Article

Research Article | Open Access

Volume 2013 |Article ID 517452 | 5 pages | https://doi.org/10.1155/2013/517452

A Global Convergence of LS-CD Hybrid Conjugate Gradient Method

Academic Editor: William J. Layton
Received25 Jun 2013
Revised03 Sep 2013
Accepted04 Sep 2013
Published22 Oct 2013

Abstract

Conjugate gradient method is one of the most effective algorithms for solving unconstrained optimization problem. In this paper, a modified conjugate gradient method is presented and analyzed which is a hybridization of known LS and CD conjugate gradient algorithms. Under some mild conditions, the Wolfe-type line search can guarantee the global convergence of the LS-CD method. The numerical results show that the algorithm is efficient.

1. Introduction

Consider the following nonlinear programs: where denotes an -dimensional Euclidean space and is continuously differentiable function.

As you know, conjugate gradient method is a line search method that takes the following form: where is a descent direction of at and is a stepsize obtained by some one-dimensional line search. If is the current iterate, we denote , , and , respectively. If is available and inverse, then leads to the Newton method and results in the steepest descent method [1]. The search direction is generally required to satisfy , which guarantees that is a descent direction of at [2]. In order to guarantee the global convergence, we sometimes require to satisfy a sufficient descent condition as follows: where is a constant and is the Euclidean norm. In line search methods, the well-known conjugate gradient method has the following form: Different conjugate gradient algorithms correspond to different choices for the parameter , where can be defined by or by other formulae. The corresponding methods are called FR (Fletcher-Reeves) [3], PRP (Polak-Ribiére-Polyak) [4, 5], DY (Dai-Yuan) [6], CD (conjugate descent [7]), LS (Liu-Storey [8]), and HS (Hestenes-Stiefel [9]) conjugate gradient method, respectively.

Although the above mentioned conjugate gradient algorithms are equivalent to each other for minimizing strong convex quadratic functions under exact line search, they have different performance when using them to minimize nonquadratic functions or when using inexact line searches. For general objective function, the FR, DY, and CD methods have strong convergence properties, but they may have modest practical performance due to jamming. On the other hand, the methods of PRP, LS, and HS in general may not be convergent, but they often have better computational performance.

Touati-Ahmed and Storey [10] have given the first hybrid conjugate algorithm; the method is combinations of different conjugate gradient algorithms; mainly it is being proposed to avoid the jamming phenomenon. Recently, some kinds of new hybrid conjugate gradient methods are given in [1117]. Based on the new method, we focus on hybrid conjugate gradient methods and analyze the global convergence of the methods with Wolfe-type line search.

The rest of this paper is organized as follows. The algorithm is presented in Section 2. In Section 3 the global convergence is analyzed. We give the numerical experiments in Section 4.

2. Description of Algorithm

Algorithm 1.
Step 0. Initialization:
given a starting point , choose parameters
Set  .
Step 1. If , stop; else go to Step 2.
Step 2. Compute step size , such that
Step 3. Let ; if , stop; otherwise, go to Step 4.
Step 4. Compute the search direction where .
Step 5. Let , and go to Step 2.

Throughout this paper, the following basic assumptions on the objective function are assumed, which have been widely used in the literature to analyze the global convergence of the conjugate gradient methods.(H2.1) The objective function is continuously differentiable and has a lower bound on the level set , where is the starting point.(H2.2) The gradient of is Lipschitz continuous in some neighborhood of ; that is, there exists a constant , such that Since is decreasing, it is clear that the sequence generated by Algorithm 1 is contained in .

3. Global Convergence of Algorithm

Now we analyze the global convergence of Algorithm 1.

Lemma 2. Suppose that assumptions (H2.1) and (H2.2) hold; the sequences and   are to be generated by Algorithm 1, if for all ; then

Proof. If , then , and we get When , multiplying by we obtain It follows from and that Therefore, the result is true.

Lemma 3. Suppose that assumptions (H2.1) and (H2.2) hold, and consider any iteration of the form (2), where is a descent direction and satisfies the Wolfe conditions (7). Then, the Zoutendijk condition holds.

Proof. From (7), we have In addition, the assumption (H2.2) gives Combing these two relations, we have which with implies that Thus, Noting that is bounded below, (15) holds.
Furthermore, from Lemma 2 and (3), we can easily obtain the following condition:

Theorem 4. Suppose that is a starting point for which assumptions (H2.1) and (H2.2) hold. Consider Algorithm 1; then, one has either for some finite or

Proof. The first statement is easy to show, since the only stopping point is in Step 3. Thus, assume that the algorithm generates an infinite sequence ; if the statement is false, there exists a constant , such that From (8), we have Squaring both sides of the above equation, we get that is, From the definitions of , and , we have Thus, we can get On the other hand, multiplying by we obtain Considering that and , we have which indicates that Dividing the above inequality (28) by , we obtain Using the above inequality recursively and noting that we have Then, from (23) and (35), it holds that Thus, it is easy to obtain This contradicts the Zoutendijk condition (15). Therefore, the conclusion holds.

4. Numerical Experiments

In this section, we give the numerical results of Algorithm 1 to show that the method is efficient for unconstrained optimization problems. We set the parameters , and and use MATLAB 7.0 to test the chosen problems on a PC with 2.10 GHz CPU processor, 1.0 GB RAM memory, and Linux operation system. We also use the condition or It-max > 5000 as the stopping criterion (It-max denotes the maximal number of iterations). When the limit of 5000 function evaluations was exceeded, the run was stopped, which is indicated by “NaN.” The problems that we tested are from [17, 19].Prob 1 ,Prob 2 ,Prob 3 ,Prob 4 ,Prob 5 ,Prob 6 .

Tables 1, 2, and 3 show the computation results.


Prob NI

1(4, 5, 10.1)(NaN, NaN, NaN)NaN5000
2(6.5, 23)(14.92256489843370, 14.92256511253481)−0.9999999999999318
3(7, 11)(0.00000007501213, −0.49999972373559)−0.2499999999999216
4(2.5, 11.9)(5.99999999984147, 15.99997571516633)−0.99999999998863160
5(7, 9.8)(NaN, NaN)NaN5000
6(−3, −1, −3, −1)(NaN, NaN, NaN, NaN)NaN5000

: the initial point; : the final point; : the final value of the objective function; NI: the number of times of iteration for each problem.

Prob NI

1(4, 5, 10.1)(−1.74444446043301, −1.74444445793385, 6.97777777040829)4.0987765965103127
2(6.5, 23)(14.92256524764775, 14.92256496282052)−0.9999999999999226
3(7, 11)(−0.00000003058419, −0.49999989218958)−0.2499999999999934
4(2.5, 11.9)(5.99999999851030, 15.99997486954042)−0.99999999998783323
5(7, 9.8)(−0.43256275978261, 0.10814076312008)0.7891770364031019
6(−3, −1, −3, −1)(2.50000015783668, 2.50000010044152, 5.24999984443432, −3.50000007174394)−79.87499999999992 29

: the initial point; : the final point; : the final value of the objective function; NI: the number of times of iteration for each problem.

Prob NI

1(4, 5, 10.1)(−1.74444443639771, −1.74444443412601, 6.97777778236907)1.9210214903710225
2(6.5, 23)(14.92256501285285, 14.92256521587144)−0.9999999999999615
3(7, 11)(0.00000000238593, −0.49999999997767)−0.2500000000000023
4(2.5, 11.9)(5.99999999856486, 15.99997486532800)−0.99999999998782311
5(7, 9.8)(−0.43256260505806, 0.10814059874208)0.7891770364031123
6(−3, −1, −3, −1)(2.49999974693426, 2.49999983895816, 5.25000008609708, −3.49999988497012)−79.8749999999998716

: the initial point; : the final point; : the final value of the objective function; NI: the number of times of iteration for each problem.

Because conjugate gradient algorithms are devised for solving large-scale unconstrained optimization problems, we chose some large-scale problems from [18] and compared the performance of the hybrid LS-CD method (Algorithm 1 in Section 2) with the LS method and CD method.

From Tables 1, 2, 3, and 4, we see that the performance of Algorithm 1 is better than that of the CD and the LS methods for some problems. Therefore, our numerical experiments show that the algorithm is efficient.


ProbDimLSCDLS-CD
NI/NF/NGNI/NF/NGNI/NF/NG

PEN 110051/142/9262/223/18251/168/125
100033/125/8352/181/16533/164/117
1000021/118/7231/157/12121/132/102
TRIG 100 305/399/398 NaN305/399/398
500 343/424/423NaN343/424/423
ROSEX 50052/112/107 92/267/238 50/186/157
1000 70/149/14598/287/255 70/246/183

Prob: the test problem name from [18]; Dim: the problem dimension; NI: the iterations number; NF: the function evaluations number; NG: the gradient evaluations number.

Acknowledgments

The authors would like to thank the anonymous referee for the careful reading and helpful comments and suggestions that led to an improved version of this paper. This work was supported in part by the Foundation of Hunan Provincial Education Department under Grant (nos. 12A077 and 13C453) and the Educational Reform Research Fund of Hunan University of Humanities, Science, and Technology (no. RKJGY1320).

References

  1. J. Nocedal and J. S. Wright, Numerical Optimization, Springer, New York, NY, USA, 1999. View at: Publisher Site | MathSciNet
  2. Y. Yuan, Numerical Methods for Nonlinear Programming, Shanghai Scientific & Technical Publishers, Shanghai, China, 1993.
  3. R. Fletcher and C. Reeves, “Function minimization by conjugate gradients,” The Computer Journal, vol. 7, pp. 149–154, 1964. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  4. E. Polak and G. Ribiére, “Note sur la convergence de méthodes de directions conjuguées,” Revue Française de Recherche Opérationnelle, no. 16, pp. 35–43, 1969. View at: Google Scholar | Zentralblatt MATH | MathSciNet
  5. B. T. Polyak, “The conjugate gradient method in extreme problems,” Computational Mathematics and Mathematical Physics, vol. 9, pp. 94–112, 1969. View at: Publisher Site | Google Scholar
  6. Y. H. Dai and Y. Yuan, “A nonlinear conjugate gradient method with a strong global convergence property,” SIAM Journal on Optimization, vol. 10, no. 1, pp. 177–182, 1999. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  7. R. Fletcher, “Unconstrained Optimization,” in Practical Methods of Optimization, vol. 1, part 1, John Wiley & Sons, New York, NY, USA, 2nd edition, 1987. View at: Google Scholar | MathSciNet
  8. Y. Liu and C. Storey, “Efficient generalized conjugate gradient algorithms. I. Theory,” Journal of Optimization Theory and Applications, vol. 69, no. 1, pp. 129–137, 1991. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  9. M. R. Hestenes and E. Stiefel, “Method of conjugate gradient for solving linear systems,” Journal of Research of the National Bureau of Standards, vol. 49, pp. 409–436, 1952. View at: Publisher Site | Google Scholar
  10. D. Touati-Ahmed and C. Storey, “Efficient hybrid conjugate gradient techniques,” Journal of Optimization Theory and Applications, vol. 64, no. 2, pp. 379–397, 1990. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  11. Y. H. Dai and Y. Yuan, “An efficient hybrid conjugate gradient method for unconstrained optimization,” Annals of Operations Research, vol. 103, pp. 33–47, 2001. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  12. N. Andrei, “A scaled BFGS preconditioned conjugate gradient algorithm for unconstrained optimization,” Applied Mathematics Letters, vol. 20, no. 6, pp. 645–650, 2007. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  13. N. Andrei, “A hybrid conjugate gradient algorithm for unconstrained optimization as a convex combination of Hestenes-Stiefel and Dai-Yuan,” Studies in Informatics and Control, vol. 17, no. 4, pp. 55–70, 2008. View at: Google Scholar | MathSciNet
  14. Y.-H. Dai and C.-X. Kou, “A nonlinear conjugate gradient algorithm with an optimal property and an improved Wolfe line search,” SIAM Journal on Optimization, vol. 23, no. 1, pp. 296–320, 2013. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  15. S. Babaie-Kafaki and N. Mahdavi-Amiri, “Two modified hybrid conjugate gradient methods based on a hybrid secant equation,” Mathematical Modelling and Analysis, vol. 18, no. 1, pp. 32–52, 2013. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  16. W. Jia, J. H. Zong, and X. D. Wang, “An improved mixed conjugate gradient method,” Systems Engineering Procedia, vol. 4, pp. 219–225, 2012. View at: Publisher Site | Google Scholar
  17. M. Sun and J. Liu, “A new conjugate method and its global convergence,” Journal of Information and Computing Science, vol. 8, no. 1, pp. 75–80, 2013. View at: Google Scholar
  18. J. J. Moré, B. S. Garbow, and K. E. Hillstrom, “Testing unconstrained optimization software,” ACM Transactions on Mathematical Software, vol. 7, no. 1, pp. 17–41, 1981. View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
  19. W. Hock and K. Schittkowski, “Test examples for nonlinear programming codes,” Journal of Optimization Theory and Applications, vol. 30, no. 1, pp. 127–129, 1981. View at: Google Scholar | Zentralblatt MATH

Copyright © 2013 Xiangfei Yang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1056 Views | 680 Downloads | 2 Citations
 PDF  Download Citation  Citation
 Download other formatsMore
 Order printed copiesOrder