Table of Contents Author Guidelines Submit a Manuscript
Journal of Applied Mathematics
Volume 2019, Article ID 8728196, 6 pages
Research Article

A New Hybrid Algorithm for Convex Nonlinear Unconstrained Optimization

1Department of Operation Research and Intelligent Techniques, College of Computer Sciences and Mathematics, University of Mosul, Iraq
2University of Telafer, Iraq

Correspondence should be addressed to Huda I. Ahmed; moc.liamg@27aeaduh

Received 24 December 2018; Revised 16 February 2019; Accepted 13 March 2019; Published 1 April 2019

Academic Editor: Urmila Diwekar

Copyright © 2019 Eman T. Hamed et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


In this study, we tend to propose a replacement hybrid algorithmic rule which mixes the search directions like Steepest Descent (SD) and Quasi-Newton (QN). First, we tend to develop a replacement search direction for combined conjugate gradient (CG) and QN strategies. Second, we tend to depict a replacement positive CG methodology that possesses the adequate descent property with sturdy Wolfe line search. We tend to conjointly prove a replacement theorem to make sure global convergence property is underneath some given conditions. Our numerical results show that the new algorithmic rule is powerful as compared to different standard high scale CG strategies.

1. Introduction

The nonlinear CG technique could be a helpful procedure to search out the minimum value of any nonlinear function through exploitation unconstrained nonlinear optimization strategies.

Let us contemplate the subsequent unconstrained minimization problem: where is a real-valued smooth function. The repetitious formula is given aswhere is associate optimum step-size computed by any line search procedure [1]. The search direction is defined asand denotes , while is a positive scalar.

Well-known established instances of square measure are from Hestenes-Stiefel, Fletcher-Reeves, Polak-Ribière, Liu-Storey, Dai-Yuan, and Dai-Liao (see [2, 3, 3, 4, 47], respectively), within the already-existing convergence analysis and implementation of the CG methodology, the weak Wolfe conditions square measure [8]:If we choose , also the strong Wolfe conditions [8] consist of (4) and Now, this allows us to review Ibrahim et al. work [9] that could be a work that considers unconstrained minimization problems. Ibrahim et al. recommend a search direction that is outlined asand is the BFGS updating matrix if step-size isThe scalar is chosen to ensure conjugacy.

Also, another addition is regarded as an equivalent unconstrained minimization problem. Ibrahim et al. [10] recommend another search direction outlined asMatrix I is the identity and <0.

In addition, Ibrahim et al. [11] propose another search direction that is outlined asThe positive scalar and are the Hestenes-Stiefel parameters.

2. A New Proposed Search Direction

In this section, we advise a replacement search direction as deduced from Ibrahim et al. [911]. The new search direction is outlined aswhereas (the approximation matrix of BFGS updating matrix) denotes approximations of Hessian matrix G, and could be a positive constant. In order to drive the value of , we have a tendency to multiply either side of (11) by to induceSince and (Perry condition [12]), thenIn order to see value of , we tend to additionally multiply either side of (11) to induceAs a result of processes of multiplication shown in (13) and (16), we tend to reach the following search directions in (17a), (17b), and (17c). The subsequent new search directions are our new projected algorithmic program:In the following step, we have a tendency to assume that each search direction got to satisfy the subsequent descent condition (, for all k). Also, there should exist a constant c>0 in order to get For all , the new direction that is outlined in (18) ought to satisfy the sufficient descent condition. The enough descent conditions are going to be used later to prove our new theorem (see Section 2.2). So to prove our new theorem, we have a tendency to necessarily use the subsequent given assumptions (see Section 2.1).

2.1. Assumptions in [9, 11, 13]

(A1) is twice continuously differentiable.(A2)f is uniformly convex; that is, m and M are positive constants, such thatfor all , and G is the Hessian matrix of f.(A3)The matrix G is Lipschitz continuous at the point ; that is, there exists the positive constant L satisfyingfor all x in a neighborhood of .

2.2. A New Theorem for Proving Sufficient Descent Property

To prove that our new projected algorithm defined in (17a), (17b), and (17c) satisfies sufficiently descent condition, we tend to suppose that assumptions in (Section 2.1) square measure are true. Additionally, the sequence is bounded. Then, sufficient descent condition (18) is true for all .

Proof. When taking (17a), (17b), and (17c) and achieving the descent condition, we can see the following:We get value , which is bounded away from zero. Therefore (18) is true.

2.3. Lemma in [10, 14]

Suppose that assumptions in Section 2.1 are true. Then, the step-size which is determined by (2) satisfies when is a positive constant.

2.4. New Theorem for Proving Global Convergence Property

Having demonstrated the important and necessary properties of regression algorithms, we now come to the proof of the necessary property to be present in all numerical optimization algorithms. Let us demonstrate the new algorithm defined in (17a), (17b), and (17c). To achieve a global convergence property, assume that the theory in Section 2.2 and the assumptions in Section 2.1 are correct. Then

Proof. By linking the theory in Section 2.2 and lemma in Section 2.3 we give the following result:Hence, from our new theorem in Section 2.2, we can define that , and we can therefore simplify (23) as . Thus, the proof is established.

3. A New Form for the Parameter in Conjugate Gradient Method

To obtain an updated version of the conjugate gradient method associated with a new parameter , we compare the standard CG-method specified in (3) with the proposed new algorithm specified in (17a), (17b), and (17c):Multiplying both sides of (24) by we getOrknowing that I is the identity matrix. Moreover, Then, the new is and . It should be noted that when using exact line searches assuming that and the matrix is the identity matrix, the new standard will be reduced to HS. This condition must also be met, with (positive constant), and sinceafter these conditions we know the new parameter isIn conjunction with both parametersand the search direction is defined as

3.1. Outlines of the New Proposed CG-Algorithm in (29a), (29b), (29c), (29d), and (29e)

By assuming , and by setting the iteration k=1, we get the following steps.

Step 1. Set , if , then stop.

Step 2. Compute by strong Wolfe line search conditions in (4) and in (6).

Step 3. Generate by implementing (29a), (29b), (29c), (29d), and (29e).

Step 4. If [15] is satisfied, then go to Step 1; if not, then continue.

3.2. Assumptions for Proving the Convergence Analysis Property of the New Algorithm in (29a), (29b), (29c), (29d), and (29e)

Let us suppose the following.(i)The level set (ii)The condition is satisfied where ; also since f is a uniformly convex function on S, then there exists a constant , such thatFrom both (19a), (19b), and (31), we can get the following:

3.3. New Theorem for Proving Sufficiently Descent Directions of the New Algorithm in (29a), (29b), (29c), (29d), and (29e)

If we have a tendency to assume that (A3) in Section 2.1 is true and if we assume that conditions (i) and (ii), outlined in (30) and in (31), respectively, are true, then the new proposed search direction defined in (29e) satisfies the sufficient descent condition.

Proof. By using the mathematical induction we demonstrate this new theorem; for initial direction (k=1) we havewhich satisfies (18).
We suppose that
And if we multiply both sides of (29a) by , then we can getsince , since :and since The above equations ensure that condition (18) is satisfied. Hence, the proof is complete.

3.4. Lemma in [1, 16]

By assuming that assumptions in Section 2.1 are true, by supposing that any CG-method with search direction could be a descent direction provided that the step-size is obtained by the strong Wolfe line search conditions, and ifthen

3.5. A New Theorem for Proving the Global Convergence Property of the New Algorithm in (29a), (29b), (29c), (29d), and (29e)

If we suppose that assumptions (i) and (ii) in (30) and in (31), respectively, are true and if we have a tendency to assume that (A3) in Section 2.1 is additionally true, then the search directions outlined in (29e) are descent provided that the step-size is computed using (4) and (6), and then

Proof. sinceand since and ,then from (A3) in part (Section 2.1), from condition (ii) in (31), and from (28) we get

4. Numerical Results and Comparisons

In this work, we have a tendency to compare our new proposed CG-method with some normal classical CG strategies like Hestenes-Stiefel [HS] and Dai-Yuan [DY] by exploitation of fifty unconstrained nonlinear cases; take a look at functions obtained from Andrei [17, 18]. As for the computer program, it was stopped when . In addition, the term , which is defined in (29a), can be computed as . Numerical results for new algorithm in (29a), (29b), (29c), (29d), and (29e) with and are for the total of 50 test problems from the CUTE library. The Sigma plotting software was used to graph the data. We adopt the performance profiles given by Dolan and Moré [19]. Thus, new, HS, and DY strategies are compared in terms of NOI, CPU, and NOF in Figures 13. For each method, we plotted the fraction of problems that were solved correctly within a factor of the best time. In the figures, the uppermost curve is the method that solves the most problems within a factor t of the best time. In Figures 13, the new method outperforms the HS algorithm and DY method in terms of NOI, CPU, and NOF. If the solution had not converged after 800 seconds, the program was terminated. Generally, convergence was achieved within this time limit; functions for which the time limit was exceeded are denoted by “F” for fail-in.

Figure 1: Performance profiles based on number of iterations.
Figure 2: Performance profiles based on CPU time.
Figure 3: Performance profiles based on function evaluation.

5. Conclusions

At the end of this work we were able to obtain a new direction of research defined in (29a), (29b), (29c), (29d), and (29e). This new trend is a hybrid trend that combines pedigree techniques with Quasi-Newton ones. Through the theories presented in the research, the new trend in (29a), (29b), (29c), (29d), and (29e) proved that it satisfies the requirement of sufficient proportions and ensures the property of global convergence. In addition, we have presented a new scalar () which ensures the sufficient descent directions. Moreover, under some conditions, we have established that the new proposed algorithm is a globally convergent algorithm for uniformly convex functions under the strong Wolfe line search conditions. The numerical results show that when we choose the value of parameters in (13) and in (16), we obtain best numerical results as compared to other similar numerical results.

Data Availability

The data used the support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.


The research is supported by College of Computer Sciences and Mathematics, University of Mosul, Republic of Iraq, under Project no. 8728196.


  1. W. Sun and Y.-X. Yuan, Optimization Theory and Methods: Nonlinear Programming, Springer, New York, NY, USA, 2006. View at MathSciNet
  2. Y. H. Dai and L. Z. Liao, “New conjugacy conditions and related nonlinear conjugate gradients methods,” Applied Mathematical Optimization, vol. 43, pp. 87–101, 2001. View at Publisher · View at Google Scholar
  3. Y. H. Dai and Y. Yuan, “A nonlinear conjugate gradient method with a strong global convergence property,” SIAM Journal on Optimization, vol. 10, no. 1, pp. 177–182, 1999. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  4. R. Fletcher and C. M. Reeves, “Function minimization by conjugate gradients,” The Computer Journal, vol. 7, pp. 149–154, 1964. View at Publisher · View at Google Scholar · View at MathSciNet
  5. M. R. Hestenes and E. Stiefel, “Methods of conjugate gradients for solving linear systems,” Journal of Research of the National Bureau of Standards, vol. 49, pp. 409–436, 1952. View at Publisher · View at Google Scholar · View at MathSciNet
  6. E. Polak and G. Ribière, “Note Sur la convergence de methods de directions Conjugue'e, ESAIM,” Mathematical Modeling and Numerical Analysis, vol. 3, no. 16, pp. 35–43, 1969. View at Google Scholar · View at MathSciNet
  7. Y. Liu and C. Storey, “Efficient generalized conjugate gradient algorithms, Part 1,” Journal of Optimization Theory and Applications, vol. 69, no. 1, pp. 129–137, 1991. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  8. P. Wolfe, “Convergence conditions for ascent methods(II): some corrections,” SIAM Review, vol. 13, no. 2, pp. 185–188, 1971. View at Publisher · View at Google Scholar
  9. M. A. Ibrahim, M. Mamat, and W. J. Leong, “BFGS method: a new search direction,” Saints Malaysian, vol. 43, pp. 1593–1599, 2014. View at Google Scholar
  10. M. A. H. Ibrahim, M. Mamat, P. L. Ghazali, and Z. Salleh, “The scaling of hybrid method in solving unconstrained optimization method,” Far East Journal of Mathematical Sciences, vol. 99, no. 7, pp. 983–991, 2016. View at Publisher · View at Google Scholar · View at Scopus
  11. M. A. H. Ibrahim, M. Mamat, and A. Z. M. Sofi, “A modified search direction of Broyden family method and its global convergence,” Journal of Engineering and Applied Sciences, vol. 12, no. 17, pp. 4504–4507, 2017. View at Google Scholar · View at Scopus
  12. A. Perry, “A modified conjugate gradient algorithm,” Operations Research, vol. 26, pp. 1073–1078, 1987. View at Google Scholar
  13. R. Jaafar, M. Mamat, and I. Mohd, “A new scaled hybrid modified BFGS algorithms for unconstrained optimization,” Applied Mathematical Sciences, vol. 7, pp. 263–270, 2013. View at Publisher · View at Google Scholar
  14. J. Nocedal and S. J. Wright, Numerical Optimization, Springer, 2006.
  15. M. J. Powell, “Restart procedure for the conjugate gradient methods,” Mathematics Programming, vol. 12, pp. 241–254, 1977. View at Publisher · View at Google Scholar
  16. H. Yabe and M. Takano, “Global convergence properties of new nonlinear conjugate gradient methods for unconstrained optimization,” Computational Optimization and Application, vol. 28, pp. 203–225, 2004. View at Publisher · View at Google Scholar
  17. N. Andrei, “An unconstrained optimization test functions collection,” Advanced Modeling and Optimization. The Electronic International Journal, vol. 10, no. 1, pp. 147–161, 2008. View at Google Scholar · View at MathSciNet
  18. N. Andrei, “Open problems in nonlinear conjugate gradient algorithms for unconstrained optimization,” Bulletin of the Malaysian Mathematical Sciences Society. Second Series, vol. 34, no. 2, pp. 319–330, 2011. View at Google Scholar · View at MathSciNet
  19. E. D. Dolan and J. J. Moré, “Benchmarking optimization software with performance profiles,” Mathematical Programming, vol. 91, no. 2, pp. 201–213, 2002. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus