The Global Convergence of a New Mixed Conjugate Gradient Method for Unconstrained Optimization

Abstract

We propose and generalize a new nonlinear conjugate gradient method for unconstrained optimization. The global convergence is proved with the Wolfe line search. Numerical experiments are reported which support the theoretical analyses and show the presented methods outperforming CGDESCENT method.

1. Introduction

This paper is concerned with conjugate gradient methods for unconstrained optimization min𝑓(𝑥),𝑥∈ℜ^𝑛, (1.1) where 𝑓(𝑥)∶ℜ^𝑛→ℜ is continuously differentiable and bounded from below. Starting from an initial point 𝑥₁, a nonlinear conjugate gradient method generates sequences {𝑥_𝑘} and {𝑑_𝑘} by the below iteration 𝑥_𝑘+1=𝑥_𝑘+𝛼_𝑘𝑑_𝑘,𝑘≥1, (1.2) where 𝛼_𝑘 is a step length which is determined by a line search and the direction 𝑑_𝑘 is generated as 𝑑_𝑘=−𝑔_𝑘,if𝑘=1,−𝑔_𝑘+𝛽_𝑘𝑑_𝑘−1,if𝑘≥2, (1.3) where 𝑔_𝑘=∇𝑓(𝑥_𝑘) is the gradient of 𝑓(𝑥) at 𝑥_𝑘 and 𝛽_𝑘 is a scalar.

Different conjugate gradient algorithms correspond to different choices for the scale parameter 𝛽_𝑘. The well-known formulae of 𝛽_𝑘 are given by 𝛽^FR_𝑘=𝑔_𝑘²𝑔_𝑘−1²,𝛽^PR_𝑘=𝑔^T_𝑘𝑔_𝑘−𝑔_𝑘−1𝑔_𝑘−1²,𝛽^DY_𝑘=𝑔_𝑘²𝑑^T_𝑘−1𝑔_𝑘−𝑔_𝑘−1,𝛽^HS_𝑘=𝑔^T_𝑘𝑔_𝑘−𝑔_𝑘−1𝑑^T_𝑘−1𝑔_𝑘−𝑔_𝑘−1, (1.4) which are called Fletcher-Reeves [1] (FR), Polak-Ribière-Polyak [2] (PRP), Dai-Yuan [3] (DY), and Hestenes-Stiefel [4] (HS), respectively. Though FR and DY have strong convergence properties, they may have modest practical performance. While PRP and HS often have better computational performance, but they may not generally be convergent.

These motivate us to derive some efficient algorithms. In this paper, we focus on mixed conjugate gradient methods. These methods are combinations of different conjugate gradient methods. The aim of this paper is to propose the new methods that possess both convergence and well numerical results.

The line search in the conjugate gradient algorithms is often based on the Wolfe inexact line search 𝑓𝑥_𝑘+𝛼_𝑘𝑑_𝑘≤𝑓𝑥_𝑘+𝛿𝛼_𝑘𝑔^T_𝑘𝑑_𝑘,𝑔𝑥_𝑘+𝛼_𝑘𝑑_𝑘^T𝑑_𝑘≥𝜎𝑔^T_𝑘𝑑_𝑘, (1.5)(1.6) where 0<𝛿<𝜎<1.

Many research on the parameter 𝛽_𝑘 have been concerned [5–7]. Such as Al-Baali [8] proved that FR global convergent with inexact line search in which 𝜎<1/2. Liu et al. [9] spread the results of [8] to the case of 𝜎=1/2. Dai and Yuan [10] gave an example when 𝜎>1/2, FR may produce a rise direction.

PRP is famous as the best performance of all conjugate gradient methods which is the restart method in nature. When the direction 𝑑_𝑘−1 is small and the factor 𝑔_𝑘−𝑔_𝑘−1 in the numerator of 𝛽^PRP_𝑘 tends to zero, the search direction 𝑑_𝑘 is close to −𝑔_𝑘. Gilbert and Nocedal [11] proposed PRP⁺ which is the most successful modified method, that is, 𝛽^PRP+_𝑘=max𝛽^PRP_𝑘,0. (1.7) Dai and Yuan [12] presented DY method and proved the global convergence when the line search satisfies the Wolfe conditions. Zheng et al. [13] derived 𝛽^new_𝑘=𝑔^T_𝑘𝑔_𝑘−𝑑_𝑘−1𝑑^T_𝑘−1𝑔_𝑘−𝑔_𝑘−1,𝛽_𝑘=𝛽^new_𝑘,if0<𝑔^T_𝑘𝑑_𝑘−11<min2,𝜎𝑔_𝑘²,𝛽^DY_𝑘,otherwise, (1.8) and discussed the properties of the new formulas.

HS is similar to PRP. It is equal to PRP when using the precision line search. HS satisfies the conjugate condition which is different from other methods.

Touati-Ahmed and Storey [14] gave 𝛽^TS_𝑘=𝛽^PRP_𝑘,if0≤𝛽^PRP_𝑘≤𝛽^FR_𝑘,𝛽^FR_𝑘,otherwise. (1.9) Dai and Chen [15] proposed 𝛽_𝑘=𝛽^HS_𝑘,if0<𝑔^T_𝑘𝑔_𝑘−11<min2,𝜎𝑔_𝑘²,𝛽^DY_𝑘,otherwise. (1.10) Dai and Ni [16] derived 𝛽_𝑘=−𝑏𝛽^DY_𝑘,if𝛽^HS_𝑘<−𝑏𝛽^DY_𝑘,𝛽^HS_𝑘,if−𝑏𝛽^DY_𝑘≤𝛽^HS_𝑘≤𝛽^DY_𝑘,𝛽^DY_𝑘,if𝛽^HS_𝑘>𝛽^DY_𝑘. (1.11) Throughout the paper, ‖⋅‖ stands for the Euclidean norm.

Hager and Zhang (CGDESCENT) [17] proposed a conjugate gradient method with guaranteed descent which corresponds to the following choice for the update parameters: 𝛽^𝑁_𝑘=max{𝛽^𝑁_𝑘,𝜂_𝑘}, where 𝜂_𝑘=−1𝑑_𝑘min𝜂,𝑔_𝑘,𝛽^𝑁_𝑘=1𝑑^T_𝑘𝑦_𝑘𝑦_𝑘−2𝑑_𝑘𝑦_𝑘²𝑑^T_𝑘𝑦_𝑘^T𝑔_𝑘+1. (1.12) Here, 𝜂>0 is a constant. The extensive numerical tests and comparisons with other methods showed that this method has advantage in some aspects.

Zhang et al. (ZZL) [18] derived a descent modified PRP conjugate method, the direction 𝑑_𝑘 is generated by 𝑑_𝑘=−𝑔_𝑘,if𝑘=1,−𝑔_𝑘+𝛽^PRP_𝑘𝑑_𝑘−𝜃_𝑘𝑦_𝑘−1,if𝑘≥2. (1.13) The numerical results suggested that the efficiency of the MPRP method is encouraging.

Consider the above mixed techniques and the properties of the classical conjugate gradient methods, the new mixed methods will be presented. The main difference between the new methods and the existed methods are the choice of 𝛽_𝑘 and giving the generalization of the new method. Moreover, the direction generated by the new methods are descent directions of the objective function under mild conditions. In the numerical results, the method’s overall performance will be given.

Firstly, we present a new formula 𝛽^new_𝑘=𝑔^T_𝑘𝑔_𝑘𝜇𝑔^T_𝑘𝑑_𝑘−1+𝑑^T_𝑘−1𝑔_𝑘−𝑔_𝑘−1. (1.14)

The rest of the paper is organized as follows. In Section 2, we give a new mixed conjugate gradient algorithm and convergence analysis. Section 3 is devoted to a generalization of the new mixed method. In the last section, numerical results and comparisons with the CGDESCENT and ZZL methods on test problems are reported and show the advantage of the new methods.

2. A New Algorithm and Convergence Analysis

We discuss a new mixed conjugate gradient method 𝛽^∗_𝑘=𝛽^new_𝑘,if𝑔_𝑘²≥𝑔^T_𝑘𝑔_𝑘−1,0,otherwise, (2.1) where 𝜇≥1.

Algorithm 2.1.  
Step 1. Give 𝑥₁∈ℜ^𝑛, 𝜀>0, 𝑑₁=−𝑔₁; 𝑘∶=1.
Step 2. If ‖𝑔_𝑘‖<𝜀, stop, else go to Step 3.
Step 3. Find 𝛼_𝑘 satisfying Wolfe conditions (1.5) and (1.6).
Step 4. Compute new iterative 𝑥_𝑘+1 by 𝑥_𝑘+1=𝑥_𝑘+𝛼_𝑘𝑑_𝑘.
Step 5. Compute 𝛽_𝑘 by (2.1), 𝑑_𝑘+1=−𝑔_𝑘+1+𝛽_𝑘+1𝑑_𝑘, 𝑘∶=𝑘+1, go to Step 2.

In order to derive the global convergence of the algorithm, we use the following assumptions.H 2.1 The objective function 𝑓(𝑥) is bounded in the level set as below 𝐿₁=𝑥∈ℜ^𝑛∣𝑓(𝑥)≤𝑓𝑥₁, (2.2) where 𝑥₁ is the starting point. H 2.2𝑓(𝑥) is continuously differentiable in a neighborhood 𝑁 of 𝐿₁ and its gradient 𝑔(𝑥) is Lipschitz continuous, there exists a constant 𝐿>0 such that 𝑔(𝑥)−𝑔𝑦≤𝐿𝑥−𝑦,∀𝑥,𝑦∈𝑁. (2.3)

Lemma 2.2 (see Zoutendijk condition [19]). Suppose that H 2.1 and H 2.2 hold. If the conjugate gradient method satisfies 𝑔^T_𝑘𝑑_𝑘<0, then ^∞_𝑘=1𝑔^T_𝑘𝑑_𝑘²𝑑_𝑘²<+∞. (2.4)

Theorem 2.3. Suppose that H 2.1 and H 2.2 hold. Let {𝑥_𝑘} and {𝑑_𝑘} be generated by (1.2) and (1.3), where 𝛽_𝑘 is computed by (2.1), 𝛼_𝑘 satisfies Wolfe line search conditions, then 𝑔^T_𝑘𝑑_𝑘<0 holds for all 𝑘≥1.

Proof. The conclusion can be proved by induction. When 𝑘=1, we have 𝑔^T₁𝑑₁=−‖𝑔₁‖²<0. Suppose that 𝑔^T_𝑘−1𝑑_𝑘−1<0 hold for 𝑘. From (1.3), we have 𝑔^T_𝑘𝑑_𝑘=−𝑔_𝑘²+𝛽_𝑘𝑔^T_𝑘𝑑_𝑘−1≤−𝑔_𝑘²+𝛽_𝑘𝑔^T_𝑘𝑑_𝑘−1. (2.5)
When 𝛽^∗_𝑘=0, it is obvious that 𝑔^T_𝑘𝑑_𝑘≤−𝑔_𝑘²<0. (2.6)
When 𝛽_𝑘=𝛽^new_𝑘, from (1.6) we have 𝑑^T_𝑘−1𝑔_𝑘−𝑔_𝑘−1=𝑔^T_𝑘𝑑_𝑘−1−𝑔^T_𝑘−1𝑑_𝑘−1≥𝜎𝑔^T_𝑘−1𝑑_𝑘−1−𝑔^T_𝑘−1𝑑_𝑘−1=(𝜎−1)𝑔^T_𝑘−1𝑑_𝑘−1>0. (2.7) Then, 𝑔^T_𝑘𝑑_𝑘≤−𝑔_𝑘²+𝑔_𝑘²𝜇𝑔^T_𝑘𝑑_𝑘−1+𝑑^T_𝑘−1𝑔_𝑘−𝑔_𝑘−1𝑔^T_𝑘𝑑_𝑘−1≤−𝑔_𝑘²+𝑔_𝑘²𝜇𝑔^T_𝑘𝑑_𝑘−1𝑔^T_𝑘𝑑_𝑘−1=1−1+𝜇𝑔_𝑘², (2.8) from 𝜇≥1, then we can deduce that 𝑔^T_𝑘𝑑_𝑘<0 holds for all 𝑘≥1.
Thus, the theorem is proved.

Theorem 2.4. Suppose that H 2.1 and H 2.2 hold. Consider Algorithm 2.1, where 𝛽_𝑘 is determined by (2.1), if 𝑔_𝑘≠0 holds for any 𝑘, then, liminf_𝑘→∞𝑔_𝑘=0. (2.9)

Proof. By contradiction, assume that (2.9) does not hold. Then there exists a constant 𝜀>0, such that 𝑔_𝑘>𝜀,∀𝑘≥1. (2.10)
From (2.1), 0≤𝛽^∗_𝑘≤𝑔_𝑘²𝑑^T_𝑘−1𝑔_𝑘−𝑔_𝑘−1=𝛽^DY_𝑘. (2.11)
By (1.3), if 𝛽_𝑘=𝛽^DY_𝑘, we derive 𝑔^T_𝑘𝑑_𝑘=−𝑔_𝑘²+𝑔_𝑘²𝑑^T_𝑘−1𝑔_𝑘−𝑔_𝑘−1𝑔^T_𝑘𝑑_𝑘−1=𝛽^DY_𝑘𝑔^T_𝑘−1𝑑_𝑘−1. (2.12)
Then, 𝛽^DY_𝑘=𝑔^T_𝑘𝑑_𝑘𝑔^T_𝑘−1𝑑_𝑘−1. (2.13)
So, 𝛽^∗_𝑘≤𝛽^DY_𝑘=𝑔^T_𝑘𝑑_𝑘𝑔^T_𝑘−1𝑑_𝑘−1. (2.14)
From (1.3), we have 𝑑_𝑘+𝑔_𝑘=𝛽_𝑘𝑑_𝑘−1. (2.15)
By squaring the two sides of (2.15) and transferring and trimming, we get 𝑑_𝑘²=−𝑔_𝑘²−2𝑔^T_𝑘𝑑_𝑘+𝛽²_𝑘𝑑_𝑘−1². (2.16) Then, 𝑑_𝑘²≤−𝑔_𝑘²−2𝑔^T_𝑘𝑑_𝑘+𝑔^T_𝑘𝑑_𝑘²𝑔^T_𝑘−1𝑑_𝑘−1²𝑑_𝑘−1²,𝑑_𝑘²𝑔^T_𝑘𝑑_𝑘²2≤−𝑔^T_𝑘𝑑_𝑘+𝑑_𝑘−1²𝑔^T_𝑘−1𝑑_𝑘−1²−𝑔_𝑘²𝑔^T_𝑘𝑑_𝑘²≤𝑑_𝑘−1²𝑔^T_𝑘−1𝑑_𝑘−1²+1𝑔_𝑘². (2.17) Since, 𝑑₁²𝑔^T₁𝑑₁²=𝑔₁²−𝑔^T₁𝑔₁²=1𝑔_𝑘²,𝑑_𝑘²𝑔^T_𝑘𝑑_𝑘²≤_𝑖≥11𝑔_𝑖². (2.18) From (2.10), we have _𝑖≥11𝑔_𝑖²≤𝑘𝜀². (2.19) Therefore, 𝑔^T_𝑘𝑑_𝑘²𝑑_𝑘²≥𝜀²𝑘_𝑘≥1,𝑔^T_𝑘𝑑_𝑘²𝑑_𝑘²=+∞. (2.20) This is a contradiction to Lemma 2.2, the global convergence is got.

3. Generalization of the New Method and Convergence

The generalization of the new mixed method is as follows: 𝛽_𝑘^∗∗=𝜆𝛽^new_𝑘,if𝑔_𝑘²≥𝑔^T_𝑘𝑔_𝑘−1,0,otherwise, (3.1) where 𝛽^new_𝑘=𝑔^T_𝑘𝑔_𝑘/(𝜇|𝑔^T_𝑘𝑑_𝑘−1|+𝑑^T_𝑘−1(𝑔_𝑘−𝑔_𝑘−1)), 𝜇>1>𝜆>0.

Algorithm 3.1.  
Step 1–Step 4 are the same as that of Algorithm 2.1.
Step 5. Compute 𝛽_𝑘 by (3.1), 𝑑_𝑘+1=−𝑔_𝑘+1+𝛽_𝑘+1𝑑_𝑘, 𝑘∶=𝑘+1, go to Step 2.

Theorem 3.2. Suppose that H 2.1 and H 2.2 hold. Let {𝑥_𝑘} and {𝑑_𝑘} be generated by (1.2) and (1.3), where 𝛽_𝑘 is computed by (3.1), 𝛼_𝑘 satisfies Wolfe line search conditions, then 𝑔^T_𝑘𝑑_𝑘<0 holds for all 𝑘≥1.

Proof. The conclusion can be proved by induction. When 𝑘=1, we have 𝑔^T₁𝑑₁=−‖𝑔₁‖²<0. Suppose that 𝑔^T_𝑘−1𝑑_𝑘−1<0 holds. For 𝑘, it is obvious that if 𝛽_𝑘^∗∗=0, then 𝑔^T_𝑘𝑑_𝑘≤−𝑔_𝑘²<0. (3.2)
When 𝛽_𝑘=𝜆𝛽^new_𝑘, from (2.5) and (3.1), we have 𝑔^T_𝑘𝑑_𝑘≤−𝑔_𝑘²+𝜆𝑔^T_𝑘𝑔_𝑘𝜇𝑔^T_𝑘𝑑_𝑘−1+𝑑^T_𝑘−1𝑔_𝑘−𝑔_𝑘−1𝑔^T_𝑘𝑑_𝑘−1≤−𝑔_𝑘²+𝜆𝑔_𝑘²𝜇𝑔^T_𝑘𝑑_𝑘−1𝑔^T_𝑘𝑑_𝑘−1≤−𝑔_𝑘²+𝜆𝜇𝑔_𝑘²=𝑔_𝑘²𝜆𝜇−1<0. (3.3)
To sum up, the theorem is proved.

Theorem 3.3. Suppose that H 2.1 and H 2.2 hold. Consider Algorithm 3.1, where 𝛽_𝑘 is determined by (3.1), if 𝑔_𝑘≠0 holds for any 𝑘, then, liminf_𝑘→∞𝑔_𝑘=0. (3.4)

Proof. By contradiction, assume that (3.4) does not hold. Then there exists a constant 𝛾>0 such that 𝑔_𝑘>𝛾,∀𝑘≥1. (3.5)
From (3.1) 0≤𝛽_𝑘^∗∗≤𝜆𝑔_𝑘²𝑑^T_𝑘−1𝑔_𝑘−𝑔_𝑘−1≤𝑔_𝑘²𝑑^T_𝑘−1𝑔_𝑘−𝑔_𝑘−1=𝛽^DY_𝑘. (3.6)
By (1.3), we have 𝑑_𝑘+𝑔_𝑘=𝛽_𝑘𝑑_𝑘−1. (3.7)
Then, 𝑑_𝑘²=−𝑔_𝑘²−2𝑔^T_𝑘𝑑_𝑘+𝛽²_𝑘𝑑_𝑘−1². (3.8) From (3.6), 𝑑_𝑘²≤−𝑔_𝑘²−2𝑔^T_𝑘𝑑_𝑘+𝑔^T_𝑘𝑑_𝑘²𝑔^T_𝑘−1𝑑_𝑘−1²𝑑_𝑘−1²,𝑑_𝑘²𝑔^T_𝑘𝑑_𝑘²2≤−𝑔^T_𝑘𝑑_𝑘+𝑑_𝑘−1²𝑔^T_𝑘−1𝑑_𝑘−1²−𝑔_𝑘²𝑔^T_𝑘𝑑_𝑘²≤𝑑_𝑘−1²𝑔^T_𝑘−1𝑑_𝑘−1²+1𝑔_𝑘²,𝑑_𝑘²𝑔^T_𝑘𝑑_𝑘²≤_𝑖≥11𝑔_𝑖². (3.9) By (3.5), we have 𝑔^T_𝑘𝑑_𝑘²𝑑_𝑘²≥𝛾²𝑘_𝑘≥1,𝑔^T_𝑘𝑑_𝑘²𝑑_𝑘²=+∞. (3.10) This is a contradiction to Lemma 2.2, and the global convergence is proved.

4. Numerical Results

This section is devoted to test the implementation of the new methods. We compare the performance of the new methods with the CGDESCENT and ZZL methods.

All tests in this paper are implemented on a PC with 1.8 MHz Pentium IV and 256 MB SDRAM using MATLAB 6.5. If 𝜀=10⁻⁶ then stop. Some classical test functions with standard starting points are selected to test the methods. These functions are widely used in the literature to test unconstrained optimization algorithms [20].

In the table, the four reported data (𝑘/nf/ng/𝑡) are iteration numbers/function evaluations/gradient evaluations/CPU time(s), and ‖𝑔_𝑘‖² stands for the square of the gradient at the final iterate. When we set 𝜇=1, 𝜆=0.5, 𝜂=0.01, 𝛿=0.01, 𝜎=0.8, the numerical results of the NEW1 and NEW2 are listed in Table 1 and the CGDESCENT and ZZL (NEW3) are listed in Table 2. When we set 𝜇=1.5, 𝜆=0.1, 𝜂=10, 𝛿=0.1, 𝜎=0.9, the numerical results of the NEW 1 (NEW4) and NEW 2 (NEW5) are listed in Table 3 and the CGDESCENT (NEW6) and ZZL (NEW7) are listed in Table 4. It can be observed from Tables 1–4 that for the most of problems, the implementation of the new methods are superior to other methods from the iteration numbers, the calls of function, and gradient evaluations.

Table 1 

Numerical results of the NEW1 and NEW2.

Table 2 

Numerical results of the CGDESCENT and NEW3.

Table 3 

Numerical results of the NEW4 and NEW5.

Table 4 

Numerical results of the NEW6 and NEW7.

Compared with the CGDESCENT method, the new methods are effective (see Table 5).

Table 5 

Using the formula 𝑁_total=nf+𝑙∗ng, where 𝑙 is fixed constant, let 𝑙=3. By 𝛾_𝑖𝑗=𝑁NEW^totalNEW𝑗𝑁_total(),CGDESCET (4.1) where 𝑗=1,2,…,7; 𝑖∈𝑆, 𝑆 is the whole of classical problems’ order. If 𝛾_𝑖(NEW(𝑗))>1, then CGDESCENT method is regarded as better performance; if 𝛾_𝑖(NEW(𝑗))=1, the methods have the same performances and if 𝛾_𝑖(NEW(𝑗))<1, the new methods are performed better.

We use 𝛾_totalNEW𝑗=Π_𝑖∈𝑆𝛾_𝑖NEW𝑗^1/|𝑆| (4.2) as a measure to compare the performance of CGDESCENT method and the new methods, where |𝑆| is the number of 𝑆. If 𝛾_total(NEW(𝑗))<1, then NEW(𝑗) method outperforms CGDESCENT method. The computational results are listed in Table 5.

It is obvious that 𝛾_total(NEW(𝑗))<1, where 𝑗=1,2,4,5, so we can deduce that the new methods outperform CGDESCENT method.

Acknowledgments

This work is supported in part by the NNSF (11171003) of China, Key Project of Chinese Ministry of Education (no. 211039), and Natural Science Foundation of Jilin Province of China (no. 201215102).