A Conjugate Gradient Method with Global Convergence for Large-Scale Unconstrained Optimization Problems

Abstract

The conjugate gradient (CG) method has played a special role in solving large-scale nonlinear optimization problems due to the simplicity of their very low memory requirements. This paper proposes a conjugate gradient method which is similar to Dai-Liao conjugate gradient method (Dai and Liao, 2001) but has stronger convergence properties. The given method possesses the sufficient descent condition, and is globally convergent under strong Wolfe-Powell (SWP) line search for general function. Our numerical results show that the proposed method is very efficient for the test problems.

1. Introduction

The conjugate gradient (CG) method has played a special role in solving large-scale nonlinear optimization due to the simplicity of their iterations and their very low memory requirements. In fact, the CG method is not among the fastest or most robust optimization algorithms for nonlinear problems available today, but it remains very popular for engineers and mathematicians who are interested in solving large problems. The conjugate gradient method is designed to solve the following unconstrained optimization problem: min𝑓(𝑥)∣𝑥∈𝑅^𝑛, (1) where 𝑓(𝑥)∶𝑅^𝑛→𝑅 is a smooth, nonlinear function whose gradient will be denoted by 𝑔(𝑥). The iterative formula of the conjugate gradient method is given by 𝑥_𝑘+1=𝑥_𝑘+𝑠_𝑘,𝑠_𝑘=𝛼_𝑘𝑑_𝑘, (2) where 𝛼_𝑘 is a step length which is computed by carrying out a line search, and 𝑑_𝑘 is the search direction defined by 𝑑_𝑘=−𝑔_𝑘if𝑘=1,−𝑔_𝑘+𝛽_𝑘𝑑_𝑘−1if𝑘≥2, (3) where 𝛽_𝑘 is a scalar and 𝑔_𝑘 denotes the gradient ∇𝑓(𝑥_𝑘). If 𝑓 is a strictly convex quadratic function, namely, 𝑓(𝑥)=12𝑥^𝑇𝐻𝑥+𝑏^𝑇𝑥, (4) where 𝐻 is a positive definite matrix and if 𝛼_𝑘 is the exact one-dimensional minimizer along the direction 𝑑_𝑘, then the method with (2) and (3) are called the linear conjugate gradient method. Otherwise, (2) and (3) is called the nonlinear conjugate gradient method. The most important feature of linear conjugate gradient method is that the search directions satisfy the following conjugacy condition: 𝑑^𝑇_𝑖𝐻𝑑_𝑗=0,𝑖≠𝑗. (5) For nonlinear conjugate gradient methods, for general objective functions, (5) does not hold, since the Hessian ∇²𝑓(𝑥) changes at different points.

Some well-known formulas for 𝛽_𝑘 are the Fletcher-Reeves (FR), Polak-Ribière (PR), Hestense-Stiefel (HS), and Dai-Yuan (DY) methods which are given, respectively, by 𝛽^FR_𝑘=𝑔_𝑘²𝑔_𝑘−1²𝛽_PR𝑘,=𝑔^𝑇_𝑘𝑔_𝑘−𝑔_𝑘−1𝑔_𝑘−1²,𝛽^HS_𝑘=𝑔^𝑇_𝑘𝑔_𝑘−𝑔_𝑘−1𝑔_𝑘−𝑔_𝑘−1^𝑇𝑑_𝑘−1,𝛽^DY_𝑘=𝑔_𝑘²𝑔_𝑘−𝑔_𝑘−1^𝑇𝑑_𝑘−1, (6)(7)(8)(9) where ‖‖ denotes the Euclidean norm. Their corresponding conjugate methods are abbreviated as FR, PR, HS, and DY methods. Although all these methods are equivalent in the linear case, namely, when 𝑓 is a strictly convex quadratic function and 𝛼_𝑘 are determined by exact line search, their behaviors for general objective functions may be far different.

For general functions, Zoutendijk [1] proved the global convergence of FR method with exact line search (here and throughout this paper, for global convergence, we mean that the sequence generated by the corresponding methods will either terminate after finite steps or contain a subsequence such that it converges to a stationary point of the objective function from a given initial point). Although one would be satisfied with its global convergence properties, the FR method performs much worse than the PR (HS) method in real computations. Powell [2] analyzed a major numerical drawback of the FR method; namely, if a small step is generated away from the solution point, the subsequent steps may be also very short. On the other hand, in practical computation, the HS method resembles the PR method, and both methods are generally believed to be the most efficient conjugate gradient methods since these two methods essentially perform a restart if a bad direction occurs. However, Powell [3] constructed a counterexample and showed that the PR method and HS method can cycle infinitely without approaching the solution. This example suggests that these two methods have a drawback that they are not globally convergent for general functions. Therefore, in the past two decades, much effort has been exceterd to find out new formulas for conjugate methods such that not only they are globally convergent for general functions but also they have good numerical performance.

Recently, using a new conjugacy condition, Dai and Liao [4] proposed two new methods. Interestingly, one of their methods is not only globally convergent for general functions but also performs better than HS and PR methods. In this paper, similar to Dai and Liao's approach, we propose another formula for 𝛽_𝑘, analyze the convergence properties for the given method, and also carry the numerical experiment which shows that the given method is robust and efficient.

The remainder of this paper is organized as follows. In Section 2, we firstly state the corresponding formula which is proposed by Dai and Liao [4] and the motivations of this paper, and then we propose the new nonlinear conjugate gradient method. In Section 3, convergence analysis for the given method is presented. Numerical results are reported in Section 4. Finally, some conclusions are given in Section 5.

2. Motivations and New Nonlinear Conjugate Gradient Method

2.1. Dai-Liao's Methods

It is well known that the linear conjugate gradient methods generate a sequence of search directions 𝑑_𝑘 such that the conjugacy condition (5) holds. Denote 𝑦_𝑘−1 to be the gradient change, which means that 𝑦_𝑘−1=𝑔_𝑘−𝑔_𝑘−1. (10) For a general nonlinear function 𝑓, we know by the mean value theorem that there exists some 𝑡∈(0,1) such that 𝑦^𝑇_𝑘−1𝑑_𝑘=𝛼_𝑘−1𝑑^𝑇_𝑘∇²𝑓𝑥_𝑘−1+𝑡𝛼_𝑘−1𝑑_𝑘−1𝑑_𝑘−1. (11) Therefore, it is reasonable to replace (5) with the following conjugacy condition: 𝑦^𝑇_𝑘−1𝑑_𝑘=0. (12) Recently, extension of (12) has been studied by Dai and Liao in [4]. Their approach is based on the Quasi-Newton techniques. Recall that, in the Quasi-Newton method, an approximation matrix 𝐻_𝑘−1 of the Hessian ∇²𝑓(𝑥_𝑘−1) is updated such that the new matrix 𝐻_𝑘 satisfies the following Quasi-Newton equation: 𝐻_𝑘𝑠_𝑘−1=𝑦_𝑘−1. (13) The search direction 𝑑_𝑘 in Quasi-Newton method is calculated by 𝑑_𝑘=−𝐻_𝑘⁻¹𝑔_𝑘. (14) Combining these two equations, we obtain 𝑑^𝑇_𝑘𝑦_𝑘−1=𝑑^𝑇_𝑘𝐻_𝑘𝑠_𝑘−1=−𝑔^𝑇_𝑘𝑠_𝑘−1. (15) The previous relation implies that (12) holds if the line search is exact since in this case 𝑔^𝑇_𝑘𝑑_𝑘−1=0. However, practical numerical algorithms normally adopt inexact line searches instead of exact line searches. For this reason, it seems more reasonable to replace the conjugacy condition (12) with the condition 𝑑^𝑇_𝑘𝑦_𝑘−1=−𝑡𝑔^𝑇_𝑘𝑠_𝑘−1,𝑡≥0, (16) where 𝑡≥0 is a scalar.

To ensure that the search direction 𝑑_𝑘 satisfies the conjugate condition (16), one only needs to multiply (3) with 𝑦_𝑘−1 and use (16), yielding 𝛽^DL1_𝑘=𝑔^𝑇_𝑘𝑦_𝑘−1−𝑡𝑠_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1. (17) It is obvious that 𝛽^DL1_𝑘=𝛽^HS_𝑘𝑔−𝑡^𝑇𝑘𝑠_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1. (18) For simplicity, we call the method with (2), (3), and (17) as DL1 method. Dai and Liao also prove that the conjugate gradient method with DL1 is globally convergent for uniformly convex functions. For general functions, Powell [3] constructed an example showing that the PR method may cycle without approaching any solution point if the step length 𝛼_𝑘 is chosen to be the first local minimizer along 𝑑_𝑘. Since the DL1 method reduces to the PR method in the case that 𝑔^𝑇_𝑘𝑑_𝑘−1=0 holds, this implies that the method with (17) need not converge for general functions. To get the global convergence, like Gilbert and Nocedal [5], who have proved the global convergence of the PR method with the restriction that 𝛽^PR_𝑘≥0, Dai and Liao replaced (17) by 𝛽^DL_𝑘𝑔=max^𝑇𝑘𝑦_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1𝑔,0−𝑡^𝑇𝑘𝑠_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1=max𝛽^HS_𝑘𝑔,0−𝑡^𝑇𝑘𝑠_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1. (19) We also call the method with (2), (3), and (19) as DL method, Dai and Liao show that DL method is globally convergent for general functions under the sufficient descent condition (21) and some suitable conditions. Besides, some numerical experiments in [4] indicate the efficiency of this method.

Similar to Dai and Liao's approach, Li et al. [6] proposed another conjugate condition and related conjugate gradient methods, and they also prove that the proposed methods are globally convergent under some assumptions.

2.2. Motivations

From the above discussions, Dai and Liao's approach is effective; the main reason is that the search directions 𝑑_𝑘 generated by DL1 method or DL method not only contain the gradient information but also contain some Hessian ∇²𝑓(𝑥) information. From (18) and (19), 𝛽^DL1_𝑘 and 𝛽^DL_𝑘 are formed by two parts; the first part is 𝛽^HS_𝑘, and the second part is −𝑡(𝑔^𝑇_𝑘𝑠_𝑘−1/𝑑^𝑇_𝑘−1𝑦_𝑘−1). So, we also can consider DL1 and DL methods as some modified forms of the HS method by adding some information of Hessian ∇²𝑓(𝑥) which is contained in the second part. The convergence properties of the HS method are similar to PR method; it does not converge for general functions even if the line search is exact. In order to get the convergence, one also needs the nonnegative restriction 𝛽_𝑘=max{𝛽^HS_𝑘,0} and the sufficient descent assumption (21). From the above discussion, the descent condition or sufficient descent condition and nonnegative property of 𝛽_𝑘 play important roles in the convergence analysis. We say that the descent condition holds if for each search directions 𝑑_𝑘𝑔^𝑇_𝑘𝑑_𝑘<0,∀𝑘≥1. (20) In addition, we say that the sufficient descent condition holds if there exists a constant 𝑐>0 such that for each search direction 𝑑_𝑘, we have 𝑔^𝑇_𝑘𝑑_𝑘≤−𝑐𝑔_𝑘²,∀𝑘≥1. (21) Motivated by the above ideal, in this paper, we focus on finding the new conjugate gradient method which possesses the following properties: (1)nonnegative property 𝛽_𝑘≥0;(2)the new formula contains not only the gradient information but also some Hessian information;(3)the search directions 𝑑_𝑘 generated by the proposed method satisfy the sufficient descent conditions (21).

2.3. The New Conjugate Gradient Method

From the structure of (6), (7), (8), and (9), the PR and HS methods have the common numerator 𝑔^𝑇_𝑘𝑦_𝑘−1, and the FR and DY methods have the common numerator ‖𝑔_𝑘‖²; and this different choice makes them have different properties. Generally speaking, FR and DY methods have better convergence properties, and PR and HS methods have better numerical experiments. Powell [3] pointed out that the FR method, with exact line search, was susceptible to jamming. That is, the algorithm could take many short steps without making significant progress to the minimum. If the line search is exact, that means 𝑔^𝑇_𝑘𝑑_𝑘−1=0, in this case, DY method will turn out to be FR method. So, these two methods have the same disadvantage. The PR and HS methods which share the common numerator 𝑔^𝑇_𝑘𝑦_𝑘−1 possess a built-in restart feature to avoid the jamming problem: when the step 𝑥_𝑘−𝑥_𝑘−1 is small, the factor 𝑦_𝑘−1 in the numerator of 𝛽_𝑘 tends to zero. Hence, the next search direction 𝑑_𝑘 is essentially the steepest descent direction −𝑔_𝑘. So, the numerical performance of these methods is better than the performance of the methods with ‖𝑔_𝑘‖² in numerator of 𝛽_𝑘.

Just as above discussions, great attentions were given to find the methods which not only have global convergent properties but also have nice numerical experiments.

Recently, Wei et al. [7] proposed a new formula 𝛽^WYL_𝑘=𝑔^𝑇_𝑘𝑦^∗_𝑘−1𝑔_𝑘−1²,𝑦^∗_𝑘−1=𝑔_𝑘−𝑔_𝑘𝑔_𝑘−1𝑔_𝑘−1. (22)

The method with formula 𝛽^WYL_𝑘 not only has nice numerical results but also possesses the sufficient descent condition and global convergence properties under the strong Wolfe-Powell line search. From the structure of 𝛽^WYL_𝑘, we know that the method with 𝛽^WYL_𝑘 can also avoid jamming: when the step 𝑥_𝑘−𝑥_𝑘−1 is small, ‖𝑔_𝑘‖/‖𝑔_𝑘−1‖ tends to 1 and the next search direction tends to the steepest descent direction which is similar to PR method. But WYL method has some advantages, such as under strong Wolfe-Powell line search, 𝛽^WYL_𝑘≥0, and if the parameter 𝜎≤1/4 in SWP, WYL method possesses the sufficient descent condition which deduces the global convergence of the WYL method.

In [8, 9], Shengwei et al. extended such modification to HS method as follows: 𝛽^MHS_𝑘=𝑔^𝑇_𝑘𝑦^∗_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1,𝑦^∗_𝑘−1=𝑔_𝑘−𝑔_𝑘𝑔_𝑘−1𝑔_𝑘−1. (23) The previous formulae 𝛽^WYL_𝑘 and 𝛽^MHS_𝑘 can be considered as the modification forms of 𝛽^PR_𝑘 and 𝛽^HS_𝑘 by using 𝑦^∗_𝑘−1 to replace 𝑦_𝑘−1, respectively. In [8, 9], the corresponding methods are proved to be globally convergent for general functions under the strong Wolfe-Powell line search and Grippo-Lucidi line search. Based on the same approach, some authors give other discussions and modifications in [10–12]. In fact, 𝑦^∗_𝑘−1 is not our point at the beginning, our purpose is involving the information of the angle between 𝑔_𝑘 and 𝑔_𝑘−1. From this point of view, 𝛽^WYL_𝑘 has the following form: 𝛽^WYL_𝑘=𝛽^FR_𝑘1−cos𝜃_𝑘, (24) where 𝜃_𝑘 is the angle between 𝑔_𝑘 and 𝑔_𝑘−1. By multiplying 𝛽^FR_𝑘 with 1−cos𝜃_𝑘, the method not only has similar convergence properties with FR method, but also avoids jamming which is similar to PR method.

The above analysis motivates us to propose the following formula to compute 𝛽_𝑘: 𝛽^MDL1_𝑘𝑔=max^𝑇𝑘𝑦^∗_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1𝑔,0−𝑡^𝑇𝑘𝑠_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1, (25) where 𝑦^∗_𝑘−1=𝑔_𝑘−(‖𝑔_𝑘‖/‖𝑔_𝑘−1‖)𝑔_𝑘−1. Since the 𝛽^MHS_𝑘 are nonnegative under the strong Wolfe-Powell line search, we omit the nonnegative restriction and propose the following formula: 𝛽^MDL_𝑘=𝑔^𝑇_𝑘𝑦^∗_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1𝑔−𝑡^𝑇𝑘𝑠_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1=𝛽^MHS_𝑘𝑔−𝑡^𝑇𝑘𝑠_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1. (26)

From (25) and (26), we know that we only substitute 𝑦_𝑘−1 in the first part of the numerator of 𝛽^DL1_𝑘𝑜𝑟𝛽^DL_𝑘 by 𝑦^∗_𝑘. The reason is that we hope the formulae (25) and (26) contain the angle information between 𝑔_𝑘 and 𝑔_𝑘−1. In fact, 𝛽^MDL_𝑘 can be expressed as 𝛽^MDL_𝑘=𝑔_𝑘²𝑑^𝑇_𝑘−1𝑦_𝑘−11−cos𝜃_𝑘𝑔−𝑡^𝑇𝑘𝑠_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1=𝛽^DY_𝑘1−cos𝜃_𝑘𝑔−𝑡^𝑇𝑘𝑠_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1. (27) For simplicity, we call the method generated by (2), (3), and (26) as MDL method and give the algorithm as follows.

Algorithm 1 (MDL). Step  1. Given 𝑥₁∈𝑅^𝑛, 𝜀≥0, set 𝑑₁=−𝑔₁, 𝑘=1; if ‖𝑔₁‖≤𝜀, then stop.
Step  2. Compute 𝑡_𝑘 by some line searches.
Step  3. Let 𝑥_𝑘+1=𝑥_𝑘+𝛼_𝑘𝑑_𝑘, and let 𝑔_𝑘+1=𝑔(𝑥_𝑘+1); if ‖𝑔_𝑘‖≤𝜀, then stop.
Step  4. Compute 𝛽_𝑘 by (26) and generate 𝑑_𝑘+1 by (3).
Step  5. Set 𝑘∶=𝑘+1 and go to Step  2.

We make the following basic assumptions on the objective functions.

Assumption A. (i) The level set Γ={𝑥∈𝑅^𝑛∶𝑓(𝑥)≤𝑓(𝑥₁)} is bounded; namely, there exists a constant 𝐵>0 such that ‖𝑥‖≤𝐵,∀𝑥∈Γ. (28)
(ii) In some neighborhood 𝑁 of Γ, 𝑓 is continuously differentiable, and its gradient is Lipschitz continuous; namely, there exists a constant 𝐿>0 such that 𝑔(𝑥)−𝑔𝑦≤𝐿𝑥−𝑦,∀𝑥,𝑦∈𝑁. (29) Under the above assumptions of 𝑓, there exists a constant 𝛾≥0 such that ∇𝑓(𝑥)≤𝛾,∀𝑥∈Γ. (30)
The step length 𝛼_𝑘 in Algorithm 1 (MDL) is obtained by some line search scheme. In conjugate gradient methods, the strong Wolfe-Powell conditions; namely, 𝑓𝑥_𝑘+𝛼_𝑘𝑑_𝑘−𝑓𝑥_𝑘≤𝛿𝛼_𝑘𝑔^𝑇_𝑘𝑑_𝑘,𝑔𝑥_𝑘+𝛼_𝑘𝑑_𝑘^𝑇𝑑_𝑘≤−𝜎𝑔^𝑇_𝑘𝑑_𝑘, (31)(32) where 0<𝛿<𝜎<1, are often imposed on the line search (SWP).

3. Convergence Analysis

Under Assumption A, based on the Zoutendijk condition in [1], for any conjugate gradient method with the strong Wolfe-Powell line search, Dai et al. in [13] proved the following general result.

Lemma 2. Suppose that Assumption A holds. Consider any conjugate gradient method in the form (2)-(3), where 𝑑_𝑘 is a descent direction and 𝛼_𝑘 is obtained by the strong Wolfe-Powell line search. If _𝑘≥11𝑑_𝑘²=∞. (33) One has that liminf_𝑘→∞𝑔_𝑘=0. (34) If the objective functions are uniformly convex, we can prove that the norm of 𝑑_𝑘 generated by Algorithm 1 (MDL) is bounded previously. Thus, by Lemma 2 one immediately has the following result.

Theorem 3. Suppose that Assumption A holds. Consider MDL method, where 𝑑_𝑘 is a descent direction and 𝛼_𝑘 is obtained by the strong Wolfe-Powell line search. If the objective functions are uniformly convex, namely, there exists a constant 𝜇>0 such that ∇𝑓(𝑥)−∇𝑓𝑦^𝑇𝑥−𝑦≥𝜇𝑥−𝑦²,∀𝑥,𝑦∈Γ. (35) One has that lim_𝑘→∞𝑔_𝑘=0. (36)

Proof. It follows from (35) that 𝑑^𝑇_𝑘−1𝑦_𝑘−1≥𝜇𝛼_𝑘−1𝑑_𝑘−1². (37) By (3), (26), (29), (30), and (37), we have 𝑑_𝑘≤𝑔_𝑘+𝛽^MDL_𝑘𝑑_𝑘−1≤𝑔_𝑘+𝑔^𝑇_𝑘𝑔_𝑘−𝑔_𝑘/𝑔_𝑘−1𝑔_𝑘−1−𝑡𝑠_𝑘−1𝜇𝛼_𝑘−1𝑑_𝑘−1²𝑑_𝑘−1=𝑔_𝑘+𝑔^𝑇_𝑘𝑔_𝑘−𝑔_𝑘−1+𝑔_𝑘−1−𝑔_𝑘/𝑔_𝑘−1𝑔_𝑘−1−𝑡𝑠_𝑘−1𝜇𝛼_𝑘−1𝑑_𝑘−1²×𝑑_𝑘−1≤𝑔_𝑘+𝑔_𝑘𝑔_𝑘−𝑔_𝑘−1+𝑔_𝑘−11−𝑔_𝑘/𝑔_𝑘−1+𝑡𝑠_𝑘−1𝜇𝛼_𝑘−1𝑑_𝑘−1²×𝑑_𝑘−1≤𝑔_𝑘+𝑔_𝑘𝑔_𝑘−𝑔_𝑘−1+𝑔_𝑘−1−𝑔_𝑘+𝑡𝑠_𝑘−1𝜇𝛼_𝑘−1𝑑_𝑘−1²𝑑_𝑘−1≤𝑔_𝑘+𝑔_𝑘2𝐿𝑠_𝑘−1+𝑡𝑠_𝑘−1𝜇𝛼_𝑘−1𝑑_𝑘−1²𝑑_𝑘−1≤𝛾1+2𝐿+𝑡𝜇=𝛾𝜇⁻¹𝜇+2𝐿+𝑡, (38) which implies the truth of (33). Therefore, by Lemma 2 we have (34), which is equivalent to (36) for uniformly convex functions. The proof is completed.

In order to prove the convergence of the MDL method, we need to state some properties of 𝛽^MHS_𝑘.

Lemma 4. In any conjugate gradient methods, if the parameter 𝛽_𝑘 is computed by (23), namely, 𝛽_𝑘=𝛽^MHS_𝑘, and 𝛼_𝑘 is determined by strong Wolfe-Powell line search of (31) and (32), then 𝛽^MHS_𝑘≥0. (39)

Proof. By SWP condition (32), we have 𝑑^𝑇_𝑘−1𝑦_𝑘−1≥𝜎𝑔^𝑇_𝑘−1𝑑_𝑘−1−𝑔^𝑇_𝑘−1𝑑_𝑘−1≥0, since 𝜎<1 and 𝑔^𝑇₁𝑑₁=−‖𝑔₁‖²<0. So we have 𝛽^MHS_𝑘=𝑔^𝑇_𝑘𝑔_𝑘−𝑔_𝑘/𝑔_𝑘−1𝑔_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1=𝑔_𝑘²𝑑^𝑇_𝑘−1𝑦_𝑘−11−cos𝜃_𝑘≥0. (40) The proof is completed.

In addition, we can also prove that, in conjugate gradient method of forms (2)-(3), if 𝛽_𝑘 is computed by 𝛽^MDL_𝑘 (26) and 𝛼_𝑘 is determined by strong Wolfe-Powell line search, then the search direction 𝑑_𝑘 satisfies the sufficient descent condition (21).

Theorem 5. In any conjugate gradient methods, in which the parameter 𝛽_𝑘 is computed by (26), namely, 𝛽_𝑘=𝛽^MDL_𝑘, and 𝛼_𝑘 is determined by strong Wolfe-Powell line search of (31) and (32), if 𝜎<1/3, then the search direction 𝑑_𝑘 satisfied the sufficient descent condition (21).

Proof. We prove this theorem by induction. Firstly, we prove the descent condition 𝑑^𝑇_𝑘𝑔_𝑘<0 as follow.
Since 𝑔^𝑇₁𝑑₁=−‖𝑔₁‖²<0, supposing that 𝑔^𝑇_𝑖𝑑_𝑖<0 holds for 𝑖≤𝑘−1, we deduce that the descent condition holds by proving that 𝑔^𝑇_𝑖𝑑_𝑖<0 holds for 𝑖=𝑘 as follow.
By SWP condition (32), we have 𝑑^𝑇_𝑘−1𝑦_𝑘−1≥(𝜎−1)𝑔^𝑇_𝑘−1𝑑_𝑘−1>0. Combining (3) and (26), we have 𝑔^𝑇_𝑘𝑑_𝑘𝑔_𝑘²=−1+1−cos𝜃_𝑘𝑑^𝑇_𝑘−1𝑦_𝑘−1𝑔^𝑇_𝑘𝑑_𝑘−1𝑔−𝑡∗^𝑇𝑘𝑑_𝑘−1²𝛼_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1𝑔_𝑘²≤−1+2−𝜎𝑔^𝑇_𝑘−1𝑑_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1≤−1+2−𝜎𝑔^𝑇_𝑘−1𝑑_𝑘−1(𝜎−1)𝑔^𝑇_𝑘−1𝑑_𝑘−1≤−1+3𝜎1−𝜎<0. (41) Equation (41) means that descent condition holds.
Secondly, we prove the following sufficient descent condition.
Set 𝑐=1−2𝜎/(1−𝜎); since the restriction 𝜎<1/3, we have 0<𝑐<1. Combining 𝑔^𝑇₁𝑑₁=−‖𝑔₁‖² and (41), the sufficient descent condition (21) holds immediately.

By Theorem 5, we can prove the following Lemma 6.

Lemma 6. Suppose that Assumption A holds. Consider MDL method, where 𝛼_𝑘 is obtained by strong Wolfe-Powell lien search with 𝜎<1/3. If there exists a constant 𝛾>0 such that 𝑔_𝑘≥𝛾,∀𝑘≥1, (42) then 𝑑_𝑘≠0 and _𝑘≥2𝑢_𝑘−𝑢_𝑘−1²<∞, (43) where 𝑢_𝑘=𝑑_𝑘/‖𝑑_𝑘‖.

Proof. Firstly, note that 𝑑_𝑘≠0; otherwise, (21) is false. Therefore, 𝑢_𝑘 is well defined. In addition, by relation (42) and Lemma 2, we have _𝑘≥11𝑑_𝑘²<∞. (44) Now, we divide formula 𝛽^MDL_𝑘 into two parts as follows: 𝛽¹_𝑘=𝑔^𝑇_𝑘𝑦^∗_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1,𝛽²_𝑘𝑔=−𝑡^𝑇𝑘𝑠_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1, (45) and define 𝑟_𝑘𝜗∶=^𝑘𝑑_𝑘,𝛿_𝑘∶=𝛽¹_𝑘𝑑_𝑘−1𝑑_𝑘, (46) where 𝜗_𝑘=−𝑔_𝑘+𝛽²_𝑘𝑑_𝑘−1.
Then by (3) we have for all 𝑘≥2, 𝑢_𝑘=𝑟_𝑘+𝛿_𝑘𝑢_𝑘−1. (47) Using the identity ‖𝑢_𝑘‖=‖𝑢_𝑘−1‖=1 and (47) we can obtain 𝑟_𝑘=𝑢_𝑘−𝛿_𝑘𝑢_𝑘−1=𝛿_𝑘𝑢_𝑘−𝑢_𝑘−1, (48) using the condition 𝛿_𝑘=𝛽^MHS_𝑘(‖𝑑_𝑘−1‖/‖𝑑_𝑘‖)≥0, the triangle inequality, and (48), it follows that 𝑢_𝑘−𝑢_𝑘−1≤(1+𝛿)𝑢_𝑘−(1+𝛿)𝑢_𝑘−1≤𝑢_𝑘−𝛿_𝑘𝑢_𝑘−1+𝛿_𝑘𝑢_𝑘−𝑢_𝑘−1=2𝑟_𝑘. (49) On the other hand, the line search condition (32) gives 𝑦^𝑇_𝑘−1𝑑_𝑘−1≥(𝜎−1)𝑔^𝑇_𝑘−1𝑑_𝑘−1. (50) Equations (50), (32), and (21) imply that 𝑔^𝑇_𝑘𝑑_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1≤𝜎.1−𝜎 (51) It follows from the definition of 𝜗_𝑘, (51), (28), and (30) that 𝜗_𝑘≤𝛿_𝑘𝑔+𝑡^𝑇𝑘𝑠_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1𝑑_𝑘−1=𝛿_𝑘𝑔+𝑡^𝑇𝑘𝑑_𝑘−1𝑑^𝑇_𝑘−1𝑦_𝑘−1𝑠_𝑘−1≤𝜎𝛾+𝑡1−𝜎2𝐵. (52) So, we have 𝑢_𝑘−𝑢_𝑘−1²≤4𝑟_𝑘²𝜗≤4^2𝑘𝑑_𝑘²≤4𝜎𝛾+𝑡1−𝜎2𝐵²1𝑑_𝑘²<∞. (53) The proof is completed.

Gilbert and Nocedal [5] introduced property (*) which is very important for the convergence properties of the conjugate gradient methods. We are going to show that method with 𝛽^MDL_𝑘 possesses such property (*).

Property (*). Consider a method of forms (2) and (3). Suppose that 0<𝛾≤𝑔_𝑘≤𝛾,∀𝑘≥1. (54) We say that the method has property (*), if for all 𝑘, there exist constants 𝑏>1, 𝜆>0 such that |𝛽_𝑘|≤𝑏 and if ‖𝑠_𝑘−1‖≤𝜆 we have |𝛽_𝑘|≤1/2𝑏.

In fact, by (50), (21), and (42), we have 𝑑^𝑇_𝑘−1𝑦_𝑘−1≥(𝜎−1)𝑔^𝑇_𝑘−1𝑑_𝑘−1≥𝑐(1−𝜎)𝑔_𝑘−1²≥(1−𝜎)𝑐𝛾². (55) Using this, (28), (29), and (30) we obtain 𝛽^MDL_𝑘≤(2𝐿+𝑡)𝑔_𝑘𝑠_𝑘−1(1−𝜎)𝑐𝛾²≤2(2𝐿+𝑡)𝛾𝐵(1−𝜎)𝑐𝛾²=∶𝑏. (56) Note that 𝑏 can be defined such that 𝑏>1. Therefore, we can say 𝑏>1. As a result, we define ()𝜆∶=1−𝜎𝑐𝛾²2𝑏(2𝐿+𝑡)𝛾, (57) we get from the first inequality in (56) that if ‖𝑠_𝑘−1‖≤𝜆, then 𝛽^MDL_𝑘≤(2𝐿+𝑡)𝜆(1−𝜎)𝑐𝜆²=1.2𝑏 (58)

Let 𝑁^∗ denote the set of positive integers. For 𝜆>0 and a positive integer Δ, denote 𝐾^𝜆_𝑘,Δ∶=𝑖∈𝑁^∗∶𝑘≤𝑖≤𝑘+Δ−1,𝑠_𝑘−1>𝜆. (59) Let |𝐾^𝜆_𝑘,Δ| denote the number of elements in 𝐾^𝜆_𝑘,Δ. From the previous property (*), we can prove the following lemma.

Lemma 7. Suppose that Assumption A holds. Consider MDL method, where 𝛼_𝑘 is obtained by the strong Wolfe-Powell line search in which 𝜎<1/3. Then if (42) holds, there exists 𝜆>0 such that, for any Δ∈𝑁^∗ and any index 𝑘₀, there is an index 𝑘≥𝑘₀ such that 𝐾^𝜆_𝑘,Δ>Δ2. (60)

The proof of this lemma is similar to the proof of Lemma 6 in [4]. In [4], authors proved that method with (19) has this property, if the search direction 𝑑_𝑘 satisfies the sufficient descent condition (21). In our paper, we do not need this assumption, since the directions generated by MDL method with strong Wolfe-Powell line search always possess the sufficient descent condition (21). So, we omit the proof of this lemma.

According to the previous lemmas and theorems, we can prove the following convergence theorem for the MDL.

Theorem 8. Suppose that Assumption A holds. Consider MDL method, if 𝛼_𝑘 is obtained by strong Wolfe-Powell line search with 𝜎<1/3. Then we have liminf_𝑘→∞‖𝑔_𝑘‖=0.

Proof. We proceed by contradiction. If liminf_𝑘→∞‖𝑔_𝑘‖>0, then (42) must hold. Then the conditions of Lemmas 6 and 7 hold. Defining 𝑢_𝑖=𝑑_𝑖/‖𝑑_𝑖‖, we have for any indices 𝑙, 𝑘, with 𝑙≥𝑘, 𝑥_𝑙−𝑥_𝑘−1=^𝑙_𝑖=𝑘𝑥_𝑖−𝑥_𝑖−1=^𝑙_𝑖=𝑘𝛼_𝑖−1𝑑_𝑖−1=^𝑙_𝑖=𝑘𝑢_𝑖−1𝑠_𝑖−1=^𝑙_𝑖=𝑘𝑠_𝑖−1𝑢_𝑘−1+^𝑙_𝑖=𝑘𝑠_𝑖−1𝑢_𝑖−1−𝑢_𝑘−1. (61) Equation (61), ‖𝑢_𝑖‖=1, and (28) give ^𝑙_𝑖=𝑘𝑠_𝑖−1≤𝑥_𝑙−𝑥_𝑘−1+^𝑙_𝑖=𝑘𝑠_𝑖−1𝑢_𝑖−1−𝑢_𝑘−1≤2𝐵+^𝑙_𝑖=𝑘𝑠_𝑖−1𝑢_𝑖−1−𝑢_𝑘−1. (62)
Let 𝜆>0 be given by Lemma 7, and define Δ∶=⌈8𝐵/𝜆⌉ to be the smallest integer not less than 8𝐵/𝜆. By Lemma 6, we can find an index 𝑘₀≥1 such that _𝑖≥𝑘0𝑢_𝑖−1−𝑢_𝑘−1²≤1.4Δ (63) With this Δ and 𝑘₀, Lemma 7 gives an index 𝑘≥𝑘₀ such that 𝐾^𝜆_𝑘,Δ>Δ2. (64) For any index 𝑖∈[𝑘,𝑘+Δ−1], by Cauchy-Schwartz inequality and (63), 𝑢_𝑖−𝑢_𝑘−1≤^𝑖_𝑗=𝑘𝑢_𝑗−𝑢_𝑗−1≤(𝑖−𝑘+1)^1/2𝑖_𝑗=𝑘𝑢_𝑗−𝑢_𝑗−1^21/2≤Δ^1/214Δ^1/2=12. (65) From these relations (65) and (64) and taking 𝑙=𝑘+Δ−1 in (62), we get 12𝐵≥2^𝑘+Δ−1_𝑖=𝑘𝑠_𝑖−1>𝜆2𝐾^𝜆_𝑘,Δ>𝜆Δ4. (66) Thus, Δ<8𝐵/𝜆, which contradicts the definition of Δ. The proof is completed.

4. Numerical Results

From (26) and (27), the MDL method can be considered as(i)form 1: a modification form of DL method;(ii)form 2: a modification form of MHS method;(iii)form 3: a modification form of DY method.

In form 1, the 𝛽^HS_𝑘 in 𝛽^DL_𝑘 is replaced by 𝛽^MHS_𝑘. By this modification, we can guarantee the nonnegativity restrictions in DL method. In form 2, 𝛽^MDL_𝑘 is obtained by 𝛽^MHS_𝑘 adding an adjusting term −𝑡(𝑔^𝑇_𝑘𝑠_𝑘−1/𝑦^𝑇_𝑘−1𝑑_𝑘−1) which contains some Hessian information of the objective function. In form 3, 𝛽^MDL_𝑘=𝛽^DY_𝑘(1−cos𝜃_𝑘)−𝑡(𝑔^𝑇_𝑘𝑠_𝑘−1/𝑑^𝑇_𝑘−1𝑦_𝑘−1) shows that 𝛽^MDL_𝑘 is obtained by multiplying 𝛽^DY_𝑘 with (1−cos𝜃_𝑘) and adding the second term −𝑡(𝑔^𝑇_𝑘𝑠_𝑘−1/𝑦^𝑇_𝑘−1𝑑_𝑘−1).

From the above convergence analysis, we know that MDL method has stronger convergent properties than DL method, and similar convergent properties with MHS method and DY method. So, in this section, we test the following four CG methods:(i)MDL method: method of the forms (2) and (3), in which 𝛽_𝑘 is computed by 𝛽^MDL_𝑘 (26);(ii)DL method: method of the forms (2) and (3), in which 𝛽_𝑘 is computed by 𝛽^DL_𝑘 (19);(iii)MHS method: method of the forms (2) and (3), in which 𝛽_𝑘 is computed by 𝛽^MHS_𝑘 (23);(iv)DY method: method of the forms (2) and (3), in which 𝛽_𝑘 is computed by 𝛽^DY_𝑘 (9).

The step length 𝛼_𝑘 in all methods is determined such that the strong Wolfe-Powell conditions (31) and (32) hold with 𝛿=0.01 and 𝜎=0.1.

The test problems are drawn from [14]. The numerical results of our tests are reported in Table 1.

Table 1 

Numerical results.

The column problem represents the problem name in [14], Dim represents the dimension of the problems. The numerical results are given in the form of 𝐼/𝐹/𝐺, where 𝐼, 𝐹, and 𝐺 denote the numbers of iterations, function evaluations and gradient evaluations, respectively. The stopping condition is ‖𝑔_𝑘‖≤10⁻⁶. Since we want to compare the performance of the different methods, in the numerical results, we omit the problems if all the four methods perform equally. The notation 𝐹 means that, for this problem, the corresponding method fails.

5. Conclusions

In this paper, based on 𝛽^DY_𝑘 and 𝛽^DL_𝑘, a new formula is proposed to compute the parameter 𝛽_𝑘 of the conjugate gradient methods. The main motivations are to improve both the convergence properties and numerical behavior of the conjugate gradient method. For general conjugate gradient methods, in order to get the global convergence results, the methods are required to possess the following major properties:(1)the generated directions 𝑑_𝑘 are descent directions;(2)the parameters 𝛽_𝑘 are nonnegative.

In addition, to ensure that the methods have robust and efficient numerical behavior, the parameter 𝛽_𝑘 needs to approach zero, when the small step 𝑠_𝑘 occurs.

From the convergence analysis of this paper, we known that the directions 𝑑_𝑘 generated by MDL method are descent directions, which is not true for DY or DL methods, and the proposed MDL method is globally convergent for general functions. In the previous section, we compare the numerical performance of the MDL method with the DY, MHS, and DL methods. From the convergence analysis and numerical results, comparing with the DL, DY, and MHS method, we can have the following.(a)MDL method versus DL method: from the computational point of view, for most of the test problems, MDL method performs quite similarly with DL method. There are 15 problems in which MDL method outperforms the DL method and 18 problems in which DL method outperforms the MDL method. But, from the convergent point of view, the MDL method outperforms the DL method.(b)MDL method versus DY method: the convergence properties of MDL method are similar to DY method. By comparing the numerical results of MDL method with DY method, there are 27 test problems in which MDL method outperforms the DY method and only 4 test problems in which DY method outperforms the MDL method. Therefore, we could say that MDL method is much better than the DY method in numerical behavior.(c)MDL method versus MHS method: they possess similar convergence properties; the numerical results show that MDL method performs little better than the MHS method.

Acknowledgments

This research was supported by Guangxi High School Foundation Grant no. 2013BYB210 and Guangxi University of Finance and Economics Science Foundation Grant no. 2013A015.