A Conjugate Gradient Method for Unconstrained Optimization Problems

Abstract

A hybrid method combining the FR conjugate gradient method and the WYL conjugate gradient method is proposed for unconstrained optimization problems. The presented method possesses the sufficient descent property under the strong Wolfe-Powell (SWP) line search rule relaxing the parameter 𝜎<1. Under the suitable conditions, the global convergence with the SWP line search rule and the weak Wolfe-Powell (WWP) line search rule is established for nonconvex function. Numerical results show that this method is better than the FR method and the WYL method.

1. Introduction

Consider the following 𝑛 variables unconstrained optimization problem: min_{𝑥∈ℜ𝑛}𝑓(𝑥), (1.1) where 𝑓∶ℜ^𝑛→ℜ is smooth and its gradient 𝑔(𝑥) is avaible. The nonlinear conjugate gradient (CG) method for (1.1) is designed by the iterative form 𝑥_𝑘+1=𝑥_𝑘+𝛼_𝑘𝑑_𝑘,𝑘=0,1,2,…, (1.2) where 𝑥_𝑘 is the 𝑘th iterative point, 𝛼_𝑘>0 is a steplength, and 𝑑_𝑘 is the search direction defined by 𝑑_𝑘=−𝑔_𝑘+𝛽_𝑘𝑑_𝑘−1,if𝑘≥1,−𝑔_𝑘,if𝑘=0, (1.3) where 𝛽_𝑘∈ℜ is a scalar which determines the different conjugate gradient methods [1, 2], and 𝑔_𝑘 is the gradient of 𝑓(𝑥) at the point 𝑥_𝑘. There are many well-known formulas for 𝛽_𝑘, such as the Fletcher-Reeves (FR) [3], Polak-Ribière-Polyak (PRP) [4], Hestenses-Stiefel (HS) [5], Conjugate-Descent (CD) [6], Liu-Storrey (LS) [7], and Dai-Yuan (DY) [8]. The CG method is a powerful line search method for solving optimization problems, and it remains very popular for engineers and mathematicians who are interested in solving large-scale problems [9–11]. This method can avoid, like steepest descent method, the computation and storage of some matrices associated with the Hessian of objective functions. Then there are many new formulas that have been studied by many authors (see [12–20] etc.).

The following formula for 𝛽_𝑘 is the famous FR method: 𝛽^FR_𝑘=‖𝑔_𝑘+1‖²‖𝑔_𝑘‖², (1.4) where 𝑔_𝑘 and 𝑔_𝑘+1 are the gradients ∇𝑓(𝑥_𝑘) and ∇𝑓(𝑥_𝑘+1) of 𝑓(𝑥) at the point 𝑥_𝑘 and 𝑥_𝑘+1, respectively, ‖⋅‖ denotes the Euclidian norm of vectors. Throughout this paper, we also denote 𝑓(𝑥_𝑘) by 𝑓_𝑘. Under the exact line search, Powell [21] analyzed the small-stepsize property of the FR conjugate gradient method and its global convergence, Zoutendijk [22] proved its global convergence for nonconvex function. Al-Baali [23] proved the sufficient descent condition and the global convergence of the FR conjugate gradient method with the SWP line search by restricting the parameter 𝜎<1/2. Liu et al. [24] extended the result to the the parameter 𝜎=1/2. Wei et al. (WYL) [17] proposed a new conjugate gradient formula: 𝛽^WYL_𝑘=𝑔^𝑇_𝑘+1(𝑔_𝑘+1−(‖𝑔_𝑘+1‖/‖𝑔_𝑘‖)𝑔_𝑘)‖𝑔_𝑘‖². (1.5) The numerical results show that this method is competitive to the PRP method, the global convergence of this method with the exact line search and Grippo-Lucidi line search conditions is proved. Huang et al. [25] proved that by restricting the parameter 𝜎<1/4, under the SWP line search rule, this method has the sufficient descent property. Then it is an interesting task to extend the bound of the parameter 𝜎 and get the sufficient descent condition.

The sufficient descent condition 𝑔^𝑇_𝑘𝑑_𝑘≤−𝑐‖𝑔_𝑘‖²,∀𝑘≥0, (1.6) where 𝑐>0 is a constant, is crucial to insure the global convergence of the nonlinear conjugate gradient method [23, 26–28]. In order to get some better results of the conjugate gradient methods, Andrei [29, 30] proposed the hybrid conjugate gradient algorithms as convex combination of some other conjugate gradient algorithms. Motivated by the ideas of Andrei [29, 30] and the above observations, we will give a hybrid method combining the FR method and the WYL method. The proposed method, relaxing the parameter 𝜎<1, under the SWP line search technique, possesses the sufficient descent condition (1.6). The global convergence with the SWP line search and the WWP line search of our method is established for the nonconvex functions. Numerical results show that the presented method is competitive to the FR and the WYL method.

In the following section, the algorithm is stated. The properties and the global convergence of the new method are proved in Section 3. Numerical results are reported in Section 4 and one conclusion is given in Section 5.

2. Algorithm

Now we describe our algorithm as follows.

Algorithm 2.1 (the hybrid method). Step 1. Choose an initial point 𝑥₀∈ℜ^𝑛, 𝜀∈(0,1), 𝜆₁≥0, 𝜆₂≥0. Set 𝑑₀=−𝑔₀=−∇𝑓(𝑥₀),𝑘∶=0.Step 2. If ‖𝑔_𝑘‖≤𝜀, then stop; otherwise go to the next step.Step 3. Compute step size 𝛼_𝑘 by some line search rules.Step 4. Let 𝑥_𝑘+1=𝑥_𝑘+𝛼_𝑘𝑑_𝑘. If ‖𝑔_𝑘+1‖≤𝜀, then stop.Step 5. Calculate the search direction 𝑑_𝑘+1=−𝑔_𝑘+1+𝛽^𝐻_𝑘𝑑_𝑘, (2.1) where 𝛽^𝐻_𝑘=𝜆₁𝛽^WYL_𝑘+𝜆₂𝛽^FR_𝑘.Step 6. Set 𝑘∶=𝑘+1, and go to Step 3.

3. The Properties and the Global Convergence

In the following, we assume that 𝑔_𝑘≠0 for all 𝑘, otherwise a stationary point has been found. The following assumptions are often used to prove the convergence of the nonlinear conjugate gradient methods (see [3, 8, 16, 17, 27]).

Assumption 3.1. (i) The function 𝑓(𝑥) has a lower bound on the level set Ω={𝑥∈ℜ^𝑛∣𝑓(𝑥)≤𝑓(𝑥₀)}, where 𝑥₀ is a given point.
(ii) In an open convex, set Ω₀ that contains Ω,𝑓 is differentiable, and its gradient 𝑔 is Lipschitz continuous, namely, there exists a constants 𝐿>0 such that
‖𝑔(𝑥)−𝑔(𝑦)‖≤𝐿‖𝑥−𝑦‖,∀𝑥,𝑦∈Ω₀. (3.1)

3.1. The Properties with the Strong Wolfe-Powell Line Search

The strong Wolfe-Powell (SWP) search rule is to find a step length 𝛼_𝑘 such that

𝑓(𝑥_𝑘+𝛼_𝑘𝑑_𝑘)≤𝑓_𝑘+𝛿𝛼_𝑘𝑔^𝑇_𝑘𝑑_𝑘,|𝑔(𝑥_𝑘+𝛼_𝑘𝑑_𝑘)^𝑇𝑑_𝑘|≤𝜎|𝑔^𝑇_𝑘𝑑_𝑘|, (3.2)(3.3) where 𝛿∈(0,1/2),𝜎∈(0,1).

The following theorem shows that the hybrid algorithm with the SWP line search possesses the sufficient condition (1.6) only under the parameter 𝜎∈(𝛿,1).

Theorem 3.2. Let the sequences {𝑔_𝑘} and {𝑑_𝑘} be generated by Algorithm 2.1, and let the stepsize 𝛼_𝑘 be determined by the SWP line search (3.2) and (3.3), if 𝜎∈(0,1),2𝜆₁+𝜆₂∈(0,1/2𝜎), then the sufficient descent condition (1.6) holds.

Proof. By the definition 𝜆₁,𝜆₂ and the formulae (1.4) and (1.5), we have 𝛽^𝐻_𝑘=𝜆₁𝛽^WYL_𝑘+𝜆₂𝛽^FR_𝑘≥𝜆₁(‖𝑔_𝑘+1‖²−(‖𝑔_𝑘+1‖/‖𝑔_𝑘‖)‖𝑔_𝑘+1‖‖𝑔_𝑘‖)‖𝑔_𝑘‖²+𝜆₂‖𝑔_𝑘+1‖²‖𝑔_𝑘‖²=𝜆₂‖𝑔_𝑘+1‖²‖𝑔_𝑘‖²,|𝛽^𝐻_𝑘|=|𝜆₁𝛽^WYL_𝑘+𝜆₂𝛽^FR_𝑘|≤𝜆₁(‖𝑔_𝑘+1‖²+(‖𝑔_𝑘+1‖/‖𝑔_𝑘‖)‖𝑔_𝑘+1‖‖𝑔_𝑘‖)‖𝑔_𝑘‖²+𝜆₂‖𝑔_𝑘+1‖²‖𝑔_𝑘‖²=(2𝜆₁+𝜆₂)‖𝑔_𝑘+1‖²‖𝑔_𝑘‖². (3.4)(3.5) Using (3.3) and the above inequality, we get |𝛽^𝐻_𝑘𝑔^𝑇_𝑘+1𝑑_𝑘|≤(2𝜆₁+𝜆₂)‖𝑔_𝑘+1‖²‖𝑔_𝑘‖²𝜎|𝑔^𝑇_𝑘𝑑_𝑘|. (3.6) By (2.1), we have 𝑔^𝑇_𝑘+1𝑑_𝑘+1‖𝑔_𝑘+1‖²=−1+𝛽^𝐻_𝑘𝑔^𝑇_𝑘+1𝑑_𝑘‖𝑔_𝑘+1‖². (3.7) We prove the descent property of {𝑑_𝑘} by induction. Since 𝑔^𝑇₀𝑑₀=−‖𝑔₀‖²<0, if 𝑔₀≠0, now we suppose that 𝑑_𝑖,𝑖=1,2,…,𝑘, are all descent directions, for example, 𝑑^𝑇_𝑖𝑔_𝑖<0.
By (3.6), we get
|𝛽^𝐻_𝑘𝑔^𝑇_𝑘+1𝑑_𝑘|≤𝜎(2𝜆₁+𝜆₂)‖𝑔_𝑘+1‖²‖𝑔_𝑘‖²(−𝑔^𝑇_𝑘𝑑_𝑘), (3.8) that is, ‖𝑔_𝑘+1‖²‖𝑔_𝑘‖²(2𝜆₁+𝜆₂)𝜎𝑔^𝑇_𝑘𝑑_𝑘≤𝛽^𝐻_𝑘𝑔^𝑇_𝑘+1𝑑_𝑘≤−‖𝑔_𝑘+1‖²‖𝑔_𝑘‖²(2𝜆₁+𝜆₂)𝜎𝑔^𝑇_𝑘𝑑_𝑘. (3.9)   However, from (3.7) together with (3.9), we deduce −1+(2𝜆₁+𝜆₂𝑔)𝜎^𝑇𝑘𝑑_𝑘‖𝑔_𝑘‖²≤𝑔^𝑇_𝑘+1𝑑_𝑘+1‖𝑔_𝑘+1‖²≤−1−(2𝜆₁+𝜆₂𝑔)𝜎^𝑇𝑘𝑑_𝑘‖𝑔_𝑘‖². (3.10) Repeating this process and using the fact 𝑑^𝑇₀𝑔₀=−‖𝑔₀‖² imply −^𝑘_𝑖=0[(2𝜆₁+𝜆₂)𝜎]^𝑖≤𝑔^𝑇_𝑘+1𝑑_𝑘+1‖𝑔_𝑘+1‖²≤−2+^𝑘_𝑖=0[(2𝜆₁+𝜆₂)𝜎]^𝑖. (3.11) By the restriction 𝜎∈(0,1) and 2𝜆₁+𝜆₂∈(0,1/2𝜎), we have (2𝜆₁+𝜆₂)𝜎∈(0,1/2). So ^𝑘_𝑖=0[(2𝜆₁+𝜆₂)𝜎]^𝑖<^∞_𝑖=0[(2𝜆₁+𝜆₂)𝜎]^𝑖=11−4𝜆₁𝜎. (3.12) Then (3.11) can be rewritten as −11−(2𝜆₁+𝜆₂≤𝑔)𝜎^𝑇𝑘+1𝑑_𝑘+1‖𝑔_𝑘+1‖²1≤−2+1−(2𝜆₁+𝜆₂.)𝜎 (3.13) Thus, by induction, 𝑔^𝑇_𝑘𝑑_𝑘<0 holds for all 𝑘≥0.
Denote 𝑐=2−1/(1−(2𝜆₁+𝜆₂)𝜎), then 𝑐∈(0,1) and (3.13) turns out to be
𝑔𝑐−2≤^𝑇𝑘𝑑_𝑘‖𝑔_𝑘‖²≤−𝑐, (3.14) this implies that (1.6) holds. The proof is complete.

Lemma 3.3. Suppose that Assumption 3.1 holds. Let the sequences {𝑔_𝑘} and {𝑑_𝑘} be generated by Algorithm 2.1, let the stepsize 𝛼_𝑘 be determined by the SWP line search (3.2) and (3.3), and let the conditions in Theorem 3.2 hold. Then the Zoutendijk condition [22] ^∞_𝑘=0(𝑔^𝑇_𝑘𝑑_𝑘)²‖𝑑_𝑘‖²<+∞ (3.15) holds.

By the same way, if Assumption 3.1 and the condition 𝑔^𝑇_𝑘𝑑_𝑘<0 (for all 𝑘) hold, (3.15) also holds for the exact line search, the Armijo-Goldstein line search, and the weak Wolfe-Powell line search. The proofs can be seen in [31, 32]. Now, we prove the global convergence theorem of Algorithm 2.1 with the SWP line search.

Theorem 3.4. Suppose that Assumption 3.1 holds. Let the sequence {𝑔_𝑘} and {𝑑_𝑘} be generated by Algorithm 2.1, let the stepsize 𝛼_𝑘 be determined by the SWP line search (3.2) and (3.3), let the conditions in Theorem 3.2 hold, and let the parameter 2𝜆₁+𝜆₂≤1. Then lim_𝑘→∞inf‖𝑔_𝑘‖=0. (3.16)

Proof. By (1.6), (3.3), and the Zoutendijk condition (3.15), we get ^∞_𝑘=0‖𝑔_𝑘‖⁴‖𝑑_𝑘‖²<+∞. (3.17) Denote 𝑡_𝑘=‖𝑑_𝑘‖²‖𝑔_𝑘‖⁴, (3.18) so (3.17) can be rewritten as ^∞_𝑘=01𝑡_𝑘<+∞. (3.19) We prove the result of this theorem by contradiction. Assume that this theorem is not true, then there exists a positive constant 𝛾>0 such that ‖𝑔_𝑘‖≥𝛾,∀𝑘≥0. (3.20) Squaring both sides of (2.1), we obtain ‖𝑑_𝑘‖²=‖𝑔_𝑘‖²−2𝛽^𝐻_𝑘−1𝑔^𝑇_𝑘𝑑_𝑘−1+(𝛽^𝐻_𝑘−1‖𝑑_𝑘−1‖)². (3.21) Dividing both sides by ‖𝑔_𝑘‖⁴, applying (3.4), (3.18), and the parameter 2𝜆₁+𝜆₂≤1, we get 𝑡_𝑘≤𝑡_𝑘−1+1‖𝑔_𝑘‖²(1+2|𝑔^𝑇_𝑘𝑑_𝑘−1|‖𝑔_𝑘−1‖²). (3.22) Using (3.3) and (1.6), we have 𝑡_𝑘≤𝑡_𝑘−1+1‖𝑔_𝑘‖²(1+2𝜎|𝑔^𝑇_𝑘−1𝑑_𝑘−1|‖𝑔_𝑘−1‖²)≤𝑡_𝑘−11+[1+2𝜎(2−𝑐)]‖𝑔_𝑘‖². (3.23) Repeating this process and using the fact 𝑡₀=1/‖𝑔₀‖², we get 𝑡_𝑘≤[1+2𝜎(2−𝑐)]^∞_𝑖=11‖𝑔_𝑖‖². (3.24) Now, combining (3.24) and (3.20), we get another formula 𝑡_𝑘≤[1+2𝜎(2−𝑐)]𝑘+1𝛾². (3.25) Thus ^∞_𝑘=01𝑡_𝑘=+∞, (3.26) this contradicts the condition (3.19). However, the conclusion of this theorem is correct.The following theorem will show that Algorithm 2.1 with the SWP line search, only under the descent condition 𝑔^𝑇_𝑘𝑑_𝑘<0,∀𝑘≥0, (3.27) is global convergence too.

Theorem 3.5. Suppose that Assumption 3.1 holds. Let the sequence {𝑔_𝑘,𝑑_𝑘} be generated by Algorithm 2.1, let the stepsize 𝛼_𝑘 be determined by the SWP line search (3.2) and (3.3), let the parameter 2𝜆₁+𝜆₂≤1, and let (3.27) hold. Then, for all 𝑘≥0, the following inequalities holds: min{𝑟_𝑘,𝑟_𝑘+11}≥2, (3.28) where 𝑟_𝑘𝑔=−^𝑇𝑘𝑑_𝑘‖𝑔_𝑘‖², (3.29) furthermore, (3.16) holds

Proof. By (2.1), (3.3), (3.27), and 2𝜆₁+𝜆₂≤1, we can deduce that (3.10) holds for all 𝜎<1 and 𝜎_∗=(2𝜆₁+𝜆₂)𝜎<1. According to the second inequality of (3.10), we have 𝜎_∗𝑟_𝑘+𝑟_𝑘+1≥1. (3.30) Similar to the way of [31], it is not difficult to get (3.28).
Secondly, using (3.30) and the Cauchy-Schwarz inequality implies that (or see [31])
𝑟²_𝑘+𝑟²_𝑘+1≥(1+𝜎²_∗)⁻¹. (3.31) Moreover, repeating the process of the first inequality of (3.10) and using 𝑟₀=1, we get 𝑟_𝑘+1<(1−𝜎_∗)⁻¹. (3.32) By (3.3), (3.22), and the above inequality, we have 𝑡_𝑘+1≤1+𝜎_∗1−𝜎_∗^𝑘+1_𝑖=01‖𝑔_𝑖‖². (3.33) Thus, using (3.20), we have 𝑡_𝑘+1≤𝑏₁(𝑘+2), (3.34) where 𝑏₁=1+𝜎_∗/(1−𝜎_∗)𝛾². By Zoutendijk condition (3.15), (3.31), and (3.34), we obtain +∞>^∞_𝑘=0(𝑔^𝑇_𝑘𝑑_𝑘)²‖𝑑_𝑘‖²=^∞_𝑘=0𝑟²_𝑘𝑡_𝑘=^∞_𝑘=0(𝑟²_2𝑘−1𝑡_2𝑘−1+𝑟²_2𝑘𝑡_2𝑘)≥^∞_𝑘=0𝑟²_2𝑘−1+𝑟²_2𝑘2𝑏₁≥(𝑘+1)^∞_𝑘=012𝑏₁(1+𝜎²_∗)(𝑘+1)=+∞. (3.35) This contradiction shows that (3.16) is true.

3.2. The Properties with the Weak Wolfe-Powell (WWP) Line Search

The weak Wolfe-Powell line search is to find a step length 𝛼_𝑘 satisfying (3.2) and 𝑔(𝑥_𝑘+𝛼_𝑘𝑑_𝑘)^𝑇𝑑_𝑘≥𝜎𝑔^𝑇_𝑘𝑑_𝑘, (3.36) where 𝛿∈(0,1/2),𝜎∈(𝛿,1).

From the computation point of view, one of the well-known formulas for 𝛽_𝑘 is the PRP method. The global convergence with the exact line search had been proved by Polak and Ribière [4] when the objective function is convex. Powell [33] gave a counter example to show that there exist nonconvex functions on which the PRP method does not converge globally even if the exact line search is used. He suggested that 𝛽_𝑘 should not be less than zero. Considering this suggestion, under the assumption of the sufficient descent condition, Gilbert and Nocedal [27] proved that the modified PRP method 𝛽⁺_𝑘=max{0,𝛽^PRP_𝑘} is globally convergent with the WWP line search technique. For the new formula 𝛽^𝐻_𝑘, we know that it is always larger than zero. Then we can also get the global convergence of the hybrid method with the WWP line search.

Lemma 3.6 (see [31, Lemma 3.3.1]). Let Assumption 3.1 hold and let the sequences {𝑔_𝑘,𝑑_𝑘} be generated by Algorithm 2.1. The stepsize 𝛼_𝑘 is determined by (3.2) and (3.36). Suppose that (3.20) is true, and the sufficient descent condition (1.6) holds. Then we have 𝑑_𝑘≠0 and ^∞_𝑘=0‖𝑢_𝑘+1−𝑢_𝑘‖²<∞, (3.37) where 𝑢_𝑘=𝑑_𝑘/‖𝑑_𝑘‖.

The following Property 1 was introduced by Gilbert and Nocedal [27], which pertains to the PRP method under the sufficient descent condition. The WYL also has this property. Now we will prove that this Property 1 pertains to the new method.

Property 1. Suppose that 0<𝛾₁≤‖𝑔_𝑘‖≤𝛾₂. (3.38) We say that the method has Property 1 if for all 𝑘, there exists constants 𝑏>1 and 𝜆>0 such that |𝛽_𝑘|≤𝑏 and ‖𝑠_𝑘‖≤𝜆⇒|𝛽_𝑘1|≤.2𝑏 (3.39)

Lemma 3.7. Let Assumption 3.1 hold, let the sufficient descent condition (1.6) hold, and let the sequences {𝑔_𝑘,𝑑_𝑘} be generated by Algorithm 2.1. Suppose that there exists a constant 𝑀>0 such that ‖𝑑_𝑘‖≤𝑀 for all 𝑘. Then this method possesses Property 1.

Proof. By (3.36), (1.6), (3.1), and (3.38), we have 𝛾₁𝑐(1−𝜎)‖𝑔_𝑘‖≤𝑐(1−𝜎)‖𝑔_𝑘‖²≤−(1−𝜎)𝑔^𝑇_𝑘𝑑_𝑘≤(𝑔_𝑘+1−𝑔_𝑘)^𝑇𝑑_𝑘≤𝐿‖𝑠_𝑘‖‖𝑑_𝑘‖≤𝐿𝑀‖𝑠_𝑘‖. (3.40) Then we get ‖𝑔_𝑘‖≤𝐿𝑀𝛾₁𝑐(1−𝜎)‖𝑠_𝑘‖. (3.41) By (3.1) again, we obtain ‖𝑔_𝑘+1‖−‖𝑔_𝑘‖≤‖𝑔_𝑘+1−𝑔_𝑘‖≤𝐿‖𝑠_𝑘‖. (3.42) Combing the above inequality and (3.41) implies that ‖𝑔_𝑘+1‖≤‖𝑔_𝑘‖+𝐿‖𝑠_𝑘‖≤(𝐿𝑀𝛾₁𝑐(1−𝜎)+𝐿)‖𝑠_𝑘‖. (3.43) By (3.5) and (3.38), we have |𝛽^𝐻_𝑘|=(2𝜆₁+𝜆₂)‖𝑔_𝑘+1‖²‖𝑔_𝑘‖²≤(2𝜆₁+𝜆₂)𝛾²₂𝛾²₁, (3.44) let 𝑏=max{2,(2𝜆₁+𝜆₂)𝛾²₂/𝛾²₁}>1,𝜆=𝛾²₁/2𝑏𝛾₂𝐿(2𝜆₁+𝑀/𝛾₁𝑐(1−𝜎)+1). If ‖𝑠_𝑘‖≤𝜆, using (3.1), (3.38), (3.43), and the above equation, we obtain |𝛽^𝐻_𝑘|≤‖𝑔_𝑘+1‖𝜆₁‖𝑔_𝑘+1−(‖𝑔_𝑘+1‖/‖𝑔_𝑘‖)𝑔_𝑘‖+𝜆₂‖𝑔_𝑘+1‖‖𝑔_𝑘‖²=𝛾₂𝜆₁‖𝑔_𝑘+1−𝑔_𝑘+𝑔_𝑘−(‖𝑔_𝑘+1‖/‖𝑔_𝑘‖)𝑔_𝑘‖+𝜆₂‖𝑔_𝑘+1‖‖𝑔_𝑘‖²≤𝛾₂𝜆₁‖𝑔_𝑘+1−𝑔_𝑘‖+𝜆₁|‖𝑔_𝑘‖−‖𝑔_𝑘+1‖|+𝜆₂‖𝑔_𝑘+1‖‖𝑔_𝑘‖²≤𝛾₂2𝜆₁𝐿‖𝑠_𝑘‖+(𝐿𝑀/𝛾₁𝑐(1−𝜎)+𝐿)‖𝑠_𝑘‖‖𝑔_𝑘‖²≤𝛾₂𝐿(2𝜆₁+𝑀/𝛾₁𝑐(1−𝜎)+1)𝛾²₁1𝜆=.2𝑏 (3.45) Therefore, the conclusion of this lemma holds.

Lemma 3.8 (see [31, Lemma 3.3.2]). Let the sequences {𝑔_𝑘} and {𝑑_𝑘} be generated by Algorithm 2.1 and the conditions in Lemma 3.7 hold. If 𝛽^𝐻_𝑘≥0 and has Property 1, then there exists 𝜆>0 such that for any Δ∈𝑁 and any index 𝑘₀, there is an index 𝑘>𝑘₀ satisfying |𝜅^𝜆_𝑘,Δ𝜆|>2, (3.46) where 𝜅^𝜆_𝑘,Δ={𝑖∈𝑁∶𝑘≤𝑖≤𝑘+Δ−1,‖𝑠_𝑖‖>𝜆},𝑁 denotes the set of positive integers, |𝜅^𝜆_𝑘,Δ| denotes the numbers of elements in 𝜅^𝜆_𝑘,Δ.

Finally, by Lemmas 3.6 and 3.8, we present the global convergence theorem of Algorithm 2.1 with the WWP line search. Similar to [31, Theorem 3.3.3], it is not difficult to prove the result, so we omit it.

Theorem 3.9. Let the sequences {𝑔_𝑘} and {𝑑_𝑘} be generated by Algorithm 2.1 with the weak Wolfe-Powell line search and the conditions in Lemma 3.7 hold. Then (3.16) holds.

4. Numerical Results

In this section, we report some results of the numerical experiments. It is well known that there exist many new conjugate gradient methods (see [1, 13–16, 18, 19, 29, 30]) which have good properties and good numerical performances. Since the given formula is the hybrid of the FR formula and the WYL formula, we only test Algorithm 2.1 under the WWP line search on problems in [34] with the given initial points and dimensions, and compare its performance with those of the FR [3] and the WYL [17] methods. The parameters are chosen as follows: 𝛿=0.1,𝜎=0.2,𝜆₁=𝜆₂=0.5. The following Himmeblau stop rule is used as follows.

If |𝑓(𝑥_𝑘)|>𝑒₁, let 𝑠𝑡𝑜𝑝1=|𝑓(𝑥_𝑘)−𝑓(𝑥_𝑘+1)|/|𝑓(𝑥_𝑘)|; otherwise, let 𝑠𝑡𝑜𝑝1=|𝑓(𝑥_𝑘)−𝑓(𝑥_𝑘+1)|.

If ‖𝑔(𝑥)‖<𝜀 or 𝑠𝑡𝑜𝑝1<𝑒₂ was satisfied, we will stop the program, where 𝑒₁=𝑒₂=10⁻⁶. We also stop the program if the iteration number is more than one thousand. All codes were written in MATLAB and run on PC with 2.60 GHz CPU processor and 256 MB memory and Windows XP operation system. The detail numerical results are listed at http://210.36.18.9:8018/publication.asp?id=36990.

Dolan and Moré [35] gave a new tool to analyze the efficiency of Algorithms. They introduced the notion of a performance profile as a means to evaluate and compare the performance of the set of solvers 𝑆 on a test set 𝑃. Assuming that there exist 𝑛_𝑠 solvers and 𝑛_𝑝 problems, for each problem 𝑝 and solver 𝑠, they defined:

𝑡_𝑝,𝑠= computing time (the number of function evaluations or others) required to solve problem 𝑝 by solver 𝑠.

Requiring a baseline for comparisons, they compared the performance on problem 𝑝 by solver 𝑠 with the best performance by any solver on this problem; that is, using the performance ratio 𝑟_𝑝,𝑠=𝑡_𝑝,𝑠min{𝑡_𝑝,𝑠.∶𝑠∈𝑆} (4.1) Suppose that a parameter 𝑟_𝑀≥𝑟_𝑝,𝑠 for all 𝑝,𝑠 is chosen, and 𝑟_𝑝,𝑠=𝑟_𝑀 if and only if solver 𝑠 does not solve problem 𝑝.

The performance of solver 𝑠 on any given problem might be of interest, but we would like to obtain an overall assessment of the performance of the solver, then they defined 𝜌_𝑠1(𝑡)=𝑛_𝑝size{𝑝∈𝑃∶𝑟_𝑝,𝑠≤𝑡}, (4.2) thus 𝜌_𝑠(𝑡) was the probability for solver 𝑠∈𝑆 that a performance ratio 𝑟_𝑝,𝑠 was within a factor 𝑡∈ℜ of the best possible ratio. Then function 𝜌_𝑠 was the (cumulative) distribution function for the performance ratio. The performance profile 𝜌_𝑠∶ℜ↦[0,1] for a solver was a nondecreasing, piecewise constant function, continuous from the right at each breakpoint. The value of 𝜌_𝑠(1) was the probability that the solver would win over the rest of the solvers.

According to the above rules, we know that one solver whose performance profile plot is on top right will win over the rest of the solvers.

Figures 1-2 show that the performances of these methods are relative to the iteration number (𝑁𝐼) and the number of the function and gradient (NFN), where the “FR” denotes the FR formula with WWP rule, the “WYL” denotes the WYL formula with WWP rule, and Algorithm 2.1 denotes the new method with WWP rule, respectively.

Figure 1 

Performance profiles of these three methods (NI).

Figure 2 

Performance profiles of these three methods (NFN).

From Figures 1-2, it is easy to see that Algorithm 2.1 is the best among the three methods, and the WYL method is much better than FR methods. Notice that the global convergence of the FR method with the WWP line search has not been established yet. In other words, the given method is competitive to the other two normal methods and the hybrid formula is notable.

5. Conclusions

This paper gives a hybrid conjugate gradient method for solving unconstrained optimization problems. Under the SWP line search, this method possesses the sufficient descent condition only with the parameter 𝜎<1. The global convergence with the SWP line search and the WWP line search is established for the nonconvex functions. Numerical results show that the given method is competitive to other two conjugate gradient methods.

For further research, we should study the new method with the nonmonotone line search technique. Moreover, more numerical experiments for large practical problems (such as the problems [36]) should be done, and the given method should be compared with other famous formulas in the future. How to choose the parameters 𝜆₁ and 𝜆₂ in the algorithm is another aspect of future investigation.

Acknowledgments

The authors are very grateful to the anonymous referees and the editors for their valuable suggestions and comments, which improved our paper greatly. This work is supported by China NSF grants 10761001 and the Scientific Research Foundation of Guangxi University (Grant no. X081082).