Abstract
The nonlinear conjugate gradient algorithms are a very effective way in solving large-scale unconstrained optimization problems. Based on some famous previous conjugate gradient methods, a modified hybrid conjugate gradient method was proposed. The proposed method can generate decent directions at every iteration independent of any line search. Under the Wolfe line search, the proposed method possesses global convergence. Numerical results show that the modified method is efficient and robust.
1. Introduction
Consider the following unconstrained optimization problem:where is a real vector with component and is a smooth function and its gradient is available. Unconstrained problem is an important problem with a broad range of scientific and operational applications.
During the last decade, the conjugate gradient methods constitute an active choice for efficiently solving the above optimization problem, especially when the dimension n is large, characterized by the simplicity of their iteration, their low memory requirement, and their excellent numerical performance. The general procedure of the iterative computational scheme is as follows.
When applied to solve problem (1), starting from an initial guess , the conjugate gradient method usually generates a sequence aswhere is the current iterate, is called a step size determined by some suitable line search, and is the search direction defined bywhere and is an important scalar parameter.
Generally inexact line search is used in order to get the global convergence of conjugate gradient method, such as the Wolfe line search or the strong Wolfe line search. That is, the step length is usually computed by the Wolfe line search:or the strong Wolfe line search:where are some fixed parameters.
Different conjugate gradient methods correspond to different values of the scalar parameter . The well-known conjugate gradient methods include the Fletcher–Reeves (FR) method [1], the Polak–Ribière–Polyak (PRP) method [2, 3], the Hestenes–Stiefel (HS) method [4], the Dai–Yuan (DY) method [5], the conjugate-descent method (CD) [6], and the Liu–Storey method (LS) [7]. The parameters in these conjugate gradient methods are specified as follows:where stands for the Euclidean norm. These methods are identical when is a strongly convex quadratic function and the line search is exact, since the gradients are mutually orthogonal, and the parameters in these methods are equal. When applied to general nonlinear functions with inexact line searches, the behavior of these methods is marked different.
It is well known that FR, DY, and CD methods have strong convergent properties, but they may not perform well in practice due to jamming. Moreover, although PRP, HS, and LS methods may not converge in general, they often perform better. Naturally, people try to devise some new methods, which have the advantages of these two kinds of methods. So far, various hybrid methods have been proposed (see [8–20]).
In [14], Wei et al. gave a variant of the PRP method, WYL method for short, where the parameter is yielded by
The above WYL method can be considered as the modification of the PRP method. It inherits the good properties of the PRP method, such as excellent numerical effect. Furthermore, Huang et al. [15] proved that the WYL method satisfies the sufficient descent condition and converges globally under the strong Wolfe line search (5) if the parameter satisfies .
Yao et al. [16] gave a modification of the HS conjugate gradient method as follows:
Under the strong Wolfe line search (5) with the parameter , it has been shown that the MHS method can generate sufficient descent directions and converges globally for general objective functions.
Jiang et al. [17] proposed a hybrid conjugate gradient method with
Under the Wolfe line search (4), the method possesses global convergence and efficient numerical performance.
In [18], Jiang et al. proposed a hybrid conjugate gradient method with
Under the parameter , it has been shown that the MDY method can generate sufficient descent directions and converges globally for general objective functions.
In [19], Jiang et al. proposed a hybrid conjugate gradient method, denote it by JHS with
Under the parameter , it has been shown that the JHS method can generate sufficient descent directions and converges globally for general objective functions.
In this paper, we introduce a new hybrid choice for parameter . This motivation mainly comes from [18, 19]. For convenience, we call the iteration method a FW method as follows:where .
It is easy to know that or or or , so is one of the hybrids of , , , and . The proposed method has attractive property of satisfying the sufficient descent condition independent of any line search and attains global convergence if the step length is yielded by the Wolfe line search (4).
This paper is organized as follows. In Section 2, we give the details of our algorithm and discuss its sufficient descent property. In Section 3, we prove the global convergence of the proposed method with Wolfe line search (4). A number of numerical experiments comparing the proposed method with other conjugate gradient methods are given in Section 4. Finally, conclusion is given in Section 5.
2. Algorithm and Its Property
In this section, first, based on the discussed above, we describe our algorithm framework (Algorithm 1) without fixed line search as follows.
|
The following lemma states that the search direction in Algorithm 1 is always sufficient descent depending on no line search.
Lemma 1. If the objective function is continuously differentiable, let be generated by Algorithm 1. Then, holds for each .
Proof. We prove this lemma by induction. For , it is easy to know that . Assume that holds for and . Now we prove that holds for .
If , then ; furthermore, we haveIf , we divide the proof into four following cases.(i)If and , then by the definition of . Noticing that , holds. From (3) and (12), we have(ii)If , , then from (12), one has . Therefore, we obtain(iii)If and , then from (12), one knows Noticing , we have , where is the angle between and . Furthermore, we obtain(iv)If and , then from (12), one getsAs in the case of (iii), we haveTherefore, holds for all .
Lemma 2. Let be generated by Algorithm 1. Then, for any , we can obtain the following relations:
Proof. From formula (12), it is easy to see thatNow we are ready to prove by considering the following four cases.(i)If and , then . In view of Lemma 1 and (14), we have(ii)If , , then . If , by the recurrence formula, we have Therefore, one gets Dividing both sides of (24) by , it follows that If , similarly, we can get that Dividing both sides of (26) by , one has(iii)If and , then . From the above formula, we have Combining with (17), we get(iv)If and , then .Noticing that , we have from the above formula thatSo,Dividing both sides of (31) by , we haveThe proof is complete.
3. Global Convergence of Algorithm
This section is devoted to the global convergence of algorithm framework under the Wolfe line search condition, i.e., the step length is yielded by condition (4). For this goal, we make the following basic assumptions in subsequent discussions. H 3.1 is bounded from below on the level set , where is the initial point. H 3.2 In some neighborhood of the level set , is continuous differentiable, and its gradient is Lipschitz continuous, that is to say, for all , there exists a constant such that
Lemma 3. Suppose that assumptions (H 3.1) and (H 3.2) hold. Let be generated by Algorithm 1, where the step length satisfies the Wolfe line search (4). Then, .
Proof. In view of (4) and (33), we haveTherefore, we getCombining (4) and (20), we can get thatLet us sum up the inequalities (36) for . We obtainThis inequality along with the assumption (H3.1) results inTherefore, Lemma 3 holds.
Theorem 1. Suppose that Assumptions (H 3.1) and (H 3.2) hold. Consider iterate by Algorithm 1; if the direction is a descent direction and the step length satisfies the Wolfe line search (4), then .
Proof. We prove this theorem by contradiction. If , in view of , there exists a constant such that . Again, from (3), it follows that . Squaring both sides of above equality, we haveDividing both sides of (39) by and in view of (20), it follows thatCombining with (40), by a recurrence of relation and , we haveTherefore, we haveHence, by recurrence formula (42), we obtainThis is a contradiction to Lemma 3. The proof is completed.
4. Numerical Results
In this part of the paper, we report some numerical experiments that indicate the efficiency of the proposed algorithm. To test and compare the computation effect of the proposed FW, a large number of testing problems from Morè et al. [21] are solved by the FW, MHS [16], MDY [18], and JHS [19]. In all conjugate gradient methods, the step length is yielded by the Wolfe line search (4). All codes were written in Matlab 7.5 and run on Dell with 2.90 GHz CPU processor, 8 GB RAM memory, and Windows 10 operating system. The parameters are set as follows: , , and . We stop the iteration if one of the following conditions is satisfied: (1) ; (2) the number of iteration Itr . If condition (2) occurs, the method is deemed to fail for solving the corresponding test problem and denote it by F. The simulation results of the proposed method were efficient and robust as compared with hybrid conjugate gradient methods (FW, MHS, MDY, and JHS). The hybrid conjugate gradient methods are listed in Table 1. Here denotes the abbreviation of the test problems, n denotes the dimension of the test problems, and Itr, NF, and NG denote the number of iteration, function evaluations, and gradient evaluations, respectively. TCpu denotes the computing time of CPU for computing the corresponding test problem (unit: seconds). To visualize the whole behaviour of the algorithms, we use the performance profiles proposed by Dolan and Morè [22] to compare the performance based on the CPU time, the number of function evaluation, the number of gradient evaluation, and the number of iteration, respectively. For each method, we plot the fraction of the problems for which the method is within a factor of the best time. The left side of the figure gives the percentage of the test problems for which a method is the fastest. Based on the theory of the performance profile above, four performance figures, i.e., Figures 1–4, can be generated according to Table 1. From the four figures, we can see that our methods perform effectively on the testing problems.
5. Conclusion
In this paper, we proposed a new hybrid conjugate gradient method for solving unconstrained optimization problems. The proposed method satisfied sufficient descent condition irrespective of the line searches condition. Moreover, the global convergence of the proposed method has been established under the Wolfe line search (4). Numerical experiments show the efficiency and robustness of the new algorithm in solving a collection of unconstrained optimization problems from [21].
Data Availability
Data will be available upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This study was supported in part by the National Natural Science Foundation of China (nos. 61967004 and 11601007) and Anhui Provincial Natural Science Foundation (no. 2008085MA01).