Journal of Optimization

Journal of Optimization / 2018 / Article

Research Article | Open Access

Volume 2018 |Article ID 5057096 | https://doi.org/10.1155/2018/5057096

Bakhtawar Baluch, Zabidin Salleh, Ahmad Alhawarat, "A New Modified Three-Term Hestenes–Stiefel Conjugate Gradient Method with Sufficient Descent Property and Its Global Convergence", Journal of Optimization, vol. 2018, Article ID 5057096, 13 pages, 2018. https://doi.org/10.1155/2018/5057096

A New Modified Three-Term Hestenes–Stiefel Conjugate Gradient Method with Sufficient Descent Property and Its Global Convergence

Academic Editor: Wlodzimierz Ogryczak
Received14 May 2018
Revised01 Aug 2018
Accepted19 Aug 2018
Published27 Sep 2018

Abstract

This paper describes a modified three-term Hestenes–Stiefel (HS) method. The original HS method is the earliest conjugate gradient method. Although the HS method achieves global convergence using an exact line search, this is not guaranteed in the case of an inexact line search. In addition, the HS method does not usually satisfy the descent property. Our modified three-term conjugate gradient method possesses a sufficient descent property regardless of the type of line search and guarantees global convergence using the inexact Wolfe–Powell line search. The numerical efficiency of the modified three-term HS method is checked using 75 standard test functions. It is known that three-term conjugate gradient methods are numerically more efficient than two-term conjugate gradient methods. Importantly, this paper quantifies how much better the three-term performance is compared with two-term methods. Thus, in the numerical results, we compare our new modification with an efficient two-term conjugate gradient method. We also compare our modification with a state-of-the-art three-term HS method. Finally, we conclude that our proposed modification is globally convergent and numerically efficient.

1. Introduction

In the field of optimization conjugate gradient methods are a well-known approach for solving large-scale unconstrained optimization problems. The conjugate gradient (CG) methods are simple and have relatively modest storage requirements. This class of methods has a vast number of applications in different areas, especially in the field of engineering [13].

Consider the unconstrained optimization problem:where is continuously differentiable and its gradient is . Normally CG methods generate a sequence defined byIn (2), is a general line search and is a search direction given by where is a parameter of the CG method. The six pioneering forms of are defined in [410].

Line searches may be exact or inexact. Exact line searches are time consuming, computationally expensive, and difficult and require large amounts of storage [1113]. Thus, inexact line search techniques are often adopted because of their efficiency and global convergence properties. Well-known inexact line search methods include the Wolfe and strong Wolfe techniques, which can be written as where , andRecently, Alhawarat and Salleh [14], Salleh and Alhawarat [15], and Alhawarat et al. [16, 17] proposed efficient CG and hybrid CG methods that fulfill the required global convergence properties. To improve the existing methods, a three-term CG technique has been introduced. Several different researchers have suggested various modifications to the three-term CG method. For instance, Beale [18] and Nazareth [19] proposed CG methods based on three terms that possess the finite termination property, but these do not perform well in practice [20, 21]. Furthermore, reports by McGuire and Wolfe [22], Deng and Li [23], Zhang et al. [24, 25], Cheng [26], Al-Bayati and Sharif [27], Zhang Xiao and Wei [28], Andrei [2931], Sugiki et al. [32], Narushima et al. [33], Babaie-Kafaki and Ghanbari [34], Al-Baali et al. [35], Sun and Liu [36], and Baluch et al. [37] discuss the global convergence and numerical results of modified three-term CG methods.

In this paper, a modified three-term Hestenes–Stiefel (HS) method is proposed. The general formula of the HS method [4] is This is known to be the first of all the CG parameters. This method ensures the global convergence of the exact line search. A nice property of the HS method is that it satisfies the conjugacy condition, regardless of whether the line search is exact or inexact [38]. However, this method does not satisfy the global convergence property when used with an inexact line search.

In this paper, the method of Zhang et al. [25] is modified with the help of another efficient CG parameter proposed by Wei et al. [39]. An attractive feature of the new three-term HS method is that it satisfies the sufficient descent condition regardless of the line search used. Furthermore, our modification is globally convergent for both convex and nonconvex functions when using an inexact line search. Numerical experiments show that the new modification is more efficient and robust than the MTTHS algorithm proposed by Zhang et al. [25]. The second aspect of this paper is to quantify the improvement of the three-term CG method over two-term approaches. To do this, we consider the efficient two-term CG method [40] given byThis DHS [40] method is one of the more efficient CG techniques, as it possesses the sufficient descent property and offers global convergence under Wolfe–Powell line search conditions. The numerical results given by this method are also convincing. Therefore, this two-term CG method is compared with our new modification to quantify the improvement offered by three-term CG methods.

The remainder of this paper is organized as follows. In Section 2, the motivation for and construction of the three-term HS CG method is discussed, and the general form is presented in Algorithm A. Section 3 is divided into two subsections, with Section 3.1 covering the sufficient descent condition and the global convergence properties for convex and nonconvex functions and Section 3.2 presenting detailed numerical results to evaluate the proposed method. Finally, Section 4 concludes this paper.

2. Motivation and Formulas

Zhang et al. [25] proposed the first three-term HS (TTHS) method. This can be written as and .

TTHS satisfies the descent property; if an exact line search is used, then it reduces to the original HS method. Further, to guarantee the global convergence properties of the search direction given by (8), a modified (MTTHS) algorithm was introduced with the search direction: where and .

As MTTHS was introduced to prove the global convergence properties of the search direction in (8), the question arises as to why (8) is not used to prove the global convergence properties. Instead of ignoring (8), it should be made efficient and globally convergent. Thus, there is room to modify (8) so as to satisfy the global convergence properties. It is expected that such a modification would outperform the MTTHS algorithm numerically.

Wei et al. [39] proposed an efficient CG parameter given byIn this parameter, the term plays an important role in satisfying the sufficient descent and global convergence properties. Thus, we take from the denominator of the above parameter and use it with (8) to construct a new modified three-term HS method. Hence,It is known that the HS method does not converge globally when the objective function is nonconvex. Further, Gilbert and Nocedal [41] showed that the parameter must be nonnegative to achieve convergence for nonconvex or nonlinear functions, i.e.,Applying the same technique to our parameter giveswhere If the line search is exact, then the parameters , , and reduce to the original parameters [4], [41], and TTHS [25]. The procedure of our proposed three-step CG method is described in Algorithm A.

Algorithm A.  

Step 0. Choose an initial point , and set ,

Step 1. For convergence, if (), then the algorithm terminates; otherwise, go to step 2.

Step 2. Compute

, and are given in (11) and (14).

Step 3. Determine the step size by the Wolfe line search (4).

Step 4. Compute the new point

Step 5. Set and go to step 1.

3. Results and Discussion

This section contains a theoretical discussion and numerical results. The first subsection considers the global convergence properties of our proposed method and the second presents the results from numerical computations.

3.1. Global Convergence Properties

Assumptions

(A1) The level set is bounded.

(A2) In some neighborhoodof , the gradient is Lipschitz continuous on an open convex set that contains , i.e., there exists a positive constant such thatAssumptions (A1) and (A2) imply that there exist positive constants and such thatWe now prove the sufficient descent condition independent of the line search and also . From (15), (11), and (14), we can writethat is,Hence, the sufficient descent condition holds regardless of the line search. Now, we prove thatAs we have , taking the modulus on both sides givesBy the Schwartz inequality, we havesoorHence, we haveThe HS method is well known for its conjugacy conditions, such asBy [15], CG methods that inherit (27) will be more efficient than other CG parameters that do not inherit this property. Dai and Liao [42] proposed the following conjugacy condition for an inexact line search: Using the exact line search , (28) reduces to the conjugacy condition in (27).

Lemma 1 (see [43]). Suppose there is an initial point for which Assumptions (A1) and (A2) hold. Now, consider the method in the form of (2), in which is a descent direction and satisfies the Wolfe line search condition (4). Then This is known as Zoutendijk’s condition and is used for proving the global convergence of a CG method. This condition together with (26) shows that

Definition 2. The function is called uniformly convex [36] on if there exists a positive constant such that We now show the global convergence of Algorithm A for uniformly convex functions.

Lemma 3. Let the sequences and be generated by Algorithm A and suppose that (31) holds. Then,where .

Proof. For details, see Lemma 2.1 of [44].

Theorem 4. Let the conditions in Assumptions (A1) and (A2) hold and the function be uniformly convex. Then,

Proof. AsThen, using the second Wolfe condition (4) and the sufficient descent condition,we haveFrom (11), (32), and (36) and Assumption (A2),Let us suppose that , where and so that . Thus, Now,Combining (38) and (39) with (15), we obtainNow, let so thatand we get . This implies thatHence, by (30), we have

We are now going to prove the global convergence of Algorithm A for nonconvex functions.

Lemma 5. Suppose that Assumptions (A1) and (A2) hold. Let the sequence be generated by Algorithm A. If there exists a constant such that for every , then where

Proof. As and , and also for all , then for all . Hence, is well defined. Ifthen , where and are unit vectors. Therefore,As ,Now, from Assumption (A2), (14), and (18),From (17), (18), and (48), there exists a constant such thatFrom (30) and (49), we obtainCombining this with (44) completes the proof.

Theorem 6. Let Assumptions (A1) and (A2) hold. Then, the sequence () generated by Algorithm A satisfies

Proof. Suppose that . Then, there exists a constant such that
The proof has two parts.
Part 1. See Theorem 2.2, step 1 in [36].
Part 2. From (15) and (49), we have In the beginning of the proof, we suppose that . Then, there exist a positive constant and some such that . Thus, which contradicts Assumption (A2), (30), and (52). Therefore,

3.2. Numerical Discussion

We now report the results of several numerical experiments. Zhang et al. [25] demonstrated the superior numerical efficiency of the MTTHS algorithm with respect to PRP+ [41], CG_DESCENT [45], and L-BFGS [46] using the Wolfe line search, while Dai and Wen [40] reported the numerical efficiency of the DHS method. Thus, we compare the efficient three-term HS method proposed in this paper (named the Bakhtawar–Zabidin–Ahmad method, BZA) with MTTHS [25] and DHS [40]. The BZA method was implemented using the Wolfe–Powell line search (4) with , , and

All codes were written in MATLAB 7.1 and run on an Intel Core i5 system with 8.0 GB RAM and a 2.60 GHz processor. Table 1 lists the numerical results given by BZA, MTTHS, and DHS for a number of test functions. In the Table 1, NI/CT/GE/FE represents number of iterations, CPU time, number of gradient evaluations and number of function evaluations.


ProblemnMTTHSDHSBZA
NI/CT/GE/FENI/CT/GE/FENI/CT/GE/FE

Extended21323/9.3448/4645/33211267/10.1753/4756/3488364/3.1079/1365/1000
Trigonometric -1504533/28.4686/15849/113153771/25.5829/14146/103742376/18.4341/8910/6533
function50011125/67.51/38848/277228163/55.0526/30616/224525904/46.5577/22140/16235

HIMMELBH(Cute)29/0.4680/77/679/0.4538/77/679/0.4471/77/67
509/0.4572/77/679/0.4574/77/679/0.4620/77/67
50009/2.0529/77/679/2.0198/77/679/1.9930/77/67

Power function23/0.3968/13/92/0.3881/9/62/0.3884/9/6
50212/2.0540/849/6363431/24.0407/13725/1029365/0.7751/261/195
10004612/33.5261/18449/13836F/F/F/F1565/11.567/6261/4695

DENSCHNF211/0.6773/64/5212/0.5399/67/548/0.4719/50/41
5014/0.5959/78/6314/0.5260/75/609/0.4739/54/44
500010/1.7879/56/4514/2.1633/75/609/1.6783/54/44

Sum Squares23/0.4203/13/92/0.3861/9/62/0.3966/9/6
function5047/0.7252/189/14139/0.6318/157/11739/0.6162/157/117
1000229/2.4542/917/6871333/11.3244/5333/3999191/1.9552/765/573

TRIDIA(Cute)23/0.4056/13/92/0.3980/9/62/0.4024/9/6
50100/1.6029/401/300193/1.7752/773/57956/0.7390/225/168
1000604/5.2318/2417/181216019/122.48/64077/48057349/3.22771397/1047

SINQUAD(Cute)225/0.4804/102/761734/12.6402/6076/434125/0.5218/101/75
5000F/F/F/FF/F/F/F411/198.6198/2837/2425
10000F/F/F/FF/F/F/F417/493.6830/2650/2232

Generalized5023/0.5659/101/7725/0.5348/109/8321/0.5187/94/72
Quartic GQ210023/0.5588/103/7931/0.5938/133/10122/0.5233/96/73
function.500026/2.9409/116/8923/2.6517/103/7925/2.7548/110/84

Generalized5050/0.7767/248/19743/0.6750/192/14842/0.6934/198/155
Triagonal-250043/1.5284/206/16246/1.4306/209/16238/1.4029/182/143
function100045/2.1144/214/16847/2.1176/214/16639/1.8380/181/141

Extended28/0.4315/32/239/0.4256/37/276/0.4158/25/18
Trigonometric -25033/0.6514/148/11426/0.5848/124/9722/0.5586/106/83
function700085/63.5473/626/54071/44.8822/440/36833/25.6757/251/217

LIARWHD10020/0.5798/136/11516/0.5065/95/7814/0.4865/84/69
500050/8.8611/390/33953/8.7426/384/33024/4.4628/189/164
1000033/14.6362/245/21153/19.3591/331/27723/9.7171/165/141

Generalized26/0.4058/27/206/0.4127/27/205/0.4008/23/17
Quartic GQ1600011/1.8086/47/3510/1.6695/43/3210/1.7246/43/32
function1000011/3.4464/47/3510/3.0683/43/3210/3.0317/43/32

Extended QP2217/0.8111/98/8021/0.6076/125/10316/0.5702/111/94
Quadratic Penalty5000226/56.4248/2247/2020F/F/F/F82/25.7327/1018/935
function10000137/95.8189/1457/1319F/F/F/F82/67.1490/1024/941

Extended Maratos220/0.6479/131/11043/0.8956/225/18130/0.8305/193/162
function100026/0.8290/176/149F/F/F/F30/0.8300/193/162
5000F/F/F/F45/5.8526/233/18732/6.0318/243/210

DENSCHNC26/0.4508/36/2910/0.4624/51/406/0.4486/34/27
507/0.4499/38/3010/0.4590/51/406/0.4696/34/27
6008/0.5625/44/3515/0.6240/130/1147/0.5445/38/30

DIXON3DQ (Cute)23/0.4835/13/92/0.3904/9/62/0.3878/9/6
20106/1.2286/424/317262/2.4917/1061/79890/1.0709/360/269
6002266/18.9637/9064/6797F/F/F/F2185/18.4756/8740/6554

EDENSCH function503094/15.3448/9288/61933094/16.4610/9288/61933089/15.3239/9273/6183
10003094/17.6514/9288/61933094/18.4918/9288/61933089/17.6263/9273/6183
50003094/198.8550/9288/61933094/199.0048/9288/61933089/198.2240/9273/6183

Diagonal 2 function50749/6.8385/2626/1876495/3.0663/1488/992490/3.3271/1473/982
100016517/126.4914/59593/430758569/53.8256/25711/171418563/53.3166/25693/17129
10000F/F/F/F16484/283.6569/49456/3297116478/279.44/49438/32959

DQDRTIC(Cute)5012/0.4358/49/365/0.4070/21/155/0.3940/21/15
500014/1.6324/57/425/0.7966/21/155/0.8084/21/15
1000014/3.7862/57/425/1.6205/21/155/1.5979/21/15

Perturbed5043/0.6470/173/12938/0.5837/153/11438/0.6313/153/114
Quadratic function1000207/2.1710/829/6211397/10.8367/5589/4196187/1.9258/749/561
5000481/44.4081/1925/14435125/435.0973/20501/15375425/39.4768/1701/1275

Diagonal 4 function504/0.3919/17/122/0.4015/9/62/0.3963/9/6
10005/0.4493/21/152/0.4005/9/62/0.4039/9/6
50005/0.8522/21/152/0.5598/9/62/0.5698/9/6

Extended Beale5015/0.4685/74/5868/0.8247/281/21215/0.4890/72/56
function10016/0.4739/74/5768/0.8245/281/21215/0.4855/72/56
50016/0.5450/74/5768/0.8840/281/21215/0.5050/72/56

DENSCHNA509/0.4336/44/3424/0.5609/102/779/0.4462/44/34
100010/0.5912/48/3726/0.7584/110/839/0.5909/44/34
500010/1.6387/46/3526/3.0388/110/839/1.4694/44/34

STAIRCASE S12079/0.9390/316/236202/1.7976/818/61572/0.8546/290/217
50215/2.0539/860/6441117/8.6778/4503/3385183/1.6893/732/548
100426/3.4107/1705/12783896/29.4293/15737/11840372/3.0840/1490/1117

NONDQUAR5028674/216.605/114767/86092F/F/F/F14756/104.86/59098/44341
10032283/244.026/129157/96873F/F/F/F17900/127.16/71670/53769
500F/F/F/FF/F/F/F32748/241.1/131043/98294

Extended Wood500128/1.5299/605/476942/8.5789/4143/320096/1.4535/551/454
1000146/1.7443/676/529946/8.8946/4100/3153101/1.7151/606/504
10000213/57.8431/979/765956/249.5361/4202/324589/31.8805/545/455

Extended Penalty2033/0.6528/144/11037/0.7031/158/12022/0.5670/98/75
300F/F/F/FF/F/F/F64/1.2248/365/300
600F/F/F/FF/F/F/F28/0.7428/172/143

Sphere function21/0.4750/5/31/0.3777/5/31/0.3902/5/3
10001/0.3904/5/31/0.3877/5/31/0.3860/5/3
50001/0.4687/5/31/0.4576/5/31/0.4616/5/3

ARGLINB(Cute)21/0.3913/5/31/0.3903/5/31/0.3840/5/3
501/0.3868/5/31/0.3941/5/31/0.3832/5/3
5002/0.4279/9/62/0.4266/9/62/0.4246/9/6

Extended White228/0.7668/192/16369/1.2885/354/28421/0.6586/136/114
and Holst function50029/0.9337/213/183102/1.5134/488/38521/0.6586/136/114
500023/4.6845/168/144125/14.3950/577/45121/3.8322/136/114

Extended Hiebert502040/33.6457/16304/14263F/F/F/F838/21.7040/9977/9138
200F/F/F/FF/F/F/F901/23.007/10513/9611
1000F/F/F/FF/F/F/FF/F/F/F

Quadratic QF15044/0.6479/177/13238/0.6068/153/11438/0.6306/153/114
500147/1.4565/589/441574/4.7399/2297/1722131/1.3814/525/393
10000744/179.6695/2977/2232F/F/F/F606/146.3228/2425/1818

Quartic5011579/66.6569/34740/2316011579/66.7991/34740/2316011574/64.593/34725/23150
50024975/148.8391/74928/4995224975/146.0726/74928/4995224968/142.57/74907/49938
100031473/201.6973/94422/6294831473/218.8475/94422/6294831466/191.54/94401/62934

Shallow function100010/0.5712/45/3448/0.7436/200/15110/0.5557/45/34
500011/1.4925/49/3748/4.9218/200/15110/1.4117/45/34
1000012/3.6303/53/4050/13.2094/209/15810/3.1569/45/34

VARDIM5028/1.1175/322/293143/4.0110/1604/146016/0.8685/221/204
10020/1.0429/286/265293/5.9711/2604/231017/0.9508/250/232
50041/2.2545/736/6941557/25.9389/11724/1016634/1.8745/645/610

DIXMAANA60008/1.8623/35/268/1.8913/35/268/1.8504/35/26
60158/1.9276/35/268/1.8546/35/268/1.9200/35/26
60308/1.9301/35/268/1.9700/35/268/1.8819/35/26

DIXMAANB97/0.4044/30/227/0.4141/30/227/0.4036/30/22
3007/0.4550/30/227/0.4776/30/228/0.4415/34/25
60008/1.8966/34/258/1.8364/34/259/2.0362/38/28

DIXMAANC96/0.4033/29/226/0.4191/29/226/0.4168/29/22
3007/0.4701/33/256/0.4721/29/227/0.4707/33/25
60007/2.0093/33/257/2.0148/33/258/2.1736/37/28

DIXMAAND907/0.4450/34/268/0.4333/38/298/0.4263/38/29
3008/0.5272/38/298/0.5406/38/298/0.5418/38/29
60009/2.4166/42/327/2.0165/34/267/2.0556/34/26

DIXMAANE919/0.5080/88/6823/0.5720/107/8318/0.4832/84/65
30084/1.5278/421/3361211/7.7407/3707/249584/2.1712/420/335
6000331/81.2267/1655/1323F/F/F/F333/83.0952/1685/1351

EG225/0.4912/25/198/0.4128/40/314/0.4139/20/15
20160/1.4413/642/4811958/14.8104/8106/6147110/1.0755/447/336
50F/F/F/F48/0.7452/228/179656/5.3312/2773/2116

EG32021/0.5445/88/6625/0.5611/102/7614/0.4851/60/45
5025/0.5965/110/8443/0.7618/207/16325/0.5236/69/53
100F/F/F/FF/F/F/F20/0.5290/88/67

Fletcher function5024/0.5879/126/10123/0.5870/123/9922/0.5687/118/95
600027/5.2874/161/13328/5.4497/168/13926/4.7950/152/125
1000027/10.4312/161/13327/10.6308/163/13526/9.8879/155/128

Extended508/0.4229/37/2811/0.4510/49/379/0.4370/41/31
Himmelblau10008/0.4706/39/3012/0. 5342/53/409/0.4775/41/31
function50009/1.4066/43/3313/1.6584/57/439/1.3377/41/31

Extended211/0.5889/75/6333/0.6806/163/12910/0.4929/69/58
Freudenstein50F/F/F/F36/0.7871/230/19310/0.4700/69/58
and Roth function20016/0.5442/102/8537/0.8501/241/20311/0.4993/73/61

Dixon and Price20132/1.6580/615/482333/3.2018/1409/1075131/1.5682/603/471
function50185/2.5157/1034/848478/4.7648/2199/1720165/2.2048/900/734
100523/6.5773/2938/24141243/11.1703/5672/4428439/5.4568/2422/1982

Raydan 1 function2106/1.3363/319/212105/0.8502/316/21099/0.8493/298/198
5059/0.8854/237/17788/1.0277/364/27558/0.7714/233/174
10070/0.9532/282/211155/1.5252/625/46970/0.8892/282/211

Raydan 2 function22/0.4129/9/62/0.3995/9/63/0.3971/13/9
502/0.3991/9/62/0.3981/9/63/0.4176/13/9
1002/0.4002/9/62/0.4123/9/63/0.3906/13/9

NONDIA(SHANO-5009/0.5483/63/5311/0.4937/70/587/0.4938/55/47
78)600014/2.7799/82/6746/6.5297/212/1659/2.3946/70/60
1000016/6.3412/96/7974/21.4446/339/26415/6.2113/92/76

Extended Block50030/1.0226/52000/4425032/0.9386/32250/2400039/1.1887/44000/34000
Diagonal BD1100039/3.6068/195000/17500033/1.3795/66500/4950039/1.6724/88500/68500
function1000038/22.3738/1070000/87500035/14.5725/705000/52500043/20.1088/965000/745000

SINCOS2000F/F/F/FF/F/F/F8/0.6960/39/30
5000F/F/F/FF/F/F/F8/1.2553/39/30
10000F/F/F/FF/F/F/F8/2.7535/39/30

DIXMAANH9058/0.8867/290/231388/2.5254/1230/84157/1.0443/286/228
30081/1.5051/408/3261264/7.8924/3872/260781/1.3140/408/326
600126/4.4107/630/5032407/40.8602/7294/4886119/4.0122/596/476

Quadratic QF25077/0.9476/330/252107/1.1580/454/34668/0.8775/293/224
function200183/1.7984/787/603337/2.9913/1419/1081139/1.5/610/470
20001150/34.4704/4862/37112903/85.3362/12110/9206540/17.4638/2404/1863

Tridiagonal double20108/1.1547/438/3291915/14.1311/7667/5751101/1.1136/410/308
Bordered50407/3.3266/1633/122511157/78.2325/44633/33475355/2.9421/1425/1069
5006278/48.1130/25119/1884013470/103.5476/53887/404164595/35.3993/18387/13791

Generalized21323/8.3265/4645/33211267/8.4042/4756/3488364/2.6867/1365/1000
Triagonal1 function5025/0.5255/103/7728/0.504/115/8624/0.5140/99/74
10024/0.5630/99/7427/0.6205/111/8323/0.5242/95/71

Extended QP2509/0.4269/49/3915/0.4503/71/559/0.4190/49/39
Quadratic penalty20013/0.4682/70/5614/0.6931/74/5912/0.4620/66/53
function3000F/F/F/FF/F/F/F17/1.5340/118/100

Extended25/0.4089/22/166/0.4254/26/195/0.4095/22/16
DENSCHNB5005/0.4222/22/167/0.4313/30/226/0.4165/26/19
function100005/1.7240/22/167/2.1577/30/226/1.9461/26/19

Extended three-27/0.4187/30/2213/0.4386/54/407/0.417/30/22
Exponential terms507/0.4211/750/55014/0.4492/1450/10757/0.4255/750/550
1007/0.4378/1500/110014/0.4705/2900/21507/0.4183/1500/1100

DIXMAANF923/0.6308/109/8523/0.5828/109/8519/0.5129/90/70
9051/0.8509/256/204386/2.4637/1229/84251/1.1421/256/204
30090/1.4444/451/3601260/7.8258/3871/261090/1.5578/451/360

DIXMAANG919/0.5318/95/7526/0.5670/127/10021/0.5403/103/81
9083/1.2240/424/340379/2.4949/1221/84156/0.8767/280/223
300137/1.8866/643/5051241/14.5812/3834/2592569/5.5495/2477/1907

Extended225/0.7100/159/13324/0.9013/120/9527/0.7803/163/135
Rosenbrock100014/0.8076/82/6726/0.6537/128/10128/0.7818/167/138
function500019/2.8693/109/8927/3.2102/132/10430/4.4233/175/144

ARWHEAD500F/F/F/FF/F/F/F10/0.4429/54/43
3000F/F/F/FF/F/F/F5/0.7593/38/32
8000F/F/F/FF/F/F/F7/2.5096/49/41

Hager function24/0.4001/17/126/0.4188/25/184/0.4008/17/12
5020/0.4789/82/6124/0.5101/98/7320/0.4738/82/61
10024/0.5275/98/73F/F/F/F24/0.5382/111/86

Extended Powell10004008/108.2/4032500/3030250F/F/F/F361/10.5/384000/293500
function30001026/86.1/3124500/2354250F/F/F/F108/10.8/379500/297750
50002213/330/11330000/8562500F/F/F/F306/48/1652500/1268750

BIGGSB1 function21/0.3861/5/31/0.3866/5/31/0.3895/5/3
2053/0.7214/213/15920/0.4869/80/5920/0.4764/80/59
50201/1.9444/804/602867/7.1960/3500/263250/0.7281/200/149

Extended Cliff10011657/87/2075150/149225039907/329/7892000/58966001674/12/293750/210000
5000F/F/F/FF/F/F/F1009/132/8857500/633250

ENGVAL827/0.4675/33/2510/0.4709/45/347/0.4756/36/28
2022/0.5657/99/7632/0.6306/138/10520/0.5382/90/69

Trecanni function26/0.4146/26/196/0.4212/26/194/0.3982/18/13

GENROSEN-2225/0.8096/159/13324/0.5864/120/9527/0.6770/163/135

Generalized Quartic function26/0.4068/27/206/0.4082/27/205/0.4081/23/17

Diagonal 1 function26/0.4015/25/187/0.4085/29/214/0.4207/18/13

Six Hump function27/0.4112/31/237/0.4271/31/236/0.4064/27/20

Three Hump function211/0.4441/52/4011/0.4335/50/3810/0.4397/47/36

Booth function23/0.3930/13/92/0.4043/9/62/0.3919/9/6

Zett1 function226/0.5067/105/7830/0.5581/126/9524/0.5046/102/77

According to Moré et al. [47], the efficiency of any method can be determined by its performance on a number of test functions. The number of test functions should not be too large or too small, with 75 considered ideal for testing the efficiency of any method. The test functions in Table 1 were taken from Andrei’s test function collection [48] with standard initial points and dimensions ranging from 2 to 10000.

If the solution had not converged after 500 seconds, the program was terminated. Generally, convergence was achieved within this time limit; functions for which the time limit was exceeded are denoted by “F” for Fail in Table 1.

The Sigma plotting software was used to graph the data. We adopt the performance profiles given by Dolan and Moré [49]. Thus, MTTHS, DHS, and BZA are compared in terms of NI/CT/GE/FE in Figures 14. For each method, we plotted the fraction of problems that were solved correctly within a factor of the best time. In the figures, the uppermost curve is the method that solves the most problems within a factor t of the best time. From Table 1 and Figures 14, the BZA method outperforms the MTTHS algorithm and DHS method in terms of NI, CT, GE, and FE.

The BZA method solves around 99.5% of the problems, and the performance of BZA is 85% better than that of DHS and 77% better than that of MTTHS. We can also conclude that, on average, three-term conjugate gradient methods are 85% better than two-term conjugate gradient methods (DHS).

4. Conclusion

We have proposed a modified three-term HS conjugate gradient method. An attractive property of the proposed method is that it produces a sufficient descent condition , regardless of the line search. The global convergence properties of the proposed method have been established under Wolfe line search conditions. Numerical results show that the proposed method is more efficient and robust than state-of-the-art three term (MTTHS) and two-term (DHS) CG methods.

Data Availability

No data were used to support this study.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

References

  1. E. Polak, Optimization: Algorithms and Consistent Approximations, Springer, New York, NY, USA, 1997. View at: MathSciNet
  2. J. Nocedal, “Conjugate gradient methods and nonlinear optimization,” in Linear and Nonlinear Conjugate Gradient Related Methods, L. Adams and J. L. Nazareth, Eds., pp. 9–23, SIAM, Philadelphia, PA, USA, 1995. View at: Google Scholar | MathSciNet
  3. R. Ziadi, R. Ellaia, and A. Bencherif-Madani, “Global optimization through a stochastic perturbation of the Polak-Ribière conjugate gradient method,” Journal of Computational and Applied Mathematics, vol. 317, pp. 672–684, 2017. View at: Publisher Site | Google Scholar
  4. M. R. Hestenes and E. Stiefel, “Methods of conjugate gradients for solving linear systems,” Journal of Research of the National Bureau of Standards, vol. 49, pp. 409–436, 1952. View at: Publisher Site | Google Scholar | MathSciNet
  5. R. Fletcher and C. M. Reeves, “Function minimization by conjugate gradients,” The Computer Journal, vol. 7, pp. 149–154, 1964. View at: Publisher Site | Google Scholar | MathSciNet
  6. E. Polak and G. Ribière, “Note sur la convergence de méthodes de directions conjuguées,” Revue Française d'Informatique et de Recherche Opérationnelle, vol. 3, no. 16, pp. 35–43, 1969. View at: Publisher Site | Google Scholar | MathSciNet
  7. B. T. Polyak, “The conjugate gradient method in extreme problems, USSR Comput,” USSR Computational Mathematics and Mathematical Physics, vol. 9, pp. 94–112, 1969. View at: Publisher Site | Google Scholar
  8. R. Fletcher, Practical Methods of Optimization, vol. I: Unconstrained Optimization, John Wiley & Sons, New York, NY, USA, 2nd edition, 1987. View at: MathSciNet
  9. Y. Liu and C. Storey, “Efficient generalized conjugate gradient algorithms, Part 1,” Journal of Optimization Theory and Applications, vol. 69, no. 1, pp. 129–137, 1991. View at: Publisher Site | Google Scholar | MathSciNet
  10. Y. H. Dai and Y. Yuan, “A nonlinear conjugate gradient method with a strong global convergence property,” SIAM Journal on Optimization, vol. 10, no. 1, pp. 177–182, 1999. View at: Publisher Site | Google Scholar | MathSciNet
  11. Z.-J. Shi and J. Guo, “A new family of conjugate gradient methods,” Journal of Computational and Applied Mathematics, vol. 224, no. 1, pp. 444–457, 2009. View at: Publisher Site | Google Scholar | MathSciNet
  12. G. Yuan, X. Lu, and Z. Wei, “A conjugate gradient method with descent direction for unconstrained optimization,” Journal of Computational and Applied Mathematics, vol. 233, no. 2, pp. 519–530, 2009. View at: Publisher Site | Google Scholar | MathSciNet
  13. Z.-J. Shi, S. Wang, and Z. Xu, “The convergence of conjugate gradient method with nonmonotone line search,” Applied Mathematics and Computation, vol. 217, no. 5, pp. 1921–1932, 2010. View at: Publisher Site | Google Scholar | MathSciNet
  14. A. Alhawarat and Z. Salleh, “Modification of nonlinear conjugate gradient method with weak Wolfe-Powell line search,” Abstract and Applied Analysis, Article ID 7238134, 6 pages, 2017. View at: Google Scholar
  15. Z. Salleh and A. Alhawarat, “An efficient modification of the Hestenes-Stiefel nonlinear conjugate gradient method with restart property,” Journal of Inequalities and Applications, vol. 2016, no. 1, Article ID 110, 2016. View at: Publisher Site | Google Scholar
  16. A. Alhawarat, Z. Salleh, M. Mamat, and M. Rivaie, “An efficient modified Polak-Ribière-Polyak conjugate gradient method with global convergence properties,” Optimization Methods and Software, vol. 32, no. 6, pp. 1299–1312, 2017. View at: Google Scholar
  17. A. Alhawarat, M. Mamat, M. Rivaie, and Z. Salleh, “An efficient hybrid conjugate gradient method with the strong wolfe-powell line search,” Mathematical Problems in Engineering, vol. 2015, Article ID 103517, 7 pages, 2015. View at: Publisher Site | Google Scholar | MathSciNet
  18. E. M. L. Beale, “A derivative of conjugate gradients,” in Numerical Methods for Nonlinear Optimization, F. A. Lootsma, Ed., pp. 39–43, Academic Press, London, UK, 1972. View at: Google Scholar | MathSciNet
  19. L. Nazareth, “A conjugate direction algorithm without line searches,” Journal of Optimization Theory and Applications, vol. 23, no. 3, pp. 373–387, 1977. View at: Publisher Site | Google Scholar | MathSciNet
  20. Y. H. Dai and Y. Yuan, Nonlinear Conjugate Gradient Methods, Shanghai Science and Technology Publisher, Shanghai, China, 2000.
  21. W. W. Hager and H. Zhang, “A survey of nonlinear conjugate gradient methods,” Pacific Journal of Optimization. An International Journal, vol. 2, no. 1, pp. 35–58, 2006. View at: Google Scholar | MathSciNet
  22. M. F. McGuire and P. Wolfe, “Evaluating a restart procedure for conjugate gradients,” Report RC-4382, IBM Research Center, Yorktown Heights, 1973. View at: Google Scholar
  23. N. Y. Deng and Z. Li, “Global convergence of three terms conjugate gradient methods,” Optimization Methods and Software, vol. 4, pp. 273–282, 1995. View at: Google Scholar
  24. L. Zhang, W. Zhou, and D. H. Li, “A descent modified Polak-Ribiere-Polyak conjugate gradient method and its global convergence,” IMA Journal of Numerical Analysis (IMAJNA), vol. 26, no. 4, pp. 629–640, 2006. View at: Publisher Site | Google Scholar | MathSciNet
  25. L. Zhang, W. Zhou, and D. Li, “Some descent three-term conjugate gradient methods and their global convergence,” Optimization Methods and Software, vol. 22, no. 4, pp. 697–711, 2007. View at: Publisher Site | Google Scholar | MathSciNet
  26. W. Cheng, “A two-term PRP-based descent method,” Numerical Functional Analysis and Optimization, vol. 28, no. 11, pp. 1217–1230, 2007. View at: Publisher Site | Google Scholar | MathSciNet
  27. A. Y. Al-Bayati and W. H. Sharif, “A new three-term conjugate gradient method for unconstrained optimization,” Canadian Journal on Science and Engineering Mathematics, vol. 1, no. 5, pp. 108–124, 2010. View at: Google Scholar
  28. J. Zhang, Y. Xiao, and Z. Wei, “Nonlinear conjugate gradient methods with sufficient descent condition for large-scale unconstrained optimization,” Mathematical Problems in Engineering, vol. 2009, Article ID 243290, 16 pages, 2009. View at: Publisher Site | Google Scholar | MathSciNet
  29. N. Andrei, “A modified Polak-Ribière-Polyak conjugate gradient algorithm for unconstrained optimization,” Optimization. A Journal of Mathematical Programming and Operations Research, vol. 60, no. 12, pp. 1457–1471, 2011. View at: Publisher Site | Google Scholar | MathSciNet
  30. N. Andrei, “On three-term conjugate gradient algorithms for unconstrained optimization,” Applied Mathematics and Computation, vol. 219, no. 11, pp. 6316–6327, 2013. View at: Publisher Site | Google Scholar | MathSciNet
  31. N. Andrei, “A simple three-term conjugate gradient algorithm for unconstrained optimization,” Journal of Computational and Applied Mathematics, vol. 241, pp. 19–29, 2013. View at: Publisher Site | Google Scholar | MathSciNet
  32. K. Sugiki, Y. Narushima, and H. Yabe, “Globally convergent three-term conjugate gradient methods that use secant conditions and generate descent search directions for unconstrained optimization,” Journal of Optimization Theory and Applications, vol. 153, no. 3, pp. 733–757, 2012. View at: Publisher Site | Google Scholar | MathSciNet
  33. Y. Narushima, H. Yabe, and J. A. Ford, “A three-term conjugate gradient method with sufficient descent property for unconstrained optimization,” SIAM Journal on Optimization, vol. 21, no. 1, pp. 212–230, 2011. View at: Publisher Site | Google Scholar | MathSciNet
  34. S. Babaie-Kafaki and R. Ghanbari, “Two modified three-term conjugate gradient methods with sufficient descent property,” Optimization Letters, vol. 8, no. 8, pp. 2285–2297, 2014. View at: Publisher Site | Google Scholar | MathSciNet
  35. M. Al-Baali, Y. Narushima, and H. Yabe, “A family of three-term conjugate gradient methods with sufficient descent property for unconstrained optimization,” Computational Optimization and Applications, vol. 60, no. 1, pp. 89–110, 2015. View at: Publisher Site | Google Scholar | MathSciNet
  36. M. Sun and J. Liu, “Three modified Polak-Ribière-Polyak conjugate gradient methods with sufficient descent property,” Journal of Inequalities and Applications, vol. 2015, no. 1, 2015. View at: Publisher Site | Google Scholar
  37. B. Baluch, Z. Salleh, A. Alhawarat, and U. A. M. Roslan, “A New Modified Three-Term Conjugate Gradient Method with Sufficient Descent Property and Its Global Convergence,” Journal of Mathematics, Article ID 2715854, 12 pages, 2017. View at: Publisher Site | Google Scholar | MathSciNet
  38. Z.-F. Dai, “Two modified HS type conjugate gradient methods for unconstrained optimization problems,” Nonlinear Analysis: Theory, Methods & Applications, vol. 74, no. 3, pp. 927–936, 2011. View at: Publisher Site | Google Scholar | MathSciNet
  39. Z. Wei, G. Li, and L. Qi, “New nonlinear conjugate gradient formulas for large-scale unconstrained optimization problems,” Applied Mathematics and Computation, vol. 179, no. 2, pp. 407–430, 2006. View at: Publisher Site | Google Scholar
  40. Z. Dai and F. Wen, “Another improved Wei–Yao–Liu nonlinear conjugate gradient method with sufficient descent property,” Applied Mathematics and Computation, vol. 218, no. 14, pp. 7421–7430, 2012. View at: Publisher Site | Google Scholar
  41. J. C. Gilbert and J. Nocedal, “Global convergence properties of conjugate gradient methods for optimization,” SIAM Journal on Optimization, vol. 2, no. 1, pp. 21–42, 1992. View at: Publisher Site | Google Scholar
  42. Y.-H. Dai and L.-Z. Liao, “New conjugacy conditions and related nonlinear conjugate gradient methods,” Applied Mathematics & Optimization, vol. 43, no. 1, pp. 87–101, 2001. View at: Publisher Site | Google Scholar | MathSciNet
  43. G. Zoutendijk, “Nonlinear programming, computational methods,” in Integer and Nonlinear Programming, J. Abadie, Ed., pp. 37–86, North-Holland, Amsterdam, The Netherlands, 1970. View at: Google Scholar | MathSciNet
  44. Z.-f. Dai and B.-S. Tian, “Global convergence of some modified PRP nonlinear conjugate gradient methods,” Optimization Letters, vol. 5, no. 4, pp. 615–630, 2011. View at: Publisher Site | Google Scholar | MathSciNet
  45. W. W. Hager and H. Zhang, “A new conjugate gradient method with guaranteed descent and an efficient line search,” SIAM Journal on Optimization, vol. 16, no. 1, pp. 170–192, 2005. View at: Publisher Site | Google Scholar | MathSciNet
  46. D. C. Liu and J. Nocedal, “On the limited memory BFGS method for large scale optimization,” Mathematical Programming, vol. 45, no. 1–3, pp. 503–528, 1989. View at: Publisher Site | Google Scholar | MathSciNet
  47. J. J. Moré, B. S. Garbow, and K. E. Hillstrom, “Testing unconstrained optimization software,” ACM Transactions on Mathematical Software, vol. 7, no. 1, pp. 17–41, 1981. View at: Publisher Site | Google Scholar | MathSciNet
  48. N. Andrei, “An unconstrained optimization test functions collection,” Advanced Modeling and Optimization, vol. 10, no. 1, pp. 147–161, 2008. View at: Google Scholar | MathSciNet
  49. E. D. Dolan and J. J. Moré, “Benchmarking optimization software with performance profiles,” Mathematical Programming, vol. 91, no. 2, pp. 201–213, 2002. View at: Publisher Site | Google Scholar | MathSciNet

Copyright © 2018 Bakhtawar Baluch et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


More related articles

818 Views | 470 Downloads | 1 Citation
 PDF Download Citation Citation
 Download other formatsMore
 Order printed copiesOrder

Related articles

We are committed to sharing findings related to COVID-19 as quickly as possible. We will be providing unlimited waivers of publication charges for accepted research articles as well as case reports and case series related to COVID-19. Review articles are excluded from this waiver policy. Sign up here as a reviewer to help fast-track new submissions.