Modified Three-Term Liu–Storey Conjugate Gradient Method for Solving Unconstrained Optimization Problems and Image Restoration Problems

Wu, Yulun; Zhang, Mengxiang; Li, Yan

doi:https://doi.org/10.1155/2020/7859286

Mathematical Problems in Engineering

On this page

Abstract Introduction Conclusions Data Availability Conflicts of Interest Acknowledgments References Copyright Related Articles

Special Issue

Machine Learning and its Applications in Image Restoration

View this Special Issue

Research Article | Open Access

Volume 2020 | Article ID 7859286 | https://doi.org/10.1155/2020/7859286

Modified Three-Term Liu–Storey Conjugate Gradient Method for Solving Unconstrained Optimization Problems and Image Restoration Problems

Yulun Wu,¹Mengxiang Zhang,¹and Yan Li²

Academic Editor: Wenjie Liu

Received26 Jul 2020

Accepted28 Sept 2020

Published19 Oct 2020

Abstract

A new three-term conjugate gradient method is proposed in this article. The new method was able to solve unconstrained optimization problems, image restoration problems, and compressed sensing problems. The method is the convex combination of the steepest descent method and the classical LS method. Without any linear search, the new method has sufficient descent property and trust region property. Unlike previous methods, the information for the function is assigned to . Next, we make some reasonable assumptions and establish the global convergence of this method under the condition of using the modified Armijo line search. The results of subsequent numerical experiments prove that the new algorithm is more competitive than other algorithms and has a good application prospect.

1. Introduction

Consider the following unconstrained optimization:

According to the research of other scholars, there are abounding effective forms to solve unconstrained optimization problems. For example, there are the steepest descent method, Newton method, and conjugate gradient method [1–8]. The nonlinear conjugate gradient method is a very effective method for solving large-scale unconstrained optimization problems. The conjugate gradient method has attracted more and more attention [9–14] because it is easy to calculate and has low memory requirement and application in many fields [15–18]. The conjugate gradient iteration formula of (1) is defined aswhere is the step size of the th iteration obtained by a certain line search rule and is search direction of step , which are two important factors for solving unconstrained optimization problems. Among them, is defined aswhere the parameter called the CG parameter is a scalar and . The method of calculating the CG parameters will affect the performance and stability of the whole algorithm, so CG parameters play an utterly significant part. Among the nonlinear conjugate gradient method, we set and let represent the Euclidean norm. Then, the classical methods include the following: HS method [19] (the parameter ) FR method [20] (the parameter ) PRP method [21, 22] (the parameter ) LS method [23] (the parameter ) CD method [24] (the parameter ) DY method [25] the parameter ()

This paper mainly studies the Liu–Storey method. At the earliest, Liu and Storey proposed a conjugate gradient method to solve the optimization problem in [9]. In this paper, they demonstrate the convergence of the method when using Wolfe line search. The experimental results show that the algorithm is feasible. Later, it was called the LS method. The LS method is the same as the PRP method with exact linear search. Therefore, the LS method and the PRP method have similar forms. As is known to all, the HS method and the PRP method are supposed be the most effective two methods in practical calculation, and a lot of results have been obtained. Consequently, we hope that the theories and analysis techniques to the PRP method can also be applied to the LS method. However, the LS method does not have the nature of automatic descent. In order to surmount the disadvantage, Li [26] proposed an improved Liu–Storey method using Crippo-Lucide search technology, where the direction of search is computed aswhere , and

Nevertheless, this method assumes a minimum step size, and the main reason is the lack of direction of the trust zone. Therefore, a number of scholars have considered the combination of conjugate gradient method and trust region property. The three-term conjugate gradient algorithm is easy to converge because it automatically has sufficient descent. Therefore, this kind of method has been concerned by many scholars. The conjugate gradient method with three terms was first proposed by Zhang et al. [27]; they defined as

Recently, a three-term conjugate gradient algorithm was proposed in Yuan’s paper [28]. The search direction of this algorithm is a based on LS method and the gradient descent method. On the basis of Yuan’s research, a modified three-term conjugate gradient method is proposed, which has more functional information than the original method, where is computed bywhere , , constant , and

The vector not only has gradient information but also has functional information, with good theoretical results and numerical performance (see [29]). One might think that the resulting method is indeed better than the original one; that is why we use instead of .

According to the meaning of and , the following inequality is obtained:

Therefore,

In [28], Yuan et al. used modified Armijo linear search, which selects the largest as the step size in set such thatwhere is the trial step length, which is often set to 1, and are given constants. As is known to all, Armijo line search technology is a basic and the cheapest method. It is used in various algorithms [30–32]. Basically, a number of other methods of line search can be regarded as a modification of Armijo line search. In an unpublished article by Yuan, a new modification of Armijo linear search was designed based on [16, 33], which is defined aswhere , , and satisfying (12). The modification of Armijo linear search is verified to be effective in improving the efficiency of the algorithm.

The following is an introduction to the paper: In the second section, we propose an improvement algorithm and establish two important properties (sufficient descent property and trust region property) of the algorithm without using linear search. Then, some other properties of the algorithm are given, and the global convergence of the algorithm is proved under appropriate assumptions. The third section gives numerical experiments on normal unconstrained optimization problems, image restoration problems, and compressive sensing problems. In the last section, the conclusion of this paper is given.

2. Conjugate Gradient Algorithm and Convergence Analysis

This section will give a new modified Liu–Storey method combining (7) and (12), as shown below:

Lemma 1. is generated by formula (7); then, it clears that

Proof. When , it is obvious that and .
When , we haveUsing (10), we have , and then, (13) holds. Also,Then, we haveSetting , we get the right half of inequality (14). By Cauchy–Schwartz inequality and (13), we obviously get the left half of inequality (14). So, we get the results we want to prove.
Then, we will focus on the global convergence of the algorithm. In order to achieve this goal, we have to make the following assumptions.

Assumption 1. (a)The level set is bounded(b)Function: is continuously differentiable and the gradient is Lipschitz continuous, and there exists a constant (Lipschitz constant) such that

Theorem 1. Assumption 1 is kept true, and then, there is a positive constant , which satisfies (12) in Algorithm 1.

Input: input parameters , , and .
Output: .
(1)	For an initial solution compute ; set ;
(2)	whiledo
(3)	Find satisfying (12).
(4)	Set .
(5)	Compute by (7).
(6)	.
(7)	End

Proof. We construct such a functionFrom Assumption 1, and are bounded, note that , and from (13), we have . Obviously, holds. Now, we discuss the following two scenarios: Case 1: if , then For all positive that are small enough, we can obtain the following inequalities: Case 2: if , then Same as Case 1, we haveSo, from (20) and (21), there exists, at least, a positive constant such that , which also implies that there must be, at least, one local minimum point such thatthat is,Thus, satisfies (12). The proof is completed. So, quite evidently, Algorithm 1 is well defined.

Theorem 2. Let Assumption 1 be satisfied and the iterative sequence be generated by Algorithm 1. After that, we obtain

Proof. We will use the contradiction to draw this conclusion. When (24) is not correct, there exists a constant such thatFrom the third step of the Algorithm (12), we havenamely, and summing up the two sides of the abovementioned inequalities and from Assumption 1 (ii), it is obtained thatThus, from (28), we haveNow, our certificates are divided into the following two cases: Case 1: the step size as . From (29), it is obvious that as . This contradicts (24). Case 2: the step size as .From Algorithm 1, for the obtained step size , it is clear that does not satisfy (12), and by (13) and (14),where ; then, according to the mean value theorem, there exists an such thatCombined with (30)–(32), we haveSo, we get the contradiction. Thus, result (24) is true, and this completes the proof.

3. Numerical Experiments

In this section, we have designed three experiments to study the computational efficiency and performance of the proposed algorithm. The first subsection is normal unconstrained optimization problems, the second subsection is the image restoration problem, and the third subsection is compressive sensing problems. All the programs are compiled with Matlab R2017a and implemented on an Intel(R) Core (TM) i7-4710MQ CPU @ 2.50 GHz, RAM 8.0 GB, and the Windows 10 operating system.

3.1. Normal Unconstrained Optimization Problems

In order to test the numerical performance of the TTMLS algorithm, the NLS algorithm [28], LS method with the normal WWP line search (LS-WWP), and PRP method with the normal WWP line search (PRP-WWP) are also experimented as the comparison group. The results can be seen in Tables 1–4. The data used in the experiment are as follows: Dimension: we choose 3000 dimensions, 6000 dimensions, and 9000 dimensions to test. Parameters: all the algorithms run with , , , , and . Stop rule (the Himmelblau stop rule [34]): if , let ; otherwise, let . If conditions or are satisfied or the iteration number of more than 1000 is satisfied, we stop the process, where . Symbol representation: NI: the iteration number. NFG: the total number of function and gradient evaluations. CPU: the CPU time in seconds. Text problems: we have tested 74 unconstrained optimization problems, and the list of problems can be seen in Yuan’s work [16].

Dolan and Moré [35] provided a way to analyze the efficiency of these algorithms. From Figures 1–3, we can see that the 3performance of the TTMLS algorithm, TTMLS algorithm, and NLS algorithm is significantly better than that of the LS-WWP method and PRP-WWP method. Figures 1 and 2 show that the TTMLS algorithm and NLS algorithm can better approximate to the target function than the LS-WWP algorithm and PRP-WWP algorithm; thus, the number of iterations and the total number of function and gradient evaluation are smaller. The reason is that the search direction of the TTMLS algorithm contains more function information. Also, the CPU time in Figure 4, TTMLS algorithm is basically the same as the NLS algorithm, which is better than the other two. To sum up, the proposed algorithm has significant advantages.

3.2. Image Restoration Problems

The image restoration problem is a difficult problem in the field of optimization. We will use the TTMLS algorithm and NLS algorithm to minimize to recover the original image from an image corrupted by impulse noise. Afterwards, we compare the performance of the two algorithms. The data used in the experiment are as follows: Parameters: all the algorithms run with , , , and . Stop rule: if or is satisfied, we stop the process. Symbol representation: CPU: the CPU time in seconds. Total: the total CPU time of the four pictures. The information of noise: , , and salt-and-pepper noise. Text problems: we restore the original image from the image destroyed by impulsive noise. The experiments chose Lena , Barbara , Man , and Baboon as the test images.

From Figures 4–6, we can Table 5 see that both algorithms can recover 30%, 45%, and 60% salt-and-pepper noise images very well. The data show that, for image restoration problems, the TTMLS algorithm has shorter CPU time than the NLS algorithm when the salt-and-pepper noise is 30%, 45%, and 60%. In conclusion, the TTMLS algorithm is promising and competitive.

3.3. Compressive Sensing Problems

The main work of this section is to accurately recover the image from a few of random projections by compressive sensing. The experimental method derives from the model proposed by Dai and Sha [36]. Then, the performance of the TTMLS algorithm and LS method with line search (12) is compared.

It is noted that the gradients and are square matrices in this experiment, and the matrix obtained by may appear as a singular matrix, which results in the invalidation of the algorithm. But, when we calculate , we only need the value of without knowing the information of this square matrix, so in this experiment, we set in . The data used in the experiment are as follows: Parameters: all the algorithms run with , , , and . Stop rule: if or the number of iterations exceeds 500 is satisfied, we stop the process. Symbol representation: PSNR: Peak Signal-to-Noise Ratio. It is an objective criterion for image evaluation. Text problems: compressive sensing problems. The experiments chose Camera man , Fruits , Lena , and Baboon as the test images.

From Figure 7 and Table 6, we can see that both algorithms are effective in compression sensing problems. Meanwhile, from the experimental data, we can see that the TTMLS algorithm has more advantages than the LS algorithm.

4. Conclusions

In this paper, based on the well-known LS method and combined with improved Armijo linear search, this paper presents a three-term conjugate gradient algorithm. Without any linear search, the search direction of the new three-term conjugate algorithm is proved to have two good properties: sufficient descent and trust region properties. Also, the global convergence of the algorithm is established. The numerical results indicate that the new algorithm is effective. The good performance of the algorithm in image restoration problems and compressive sensing problems also proves that the algorithm is competitive.

Data Availability

Data used in this study can be obtained from the corresponding author on reasonable request.

Conflicts of Interest

There are no potential conflicts of interest.

Acknowledgments

The authors want to thank the support of the funds. This work was supported by the High Level Innovation Teams and Excellent Scholars Program in Guangxi Institutions of Higher Education (Grant no. [2019] 32), the National Natural Science Foundation of China (Grant no. 11661009), the Guangxi Natural Science Foundation (No. 2020GXNSFAA159069), and the Guangxi Natural Science Key Foundation (No. 2017GXNSFDA198046).

References

Y.-H. Dai, “New properties of a nonlinear conjugate gradient method,” Numerische Mathematik, vol. 89, no. 1, pp. 83–98, 2001.
View at: Publisher Site | Google Scholar
L. Grippo and S. Lucidi, “A globally convergent version of the Polak-Ribière conjugate gradient method,” Mathematical Programming, vol. 78, no. 3, pp. 375–391, 1997.
View at: Publisher Site | Google Scholar
J. Nocedal and S. Wright, “Numerical optimization,” in Springer Series in Operations Research, Springer, Berlin, Germany, 2nd edition, 2006.
View at: Google Scholar
Z. Shi, “Restricted PR conjugate gradient method and its global convergence,” Advances in Mathematics, vol. 31, pp. 47–55, 2002.
View at: Google Scholar
Z. Wei, S. Yao, and L. Liu, “The convergence properties of some new conjugate gradient methods,” Applied Mathematics and Computation, vol. 183, no. 2, pp. 1341–1350, 2006.
View at: Publisher Site | Google Scholar
G. Yuan, X. Wang, and Z. Sheng, “Family weak conjugate gradient algorithms and their convergence analysis for nonconvex functions,” Numerical Algorithms, vol. 84, no. 3, pp. 935–956, 2020.
View at: Publisher Site | Google Scholar
W. Zhou, “A short note on the global convergence of the unmodified PRP method,” Optimization Letters, vol. 7, no. 6, pp. 1367–1372, 2013.
View at: Publisher Site | Google Scholar
D. Li and M. Fukushima, “On the global convergence of the BFGS method for nonconvex unconstrained optimization problems,” SIAM Journal on Optimization, vol. 11, pp. 1054–1064, 1999.
View at: Publisher Site | Google Scholar
N. Andrei, “An unconstrained optimization test functions collection,” Advanced Modeling Optimization, vol. 10, pp. 147–161, 2008.
View at: Google Scholar
J. C. Gilbert and J. Nocedal, “Global convergence properties of conjugate gradient methods for optimization,” Siam Journal on Optimization, vol. 2, no. 1, pp. 21–42, 1992.
View at: Publisher Site | Google Scholar
X. Li and Q. Ruan, “A modified PRP conjugate gradient algorithm with trust region for optimization problems,” Numerical Functional Analysis and Optimization, vol. 32, no. 5, pp. 496–506, 2011.
View at: Publisher Site | Google Scholar
Z. Wei, G. Li, and Q. Li, “Global convergence of the Polak-Ribière-Polyak conjugate gradient method with an Armijo-type inexact line search for nonconvex unconstrained optimization problems,” Mathematics of Computation, 2008.
View at: Google Scholar
G. Yu, L. Guan, and Z. Wei, “Globally convergent Polak-Ribière-Polyak conjugate gradient methods under a modified Wolfe line search,” Applied Mathematics and Computation, vol. 215, no. 8, pp. 3082–3090, 2009.
View at: Publisher Site | Google Scholar
L. Zhang, W. Zhou, and D. Li, “Global convergence of a modified Fletcher-Reeves conjugate gradient method with Armijo-type line search,” Numerische Mathematik, vol. 104, no. 4, pp. 561–572, 2006.
View at: Publisher Site | Google Scholar
F. Wen and X. Yang, “Skewness of return distribution and coefficient of risk premium,” Journal of Systems Science and Complexity, vol. 22, no. 3, pp. 360–371, 2009.
View at: Publisher Site | Google Scholar
G. Yuan, J. Lu, and Z. Wang, “The PRP conjugate gradient algorithm with a modified WWP line search and its application in the image restoration problems,” Applied Numerical Mathematics, vol. 152, pp. 1–11, 2020.
View at: Publisher Site | Google Scholar
G. Yuan, Z. Wei, and Y. Yang, “The global convergence of the Polak-Ribiere-Polyak conjugate gradient algorithm under inexact line search for nonconvex functions,” Journal of Computational and Applied Mathematics, vol. 362, pp. 262–275, 2019.
View at: Publisher Site | Google Scholar
W. Zhou and F. Wang, “A PRP-based residual method for large-scale monotone nonlinear equations,” Applied Mathematics and Computation, vol. 261, pp. 1–7, 2015.
View at: Publisher Site | Google Scholar
M. R. Hestenes and E. Stiefel, “Methods of conjugate gradients for solving linear systems,” Journal of Research of the National Bureau of Standards, vol. 49, no. 6, pp. 409–436, 1952.
View at: Publisher Site | Google Scholar
R. Fletcher and C. Reeves, “Function minimization by conjugate gradients,” The Computer Journal, vol. 7, no. 2, pp. 149–154, 1964.
View at: Publisher Site | Google Scholar
E. Polak and G. Ribière, “Note sur la convergence de méthodes de directions conjuguées,” Revue Française d’informatique et de Recherche Opérationnelle. Série Rouge, vol. 3, no. 16, pp. 35–43, 1969.
View at: Publisher Site | Google Scholar
B. T. Polyak, “The conjugate gradient method in extremal problems,” USSR Computational Mathematics and Mathematical Physics, vol. 9, no. 4, pp. 94–112, 1969.
View at: Publisher Site | Google Scholar
Y. Liu and C. Storey, “Efficient generalized conjugate gradient algorithms, part 1: theory,” Journal of Optimization Theory and Applications, vol. 69, no. 1, pp. 129–137, 1991.
View at: Publisher Site | Google Scholar
R. Fletcher, Practical Method of Optimization Vol. I: Unconstrained Optimization, John Wiley & Sons, New York, NY, USA, 1987.
Y. H. Dai and Y. Yuan, “A nonlinear conjugate gradient method with a strong global convergence property,” SIAM Journal on Optimization, vol. 10, no. 1, pp. 177–182, 1999.
View at: Publisher Site | Google Scholar
Z. Li, “A new Liu-Storey type nonlinear conjugate gradient method for unconstrained optimization problems,” Journal of Computational and Applied Mathematics, vol. 225, no. 1, pp. 146–157, 2009.
View at: Publisher Site | Google Scholar
L. Zhang, W. Zhou, and D.-H. Li, “A descent modified Polak-Ribière-Polyak conjugate gradient method and its global convergence,” IMA Journal of Numerical Analysis, vol. 26, no. 4, pp. 629–640, 2006.
View at: Publisher Site | Google Scholar
G. Yuan, T. Li, and W. Hu, “A conjugate gradient algorithm for large-scale nonlinear equations and image restoration problems,” Applied Numerical Mathematics, vol. 147, pp. 129–141, 2020.
View at: Publisher Site | Google Scholar
G. Yuan and Z. Wei, “Convergence analysis of a modified BFGS method on convex minimizations,” Computational Optimization and Applications, vol. 47, no. 2, pp. 237–255, 2010.
View at: Publisher Site | Google Scholar
Z. Dai and F. Wen, “Global convergence of a modified hestenes-stiefel nonlinear conjugate gradient method with armijo line search,” Numerical Algorithms, vol. 59, no. 1, pp. 79–93, 2012.
View at: Publisher Site | Google Scholar
M. Li and A. Qu, “Some sufficient descent conjugate gradient methods and their global convergence,” Computational and Applied Mathematics, vol. 33, pp. 333–347, 2004.
View at: Publisher Site | Google Scholar
L. Zhang, W. Zhou, and D. Li, “Global convergence of the dy conjugate gradient method with armijo line search for unconstrained optimization problems,” Optimization Methods and Software, vol. 22, no. 3, pp. 511–517, 2007.
View at: Publisher Site | Google Scholar
G. Yuan, Z. Wei, and X. Lu, “Global convergence of BFGS and PRP methods under a modified weak Wolfe-Powell line search,” Applied Mathematical Modelling, vol. 47, pp. 811–825, 2017.
View at: Publisher Site | Google Scholar
Y. Yuan and W. Sun, Theory and Methods of Optimization, Science Press of China, Beijing, China, 1999.
E. Dolan and J. Moré, “Benchmarking optimization software with performance profiles,” Mathematical Programming, vol. 91, no. 2, pp. 201–213, 2001.
View at: Publisher Site | Google Scholar
Q. Dai and W. Sha, “The physics of compressive sensing and the gradient-based recovery algorithms,” Mathematics, 2009.
View at: Google Scholar

Copyright

Copyright © 2020 Yulun Wu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

562

Downloads

644

Citations