Mathematical Problems in Engineering

Mathematical Problems in Engineering / 2020 / Article
Special Issue

Machine Learning and its Applications in Image Restoration

View this Special Issue

Research Article | Open Access

Volume 2020 |Article ID 4072651 | https://doi.org/10.1155/2020/4072651

Zhou Sheng, Dan Luo, "A Cauchy Point Direction Trust Region Algorithm for Nonlinear Equations", Mathematical Problems in Engineering, vol. 2020, Article ID 4072651, 9 pages, 2020. https://doi.org/10.1155/2020/4072651

A Cauchy Point Direction Trust Region Algorithm for Nonlinear Equations

Guest Editor: Maojun Zhang
Received13 Jul 2020
Accepted10 Aug 2020
Published24 Aug 2020

Abstract

In this paper, a Cauchy point direction trust region algorithm is presented to solve nonlinear equations. The search direction is an optimal convex combination of the trust region direction and the Cauchy point direction with the sufficiently descent property and the automatic trust region property. The global convergence of the proposed algorithm is proven under some conditions. The preliminary numerical results demonstrate that the proposed algorithm is promising and has better convergence behaviors than the other two existing algorithms for solving large-scale nonlinear equations.

1. Introduction

The system of nonlinear equations is one of the most important mathematical models, with extensive applications in chemical equilibrium problems [1], problem of hydraulic circuits of power plants [2], L-shaped beam structures [3], and image restoration problems [4]. Some -norm regularized optimization problems in compressive sensing [5] are also obtained by reformulating the systems of nonlinear equations. In this paper, consider the following nonlinear system of equation:where is continuously differentiable. Letwhere . It is clear that the nonlinear system of equations (1) is equivalent to the unconstrained optimization problem as follows:

Many literature studies have studied a variety of numerical algorithms to solve the nonlinear system of equations, such as the conjugate gradient algorithm [69], Levenberg–Marquardt algorithm [1013], and trust region algorithm [1421]. The trust region method plays an important role in the area of nonlinear optimization. It was proposed by Powell in [22] to solve the nonlinear optimization problem by using an iterative structure. Recently, some trust region-based methods have been developed to solve nonlinear equations and have shown very promising results. For instance, Fan and Pan [16] discussed a regularized trust region subproblem with an adaptive trust region radius, which possesses quadratic convergence under the local error bound condition. Yuan et al. [17] proposed a trust region-based algorithm, in which the Jacobian is updated by the BFGS formula, for solving nonlinear equations. The algorithm showed global convergence at a quadratic convergence rate. This algorithm does not compute the Jacobian matrix in each interrogation, which significantly reduces the computation burden. Esmaeili and Kimiaei [19] studied an adaptive trust region algorithm with the moderate radius size by employing the nonmonotone technique, and the global and quadratic local convergence was given. However, the maximal dimension of the test problems was only 300. Qi et al. [23] proposed an asymptotic Newton search direction method which is an adequate combination of the projected gradient direction and the projected trust region direction for solving box-constrained nonsmooth equations and proved the global and quadratic local convergence. Motivated by the search direction by Qi et al., Yuan et al. [24] used the idea of the search direction to solve nonsmooth unconstrained optimization problems and also showed global convergence.

Inspired by the fact that the search direction by Qi et al. [23] speeds up the local convergence rate compared to the gradient direction, we propose an efficient Cauchy point direction trust region algorithm to solve (1) in this paper. The search direction is an optimal convex combination of the trust region direction and the Cauchy point direction with the automatic trust region property and the sufficiently descent property, which can improve the numerical performance. We also show the global convergence of the proposed algorithm. Numerical results indicate that this algorithm has better behavior convergence compared with that of the classical trust region algorithm and the adaptive regularized trust region algorithm. However, the local convergence is not given.

The remainder of this paper is organized as follows. In Section 2, we introduce our motivation and a new search direction. In Section 3, we give a Cauchy point direction trust region algorithm for solving problem (1). The global convergence of the proposed algorithm is proven in Section 4. Preliminary numerical results are reported in Section 5. We conclude our paper in Section 6.

Throughout this paper, unless otherwise specified, denotes the Euclidean norm of vectors or matrices.

2. Motivation and Search Direction

In this section, motivated by the definition of the Cauchy point which is parallel to the steepest descent direction, we present a search direction with the Cauchy point. Let be the trust region radius and .

2.1. Cauchy Point Direction

Noting that is a semidefinite matrix, we define the Cauchy point at the current point :where, and is the Jacobian of or its approximation.

The following lemma gives a nice property of the Cauchy point direction .

Proposition 1 (automatic trust region property). Suppose that is defined by (4), then .

Proof. It is easy to get the result of this proposition from . This completes the proof.
From the definition of and the above proposition, we can determine the descent property and the automatic trust region property of the Cauchy point direction .

2.2. Trust Region Direction

Let the trial step be a solution of the following trust region subproblem:where is the same as the above definition.

From the famous result of Powell [22], we give the following lemma and omit its proof.

Lemma 1. Let be a solution of the trust region subproblem (6), then

As mentioned by Qi et al. [23] and Yuan et al. [24], when is far from the optimization solution, the trust region direction may not possess the descent property. However, it follows from the definition of the Cauchy point direction that always possesses the descent property. On the contrary, if the search direction is defined by using a combination of and , it will be a descent direction. Hence, we give a new search direction by using a convex combination form in the next section.

2.3. Search Direction

Let be a solution of the one-dimensional minimization problem as follows:and the search direction is defined as

For the above one-dimensional minimization problem, we can use the gold section method, Fibonacci method, parabolic method [25], etc. The idea of the search direction follows from Qi et al. [23]. However, we use the Cauchy point direction which possesses the automatic trust region property to get optimal by using the gold section method for problem (6) in this paper.

3. New Algorithm

In this section, let be the optimal solution of (3). Then, the actual reduction is defined asand the predicted reduction is

Thus, is defined by the ratio between and :

We now list the detailed steps of the Cauchy point direction trust region algorithm in Algorithm 1.

Algorithm 1. Cauchy point direction trust region algorithm for solving (3).Step 1. Given initial point and parameters , , , , and . Let .Step 2. If the termination condition is satisfied at the iteration point , then stop. Otherwise, go to Step 3.Step 3. Solve the trust region subproblem (6) to obtain and compute and from (4) and (5), respectively.Step 4. To solve the one-dimensional minimization problem (8) giving by using Algorithm 2, calculate the search direction from (6).Step 5. Compute , ifare satisfied; let , and update byStep 6. If , return to Step 3, and let . Otherwise, return to Step 2. Let .In Step 4 of Algorithm 1, we use the gold section algorithm as given in Algorithm 2.

Algorithm 2. Gold section algorithm [25] for solving problem (8).Step 1. For the given initial parameters , computeand . Let .Step 2. If , go to Step 3. Otherwise, go to Step 4.Step 3. If the termination condition is satisfied, stop. Otherwise, letand compute and go to Step 5.Step 4. If the termination condition is satisfied, stop. Otherwise, letand compute and go to Step 5.Step 5. Let ; return to Step 2.

4. Global Convergence

In this section, the global convergence of Algorithm 1 is proven. Firstly, it is easy to have the following lemma from Lemma 1.

Lemma 2. Let be defined by (9), where is a solution of problem (8). Then,

Proof. From the definition of , we have . It follows from the definition of and Lemma 1 that holds. Thus, (16) holds, and therefore, the proof is completed.
The next lemma indicates an important property of the Cauchy point direction , which is the sufficiently descent property.

Lemma 3. holds, for any .

Proof. For any , from (9), we haveand note that ; thus,This completes the proof.
To derive the global convergence of Algorithm 1, the following assumption is considered.

Assumption 1. The Jacobian of is bounded; i.e., there exists a positive constant such that , for any .
In the sequel, we show that “Steps 3–6” of Algorithm 1 do not cycle in the inner loop infinitely.

Lemma 4. Suppose that Assumption 1 holds, let be generated by Algorithm 1. If is not a stationary point of problem (3), then Algorithm 1 is well defined; i.e., the inner cycle of Algorithm 1 can be left in a finite number of internal iterates.

Proof. It follows from which is not a stationary point that there exists a positive constant such thatSince and Assumption 1 holds, there exists a positive constant such thatLetting , then from (4) and (22), we obtainand noting (21), we have , which impliesand therefore, (13) holds. Next, we prove that (14) is also satisfied. By contradiction, we assume that “Step 3–Step 6” of Algorithm 1 do cycle in the inner loop infinitely; then, .
From the definition of , we getIt follows from (13) and Lemma 3 that there exists a positive constant such thatBy the definition , we have , and noting Proposition 1 and , further, we getThus, from (26) and (27), we haveand therefore , which means that when is sufficiently small. However, the definition of in Step 5 of Algorithm 1 contradicts the fact that . Hence, the cycling between Step 3 and Step 6 terminates finitely. This completes the proof.
We can get the global convergence of Algorithm 1 based on the above lemmas.

Theorem 1. Suppose that Assumption 1 holds and be generated by Algorithm 1, then

Proof. Suppose that the conclusion is not true, then there exists a positive constant such that for any ,From Lemma 2, Assumption 1, and the above inequation, we haveNoting the updating rule of , , (31), and Lemma 4, we deduce thatwhich yields a contradiction that this means (30) is not true; i.e., the result (29) of the theorem holds, and therefore, the proof is completed.

5. Numerical Results

In this section, the numerical results of the proposed algorithm for solving problem (1) are reported. We denote the proposed Algorithm 1 by CTR and compare it with some existing algorithms. The adaptive trust region algorithm by Fan and Pan [16] and the classical trust region algorithm [26] are denoted by ATR and TTR, respectively. In numerical experiments, all parameters of Algorithm 1 are chosen as follows: , , , , , and . We choose in Algorithm 2. Steihaug method [27] is employed to obtain the approximative trial step for solving the trust region subproblem (6). We will terminate algorithms if the number of iterations is larger than 2000.

All codes were implemented in MATLAB R2010b. The numerical experiments were performed on a PC with an Intel Pentium (R) Dual-Core CPU at 3.20 GHz and 2.00 GB of RAM and using the Windows 7 operating system.

We list the test problems in Table 1 that can also be found in [28]. In Tables 13, “No.” denotes the number of the test problem, “Problem name” denotes the name of the test problem, “” denotes the initial point, “” denotes the dimension of the problem, “NI” is the total number of iterations, “NF” is the number of the function evaluations, “” refers to the optimum function values, and “” denotes the mean value of . It is worth mentioning that “” represents the stopping of the algorithms in situations that the number of iterations exceeds the maximal iterations, but the terminal conditions cannot be satisfied yet.


No.Problem name

1Exponential function 1
2Exponential function 2
3Extended Rosenbrock function
4Trigonometric function
5Singular function
6Logarithmic function
7Broyden tridiagonal function
8Trigexp function
9Function 15
10Strictly convex function 1
11Strictly convex function 2
12Zero Jacobian function
13Linear function-full rank
14Penalty function
15Brown almost-linear function
16Extended Powell singular function
17Tridiagonal system
18Extended Freudenstein and Roth function
19Discrete boundary value problem
20Troesch problem


No.nCTRATRTTR
NI/NFNI/NFNI/NF

15006/73.18E − 063.32E − 077/86.73E − 066/73.18E − 06
10005/68.95E − 063.32E − 077/84.74E − 065/68.95E − 06
20005/66.32E − 063.32E − 077/83.34E − 065/66.32E − 06
25002/33.50E − 061.57E − 062/33.38E − 062/33.38E − 06
10002/36.67E − 063.32E − 072/36.69E − 062/36.69E − 06
20003/49.13E − 063.32E − 073/88.39E − 063/58.44E − 06
350012/162.16E − 073.31E − 0112/203.89E − 0616/219.44E − 10
100011/155.92E − 072.96E − 0112/205.51E − 0616/211.39E − 09
200011/151.78E − 073.50E − 0112/207.82E − 0616/211.84E − 09
41009/117.33E − 065.28E − 012001/22932001/2270
550022/254.39E − 062.96E − 0118/195.84E − 0619/208.32E − 06
100025/276.37E − 064.35E − 0123/585.68E − 0623/244.33E − 06
200024/285.24E − 064.50E − 0119/205.53E − 0621/235.23E − 06
65005/62.12E − 069.81E − 045/64.91E − 087/82.74E − 08
10005/61.44E − 065.05E − 015/61.91E − 077/81.58E − 07
20006/77.69E − 076.08E − 015/67.56E − 087/85.08E − 08
75006/72.57E − 091.59E − 016/71.51E − 096/74.28E − 11
10006/72.63E − 061.63E − 016/71.03E − 086/73.83E − 06
20006/75.50E − 062.26E − 016/72.60E − 086/75.43E − 06
810014/164.31E − 082.92E − 012001/22812001/2268
9100014/185.42E − 061.46E − 0214/202.76E − 0716/221.36E − 09
105006/75.54E − 084.14E − 016/71.33E − 076/71.09E − 07
10006/76.04E − 083.71E − 016/72.49E − 076/79.91E − 07
20006/71.16E − 074.47E − 016/74.33E − 077/82.32E − 07
111008/93.22E − 072.21E − 018/93.38E − 088/91.15E − 06
125005/63.62E − 128.36E − 0116/175.81E − 0618/195.96E − 06
10005/64.15E − 118.82E − 0116/177.38E − 0618/199.31E − 06
20006/83.42E − 078.29E − 0116/178.23E − 0619/202.82E − 06
135009/102.41E − 077.19E − 014/57.67E − 069/101.04E − 08
10009/103.93E − 077.62E − 014/58.32E − 069/103.37E − 08
200010/117.01E − 067.30E − 015/61.00E − 1010/111.35E − 07
145004/56.96E − 061.09E − 027/105.22E − 065/65.34E − 06
100012/131.79E − 071.77E − 018/102.39E − 066/82.65E − 06
200010/115.39E − 061.15E − 0111/161.70E − 0610/115.32E − 07
155003/42.85E − 093.61E − 013/42.48E − 064/52.49E − 06
10004/51.32E − 075.56E − 013/46.34E − 076/78.80E − 08
20004/51.40E − 085.04E − 014/53.74E − 076/73.63E − 07
165001/21.10E − 063.32E − 071/21.10E − 061/21.10E − 06
10001/21.55E − 063.32E − 071/21.55E − 061/21.55E − 06
20001/22.20E − 063.32E − 071/22.20E − 061/22.20E − 06
17100034/449.22E − 093.58E − 0190/2196.85E − 0963/754.89E − 10


No.nCTRATRTTR
NI/NFNI/NFNI/NF

185006/83.57E − 074.13E − 017/85.84E − 087/86.12E − 09
10008/101.14E − 063.17E − 017/87.87E − 088/97.10E − 08
20008/103.08E − 063.17E − 017/81.07E − 078/92.29E − 08
195002/39.77E − 093.32E − 073/46.77E − 092/39.77E − 09
10002/39.10E − 103.32E − 073/46.17E − 102/39.09E − 10
20002/34.47E − 103.32E − 073/43.02E − 102/34.47E − 10
2010004/51.90E − 073.32E − 078/98.94E − 084/51.87E − 07

It is shown from Tables 2 and 3 that the proposed algorithm is efficient to solve these test problems and is competitive with the other two algorithms in NI and NF for most problems. For problems 1, 2, 16, 19, and 20, is close to zero, which implies that the trust region direction was used many times. Moreover, the number of iteration is the same as that of algorithm TTR, which means that our algorithm speeds up the local convergence of the iterates. It can be seen that is not too large for other test problems. However, for problems 4 and 8, algorithms ATR and TTR are not efficient to get the optimal solution when . We used the performance profiles by Dolan and Moré [29] to compare these algorithms with respect to NI and NF again, as shown in Figures 1 and 2. Therefore, it is noted from Tables 2 and 3 and Figures 1 and 2 that the proposed algorithm is well promising and has better performance in test problems in comparison with two existing algorithms. In order to investigate the behavior of the convergence rate of three algorithms, we plot the curve on problem 10 with and in Figure 3, where is the optimal point. Figure 3 indicates that Algorithm 1 converges faster than others.

6. Conclusions

In this paper, we presented a Cauchy point direction trust region algorithm for solving the nonlinear system of equations. This algorithm combines the trust region direction and the Cauchy point direction with the descent property and the automatic trust region. The optimal convex combination parameters were determined by using the gold section algorithm which is an exact one-dimensional linear search algorithm. The new algorithm was proven to have the descent property and the trust region property. We also showed the global convergence of the proposed algorithm under suitable condition. More importantly, the numerical results demonstrated the faster convergence of the proposed algorithm over two existing methods, which is very promising to solve large-scale nonlinear equations. However, the convergence rate of the proposed algorithm remains unclear. Therefore, the demonstration of the theoretical convergence rate of the proposed algorithm, application in image restoration problems, and compressive sensing will be the topics among others in our future study.

Data Availability

The data used to support the findings of this study are included within the article.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (11261006 and 11661009) and the Guangxi Natural Science Foundation (2020GXNSFAA159069).

References

  1. K. Meintjes and A. P. Morgan, “Chemical equilibrium systems as numerical test problems,” ACM Transactions on Mathematical Software, vol. 16, no. 2, pp. 143–151, 1990. View at: Publisher Site | Google Scholar
  2. A. A. Levin, V. Chistyakov, E. A. Tairov, and V. F. Chistyakov, “On application of the structure of the nonlinear equations system, describing hydraulic circuits of power plants, in computations,” Bulletin of the South Ural State University. Series “Mathematical Modelling, Programming and Computer Software, vol. 9, no. 4, pp. 53–62, 2016. View at: Publisher Site | Google Scholar
  3. F. Georgiades, “Nonlinear equations of motion of L-shaped beam structures,” European Journal of Mechanics—A/Solids, vol. 65, pp. 91–122, 2017. View at: Publisher Site | Google Scholar
  4. G. Yuan, T. Li, and W. Hu, “A conjugate gradient algorithm for large-scale nonlinear equations and image restoration problems,” Applied Numerical Mathematics, vol. 147, pp. 129–141, 2020. View at: Publisher Site | Google Scholar
  5. Y. Xiao and H. Zhu, “A conjugate gradient method to solve convex constrained monotone equations with applications in compressive sensing,” Journal of Mathematical Analysis and Applications, vol. 405, no. 1, pp. 310–319, 2013. View at: Publisher Site | Google Scholar
  6. Z. Papp and S. Rapajić, “FR type methods for systems of large-scale nonlinear monotone equations,” Applied Mathematics and Computation, vol. 269, pp. 816–823, 2015. View at: Publisher Site | Google Scholar
  7. W. Zhou and F. Wang, “A PRP-based residual method for large-scale monotone nonlinear equations,” Applied Mathematics and Computation, vol. 261, pp. 1–7, 2015. View at: Publisher Site | Google Scholar
  8. G. Yuan and M. Zhang, “A three-terms Polak-Ribière-Polyak conjugate gradient algorithm for large-scale nonlinear equations,” Journal of Computational and Applied Mathematics, vol. 286, pp. 186–195, 2015. View at: Publisher Site | Google Scholar
  9. G. Yuan, Z. Meng, and Y. Li, “A modified Hestenes and Stiefel conjugate gradient algorithm for large-scale nonsmooth minimizations and nonlinear equations,” Journal of Optimization Theory and Applications, vol. 168, no. 1, pp. 129–152, 2016. View at: Publisher Site | Google Scholar
  10. J. Fan and J. Pan, “Convergence properties of a self-adaptive Levenberg-Marquardt algorithm under local error bound condition,” Computational Optimization and Applications, vol. 34, no. 1, pp. 47–62, 2006. View at: Publisher Site | Google Scholar
  11. J. Fan, “Accelerating the modified Levenberg-Marquardt method for nonlinear equations,” Mathematics of Computation, vol. 83, no. 287, pp. 1173–1187, 2013. View at: Publisher Site | Google Scholar
  12. X. Yang, “A higher-order Levenberg-Marquardt method for nonlinear equations,” Applied Mathematics and Computation, vol. 219, no. 22, pp. 10682–10694, 2013. View at: Publisher Site | Google Scholar
  13. K. Amini and F. Rostami, “A modified two steps Levenberg-Marquardt method for nonlinear equations,” Journal of Computational and Applied Mathematics, vol. 288, pp. 341–350, 2015. View at: Publisher Site | Google Scholar
  14. J.-l. Zhang and Y. Wang, “A new trust region method for nonlinear equations,” Mathematical Methods of Operations Research (ZOR), vol. 58, no. 2, pp. 283–298, 2003. View at: Publisher Site | Google Scholar
  15. J. Fan, “Convergence rate of the trust region method for nonlinear equations under local error bound condition,” Computational Optimization and Applications, vol. 34, no. 2, pp. 215–227, 2006. View at: Publisher Site | Google Scholar
  16. J. Fan and J. Pan, “An improved trust region algorithm for nonlinear equations,” Computational Optimization and Applications, vol. 48, no. 1, pp. 59–70, 2011. View at: Publisher Site | Google Scholar
  17. G. Yuan, Z. Wei, and X. Lu, “A BFGS trust-region method for nonlinear equations,” Computing, vol. 92, no. 4, pp. 317–333, 2011. View at: Publisher Site | Google Scholar
  18. H. Esmaeili and M. Kimiaei, “A new adaptive trust-region method for system of nonlinear equations,” Applied Mathematical Modelling, vol. 38, no. 11-12, pp. 3003–3015, 2014. View at: Publisher Site | Google Scholar
  19. H. Esmaeili and M. Kimiaei, “An efficient adaptive trust-region method for systems of nonlinear equations,” International Journal of Computer Mathematics, vol. 92, no. 1, pp. 151–166, 2015. View at: Publisher Site | Google Scholar
  20. M. Kimiaei and H. Esmaeili, “A trust-region approach with novel filter adaptive radius for system of nonlinear equations,” Numerical Algorithms, vol. 73, no. 4, pp. 999–1016, 2016. View at: Publisher Site | Google Scholar
  21. H. Esmaeili and M. Kimiaei, “A trust-region method with improved adaptive radius for systems of nonlinear equations,” Mathematical Methods of Operations Research, vol. 83, no. 1, pp. 109–125, 2016. View at: Publisher Site | Google Scholar
  22. M. J. D. Powell, “Convergence properties of a class of minimization algorithms,” in Nonlinear Programming, pp. 1–27, Elsevier, Amsterdam, Netherlands, 1975. View at: Google Scholar
  23. L. Qi, X. J. Tong, and D. H. Li, “Active-set projected trust-region algorithm for box-constrained nonsmooth equations,” Journal of Optimization Theory and Applications, vol. 120, no. 3, pp. 601–625, 2004. View at: Publisher Site | Google Scholar
  24. G. Yuan, Z. Wei, and Z. Wang, “Gradient trust region algorithm with limited memory BFGS update for nonsmooth convex minimization,” Computational Optimization and Applications, vol. 54, no. 1, pp. 45–64, 2013. View at: Publisher Site | Google Scholar
  25. Y. X. Yuan and W. Sun, Optimization Theory and Methods, Science Press, Beijing, China, 1997.
  26. A. R. Conn, N. I. M. Gould, and P. L. Toint, Trust-region Methods, Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2000.
  27. T. Steihaug, “The conjugate gradient method and trust regions in large scale optimization,” SIAM Journal on Numerical Analysis, vol. 20, no. 3, pp. 626–637, 1983. View at: Publisher Site | Google Scholar
  28. W. La Cruz, J. M. Martínez, and M. Raydan, “Spectral residual method without gradient information for solving large-scale nonlinear systems of equations,” Mathematics of Computation, vol. 75, no. 255, pp. 1429–1449, 2006. View at: Publisher Site | Google Scholar
  29. E. D. Dolan and J. J. Moré, “Benchmarking optimization software with performance profiles,” Mathematical Programming, vol. 91, no. 2, pp. 201–213, 2002. View at: Publisher Site | Google Scholar

Copyright © 2020 Zhou Sheng and Dan Luo. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


More related articles

 PDF Download Citation Citation
 Download other formatsMore
 Order printed copiesOrder
Views337
Downloads258
Citations

Related articles

Article of the Year Award: Outstanding research contributions of 2020, as selected by our Chief Editors. Read the winning articles.