Journal of Applied Mathematics

Journal of Applied Mathematics / 2014 / Article

Research Article | Open Access

Volume 2014 |Article ID 610612 | 8 pages | https://doi.org/10.1155/2014/610612

A Novel Self-Adaptive Trust Region Algorithm for Unconstrained Optimization

Academic Editor: Kazutake Komori
Received28 Aug 2013
Revised18 Mar 2014
Accepted19 Mar 2014
Published15 Apr 2014

Abstract

A new self-adaptive rule of trust region radius is introduced, which is given by a piecewise function on the ratio between the actual and predicted reductions of the objective function. A self-adaptive trust region method for unconstrained optimization problems is presented. The convergence properties of the method are established under reasonable assumptions. Preliminary numerical results show that the new method is significant and robust for solving unconstrained optimization problems.

1. Introduction

Consider the following unconstrained optimization problem: where is continuously differentiable.

The trust region methods calculate a trial step by solving the subproblem at each iteration, where and is symmetric matric approximating the Hessian of at , and is the current trust region radius. Throughout this paper, denotes the -norm. We define the ratio, and the numerator and the denominator are called the actual reduction and the predicted reduction, respectively. For basic trust region (BTR) method, if the sufficient reduction predicted by the model is realized by the objective function, the trial point is accepted as the next iterate and the trust region is expanded or kept the same. If the model reduction turns out to be a poor predictor of the actual behavior of the objective function, the trial point is rejected and the trust region is contracted, with the hope that the model provides a better prediction in the smaller region. More formally, basic trust region radius update rule can be usually summarized as follows: where the constants ,,, and satisfy

The iteration is said to be successful if . If not, the iteration is unsuccessful, and the trial point is rejected. If , the iteration is said to be very successful iteration [1]. If is significantly larger than one, that is, , the iteration is called too successful iteration [2].

Sartenaer [3] developed an elaborate strategy which can automatically determine an initial trust region radius. The basic idea is to determine a maximal initial radius through many repeated trials in the steepest descent direction in order to guarantee a sufficient agreement between the model and the objective function. This strategy requires additional evaluations of the objective function. Zhang et al. [4] presented another strategy of determining the trust region radius. Their basic idea is originated from the following subproblem in an artificial neural network research [5]: where , ,, and is a nonnegative integer. Therefore, instead of adjusting , one adjusts at each iteration. Motivated by this technique, they solved the trust region subproblem with where , is a nonnegative integer, and is a positive definite matrix for some . However, their method needs to estimate or at each iteration, which leads to some additional cost of computation. As a result, Shi and Guo [6] proposed a simple adaptive trust region method. The new trust region model is more consistent with the objective function at the current iteration. Fu et al. [7] developed an adaptive trust region method based on the conic model by using the above adaptive strategy [4]. Sang and Sun [8] gave a new self-adaptive adjustment strategy to update the trust region radius, which makes full use of the information at the current point. Yu and Li [9] proposed a self-adaptive trust region algorithm for solving this nonsmooth equation. Some authors [1, 10] adopted different values for parameters (5) but seldom questioned the radius update formula (4). In addition, many adaptive nonmonotonic trust region methods have been proposed in [1118]. Hei [19] proposed a self-adaptive update method, in which trust region radius is a product of a so-called -function and ; that is, , where is the -function. As the ratio is larger than one, is nondecreasing and bounded. However, as the iteration is very successful, the ratio is larger than one; it implies that the local approximation of the objective function by the model function is not good. Walmag and Delhez [2] suggested that it is not overconfident in the model at too successful iterations. They presented a self-adaptive update method, in which trust region radius is , where is the -function. If is significantly larger than one, -function is nonincreasing. However, they took but close to one to match the convergence criteria presented by Conn et al. [1].

In our opinion, the agreement between the model and the objective function is not good enough at too successful iterations. We take that the updated trust region radius is less than and is bounded lower away from zero. It implies that is not always necessary. This strategy can also match the convergence criteria presented by Conn et al. [1].

In the paper, the -function is introduced, which is a variant of -function. A new self-adaptive trust region method is proposed, in which the trust region radius is . The new method is more efficient at too successful iterations.

The rest of the paper is organized as follows. In Section 2, we define -function to introduce new update rules and a new self-adaptive trust region algorithm is presented. In Section 3, the convergence properties of proposed algorithm are investigated. In Section 4, numerical results are given. In Section 5 conclusions are summarized.

2. -Function and the New Self-Adaptive Trust Region Algorithm

To obtain the new trust region radius update rules, we define -function ,.

Definition 1. A function is called an -function if it satisfies the following:(1) is nondecreasing in and nonincreasing in , , for ,(2),(3),(4),(5),(6), for ,(7)and ,
where the constants ,,,,,, are positive constants such that

It is easy to prove that the -function is a bounded function in . In the following, we show the differences between the usual empirical rule, the -function rule, the -function rule, and the -function rule.

The usual empirical rules ([1, 20, 21]) (Figure 1(a)) can be usually summarized as follows: where ,,, and are predefined constants such that

The -function rule and the -function rule can be described as follows: where the -function (Figure 1(b)) proposed by Hei [19] is chosen aswhere , , , , and are constants, and the -function (Figure 1(c)) proposed by Walmag and Delhez [2] is chosen as where ,,, and are constants.

The -function rule can be described as follows: where the -function (Figure 2) is chosen as

where , , , , , and are constants.

The -function generalizes the -function and the -function. It contains some favorable features of the -function [19] and the -function [2].

Now describe the new self-adaptive trust region algorithm with improved update rules.

Algorithm 2. One has the following.
Step 1. Given , , ,,, and ; ; set .
Step 2. If or , stop. Otherwise solve subproblem (2) to get .
Step 3. Compute
Compute as follows:
Update the trust region radius where is defined by (15).

Step 4. Compute and ; set ; go to step 2.

3. Convergence of Algorithm 2

In the section, we investigate the convergence properties of Algorithm 2. Since it can be considered as a variant of the basic trust region method of Conn et al. [1], we expect similar results and significant similarities in their proofs under the following assumptions.

Assumption 3. Consider the following.(i)The sequence is uniformly bounded in norm; that is , for some constant .(ii)The function is bounded on the level set .(iii)The solution of the subproblem (2) satisfies where .(iv)The solution of the subproblem (2) satisfies for positive constant .

Lemma 4. Suppose that Assumption 3 holds. Then where arbitrarily decreases with decreasing.

Proof. Since from Taylor theorem, we have that it follows from the definition of in (2) that

By Algorithm 2, we are capable of showing that the iteration must be very successful but not too successful if the trust region radius is sufficiently small enough and also that the trust region radius has to increase in this case. The following lemma’s proof is a bit different from the proof of Theorem in [1].

Lemma 5. Suppose that Assumption 3 holds. If and there exists a constant such that then

Proof . Using Assumption 3 and and assuming that there is such that , we obtain that Combining (21) and (26), we have By (24), we can choose sufficient small , such that
Using (27) and (28), we have ; that is, . Since , then . And so, by Algorithm 2, we have , where falls below the threshold .

The proof of Lemma 5 efficiently uses the conditions to explain the case of too successful iteration, as distinguished from the proof of Theorem in [1].

Theorem 6. Suppose that Assumption 3 holds. Let the sequence be generated by Algorithm 2. Then

Proof. Assume, for the purpose of deriving a contradiction, that, for all ,
Suppose that there is an infinite iteration subsequence such that . Using Algorithm 2 and (21), we have where . Since is bounded below, then which contradicts (25). Hence (30) holds.

In our strategy of the trust region radius’ adjustment, most of the iterations are indeed very successful but not too successful; the trust region constraint becomes irrelevant in the local subproblem. Therefore, superlinear convergence of trust region algorithm is preserved by the proposed self-adaptive radius update.

4. Numerical Experiments

In this section, we present preliminary numerical results to illustrate that the performance of Algorithm 2, denoted by LATR, the basic trust region method in [1], denoted by BTR, and the parameters needed in (9) are chosen that ,,, and ; the adaptive trust region method in [19], denoted by RATR, and the parameters needed in (12) are chosen that ,,, and ; and the adaptive trust region method in [2], denoted by ATR, and the parameters needed in (12) are chosen that ,,, and . In Algorithm 2, , , , , , , and . All tests are implemented by using MATLAB R2012a on a PC with CPU 2.67 GHz and 8.00 GB RAM. The first eleven problems are taken from [22]; others are from the CUTEr collection [15, 23]. The discrete Newton method is used to update the approximate Hessian matrix . To stabilize the algorithms, the approximate Hessian matrix can be chosen as follows: where is obtained by forward difference formula at (see [20]) and is an identity matrix.

In all trust region algorithms in this paper, the trial step is computed by CG-Steihaug algorithm in [20]. The iteration is terminated by the following condition: except for problem Waston, which will exceed 500 iterations. For the problem, the stopping criterion is In Table 1, we give the dimension (Dim) of each test problem, the number of function evaluations (nf), and the number of gradient evaluations (ng). In many cases, algorithm LATR is superior than algorithms BTR, RATR, and ATR, especially for solving problems (1), (7), (8), (13), (19), and (28), although the numbers of gradient evaluations are a bit more than others sometimes. Further result is shown by Figure 3, which is characterized by means of performance profile proposed in [24]. Consider the following performance profile function: where and is the performance ratio of a solver on a problem; denotes the number of the tested problems, the number of the solvers, and the number of the function evaluations (or the CPU time, the number of gradient evaluations, number of iterations, etc.) required to solve the problem by the solver .


Problem Dim
BTR RATR ATR LATR
nf ng nf ng nf ng nf ng

(1) Trigonometric 200 214 17 205 10 78 9

(2) Extended Powell singular200 28 20 26 20 32 21 30 24

(3) Schittkowski function 302 200 239 160 363 275 117 24 83 65

(4) Linear function full rank 200 23 4 21 4 15 3 11 7

(5) Watson 200 30 14 22 10 16 10 18 11
300 39 15 115 78 73 49 67 48
400 84 39 27 12 38 25 107 78
500 95 43 29 14 68 48 39 26

(6) Nearly separable 200 41 21 27 15 23 16 24 17
300 46 25 34 18 24 16 23 16
400 52 25 39 23 33 22 26 19
500 37 26

(7) Yang tridiagonal 200 67 40 61 44 99 32 51 41
300 151 113 72 57 105 38 74 61
400 133 93 141 106 151 68 109 90
500 148 97 120 95 257 149 109 90

(8) Allgower 200 40 23 53 38 57 2 83 69
300 18 1 17 1 24 1 7 1
400 25 15 17 3 56 2 21 17
500 20 3 17 3 52 3 9 2

(9) Linear function rank 1 200 101 42 62 27 285 30 39 24
300 102 40 66 25 300 30 35 16
400 110 45 69 25 319 32 39 20
500 108 42 70 25 317 31 47 32

(10) Linear function rank 1 with zero columns and rows 200 64 3 60 2 21 2 22 2
300 67 3 64 3 21 1 21 1
400 66 1 65 1 29 5 22 1
500 78 6 67 2 24 2 26 4

(11) Discrete integral equation 200 54 19 24 4 39 14 27 18
300 31 5 70 46 42 22 30 28
400 44 12 43 32 32 16 31 29
500 49 15 37 17 45 23 31 29

(12) CRAGGLVY 200 36 9 35 9 22 10 19 9

(13) GENHUMPS 200 26 0 26 0 74 0 9 0

(14) BROYDNBD 200 32 7 20 14 47 0 17 15

(15) PENALTY 200 71 33 69 33 59 33 59 37
300 75 35 74 35 63 38 59 35
400 74 33 76 35 64 36 61 37
500 77 35 80 37 58 36 62 38

(16) PENALTY2 200 46 8 42 8 88 9 25 11
300 149 72 159 129 78 50 150 116
400 434 208 465 390
500 330 228

(17) CHEBYQAD 200 75 55 79 70 39 7 83 67
300 28 12 83 70 41 6 89 70
400 95 63 99 85 108 64 104 82
500 32 16 37 22 119 52 103 83

(18) GENROSE 200 194 161 253 248 265 172 241 205
300 308 235 381 373 394 248 357 303
400 405 301 487 479 499 323 468 399
500

(19) INTEGREQ 200 15 1 14 1 214 0 6 1
300 15 1 14 1 206 0 6 1
400 15 1 14 1 203 0 6 1
500 15 1 14 1 201 0 6 1

(20) FLETCHCR 200 113 88 151 146 157 91 158 132
300 147 122 218 213 224 132 233 195
400 183 157 292 284 295 175 309 258
500 217 190 357 351 362 211 384 321

(21) ARGLINB 200 6 5 6 5 6 5 6 5
300 7 6 7 6 7 6 7 6
400 7 6 7 6 300 5 7 6
500 7 6 7 6 424 5 7 6

(22) NONDIA 200 7 6 7 6 93 6 7 6
300 7 6 7 6 7 6 7 6
400 6 5 6 5 6 5 6 5
500 6 5 6 5 6 5 6 5

(23) EG2 200 14 9 11 9 59 23 10 8
300 14 9 12 9 60 27 6 5
400 9 7 7 6 180 30 8 6
500 12 5 17 13 111 34 14 11

(24) CURLY20 200 13 11 19 17 16 14 11 10
300 13 11 21 18 17 15 13 12
400 10 9 11 11 11 10 9 9
500 10 9 11 11 11 10 9 9

(25) CUBE 200 52 20 39 25 95 16 37 29
300 50 16 77 65 98 20 40 32
400 52 16 45 31 83 16 35 28
500 52 16 52 38 128 14 35 28

(26) EXPLIN1 200 30 2 28 2 12 3 13 4
300 29 1 28 1 33 4 13 4
400 31 2 29 2 23 4 21 16
500 32 2 30 2 15 3 13 4

(27) SINQUAD 200 68 55 18 17 119 14 19 17
300 186 175 16 15 225 20 64 60
400 25 12 21 19 224 25 48 45
500 31 15 43 41 229 40 35 33

(28) LIARWHD 200 29 1 28 1 50 2 14 4
300 32 2 29 2 18 2 12 2
400 30 1 30 1 30 2 12 1
500 30 1 30 1 33 2 11 1

— means that the algorithm reaches 500 iterations.

From Table 1, we know that and ; then performance profile is given on the sum of the number of function and gradient evaluations to solve the problem. As we can see on Figure 3, the new self-adaptive algorithm is superior to the other three algorithms.

5. Conclusion

This paper presents a new self-adaptive trust region algorithm according to the new self-adaptive radius update rule. As the iteration istoo successful, we suggest reducing the trust region radius with the new rules. The convergence properties of the method are established under reasonable assumptions. Numerical experiments show that the new algorithm for solving unconstrained optimization problems is significant and robust.

For future research, we should investigate how to select an appropriate -function to conduct numerical experiments.

Conflict of Interests

The authors declare that they have no financial nor personal relationships with other people or organizations that can inappropriately influence their work; there is no professional nor another personal interest of any nature or kind in any product, service, and/or company that could be construed as influencing the position presented in or the review of the paper.

Acknowledgments

This research is partly supported by Chinese NSF under Grant 11171003, Chinese Ministry of Education, Science and Technology Research Key Project no. 2011039, and Department of Education of Jilin Province Project no. 2009158 and no. 201215102.

References

  1. A. R. Conn, N. I. M. Gould, and P. L. Toint, Trust-Region Methods, MPS/SIAM Series on Optimization, Society for Industrial and Applied Mathematics (SIAM); Mathematical Programming Society (MPS), Philadelphia, Pa, USA, 2000. View at: Publisher Site | MathSciNet
  2. J. M. B. Walmag and E. J. M. Delhez, “A note on trust-region radius update,” SIAM Journal on Optimization, vol. 16, no. 2, pp. 548–562, 2005. View at: Publisher Site | Google Scholar | MathSciNet
  3. A. Sartenaer, “Automatic determination of an initial trust region in nonlinear programming,” SIAM Journal on Scientific Computing, vol. 18, no. 6, pp. 1788–1803, 1997. View at: Publisher Site | Google Scholar | MathSciNet
  4. X. Zhang, J. Zhang, and L. Liao, “An adaptive trust region method and its convergence,” Science in China A, vol. 45, no. 5, pp. 620–631, 2002. View at: Publisher Site | Google Scholar | MathSciNet
  5. X. S. Zhang, “Trust region method in neural network,” Acta Mathematicae Applicatae Sinica, vol. 12, pp. 1–10, 1996. View at: Google Scholar
  6. Z.-J. Shi and J.-H. Guo, “A new trust region method for unconstrained optimization,” Journal of Computational and Applied Mathematics, vol. 213, no. 2, pp. 509–520, 2008. View at: Publisher Site | Google Scholar | MathSciNet
  7. J. Fu, W. Sun, and R. J. B. de Sampaio, “An adaptive approach of conic trust-region method for unconstrained optimization problems,” Journal of Applied Mathematics & Computing, vol. 19, no. 1-2, pp. 165–177, 2005. View at: Publisher Site | Google Scholar | MathSciNet
  8. Z. Sang and Q. Sun, “A self-adaptive trust region method with line search based on a simple subproblem model,” Journal of Computational and Applied Mathematics, vol. 232, no. 2, pp. 514–522, 2009. View at: Publisher Site | Google Scholar | MathSciNet
  9. Z. Yu and Q. Li, “A self-adaptive trust region method for the extended linear complementarity problems,” Applications of Mathematics, vol. 54, no. 1, pp. 53–65, 2009. View at: Publisher Site | Google Scholar | MathSciNet
  10. N. I. M. Gould, D. Orban, A. Sartenaer, and P. L. Toint, “Sensitivity of trust-region algorithms to their parameters,” 4OR. A Quarterly Journal of Operations Research, vol. 3, no. 3, pp. 227–241, 2005. View at: Publisher Site | Google Scholar | MathSciNet
  11. J.-L. Zhang and X.-S. Zhang, “A nonmonotone adaptive trust region method and its convergence,” Computers & Mathematics with Applications, vol. 45, no. 10-11, pp. 1469–1477, 2003. View at: Publisher Site | Google Scholar | MathSciNet
  12. J. Fu and W. Sun, “Nonmonotone adaptive trust-region method for unconstrained optimization problems,” Applied Mathematics and Computation, vol. 163, no. 1, pp. 489–504, 2005. View at: Publisher Site | Google Scholar | MathSciNet
  13. J. Zhang, K. Zhang, and S. Qu, “A nonmonotone adaptive trust region method for unconstrained optimization based on conic model,” Applied Mathematics and Computation, vol. 217, no. 8, pp. 4265–4273, 2010. View at: Publisher Site | Google Scholar | MathSciNet
  14. M. Ahookhosh and K. Amini, “A nonmonotone trust region method with adaptive radius for unconstrained optimization problems,” Computers & Mathematics with Applications, vol. 60, no. 3, pp. 411–422, 2010. View at: Publisher Site | Google Scholar | MathSciNet
  15. N. Andrei, “An unconstrained optimization test functions collection,” Advanced Modeling and Optimization, vol. 10, no. 1, pp. 147–161, 2008. View at: Google Scholar | MathSciNet
  16. Z. Shi and S. Wang, “Nonmonotone adaptive trust region method,” European Journal of Operational Research, vol. 208, no. 1, pp. 28–36, 2011. View at: Publisher Site | Google Scholar | MathSciNet
  17. Z. Sang and Q. Sun, “A new non-monotone self-adaptive trust region method for unconstrained optimization,” Journal of Applied Mathematics and Computing, vol. 35, no. 1-2, pp. 53–62, 2011. View at: Publisher Site | Google Scholar | MathSciNet
  18. Z. Cui and B. Wu, “A new modified nonmonotone adaptive trust region method for unconstrained optimization,” Computational Optimization and Applications, vol. 53, no. 3, pp. 795–806, 2012. View at: Publisher Site | Google Scholar | MathSciNet
  19. L. Hei, “A self-adaptive trust region algorithm,” Journal of Computational Mathematics, vol. 21, no. 2, pp. 229–236, 2003. View at: Google Scholar | MathSciNet
  20. J. Nocedal and S. T. Wright, Numerical Optimization, Springer, Berlin, Germany, 2000.
  21. W. Sun and Y.-X. Yuan, Optimization Theory and Methods, Nonlinear programming, vol. 1 of Springer Optimization and Its Applications, Springer, New York, NY, USA, 2006. View at: MathSciNet
  22. J. J. Moré, B. S. Garbow, and K. E. Hillstrom, “Testing unconstrained optimization software,” Association for Computing Machinery. Transactions on Mathematical Software, vol. 7, no. 1, pp. 17–41, 1981. View at: Publisher Site | Google Scholar | MathSciNet
  23. N. I. M. Gould, D. Orban, and P. L. Toint, “GALAHAD, a library of thread-safe Fortran 90 packages for large-scale nonlinear optimization,” Association for Computing Machinery. Transactions on Mathematical Software, vol. 29, no. 4, pp. 353–372, 2003. View at: Publisher Site | Google Scholar | MathSciNet
  24. E. D. Dolan and J. J. Moré, “Benchmarking optimization software with performance profiles,” Mathematical Programming, vol. 9, pp. 1201–1213, 2002. View at: Google Scholar

Copyright © 2014 Yunlong Lu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


More related articles

760 Views | 539 Downloads | 1 Citation
 PDF  Download Citation  Citation
 Download other formatsMore
 Order printed copiesOrder

Related articles

We are committed to sharing findings related to COVID-19 as quickly and safely as possible. Any author submitting a COVID-19 paper should notify us at help@hindawi.com to ensure their research is fast-tracked and made available on a preprint server as soon as possible. We will be providing unlimited waivers of publication charges for accepted articles related to COVID-19. Sign up here as a reviewer to help fast-track new submissions.