Abstract
A new self-adaptive rule of trust region radius is introduced, which is given by a piecewise function on the ratio between the actual and predicted reductions of the objective function. A self-adaptive trust region method for unconstrained optimization problems is presented. The convergence properties of the method are established under reasonable assumptions. Preliminary numerical results show that the new method is significant and robust for solving unconstrained optimization problems.
1. Introduction
Consider the following unconstrained optimization problem: where is continuously differentiable.
The trust region methods calculate a trial step by solving the subproblem at each iteration, where and is symmetric matric approximating the Hessian of at , and is the current trust region radius. Throughout this paper, denotes the -norm. We define the ratio, and the numerator and the denominator are called the actual reduction and the predicted reduction, respectively. For basic trust region (BTR) method, if the sufficient reduction predicted by the model is realized by the objective function, the trial point is accepted as the next iterate and the trust region is expanded or kept the same. If the model reduction turns out to be a poor predictor of the actual behavior of the objective function, the trial point is rejected and the trust region is contracted, with the hope that the model provides a better prediction in the smaller region. More formally, basic trust region radius update rule can be usually summarized as follows: where the constants ,,, and satisfy
The iteration is said to be successful if . If not, the iteration is unsuccessful, and the trial point is rejected. If , the iteration is said to be very successful iteration [1]. If is significantly larger than one, that is, , the iteration is called too successful iteration [2].
Sartenaer [3] developed an elaborate strategy which can automatically determine an initial trust region radius. The basic idea is to determine a maximal initial radius through many repeated trials in the steepest descent direction in order to guarantee a sufficient agreement between the model and the objective function. This strategy requires additional evaluations of the objective function. Zhang et al. [4] presented another strategy of determining the trust region radius. Their basic idea is originated from the following subproblem in an artificial neural network research [5]: where , ,, and is a nonnegative integer. Therefore, instead of adjusting , one adjusts at each iteration. Motivated by this technique, they solved the trust region subproblem with where , is a nonnegative integer, and is a positive definite matrix for some . However, their method needs to estimate or at each iteration, which leads to some additional cost of computation. As a result, Shi and Guo [6] proposed a simple adaptive trust region method. The new trust region model is more consistent with the objective function at the current iteration. Fu et al. [7] developed an adaptive trust region method based on the conic model by using the above adaptive strategy [4]. Sang and Sun [8] gave a new self-adaptive adjustment strategy to update the trust region radius, which makes full use of the information at the current point. Yu and Li [9] proposed a self-adaptive trust region algorithm for solving this nonsmooth equation. Some authors [1, 10] adopted different values for parameters (5) but seldom questioned the radius update formula (4). In addition, many adaptive nonmonotonic trust region methods have been proposed in [11–18]. Hei [19] proposed a self-adaptive update method, in which trust region radius is a product of a so-called -function and ; that is, , where is the -function. As the ratio is larger than one, is nondecreasing and bounded. However, as the iteration is very successful, the ratio is larger than one; it implies that the local approximation of the objective function by the model function is not good. Walmag and Delhez [2] suggested that it is not overconfident in the model at too successful iterations. They presented a self-adaptive update method, in which trust region radius is , where is the -function. If is significantly larger than one, -function is nonincreasing. However, they took but close to one to match the convergence criteria presented by Conn et al. [1].
In our opinion, the agreement between the model and the objective function is not good enough at too successful iterations. We take that the updated trust region radius is less than and is bounded lower away from zero. It implies that is not always necessary. This strategy can also match the convergence criteria presented by Conn et al. [1].
In the paper, the -function is introduced, which is a variant of -function. A new self-adaptive trust region method is proposed, in which the trust region radius is . The new method is more efficient at too successful iterations.
The rest of the paper is organized as follows. In Section 2, we define -function to introduce new update rules and a new self-adaptive trust region algorithm is presented. In Section 3, the convergence properties of proposed algorithm are investigated. In Section 4, numerical results are given. In Section 5 conclusions are summarized.
2. -Function and the New Self-Adaptive Trust Region Algorithm
To obtain the new trust region radius update rules, we define -function ,.
Definition 1. A function is called an -function if it satisfies the following:(1) is nondecreasing in and nonincreasing in , , for ,(2),(3),(4),(5),(6), for ,(7)and ,
where the constants ,,,,,, are positive constants such that
It is easy to prove that the -function is a bounded function in . In the following, we show the differences between the usual empirical rule, the -function rule, the -function rule, and the -function rule.
The usual empirical rules ([1, 20, 21]) (Figure 1(a)) can be usually summarized as follows: where ,,, and are predefined constants such that
(a)
(b)
(c)
The -function rule and the -function rule can be described as follows: where the -function (Figure 1(b)) proposed by Hei [19] is chosen aswhere , , , , and are constants, and the -function (Figure 1(c)) proposed by Walmag and Delhez [2] is chosen as where ,,, and are constants.
The -function rule can be described as follows: where the -function (Figure 2) is chosen as
where , , , , , and are constants.
The -function generalizes the -function and the -function. It contains some favorable features of the -function [19] and the -function [2].
Now describe the new self-adaptive trust region algorithm with improved update rules.
Algorithm 2.
One has the following.
Step 1. Given , , ,,, and ; ; set .
Step 2. If or , stop. Otherwise solve subproblem (2) to get .
Step 3. Compute
Compute as follows:
Update the trust region radius
where is defined by (15).
Step 4. Compute and ; set ; go to step 2.
3. Convergence of Algorithm 2
In the section, we investigate the convergence properties of Algorithm 2. Since it can be considered as a variant of the basic trust region method of Conn et al. [1], we expect similar results and significant similarities in their proofs under the following assumptions.
Assumption 3. Consider the following.(i)The sequence is uniformly bounded in norm; that is , for some constant .(ii)The function is bounded on the level set .(iii)The solution of the subproblem (2) satisfies where .(iv)The solution of the subproblem (2) satisfies for positive constant .
Lemma 4. Suppose that Assumption 3 holds. Then where arbitrarily decreases with decreasing.
Proof. Since from Taylor theorem, we have that it follows from the definition of in (2) that
By Algorithm 2, we are capable of showing that the iteration must be very successful but not too successful if the trust region radius is sufficiently small enough and also that the trust region radius has to increase in this case. The following lemma’s proof is a bit different from the proof of Theorem in [1].
Lemma 5. Suppose that Assumption 3 holds. If and there exists a constant such that then
Proof . Using Assumption 3 and and assuming that there is such that , we obtain that
Combining (21) and (26), we have
By (24), we can choose sufficient small , such that
Using (27) and (28), we have ; that is, . Since , then . And so, by Algorithm 2, we have , where falls below the threshold .
The proof of Lemma 5 efficiently uses the conditions to explain the case of too successful iteration, as distinguished from the proof of Theorem in [1].
Theorem 6. Suppose that Assumption 3 holds. Let the sequence be generated by Algorithm 2. Then
Proof. Assume, for the purpose of deriving a contradiction, that, for all ,
Suppose that there is an infinite iteration subsequence such that . Using Algorithm 2 and (21), we have
where . Since is bounded below, then
which contradicts (25). Hence (30) holds.
In our strategy of the trust region radius’ adjustment, most of the iterations are indeed very successful but not too successful; the trust region constraint becomes irrelevant in the local subproblem. Therefore, superlinear convergence of trust region algorithm is preserved by the proposed self-adaptive radius update.
4. Numerical Experiments
In this section, we present preliminary numerical results to illustrate that the performance of Algorithm 2, denoted by LATR, the basic trust region method in [1], denoted by BTR, and the parameters needed in (9) are chosen that ,,, and ; the adaptive trust region method in [19], denoted by RATR, and the parameters needed in (12) are chosen that ,,, and ; and the adaptive trust region method in [2], denoted by ATR, and the parameters needed in (12) are chosen that ,,, and . In Algorithm 2, , , , , , , and . All tests are implemented by using MATLAB R2012a on a PC with CPU 2.67 GHz and 8.00 GB RAM. The first eleven problems are taken from [22]; others are from the CUTEr collection [15, 23]. The discrete Newton method is used to update the approximate Hessian matrix . To stabilize the algorithms, the approximate Hessian matrix can be chosen as follows: where is obtained by forward difference formula at (see [20]) and is an identity matrix.
In all trust region algorithms in this paper, the trial step is computed by CG-Steihaug algorithm in [20]. The iteration is terminated by the following condition: except for problem Waston, which will exceed 500 iterations. For the problem, the stopping criterion is In Table 1, we give the dimension (Dim) of each test problem, the number of function evaluations (nf), and the number of gradient evaluations (ng). In many cases, algorithm LATR is superior than algorithms BTR, RATR, and ATR, especially for solving problems (1), (7), (8), (13), (19), and (28), although the numbers of gradient evaluations are a bit more than others sometimes. Further result is shown by Figure 3, which is characterized by means of performance profile proposed in [24]. Consider the following performance profile function: where and is the performance ratio of a solver on a problem; denotes the number of the tested problems, the number of the solvers, and the number of the function evaluations (or the CPU time, the number of gradient evaluations, number of iterations, etc.) required to solve the problem by the solver .
From Table 1, we know that and ; then performance profile is given on the sum of the number of function and gradient evaluations to solve the problem. As we can see on Figure 3, the new self-adaptive algorithm is superior to the other three algorithms.
5. Conclusion
This paper presents a new self-adaptive trust region algorithm according to the new self-adaptive radius update rule. As the iteration istoo successful, we suggest reducing the trust region radius with the new rules. The convergence properties of the method are established under reasonable assumptions. Numerical experiments show that the new algorithm for solving unconstrained optimization problems is significant and robust.
For future research, we should investigate how to select an appropriate -function to conduct numerical experiments.
Conflict of Interests
The authors declare that they have no financial nor personal relationships with other people or organizations that can inappropriately influence their work; there is no professional nor another personal interest of any nature or kind in any product, service, and/or company that could be construed as influencing the position presented in or the review of the paper.
Acknowledgments
This research is partly supported by Chinese NSF under Grant 11171003, Chinese Ministry of Education, Science and Technology Research Key Project no. 2011039, and Department of Education of Jilin Province Project no. 2009158 and no. 201215102.