Abstract
An adaptive projected affine scaling algorithm of cubic regularization method using a filter technique for solving box constrained optimization without derivatives is put forward in the passage. The affine scaling interior-point cubic model is based on the quadratic probabilistic interpolation approach on the objective function. The new iterations are obtained by the solutions of the projected adaptive cubic regularization algorithm with filter technique. We prove the convergence of the proposed algorithm under some assumptions. Finally, experiments results showed that the presented algorithm is effective in detail.
1. Introduction
In this passage, we investigate the following optimization:where is smooth, but its derivative information is unavailable or unreliable. For each , is not greater than , and there exists some such that is less than strictly. Define the feasible set and the strict interior by , , respectively.
The structure of this passage is as shown below. The relevant information is reviewed and our algorithm for problem (1) is introduced in Section 2. The corresponding analysis are investigated in Sections 3 and 4, respectively. Then, the numerical experiments are listed in details in Section 5.
2. Literature Review
Optimization (1) forms an ordinary but important class in various circumstances. There are lots of researchers using different tools to solve it ([1–4]). In order to complements the algorithm library for solving this kind of optimization, we extent the cubic regularization algorithm (shorted by ARC), which is presented by Cartis et al. in [5, 6], with filter technique to solve (1) by projection measure based on probabilistic models.
The numerical results of classical ARC showed excellent performance. In [7], Cartis et al. extended ARC for unconstrained optimizations to finite-difference version. In [8], the authors extended the bound of [5, 6] to nonlinear problems with convex constraints. Gould et al., in [9], updated the parameter in the large-scale least-squares nonlinear optimization. Huang and Zhu, in [3], presented an affine scaling ARC algorithm for derivative-free box constrained minimization using backtracking line search technique to obtain the step size. In the framework of ARC, the authors did not use box constraints’ information; thus, we turn to an affine scaling technique in [10], which usually combine with the methods of trust region [10, 11]. However, for various reasons, there are many examples in computational science and engineering; their (at least some) derivatives were unavailable or unreliable. However, they may still be desirable to get the optimizations. This situation led researchers to look for the way to solve (1). Recently, there are many papers proposing various different techniques for the optimization without derivatives. Algorithms for bound constrained optimization with global convergence results using the trust region derivative-free methods were presented in [12, 13]. Powell, in [14, 15], proposed the NEWUOA algorithm, which employs a quadratic polynomial interpolation for the unconstrained or box constrained optimization with excellent experiment performance. In [16], Conn et al. gave a derivative-free version of the trust region method. The corresponding model was constructed by using the polynomial interpolation techniques. Li and Zhu, in [17], proposed an affine trust region method to solve (1) by using a backtracking relevant condition and a filter technique.
Motivated by these ideas, we will try our best to introduce the ARC method using filter technique to solve (1) without derivatives under certain assumptions in this paper. In absence of derivatives, we use probabilistic polynomials’ interpolation to construct models of . The presented algorithm has two merits. The first merit is that to solve the subproblem which is only needed once at every point under some conditions, but the ARC algorithms need to solve repeatedly in [5, 6], in which solving a subproblem is computationally more expensive than using line search to get the trial step. The other merit is that we take a backing line search technique to make the new iteration strictly feasible.
In this paper, unless otherwise noted, we write for brevity.
2.1. Notations
There are many constants and notations displayed in the next sections. For convenience, we collect the following: is bound on is bound on is bound on is bound on affines scaling matrix for affines scaling matrix for is bound on are constants related to is projected gradient for is projected gradient for is the step size at is the th trial step size at
3. Development of the Algorithm
At first, we review the elements of our presented algorithm.
Following the notations proposed by Cartis et al., in [5, 6], we give the ARC subproblem as follows:where is a dynamic positive parameter.
Definition 1. A mapping is called the projection onto ifwhere and is closed. since in (1) is a box. is a well-known criticality measure.
Definition 2 (see [18, 19]). Function is said fully quadratic of in if, for all , there exist some positive constants , such thatNext, random models will be considered; is denoted for their realizations. and will be used for random quantities, while their realizations are denoted by and , respectively.
Definition 3 (see [18, 19]). A sequence is said -probabilistically fully quadratic for sequence if satisfy , in which is generated by . The random models are probabilistically fully quadratic if .
Now, we expand (2) to the derivative-free version to solve (1):where . And, in the derivative-free case, so .
Filter technique was proposed firstly by Fletcher and Leyffer for constrained nonlinear optimization. The idea of using filter is to interpret the system of (1) as a biobjective optimization with two goals: minimizing and minimizing , where , , andThen, we also definite is the derivative version of .
In order to ensure or decreasing sufficiently, we give such a strategy:orwhere and are small positive constants.
If (9) or (10) holds, the current filter accepts . The filter remains unchanged. Otherwise, neither (9) nor (10) holds; the current filter rejects ; the filter can be augmented asAt the beginning, initialize .
Now, we present a projected adaptive cubic regularization algorithm with filter technique (shorted by PFARC) for the solutions of derivative-free box constraint optimization:
3.1. Initialization Step
An initial point . Given a positive parameter , a radius , and constants , they satisfy that
Initialize the filter . Choose . Set .
3.2. Main Step
Step 1: set a interpolation points set such that ; then, apply Algorithms 6.1 and 6.3 in [20] to construct the fully quadratic model on . Set Step 2: if , stop at . Step 3: solve (7) and obtain . Step 4: line search. Step 4.1: let and . Step 4.2: set . Step 4.3: if accepts , go to Step 4.4, else go to Step 4.6. Step 4.4: if , holds, and accept and go to Step 4.7, else go to Step 4.5. Step 4.5: if , holds, and accept and go to Step 4.7, else go to Step 4.6. Step 4.6: set ; let ; then, go to Step 4.2. Step 4.7: set and where , for some and . Step 5: let . If , set , and apply Algorithm 6.1 and 6.3 in [20] to construct fully quadratic model in . Set . Go to Step 3 to compute . Until . Update , , and , and compute . Step 6: set . Compute . Step 7: compute ratio . Set Step 8: if neither (9) nor (10) holds, augment the filter set, else the filter does not change. Step 9: set . Then, the sample set is constructed in . Compute , . Set .
Remark 1. Now, we describe a concrete implementation in Step 3 using a Cauchy step which is constructed by an affine scaled gradient [21]. The search direction satisfies , whereAs we all know that the computation of Cauchy condition usually is much cheaper than the computation in Step 3, we will give a relationship between the critical measure and the affine gradient. Therefore, we can replace the predicted decrease by the Cauchy step at each criticality measure satisfying this relationship.
Remark 2. Similar to Lemma 5.1 in [16], we can prove that Step 5 will terminate in a finite number if .
Remark 3. In Step 4, the scalar with denotes thatin which , and represent the th elements of , and , respectively.
4. Global Convergence Analysis
Next, we will present some properties of the algorithm PFARC which will be referenced in analysis of global and local convergence. Denote the indices set of the augmented filter by . First, we make some assumptions as following.
Assumption 1. The iterates remain in , where is the level set of , .
Assumption 2. Let be twice continuously differentiable and bounded from below, and its gradient is Lipschitz continuous with Lipschitz constants .
Assumption 3. Assume that , , , and , for all , respectively, where , , and are positive constants.
Let be a solution to (7); the following equations,are the first-order necessary conditions of (7).
Consequently, from Assumptions 1–3, (5), and the choice of in Step 5, we can get thatLet denote any criticality measure for the derivative-free model such thatholds on , for some . Certainly, and a natural choice for satisfying (20) is . And, there exist an such that . First, we show that is a criticality measure, which can be found in [21].
Lemma 1. The function defined in (20) is a critical measure.
Lemma 2. Let satisfy the Cauchy decrease condition in Remark 1. If , there is a constant which only depends on , and such that the following holds:
Proof. Set and . First, we derive an upper bound to ; then, apply . It is easy to see that , since, by (20), the maximum step-size arrived at the lower bounds to isThe step-size arrived at the upper bounds to is calculated in the same way:Let . Noting that and , the function attains its global minimum at , whereWe have . If , thenand furthermore,Then, if is positive semidefinite, that is, , we have thatThat is,Otherwise, , we can get that , from , which is equivalent to (25).
First, assume that . Therefore, from (25), we haveNoting that , we can also obtainBy combining , (26), and (30), we can get thatNow, assume that ; then,On the contrary, , and we obtain thatIf ,If , we can have that from , and then,Then, combining and (20) and (22)–(35), holds, where is a constant which only depends on , and . The proof is completed.
The following lemma from [5] gives a useful bound on the step .
Lemma 3. If is a solution in Step 3, then
Assumption 4. Suppose that , where .
Lemma 4. Let Assumptions 1–4 hold and be fully quadratic. Ifthen and .
Proof. From (37), we will get that because, otherwise, the algorithm would have terminated; then, will conflict with (37). However,Firstly, using a Taylor expansion, for some , the first term of the numerator in the fraction above becomes thatwhere Assumption 2, Lemma 3, and (37) are used.
Next, we consider the second term of the numerator in the fraction:However, the denominator is from Lemma 2.
Thus, following with (37)–(40), we get thatThus, and are from Step 7.
Assumption 5. Let ; then, as .
Assumption 5, which is reasonable when we use Armijo backing line search or Wolfe condition (ii) for global convergence, is important to promote global convergence.
If satisfies Armijo condition, , where is in , we assume that , where is the first integer such that satisfies Armijo condition, and we can get thatThen, we can get thatwhere . From Assumption 2 and (5), thenDividing (44) by and applying Lemma 2,If , using Assumption 3 and the fact that is bounded, , we can infer that as from (45).
Then, we show that Assumption 5 can be also achieved when satisfies the Wolfe condition (ii):We rewrite it asFrom Assumption 2 and (5) again, we have thatUsing Lemma 2 and Assumption 3, the following inequality,holds. Consequently, if .
Assumption 6. Assume that , when .
Lemma 5. Let Assumptions 1–6 hold. If and after many steps with a KKT points of (1), algorithm PFARC does not stop; then, .
Proof. Choose an integer that satisfy that the filter is not augmented in iteration for all iteration . Step 4 in algorithm PFARC gives that either (9) or (10) holds. If (9) holds for , then since is in . Recall that , we infer that holds. If (9) does not hold for certain sufficiently large, algorithm DFFARC implies that, for each which is greater than ,Hence, for all ,To have a contradiction, we let . Then, and . Since is bounded from below, the right-hand side series is bounded as ; then, .
Let be given in Step 4 with . It is easy to see thatwhich is the KKT-conditions of (1). From the continuity of , Assumption 6, and (5), we can get that has the same sign when is close to ; then, as .
Furthermore, if is obtained from the reduced conditions in Step 4, we have that from Assumption 5. Recall that , we can have that, for sufficiently large , ; then, from Lemma 4. According to the updating role in , if , then from Lemma 4. By induction, . Suppose the claim that holds. It is obvious that if . If , then ; consequently, since . The claim , which contradicts to . Hence, , and our hypothesis is impossible. Thus, the conclusion holds.
Lemma 6. Let Assumptions 1–6 hold and . If algorithm PFARC does not stop after finite steps at a KKT points of in (1), then
Proof. To get a contradiction, we suppose that, for all large , , where is a subsequence of and is a constant. By the definition of , the pair of is augmented to . Thus, there exists an infinite subsequence such that from the compactness theorem, which contradicts to . Hence, (53) holds.
Lemma 7. Let Assumptions 1–6 hold, and sequence is probabilistically fully quadratic. is a sequence of random iterates generated by algorithm PFARC. If any limit point of satisfies the strict complementarity, then almost surely.
Proof. Combining the affine scaling technique with the proof in Theorem 4.2 in [18], it is easy to derive the conclusion similar to Theorem 4.8 in [19]. We have omitted this section for reasons of length.
Theorem 1. Let Assumptions 1–5 hold, and algorithm PFARC does not stop after finite steps at a KKT points of in (1); thus,
Proof. From Lemmas 5 and 6, we just think about this one case in which there are with and , and are larger than integer . If (54) does not hold, there is a subsequence such that . From Lemma 6, we also have, for every , that ; the iterate is the first iterate after such that is augmented to , that is, . Thus,Obviously, is infinite, where and are defined as above.
For every , we have that either (9) or (10) holds. If (9) holds, then . Since and and , then which contradicts to (55). Then, algorithm PFARC implies that (10) holds. Thus, .
Moreover, is monotonically decreasing and . Consequently,This ensures that , with some , because otherwise the inequality above will give thatSimilar to proof of Lemma 5, we can to prove that , where . Furthermore, from Assumption 5, we can get as . Recall that and ; thus, , and then, , where is defined in Step 4, so . However,Thus, combining (58) and inequalities above, we can have for sufficiently large thatThen,Consequently,Since is convergent, we can get from (61). However, is uniform continuous on . Thus, holds, which conflicts to the fact that . Therefore, our assumption is wrong and .
Combining (57) with , we get that , which conflicts to is bounded below. Thus, there is a subsequence such thatSince is not in and , it follows from (7) and (62) thatFor all , , Lemma 6 implies . Consequently, from (63), we get that ; it contradicts to that since . Thus, the claim holds.
5. Properties of the Local Convergence
We can know that any limit point of (1) is a KKT point from Theorem 1. In the following section, we consider the local convergence of algorithm PFARC, where the following assumptions are required.
Assumption 7. Assume that as .
Assumption 8. Assume that satisfiesNoting that is equivalent to , then is equivalent to the fact . Let . From (5), the fact is equivalent to the fact , as . To illustrate the properties of the local convergence, we substitute for in Step 4.4. That is, we accept the trial step ifholds in Step 4.4 in the following sections.
Theorem 2. Assume that Assumptions 1–8 hold and algorithm PFARC does not stop finite step at a KKT points of (1); then, for large and all iterations eventually satisfy is bounded from above as .
Proof. Theorem 1 gives that from the assumptions in Theorem 2. Consequently, Assumption 7 implies that .
Thus, for large ,in which appears in Assumption 8. And, we also have that for large from Assumptions 2 and 8. Then, we can infer that . As a consequence, . So, is positive definite. And,Since , the first and last terms above giveOtherwise, , from Lemma 2. This, however, contradicts since . Recall thatand we can get , from . Thus, (68) gives that . Furthermore, , in which appears in Step 4. Then, for large .
Assumption 6 provides that and is the same for large if . On the other side, (5) gives that and are also the same for large if for some . Thus, for large .
Now let us think about . From Step 7, .
However, using a Taylor expansion, for some ,Together with the inequalities above, for large ,where the last inequality is from and Assumptions 2 and 7. Recall the updating role in in Step 7, and we get thatThus, is bounded from above and . Clearly, . Set , where is sufficiently small such thatwhenever . Let , where is sufficiently small such thatwhenever . Such exists by Lemma 1.2 in [22].
Then, it follows thatwhenever , where .
Let be given by in Step 4. Similar to the proof of Lemma 5, we can also get that . Therefore, for large .
Corollary 1. Under the conditions of Theorem 2, then
Proof. Recalling (5) and for large , we obtain the following inequalities:Hence,as . The limit in (79) is from (75). In fact, (75) gives that since . Thus, , Furthermore, . Noting that is equivalently to , we have that (76) holds.
Since is the limit point of , there exist positive scales , such thatConsequently, (77) holds from the inequalities above and (76).
6. Numerical Analysis
The main task of this section is to test the algorithm PFARC constructed in this paper by solving actual test problems. We use MATLAB 2014a to program the algorithm, solve all the test problems in Table 1, and record the corresponding results. The initial parameters given by the algorithm PFARC are as follows:
In this paper, the termination precision of PFARC is set to .
In the following table, every test problem comes from [23], where represents the dimension of the problems and and represent the lower bound and upper bound of the problems, respectively.
Next, we will use PFARC and BOBYQA to solve the same problems in Table 1, respectively, and the results are recorded in Table 2, where and represent the number of iterations in PFARC and BOBYQA, respectively.
From the comparison results in Table 2, we can easily see that our algorithm PFARC can use a few iterations to get the optimal solution when solving the test problem given in Table 1, while BOBYQA algorithm needs a large number of iterations, which shows that PFARC has advantages in calculating the given test problems in Table 1.
In order to further test our algorithm, we will use PFARC given by this paper to calculate the problems given in [3], and the comparison results are recorded in Table 3. In Table 3, represents the dimension of the problems, represents the number of iterations in PFARC, and represents the number of iterations for the algorithm given in [3]. In addition, the questions in Table 3 are from [24].
From the results in Table 3, it can be seen that the same problem is solved by the ARC method using the interpolation technology without derivative. The number of iterations required by PFARC is less than the number of iterations in [3]. The import reason is that filter technique implemented in PFARC has two goals: minimization of and , but a line search technique in [3] only performed the minimization of . Therefore, our algorithm PFARC is more effective for solving bounded constrained derivative-free optimization problems.
In this paper, we propose a new algorithm PFARC for solving box constrained optimization problems without derivatives, which further complements the algorithm library for solving this kind of optimization. Since box constrained optimization is a classical problem, the proposed algorithm is an evolution of the existing algorithms, which further improved the existing ones. Box constrained optimization can be used in many fields, such as finance, science and technology, and military industry. Because the derivative information observed often is unavailable or unreliable, the derivative-free algorithm we proposed is closer to the reality. In the future, we can use the idea of this algorithm PFARC to solve other optimization problems, which is what we are going to do later.
Data Availability
The data used to support the findings of the study can be obtained from W. Hock, K. Schittkowski, Test examples for nonlinear programming codes, Lecture Notes in Economics and Mathematics System, 187. Springer (1981), and K. Schittkowski, MORE test examples for nonlinear programming codes, Lecture Notes in Economics and Mathematics System, 187. Springer (1981).
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
The authors gratefully acknowledge the partial supports of the Natural Science Foundation of Hainan Province (117107).