A Simple Sufficient Descent Method for Unconstrained Optimization

Abstract

We develop a sufficient descent method for solving large-scale unconstrained optimization problems. At each iteration, the search direction is a linear combination of the gradient at the current and the previous steps. An attractive property of this method is that the generated directions are always descent. Under some appropriate conditions, we show that the proposed method converges globally. Numerical experiments on some unconstrained minimization problems from CUTEr library are reported, which illustrate that the proposed method is promising.

1. Introduction

In this paper, we consider the unconstrained optimization problemmin𝑓(𝑥),𝑥∈ℝ^𝑛, (1.1) where 𝑓∶ℝ^𝑛→ℝ is a continuously differentiable function, and its gradient at point 𝑥_𝑘 is denoted by 𝑔(𝑥_𝑘), or 𝑔_𝑘 for the sake of simplicity. 𝑛 is the number of variables, which is automatically assumed to be large. Large-scale optimization is one of the important research areas both in optimization theory and algorithm design. There exist some kinds of effective methods available for solving (1.1), as for instance, inexact Newton, limited memory quasi-Newton, conjugate gradient, spectral gradient, and subspace methods [1–4].

The iterative formula the method is given by𝑥_𝑘+1=𝑥_𝑘+𝛼_𝑘𝑑_𝑘, (1.2) where 𝛼_𝑘 is a steplength, and 𝑑_𝑘 is a search direction. Generally, we say that 𝑑_𝑘 is a descent direction of 𝑓 at 𝑥_𝑘 if 𝑑^⊤_𝑘𝑔_𝑘<0. Furthermore, if there exists a positive constant 𝐶>0 such that 𝑑^⊤_𝑘𝑔_𝑘≤−𝐶‖𝑔_𝑘‖², then 𝑑_𝑘 is often called a sufficient descent direction. The descent property is very important for the iterative method to be global convergent, especially for the conjugate gradient method [5]. Unfortunately, for the earliest spectral gradient method [6] and the PRP conjugated gradient method, their generated directions are not always descent. Therefore, in the last years, much effort has been done to smooth away the drawback and develop new methods, for example, [7].

As is wellknown, if Armijo line search is used, the standard PRP method can cycle infinitely without approaching any optimal point when solving nonconvex minimization problems. To overcome this drawback, Hager and Zhang [8] made a slight modification on the PRP formula. Consequently, the resulting method own the remarkable property: for any line search, the generated direction always satisfies the sufficient descent condition 𝑑^⊤_𝑘𝑔_𝑘≤−(7/8)‖𝑔_𝑘‖². Moreover, Zhang et al. [9] further studied the PRP formula and proposed a three-term modified PRP method, in which the generated directions satisfy𝑑^⊤_𝑘𝑔_𝑘=−𝑔_𝑘², (1.3) and this property is independent of any line search. Additionally, for improving the numerical performance of the standard FR method, Zhang et al. [10] developed a modified FR method where the direction also satisfy (1.3).

Although much progress has been made in designing a sufficient descent direction in a conjugate gradient method, it seemingly receive little attention in other methods. Very recently, An et al. [5] proposed a robust technique to construct a sufficient descent direction for unconstrained optimization. The descent direction is a linear combination of the steepest descent direction and the projection of the original direction, that is,𝑑_𝑘=−𝑔_𝑘+𝜆_𝑘𝑔𝐼−^𝑘𝑔^⊤_𝑘𝑔_𝑘²𝑑_𝑘, (1.4) where 𝜆_𝑘 is a scalar and 𝑑_𝑘 is an original direction. Based on the definition of 𝑑_𝑘, it is easy to deduce that the sufficient descent condition (1.3) holds. This technique has been applied to the PSB updated formula and showed that the resulting method converges globally and superlinearly for an uniformly convex function. Moreover, the direction in [11] is truly a special case of (1.4).

We note that (1.4) can be considered as a general form of a nonlinear conjugate gradient method, in which𝑑_𝑘=−𝑔_𝑘+𝛽_𝑘𝑑_𝑘−1, (1.5) where 𝛽_𝑘 is a scalar. Comparing with (1.4), we see that the scalar 𝛽_𝑘 takes the place of the coefficient matrix of 𝑑_𝑘. In this paper, we further study the sufficient descent technique (1.4), and propose a simple sufficient descent method for solving unconstrained optimization problems. Our motivation is simple, we choose 𝑑_𝑘 as the gradient of the previous step in (1.4). The search direction in this proposed method always satisfies a sufficient descent condition. Under some conditions, we show that the algorithm converges globally by using a special line search. The performance of the method on some CUTEr test problems indicates that it is encouraging.

We organize this paper as follows. In the next section, we construct the sufficient descent direction. In Section 3, we state the steps of our new algorithm with a special line search. We also report some experiments on some large-scale unconstrained optimization problems. Finally, we conclude this paper with some remarks in the last section. Throughout this paper, the symbol ‖⋅‖ denotes the Euclidean norm of a vector.

2. New Search Direction

This section aims to state the new direction formula and investigate its properties. Now, if we take 𝑔_𝑘−1 as 𝑑_𝑘 and set 𝜆_𝑘≡1 in (1.4), we get𝑑_𝑘=−𝑔_𝑘+𝑔𝐼−^𝑘𝑔^⊤_𝑘𝑔_𝑘²𝑔_𝑘−1. (2.1) Obviously, the direction is a linear combination of the gradient at the current and the previous steps. Additionally, to obtain the global convergence of PRP method, Cheng [11] introduced a descent direction defined as𝑑_𝑘=−𝑔_𝑘+𝛽^PRP_𝑘𝑔𝐼−^𝑘𝑔^⊤_𝑘𝑔_𝑘²𝑑_𝑘−1. (2.2) We note that (2.2) is only a special choice of (1.4), in which 𝑑_𝑘 is replaced by 𝑑_𝑘−1 and 𝜆_𝑘 is chosen as the scalar 𝛽^PRP_𝑘. If we denote𝐻_𝑘𝑔=𝐼−^𝑘𝑔^⊤_𝑘𝑔_𝑘², (2.3) then it is easy to see that for any 𝑦≠0, we have 𝑦^⊤𝐻_𝑘𝑦=𝑦^⊤𝐻_𝑘𝐻^⊤_𝑘𝑦=𝐻^⊤_𝑘𝑦²≥0, (2.4) which indicates that 𝐻_𝑘 is a symmetric and positive semidefinite matrix.

3. Sufficient Descent Method

As we have stated in the previous section, the directions in (2.1) satisfy a sufficient descent condition. In this section, we list our algorithm and establish its global convergence.

Firstly, we state the steps of the simple sufficient descent method. In order to achieve the global convergence of our method, we consider the backtracking line search of Grippo and Lucidi (GL) [12]. That is, for given 𝛿∈(0,1), 𝛽>0, 𝜌∈(0,1), find the first 𝑗_𝑘 (𝑗=1,2,…) such that 𝛼_𝑘=𝛽𝜌^𝑗𝑘 satisfies𝑓𝑥_𝑘+𝛼_𝑘𝑑_𝑘≤𝑓𝑥_𝑘−𝛿𝛼²_𝑘𝑑_𝑘². (3.1)

Now, we are ready to state the steps of the Simple Sufficient Descent (SSD) method.

Algorithm 3.1 (SSD). We have the following steps.Step 1. Given 𝑥₀∈ℝ^𝑛. Let 0<𝛿<1, and 𝑑₀=−𝑔₀. Set 𝑘∶=0. Step 2. If ‖𝑔_𝑘‖=0, then stop.Step 3. Compute 𝑑_𝑘 by (2.1).Step 4. Find the steplength 𝛼_𝑘 satisfying (3.1). Let 𝑥_𝑘+1=𝑥_𝑘+𝛼_𝑘𝑑_𝑘.Step 5. Set 𝑘∶=𝑘+1, go to Step 2.

The remainder of this section is devoted to investigate the global convergence of Algorithm 3.1. We first state some assumptions.

Assumption. Function 𝑓 is continuously differentiable and the level set ℱ={𝑥∈ℝ^𝑛∣𝑓(𝑥)≤𝑓(𝑥₀)} is bounded.

Assumption. The gradient of 𝑓 is Lipschitz continuous, that is, there exists a constant 𝐿>0 such that 𝑔(𝑥)−𝑔𝑦≤𝐿𝑥−𝑦,∀𝑥,𝑦∈ℝ^𝑛. (3.2)

We first prove the following lemma.

Lemma 3.4. Let the sequence {𝑥_𝑘} be generated by Algorithm 3.1. Suppose that Assumptions 3.2 and 3.3 hold. Then, ^∞_𝑘=0𝑔_𝑘²𝑑_𝑘²<∞. (3.3)

Proof. We have the following cases.Case 1. Consider firstly the GL line search (3.1). If 𝛼_𝑘≠𝛽, then 𝜌⁻¹𝛼_𝑘 may not satisfy (3.1). That is, 𝑓𝑥_𝑘+𝜌⁻¹𝛼_𝑘𝑑_𝑘>𝑓𝑥_𝑘−𝛿𝜌⁻²𝛼²_𝑘𝑑_𝑘². (3.4) By the mean value theorem and Assumption 3.3, there exists 𝜃_𝑘∈(0,1) such that 𝑓𝑥_𝑘+𝜌⁻¹𝛼_𝑘𝑑_𝑘−𝑓𝑥_𝑘=𝜌⁻¹𝛼_𝑘𝑔𝑥_𝑘+𝜃_𝑘𝜌⁻¹𝛼_𝑘𝑑_𝑘^⊤𝑑_𝑘=𝜌⁻¹𝛼_𝑘𝑔^⊤_𝑘𝑑_𝑘+𝜌⁻¹𝛼_𝑘𝑔𝑥_𝑘+𝜃_𝑘𝜌⁻¹𝛼_𝑘𝑑_𝑘^⊤−𝑔_𝑘𝑑_𝑘≤𝜌⁻¹𝛼_𝑘𝑔^⊤_𝑘𝑑_𝑘+𝐿𝜌⁻²𝛼²_𝑘𝑑_𝑘², (3.5) where 𝐿>0 is the Lipschitz constant of 𝑔. Substituting the last inequality into (3.4), we get 𝛼_𝑘1≥−𝜌⁻¹()𝑔𝐿+𝛿𝑑^𝑘2_𝑘²=𝑐₁𝑔_𝑘²𝑑_𝑘². (3.6) This implies that there is a constant 𝑀>0 such that 𝑔_𝑘⁴𝑑_𝑘²≤𝑀𝑓𝑥_𝑘−𝑓𝑥_𝑘+1. (3.7)Case 2. If 𝛼=𝛽, then we have 𝑔_𝑘⁴𝑑_𝑘²≤𝑔_𝑘²≤𝑑_𝑘²≤𝛿⁻¹𝛽⁻²𝑓𝑥_𝑘−𝑓𝑥_𝑘−1, (3.8) which shows that (3.7) holds with 𝑀=𝛿⁻¹𝛽⁻². Moreover, (3.7) indicates the lemma claims.

Theorem 3.5. Let {𝑥_𝑘} be generated by Algorithm 3.1 with GL backtracking line search (3.1). Then liminf_𝑘→∞𝑔_𝑘=0. (3.9)

Proof. From Assumption 3.3, we know that there exists a positive constant 𝛾 such that 𝑔_𝑘≤𝛾. (3.10) By the definition of 𝑑_𝑘 in (2.1), we have 𝑑_𝑘≤𝑔_𝑘+𝐻_𝑘𝑔_𝑘−1≤𝑔_𝑘+𝑔_𝑘−1≤2𝛾, (3.11) which shows that ‖𝑑_𝑘‖ is bounded. We get from (3.1) that ^∞_𝑘=0𝛼²_𝑘𝑑_𝑘²<∞. (3.12) Consequently, we have lim_𝑘→∞𝛼_𝑘𝑑_𝑘=0. (3.13) For the sake of contradiction, we suppose that (3.9) does not hold. Then there is a constant 𝜖>0 such that ‖𝑔_𝑘‖≥𝜖 for all 𝑘≥0.
If liminf_𝑘→∞𝛼_𝑘>0, we get from (3.13) that liminf_𝑘→∞‖𝑑_𝑘‖=0, which shows liminf_𝑘→∞‖𝑔_𝑘‖=0. This contradicts the assumption (3.9).
If liminf_𝑘→∞𝛼_𝑘=0, that is, there is an infinite index set 𝐾 such thatlim_{𝑘∈𝐾,𝑘→∞}𝛼_𝑘=0. (3.14) It follows from the line search step in Algorithm 3.1 that when 𝑘∈𝐾 is sufficiently large, 𝜌⁻¹𝛼_𝑘 does not satisfy (3.1). Then from the first part of the proof of Lemma 3.4 we see that (3.6) holds. Since ‖𝑑_𝑘‖ is bounded and lim_{𝑘∈𝐾,𝑘→∞}𝛼_𝑘=0, (3.6) implies liminf_𝑘→∞‖𝑔_𝑘‖=0. This also yields a contradiction. The proof is complete.

4. Numerical Results

In this section, we test the feasibility and effectiveness of the Algorithm SSD. The algorithm is implemented in Fortran77 code using double precision arithmetic. All runs are performed on a PC ( Intel Pentium Dual E2140 1.6 GHz, 256 MB SDRAM) with Linux operations system. The algorithm stops if the infinity-norm of the final gradient is below 10⁻⁵, that is∇𝑓(𝑥)≤10⁻⁵. (4.1) The iteration is also stopped if the number of iterations exceeds 10000 or the number of function evaluations reaches 20000. Our experiments are performed on the subset of the nonlinear unconstrained problems from CUTEr [13] collection. The second-order derivatives of all the selected problems are available. Since we are interested in large problems, we refined this selection by considering only problems where the number of variables is at least 50. The parameter in line search (3.1) is taken as: 𝛽=1, 𝜌=0.1, and 𝛿=10⁻⁴. The numerical results of the algorithms SSD are listed in Table 1. The columns have the following meanings: Problem: name of the test problem,Dim: dimension of the test problem,Iter: number of iterations,Nf: number of function evaluations,Time: CPU time in seconds,Fv: final function value,Norm-2: 𝑙₂-norm of the final gradient, and Norm-0: ∞-norm of the final gradient.

Table 1 

Test results for SSD method.

In addition, we also present extensive numerical results of the state-of-the-art algorithm CG_DESCENT [8, 14]. CG_DESCENT is a conjugate gradient descent method for solving large-scale unconstrained optimization problems. A new nonmonotone line search were used in CG_DESCENT which makes this algorithm very efficient. The Fortran code can be obtained from http://www.math.ufl.edu/~hager/. When running of CG_DESCENT, default values are used for all parameters. The results are summarized in Table 2.

Table 2 

Test results for CG_DESCENT.

Observing the tables, we see that SSD works well, as it reached a stationary point based on the stopping criterion (4.1) for almost all these test problems. Although SSD requires large number of iterations or function evaluations, it needs less time consuming comparatively. However, the numerical comparisons tell us that the state-of-the-art algorithm CG_DESCENT performs a little better than SSD, but for some specific problems, the enhancement of SSD is still noticeable. The numerical experiments also show that SSD can potentially used to solve unconstrained optimization problems with higher dimensions.

5. Concluding Remarks

In this paper, motivated by a descent technique, we have proposed a simple sufficient descent method for solving large-scale unconstrained optimization problems. At each iteration, the generated directions are only related to the gradient information of two successive points. Under some mild conditions, we have shown that this method is global convergent. The numerical results indicate that it works well on some selected problems from CUTEr library. We think there are several other issues that could be lead to improvements. A first point that should be considered is probably the choice of a nonmonotone line search technique. We have adopted the monotone scheme of Grippo and Lucidi (3.1), but this is not the only possible choice, and the framework developed in [15, 16] may allow one to achieve a better performance. Similar to [5], another important point that should be further investigated is a modification of the search direction without violating the nice sufficient descent property. To this end, it is interesting to note that this method is capable of solving nonlinear systems of equations without Jacobian information and cheaply computing.

Acknowledgments

The authors thank Professor Zengxin Wei of Guangxi University for his careful reading. The authors are also grateful two anonymous referees for their useful comments and suggestions to improve the paper. The work is supported by the Chinese Postdoctoral Foundation Grant no. 20090461094, the Natural Science Foundation of China Grant no. 11001075, the Natural Science Foundation of Henan Province Eduction Department Grant no. 2010B110004.