A New BFGS Algorithm Using the Decomposition Matrix of the Correction Matrix to Obtain the Search Directions

Abstract

We present an improved method for determining the search direction in the BFGS algorithm. Our approach uses the equal inner product decomposition method for positive-definite matrices. The decomposition of an approximated Hessian matrix expresses a correction formula that is independent from the exact line search. This decomposed matrix is used to compute the search direction in a new BFGS algorithm.

1. Introduction

Using the equal inner product decomposition algorithm [1–3] for positive-definite matrices with a vector, a novel quasi-Newton method has been proposed [4, 5]. In this algorithm, the equal inner product decomposition matrix of the correction matrix is utilized to obtain the search directions. That is, for the unrestricted optimization problem where is a continuously differentiable function; the equal inner product decomposition correction formula for the approximate Hessian matrix with the objective function gradient vector is used to compute the search directions.

The BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm [6–9] is a typical quasi-Newton method. If we consider to be an iteration point, and , and , , then the correction formulae for the approximate Hessian matrix and its inverse can be expressed as In computing the search direction at the iteration point , we generally solve the equation: but we would rather avoid using (3) to calculate There are clearly more computations involved in (4) than in (5). We take this apparently more complex route because may appear to be non-positive-definite or even singular when we calculate (3). Equation (2) can be transformed to Using this form, Gill and Murray [10] developed a method for solving (4) by Cholesky decomposition. First, they found the Cholesky decomposition of  : where is a unit lower triangular matrix and is a diagonal matrix, and then they successively solved Because (6) establishes the iterative formula from and to and , the Cholesky decomposition of need not be repeated on each iteration. The detailed calculation method is described in [10–12].

Unlike the above approach, the algorithm proposed in [4, 5] utilizes the equal inner product decomposition algorithm for a positive-definite matrix with a vector. In this improved technique, the matrix is computed. This is the equal inner product decomposition matrix of with ; that is, where , . The search direction at the iteration point is then given by where is the sum of the elements in the th column of .

Instead of using the correction formula (2), the method of [4, 5] directly establishes the correction formula from to , which are equal inner product decomposition matrixes of with the objective function gradient vectors. When the correction matrix is calculated, the search direction is immediately given by (10).

Thus, [4, 5] presented a novel approach to the computation of search directions. However, all the conclusions in [4, 5] relate to the exact line search condition. Therefore, this algorithm is feasible in theory but has restricted applicability.

The BFGS algorithm with the Wolfe line search has global, superlinear convergence [13–15]. Compared with other quasi-Newton methods, BFGS is the most popular and efficient algorithm. In this paper, we focus on the BFGS algorithm and set up a correction formula expressed by the decomposition matrix that is independent of the exact line search. This leads to a new BFGS algorithm that computes the search directions using the equal inner product decomposition matrix of the approximate Hessian matrix.

In Section 3, the procedures of computing search directions in the usual BFGS and in the new BFGS have been compared by a computing example.

2. Correction Formula Expressed by Decomposition Matrix

At the iteration point , we use the method of [3] to directly compute the equal inner product decomposition matrix of   with . However, this decomposition method is computationally expensive. One alternative is to set up a similar correction formula between and as that between and or between and .

We denote , , and . By Theorem 2.2 in [4], if the equal inner product decomposition matrix of with has already been computed in the exact line search condition for the BFGS correction formula (2), then the equal inner product decomposition matrix of with can be expressed as where , ,, and . By Theorem 2.3 in [4], the equal inner product decomposition matrix of the initial correction matrix with is where .

As defined by (11), only satisfies under the exact line search condition, that is, when . If the iteration point from to is obtained through an inexact line search, then obviously in (11) is no longer the decomposition matrix of defined by (2). The following conclusions are independent of the exact line search but are dependent on the correction matrix definition for BFGS.

2.1. Main Results

Lemma 1. Suppose . If  , , , then the matrix is nonsingular and

The proof is simple and is therefore omitted from this paper.

Lemma 2. For the BFGS correction formula (11), if   is the equal inner product decomposition matrix of   with , we denote Then, and

Proof. Because , where is the step length, we have . By a direct calculation, we have Therefore, holds. Since Lemma 1 implies Thus, we have . This completes the proof.

Lemma 3. Writing , where , if , one has .

Proof. A direct computation gives . Thus, we have This completes the proof.

Writing where , , we can state the following results.

Theorem 4. For the BFGS correction formula (2), if the equal inner product decomposition matrix of with has already been obtained, then the equal inner product decomposition matrix of   with is where and .

Proof. By Lemma 2 and (21), we have Because is the Householder matrix, . Thus, we have Therefore, Lemma 2 gives . By Lemma 3, we get where . Thus, defined by (21) is the equal inner product decomposition matrix of with . This completes the proof.

From the above proof, it can be found that in (21) is the equal inner product decomposition matrix of defined by (2), which is independent of the exact line search.

Corollary 5. Under the exact line search condition, (21) is equivalent to (11); that is, (11) is a special case for (21).

Proof. Under the exact line search condition, we have and .
From (20), we have Thus, , where .
Therefore, . Under the exact line search condition, . From the above results, (11) can be directly deduced from (21). This completes the proof.

2.2. Extension

We can extend the conclusion of Theorem 4 to the following BFGS correction formula using self-scaling [16–18]: where and . When , , so (26) can be written as Equation (27) is well known for its practical applicability. Here, we have only transformed (27) into the form expressed by the decomposition correction matrix. When , similar methods can be used.

Applying the Sherman-Morrison formula [13] to (27), we can deduce the correction matrix for : Writing , where and , we can state the following for the inexact line search condition.

Theorem 6. For the self-scaling BFGS correction formula (27), if the equal inner product decomposition matrix of with is already known, then the equal inner product decomposition matrix of   with is where , , and for step length .

Proof. The proof is similar to that for Theorem 4.
Using (29), we have Because of (30) gives Therefore, and we have that is, , where . This completes the proof.

Theorem 6 admits the following corollary.

Corollary 7. Under the exact line search condition, if the equal inner product decomposition matrix of with for the self-scaling BFGS correction formula (27) is already known, then the equal inner product decomposition matrix of   with is where , , and   for step length .

Proof. The proof is simple and is therefore omitted.

3. A New BFGS Algorithm

3.1. Algorithm

Replacing the corresponding steps of the BFGS algorithm by (12), (21), and (10) establishes a new BFGS algorithm that uses the decomposition correction matrix to determine the search direction.

The New BFGS Algorithm

Step 1. Choose a start point , , and compute . If , terminate the algorithm; otherwise, go to Step 2.

Step 2. Let . For the initial matrix , form the equal inner product decomposition with to give the decomposition matrix and .

Step 3. Compute the search direction as where , is the sum of the elements in the th column of  .

Step 4. Perform a one-dimensional line search (exact line search or Wolfe line search), and compute the step length factor .

Step 5. Let . Compute . If , terminate the algorithm; otherwise, go to Step 6.

Step 6. Using (21), calculate the equal inner product decomposition matrix of with and the coefficient .

Step 7. Let and go to Step 3.

3.2. Example

Here, by an example, we compare the two methods of computing the search directions between the usual BFGS algorithm [10–12] and the new BFGS algorithm. Considering the optimization problem [19], when solving the problems with BFGS algorithm, from the second iterative computation, all the following iteration procedures are similar, so the difference of the two algorithms can be clearly seen by one iterative computing procedure. Here we assume that the first iterative computing procedure is finished and at the first iterative computing, we choose the initial point , the initial matrix . Noting , the initial searching direction is At the point , we have done the Wolfe-Powell search along the and obtained the step length which satisfies the Wolfe-Powell conditions [13]. Thus we get

Now we compare the two different algorithms of computing search directions by the second iterative computation.

We first compute the search direction with the usual BFGS algorithm.

Step 1. Compute Cholesky decomposition of  .
Obviously, the Cholesky decomposition of   is
Let , definite , and .
Compute the following in turn [13]: Hence, Then let . Computing the equation we get .
Compute the following in turn [13]: Thus we obtain Therefore, we get the Cholesky decomposition of :

Step 2. Solve the following equations: From , there is Solving the equations, we get . Then from , there is Solving the equations, we get the search direction

We then use the new BFGS algorithm to compute the search direction .

Step 1. First compute the initial correction matrix and the coefficient .
For , we get

Step 2. By , we compute .
Make use of known data, and we get Thus there is Therefore we have Because we obtain And , , so Thus we can obtain , in which Therefore, the search direction is

3.3. Analysis

The above example clearly shows the difference between the two algorithms in computing the search directions. In the new algorithm, correction matrixes are directly used to compute search directions. There is no procedure of Cholesky decomposition and no procedure of solving equations. Thus it simplifies the computing procedure of the search directions.

To compute the decomposition correction matrix at point using the new BFGS algorithm, we first need to compute a Householder matrix . This increases the computational complexity. However, the search direction can be directly computed by (10) at the same time. Consequently, the two steps of the correction matrix calculation and search direction calculation are performed concurrently. In practice, the computation of  (21) can be simplified using the properties of the Householder matrix.

Let the row vectors of be denoted as ; that is, Thus, we can write where are the components of . Therefore, we have where . Hence, As a result, after and have been computed, the decomposition correction matrix can be directly obtained from (62), removing the need to compute the Householder matrix .

We can also compare the condition numbers of   and in the correction formulae (3) and (21). The condition number is directly related to the possible magnification in relative error of the matrix calculations. Thus, a smaller condition number implies better numerical stability.

Because the correction matrix is symmetric and positive-definite, if the eigenvalues , of have size order then the condition number of in terms of the spectral norm [20] is Because , and therefore Clearly, only when is a diagonal matrix whose diagonal elements are all the same. Otherwise, . Furthermore, and , which is beneficial to the calculation stability of (21). Note that (66) holds for all , so the numerical stability of correction formula (21) should be much better than that of (3).

In addition, the iterative calculation of the decomposition correction matrix involves only nonsingular matrices. Thus, the calculation accuracy requirements should be lower than in the original BFGS algorithm, which considers symmetric positive-definite matrices.

Conflict of Interests

The author declares that there is no conflict of interests regarding the publication of this paper.

Acknowledgment

This research was financially supported by Twelfth Five-Year-Plan Science and Technology Research Project of Educational Department, Jilin Province (2014598).