Eigenvalue Decomposition-Based Modified Newton Algorithm

Abstract

When the Hessian matrix is not positive, the Newton direction may not be the descending direction. A new method named eigenvalue decomposition-based modified Newton algorithm is presented, which first takes the eigenvalue decomposition of the Hessian matrix, then replaces the negative eigenvalues with their absolute values, and finally reconstructs the Hessian matrix and modifies the searching direction. The new searching direction is always the descending direction. The convergence of the algorithm is proven and the conclusion on convergence rate is presented qualitatively. Finally, a numerical experiment is given for comparing the convergence domains of the modified algorithm and the classical algorithm.

1. Introduction

Newton algorithm (also known as Newton’s method) is commonly used in numerical analysis, especially in nonlinear optimization [1]. But, in the definition domain of the object function, the method requires that (1) object function 𝑓(𝐱) is twice differentiable, (2) the Hessian matrix must be positive, and (3) the initial solution 𝐱⁰ should be near the extreme point 𝐱^𝑘 [2]. If there is a point in the definition domain whose Hessian matrix is not positive, then the Newton direction of this point is not the descending direction. So the classical algorithm may not be convergent in the definition domain.

There are several improved forms of the classical Newton algorithm on the above problem: (1) combining Newton direction with the steepest descent direction [3]; (2) combining Newton direction with the negative curvature direction [4]; (3) adding a diagonal matrix 𝑣_𝑘𝐈 to the Hessian matrix so that the sum matrix is positive, where 𝑣_𝑘 is a positive number bigger than the absolutely value of the smallest eigenvalue [5]. This paper presents a new algorithm to handle the “nonpositive problem.”

2. Newton Algorithm

We discuss the extreme value problems of multi-function. Suppose object function 𝑓(𝐱) is a twice differentiable function, and the extreme value problem of 𝑓(𝐱) is min_𝐱𝑓(𝐱). (1)

If the Hessian matrix of 𝑓(𝐱) is positive, the Taylor expansion to the second order of 𝑓(𝐱) is: 𝑓(𝐱)≈𝑓𝐱^𝑘+∇𝑓𝐱^𝑘𝐱−𝐱^𝑘+12𝐱−𝐱^𝑘𝑇∇²𝑓𝐱^𝑘𝐱−𝐱^𝑘. (2)

As we know, ∇_𝐱𝑓(𝐱)=0 is the necessary condition, so the derivative form of (2) is ∇_𝐱𝑓(𝐱)=0+∇𝑓𝐱^𝑘+𝐱−𝐱^𝑘𝑇∇²𝑓𝐱^𝑘=0. (3)

Equation (3) is usually called Newton's equation. When matrix ∇²𝑓(𝐱^𝑘) is positive, and the inverse matrix [∇²𝑓(𝐱^𝑘)]⁻¹ exists, we can get an iterative equation as follows: 𝐱=𝐱^𝑘+∇²𝑓(𝐱^𝑘)⁻¹−∇𝑓𝐱^𝑘. (4)

Equation (4) is usually called Newton iterative algorithm format, where −∇𝑓(𝐱^𝑘) is called negative gradient direction, and [∇²𝑓(𝐱^𝑘)]⁻¹[−∇𝑓(𝐱^𝑘)] is called Newton direction.

Next, we must give a proper step length to ensure the value of function declining. A well-known algorithm is “Damped Newton Algorithm,” which gets the step length by solving the following optimization problem: 𝛼=argmin_𝛼𝐱^𝑘+𝛼∇²𝑓(𝐱^𝑘)⁻¹−∇𝑓𝐱^𝑘. (5)

3. Modified Newton Algorithm Based on Eigenvalue Decomposition

3.1. Modified Algorithm

First, take eigenvalue decomposition on the Hessian matrix [∇²𝑓(𝐱^𝑘)]⁻¹: ∇²𝑓(𝐱^𝑘)⁻¹=𝐔Σ𝐔^𝐻, (6) where 𝐔 is a unitary matrix; Σ is a diagonal matrix. Because matrix [∇²𝑓(𝐱^𝑘)]⁻¹ is not positive, there must be a few negative eigenvalues in Σ. We replace the negative eigenvalues with their absolutely values, so the modified searching direction is: 𝐝^𝑘=𝐔|Σ|𝐔^𝐻−∇𝑓𝐱^𝑘. (7)

The inner product of the negative gradient direction with 𝐝^𝑘 is always positive, because the right part of the next equation is a quadratic form [6]: −∇𝑓𝐱^𝑘,𝐔|Σ|𝐔^𝐻−∇𝑓𝐱^𝑘=−∇𝑓𝐱^𝑘𝑇𝐔|Σ|𝐔^𝐻−∇𝑓𝐱^𝑘>0. (8) Equation (8) shows that the angle between the negative gradient direction and 𝐝^𝑘 is less than 90°, so the new direction 𝐝^𝑘 is always a descent direction. Then we can get the similar iterative equation by replacing the Newton direction with 𝐝^𝑘, that is, 𝐱=𝐱^𝑘+𝛼𝐝^𝑘, (9) where 𝛼 is the step length obtained by solving optimization problem (5).

3.2. Convergence Conclusion

We prove that the modified Newton algorithm based on eigenvalue decomposition is convergent. First, we talk about Wolfe-Powell condition [7].

If step length 𝛼_𝑘>0 satisfies the two following equations: 𝑓𝐱^𝑘+𝛼_𝑘𝐝^𝑘−𝑓𝐱^𝑘≤𝜌𝛼_𝑘∇𝑓𝐱^𝑘𝑇𝐝^𝑘,∇𝑓𝐱^𝑘+𝛼_𝑘𝐝^𝑘𝑇𝐝^𝑘≥𝜎∇𝑓𝐱^𝑘𝑇𝐝^𝑘. (10)

Equation (10) is named Wolfe-Powell condition, where 𝜌∈(0,1) and 𝜎∈(𝜌,1).

Theorem 1. Let 𝑓 be a continuously differentiable function, and let 𝛼 be defined by 𝛼=min𝛼≥0∣𝑓𝐱^𝑘+𝛼𝐝^𝑘=𝑓𝐱^𝑘+𝛼𝜌∇𝑓𝐱^𝑘𝑇𝐝^𝑘, (11) where 𝛼 has a bound. Then there must be a finite interval (𝑎,𝑏)⊆[0,𝛼], for all 𝛼_𝑘∈(𝑎,𝑏), the Wolfe-Powell conditions are right.

Proof. Let 𝜙(𝑡)≜𝑓(𝐱^𝑘+𝑡𝐝^𝑘), as 𝛼>0 and satisfies the following equation: 𝜙(𝛼)≜𝜙(0)+𝛼𝜌𝜙^′(0). (12) Then 𝜙(𝛼)<𝜙(0)+𝛼𝜌𝜙^′(0)∀𝛼∈(0,𝛼). (13) According to the Mean Value Theorem, there must be 𝛼∈(0,𝛼) satisfies the following equation: 𝜙(𝛼)−𝜙(0)𝛼<𝜙^′(𝛼)<0. (14) Notice 𝜌𝜙^′(0)=𝜙^′(𝛼), and 𝜌<𝜎, so we can get: 𝜙^′(𝛼)>𝜎𝜙^′(0). (15) Notice 𝜙 is a continuously differentiable operator, so there is an open set (𝑎,𝑏) satisfies (𝑎,𝑏)⊂(0,𝛼), where 𝛼∈(𝑎,𝑏). For any 𝛼_𝑘 when 𝛼_𝑘∈(𝑎,𝑏), the following equations are right: (1)𝜙𝛼_𝑘≤𝜙(0)+𝛼_𝑘𝜙^′(0),(2)𝜙^′𝛼_𝑘≥𝜎𝜙^′(0). (16)
Finally, we can get the Wolfe-Powell condition.

Theorem 2. Let 𝑓∶𝐑^𝑛→𝐑 be a continuously differentiable function, so 𝑓 has lower bounded in 𝐑^𝑛; ∇𝑓 is uniformly continuous on 𝑂, which contains a horizontal set 𝐿𝑒𝑣_𝛼0∶={𝐱∈𝐑^𝑛∣𝑓(𝐱)≤𝛼₀}, where 𝛼₀=𝑓(𝐱⁰) and 𝐱⁰∈𝐑^𝑛. If 𝐝^𝑘 satisfy the following conditions: 𝜃^𝑘≤𝜋/2−𝜇, where 𝜃^𝑘 stands for the angle between −∇𝑓(𝐱^𝑘) and 𝐝^𝑘, and 𝜇∈(0,𝜋/2) is a constant, then there must be a 𝑘 satisfying ∇𝑓(𝐱^𝑘)=0 or ∇𝑓(𝐱^𝑘)→0 by Wolfe-Powell conditions.

Proof. We proof the theorem by contradiction. Suppose ∇𝑓(𝐱^𝑘)≠0 or ∇𝑓(𝐱^𝑘)↛0 for all 𝑘, and there is a set {𝑘_𝑖} and a constant 𝜀>0 satisfy ‖∇𝑓(𝐱^𝑘𝑖)‖≥𝜀 for all 𝑖. Because 𝐝^𝑘 satisfies the angle condition 𝜃^𝑘≤𝜋/2−𝜇, so ∇𝑓𝐱^𝑘𝑇𝐝^𝑘=−∇𝑓𝐱^𝑘𝐝^𝑘cos𝜃_𝑘≤−∇𝑓𝐱^𝑘𝐝^𝑘sin𝜇<0. (17)
Equation (17) shows that 𝐝^𝑘 is a descending direction, so {𝑓(𝐱^𝑘)} is a decreasing sequence. Suppose the limit of {𝑓(𝐱^𝑘)} is 𝜉, and 𝜉>−∞. That is 𝑓(𝐱^𝑘)→𝜉, so 𝑓(𝐱^𝑘)−𝑓(𝐱^𝑘+1)→0, take the Wolfe-Powell condition (16)(1) on index 𝑘_𝑖, 𝑓𝐱^𝑘𝑖−𝑓𝐱^𝑘𝑖+1≥−𝜌𝛼_𝑘𝑖∇𝑓𝐱^𝑘𝑖𝑇𝐝^𝑘𝑖≥∇𝑓𝐱^𝑘𝑖𝛼_𝑘𝑖𝐝^𝑘𝑖cos𝜃_𝑘𝑖≥𝜀sin𝜇𝛼_𝑘𝑖𝐝^𝑘𝑖. (18) It is easy to get ‖𝛼_𝑘𝑖𝐝^𝑘𝑖‖→0, again by the Wolfe-Powell condition (16)(2), ∇𝑓(𝐱^𝑘𝑖+1)−∇𝑓(𝐱^𝑘𝑖)^𝑇𝐝^𝑘𝑖≥−(1−𝜎)∇𝑓𝐱^𝑘𝑖𝑇𝐝^𝑘𝑖. (19) Further ∇𝑓𝐱^𝑘𝑖+1−∇𝑓𝐱^𝑘𝑖≥𝜀(1−𝜎)sin𝜇. (20)
Because of ‖𝐱^𝑘𝑖+1−𝐱^𝑘𝑖‖=‖𝛼_𝑘𝑖𝐝^𝑘𝑖‖→0, (20) is in contradiction with the assumption that ∇𝑓 is uniformly continuous on 𝐿𝑒𝑣_𝛼0. The contradiction suggests that ∇𝑓(𝐱^𝑘)→0.

4. Simulations on Convergence Properties

As the new searching direction 𝐝^𝑘 satisfies the following equation: 𝐝^k=𝐔Σ𝐔^𝐻−∇𝑓𝐱^𝑘if∇²𝑓𝐱^𝑘𝐔|Σ|𝐔>0_𝐻−∇𝑓𝐱^𝑘else. (21)

So, if the Hessian matrix of all points in definition domain is positive, then the convergence rate of modified algorithm equals the classical Newton algorithm. The other side, if there are any points with its Hessian matrix negative, then the convergence rate at most equals the one of classical Newton algorithm.

In addition, in order to verify that the improved algorithm has a wider convergence domain than the classical Newton algorithm, we employ a convex function: 𝑓(𝑥,𝑦)=−3(1−𝑥)²𝑒^{−𝑥2−(𝑦+1)2} as the objective function. Let the definition domain be (𝑥,𝑦)∈{[−3,1],[−3,1]}, and we choose the grid points by 0.1 interval, then use both classical Newton's algorithm and eigenvalue decomposition-based algorithm to find the minimum value. The results are drawn in Figure 1.

(a)

(b)

(c)

(d)

Figure 1 

Results of numerical experiment (a) objective function; (b) contour map; (c) convergence domain of Newton's algorithm; (d) convergence domain of the new algorithm.

In Figure 1(c), the green point means that the algorithm has found the minimum value using this point as the initial solution; and the red point means that the algorithm did not converge or has not found the minimum value. Obviously, the convergence domain of eigenvalue decomposition algorithm is bigger than the one of the classical Newton algorithm.

5. Conclusion

A new algorithm is put forward based on eigenvalue decomposition and classical Newton algorithm. The eigenvalue decomposition based-algorithm has a wider convergence domain than the classical Newton algorithm.

Acknowledgments

This work is supported by the Natural Science Foundation of China (60974124), the Program for New Century Excellent Talents in University, and the Project sponsored by SRF for ROCS, SEM in China, the key lab open foundation for Space Flight Dynamics technique (SFDLXZ-2010-004).