A Modified Levenberg-Marquardt Method for Nonsmooth Equations with Finitely Many Maximum Functions

Abstract

For solving nonsmooth systems of equations, the Levenberg-Marquardt method and its variants are of particular importance because of their locally fast convergent rates. Finitely many maximum functions systems are very useful in the study of nonlinear complementarity problems, variational inequality problems, Karush-Kuhn-Tucker systems of nonlinear programming problems, and many problems in mechanics and engineering. In this paper, we present a modified Levenberg-Marquardt method for nonsmooth equations with finitely many maximum functions. Under mild assumptions, the present method is shown to be convergent Q-linearly. Some numerical results comparing the proposed method with classical reformulations indicate that the modified Levenberg-Marquardt algorithm works quite well in practice.

1. Introduction

In the past few years, there has been a growing interest in the study of nonlinear equations (see, e.g., [1, 2]) and nonsmooth equations, which have been proposed in the study of the nonlinear complementarity problem, the variational inequality problem, equilibrium problem and engineering mechanics (see, e.g., [3–10]).

Finitely many maximum functions systems are very useful in the study of nonlinear complementarity problems, variational inequality problems, Karush-Kuhn-Tucker systems of nonlinear programming problems, and many problems in mechanics and engineering. In the present paper, we study a new method for nonsmooth equations with finitely many maximum functions system proposed in [11] max_𝑗∈𝐽1𝑓_1𝑗⋮(𝑥)=0,max_{𝑗∈𝐽𝑛}𝑓_𝑛𝑗(𝑥)=0, (1.1) where 𝑓_𝑖𝑗∶𝑅^𝑛→𝑅 for 𝑗∈𝐽_𝑖,𝑖=1,…,𝑛 are continuously differentiable, 𝐽_𝑖 for 𝑖=1,…,𝑛 are finite index sets. Denote 𝐻(𝑥)=(𝑓₁(𝑥),…,𝑓_𝑛(𝑥))^𝑇,𝑥∈𝑅^𝑛, (1.2)where𝑓_𝑖(𝑥)=max_{𝑗∈𝐽𝑖}𝑓_𝑖𝑗(𝑥),𝑥∈𝑅^𝑛𝐽,𝑖=1,…,𝑛,_𝑖(𝑥)={𝑗_𝑖∈𝑁∣𝑓_𝑖𝑗(𝑥)=𝑓_𝑖(𝑥)},𝑥∈𝑅^𝑛,𝑖=1,…,𝑛. (1.3) Then (1.1) can be rewritten as follows:𝐻(𝑥)=0, (1.4)where 𝐹∶𝑅^𝑛→𝑅^𝑛 is a nonsmooth function. By using the following subdifferential for the function 𝐻(𝑥) given in (1.2),𝜕_⋆𝐻(𝑥)={(∇𝑓_1𝑗1,…,∇𝑓_𝑛𝑗𝑛)^𝑇∣𝑗₁∈𝐽₁(𝑥),…,𝑗_𝑛∈𝐽_𝑛(𝑥)},𝑥∈𝑅^𝑛, (1.5)Gao gave Newton method for (1.4) with the superlinear convergence in [11].

Based on [5, 11], we present a modification of the Levenberg-Marquardt method for solving nonsmooth equations. In Section 2, we recall some results of generalized Jacobian and semismoothness. In Section 3, we give the Levenberg-Marquardt method which has been proposed in [5] and the new modified Levenberg-Marquardt method for the system of nonsmooth equations with finitely many maximum functions. The convergence of the modified Levenberg-Marquardt algorithm is also given. In Section 4, some numerical tests comparing the proposed modified Levenberg-Marquardt algorithm with the original method show that our algorithm works quite well.

2. Preliminaries

We start with some notions and propositions, which can be found in [8–11].

Let 𝐹(𝑥) be locally Lipschitzian. Then, 𝐹(𝑥) is almost everywhere F-differentiable. Let the set of points where 𝐹(𝑥) is F-differentiable be denoted by 𝐷_𝐹. Then for 𝑥∈𝑅^𝑛,𝜕_𝐵𝐹(𝑥)={𝑉∈𝑅^𝑛×𝑛∣∃{𝑥_𝑘}∈𝐷_𝐹,𝑥_𝑘→𝑥,𝐹^′(𝑥_𝑘)→𝑉}. (2.1)The general Jacobian of 𝐹(𝑥)∶𝑅^𝑛→𝑅^𝑛 at 𝑥 in the sense of Clarke is defined as𝜕𝐹(𝑥)=conv𝜕_𝐵𝐹(𝑥). (2.2)

Proposition 2.1. 𝜕_𝐵𝐹(𝑥) is a nonempty and compact set for any 𝑥; the point to set 𝐵-subdifferential map is upper semicontinuous.

Proposition 2.2. 𝜕_⋆𝐻(𝑥) is a nonempty and compact set for any 𝑥 and upper semicontinuous.

Proof. From the fact that 𝜕_⋆𝐻(𝑥) is a finite set of points in 𝑅^𝑛×𝑛 and can be calculated by determining the index sets 𝐽_𝑖(𝑥),𝑖=1,…,𝑛 and evaluating the gradients ∇𝑓_𝑖𝑗𝑖(𝑥), 𝑗_𝑖∈𝐽_𝑖(𝑥),𝑖=1,…,𝑛.

Definition 2.3. 𝐹(𝑥) is semismooth at 𝑥 if 𝐹(𝑥) is locally Lipschitz at 𝑥 andlim_{𝑉∈𝜕𝐹(𝑥+𝑡ℎ′)ℎ′→ℎ,𝑡↓0}𝑉ℎ^′ (2.3)exists for all ℎ∈𝑅^𝑛. If 𝐹(𝑥) is semismooth at 𝑥, one knows 𝑉ℎ−𝐹^′(𝑥;ℎ)=𝑜(‖ℎ‖),∀𝑉∈𝜕𝐹(𝑥+ℎ),ℎ→0. If for all 𝑉∈𝜕𝐹(𝑥+ℎ),ℎ→0, 𝑉ℎ−𝐹^′(𝑥;ℎ)=𝑜(‖ℎ‖²), one calls the function 𝐹(𝑥) is strongly semismooth at 𝑥.

Proposition 2.4. (I) If 𝐹(𝑥)∶𝑅^𝑛→𝑅^𝑛 is locally Lipschitz continuous and semismooth at 𝑥, thenlim_{𝑉∈𝜕𝐹(𝑥+𝑡ℎ)ℎ→0}‖𝐹(𝑥+ℎ)−𝐹(𝑥)−𝑉ℎ‖‖ℎ‖=0. (2.4)
(II) If 𝐹(𝑥)∶𝑅^𝑛→𝑅^𝑛 is locally Lipschitz continuous, strongly semismooth at 𝑥, and directionally differentiable in a neighborhood of 𝑥, thenlimsup_{𝑉∈𝜕𝐹(𝑥+𝑡ℎ)ℎ→0}‖𝐹(𝑥+ℎ)−𝐹(𝑥)−𝑉ℎ‖‖ℎ‖²<∞. (2.5)

Lemma 2.5. Equation of maximum functions (1.4) is a system of semismooth equations.

In the study of algorithms for the local solution of semismooth systems of equations, similar to [11], one also has the following lemmas.

Lemma 2.6. Suppose that 𝐻(𝑥) and 𝜕_⋆𝐻(𝑥) are defined by (1.4) and by (1.5), respectively, and all 𝑉∈𝜕_⋆𝐻(𝑥) are nonsingular. Then there exists a constant 𝑐 such that‖𝑉⁻¹‖≤𝑐,∀𝑉∈𝜕_⋆𝐻(𝑥). (2.6)

The proof is similar to [11, Lemma 2.1], from the fact that 𝜕_⋆𝐻(𝑥) is a finite set of points.

Lemma 2.7. Suppose that 𝑥^⋆ is a solution of (1.1), then‖diag(𝜆_𝑖^(𝑘)𝑓_𝑖(𝑥_𝑘))‖≤𝑀, (2.7)for all 𝑥 in some neighborhood of 𝑥^⋆ and 𝜆_𝑖^(𝑘)∈𝑅 and 0<|𝜆_𝑖^(𝑘)|<+∞ for 𝑖=1,…,𝑛, 𝑘=0,1,2,….

Since each 𝑓_𝑖𝑗 of (1.1) is continuous, one gets the lemma immediately.

3. Modified Levenberg-Marquardt Method and Its Convergence

In this section, we briefly recall some results on the Levenberg-Marquardt-type method for the solution of nonsmooth equations and their local convergence (see, e.g., [5, 9]). We also give the modified Levenberg-Marquardt method and analyze its local behavior. Now we consider exact and inexact versions of Levenberg-Marquardt method.

Given a starting vector 𝑥₀∈𝑅^𝑛, let𝑥_𝑘+1=𝑥_𝑘+𝑑_𝑘, (3.1)where 𝑑_𝑘 is the solution of the system((𝑉_𝑘)^𝑇𝑉_𝑘+𝜎_𝑘𝐼)𝑑=−(𝑉_𝑘)^𝑇𝐻(𝑥_𝑘),𝑉_𝑘∈𝜕_𝐵𝐻(𝑥_𝑘),𝜎_𝑘≥0. (3.2)In the inexact versions of this method 𝑑_𝑘 can be given by the solution of the system((𝑉_𝑘)^𝑇𝑉_𝑘+𝜎_𝑘𝐼)𝑑=−(𝑉_𝑘)^𝑇𝐻(𝑥_𝑘)+𝑟_𝑘,𝑉_𝑘∈𝜕_𝐵𝐻(𝑥_𝑘),𝜎_𝑘≥0, (3.3)where 𝑟_𝑘 is the vector of residuals and we can assume ‖𝑟_𝑘‖≤𝛼_𝑘‖(𝑉_𝑘)^𝑇𝐻(𝑥_𝑘)‖ for some 𝛼_𝑘≥0.

We now give the modified Levenberg-Marquardt method for (1.1) as follows.

Modified Levenberg-Marquardt Method
Step 1. Given 𝑥₀, 𝜖>0,𝜆^𝑘_𝑖∈𝑅^𝑛,0<|𝜆^𝑘_𝑖|<+∞.
Step 2. Solve the system to get 𝑑_𝑘,((𝑉_𝑘)^𝑇𝑉_𝑘+diag(𝜆_𝑖^(𝑘)𝑓_𝑖(𝑥_𝑘)))𝑑_𝑘=−(𝑉_𝑘)^𝑇𝐻(𝑥_𝑘)+𝑟_𝑘,𝑉_𝑘∈𝜕_⋆𝐻(𝑥), (3.4)for 𝑖=1,…,𝑛 and 𝑟_𝑘 is the vector of residuals‖𝑟_𝑘‖≤𝛼_𝑘‖(𝑉_𝑘)^𝑇𝐻(𝑥_𝑘)‖,𝛼_𝑘≥0. (3.5)
Step 3. Set 𝑥_𝑘+1=𝑥_𝑘+𝑑_𝑘, if ‖𝐻(𝑥_𝑘)‖≤𝜖, terminate. Otherwise, let 𝑘∶=𝑘+1, and go to Step 2.
Based upon the above analysis, we give the following local convergence result.

Theorem 3.1. Suppose that {𝑥_𝑘} is a sequence generated by the above method and there exist constants 𝑎>0, 𝛼_𝑘≤𝑎 for all 𝑘. Let 𝑥^⋆ be a solution of 𝐻(𝑥)=0, and let all 𝑉∈𝜕_⋆𝐻(𝑥^⋆) be nonsingular. Then the sequence {𝑥_𝑘} converges Q-linearly to 𝑥^⋆ for ‖𝑥₀−𝑥^⋆‖≤𝜖.
Proof. By Lemma 2.6 and the continuously differentiable of 𝑓_𝑖(𝑥), there is a constant 𝐶>0 such that for all 𝑥_𝑘 sufficiently close to 𝑥^⋆(𝑉_𝑘)^𝑇𝑉_𝑘+diag(𝜆_𝑖^(𝑘)𝑓_𝑖(𝑥_𝑘)) are nonsingular with‖[(𝑉_𝑘)^𝑇𝑉_𝑘+diag(𝜆_𝑖^(𝑘)𝑓_𝑖(𝑥_𝑘))]⁻¹‖≤𝐶. (3.6)Furthermore, by Proposition 2.4, there exists 𝛿>0, which can be taken arbitrarily small, such that‖𝐻(𝑥_𝑘)−𝐻(𝑥^⋆)−𝑉_𝑘(𝑥_𝑘−𝑥^⋆)‖≤𝛿‖𝑥_𝑘−𝑥^⋆‖ (3.7)for all 𝑥_𝑘 in a sufficiently small neighborhood of 𝑥^⋆ depending on 𝛿. By Proposition 2.2 the upper semicontinuity of the 𝜕_⋆𝐻(𝑥), we also know‖(𝑉_𝑘)^𝑇‖≤𝑐₁, (3.8)for all 𝑉_𝑘∈𝜕_⋆𝐻(𝑥) and all 𝑥_𝑘 sufficiently close to 𝑥^⋆, with 𝑐₁>0 being a suitable constant. From the locally Lipschitz continuous of 𝐻(𝑥), we have‖(𝑉_𝑘)^𝑇𝐻(𝑥_𝑘)‖≤‖(𝑉_𝑘)^𝑇‖‖𝐻(𝑥_𝑘)−𝐻(𝑥^⋆)‖≤𝑐₁𝐿‖𝑥_𝑘−𝑥^⋆‖, (3.9)for all 𝑥_𝑘 in a sufficiently small neighborhood of 𝑥^⋆ and a constant 𝐿>0. From (3.4), we also know[(𝑉_𝑘)^𝑇𝑉_𝑘+diag(𝜆_𝑖^(𝑘)𝑓_𝑖(𝑥_𝑘))](𝑥_𝑘+1−𝑥^⋆)=[(𝑉_𝑘)^𝑇𝑉_𝑘+diag(𝜆_𝑖^(𝑘)𝑓_𝑖(𝑥_𝑘))](𝑥_𝑘−𝑥^⋆)−(𝑉_𝑘)^𝑇𝐻(𝑥_𝑘)+𝑟_𝑘=(𝑉_𝑘)^𝑇[𝐻(𝑥^⋆)−𝐻(𝑥_𝑘)+𝑉_𝑘(𝑥_𝑘−𝑥^⋆)]+diag(𝜆_𝑖^(𝑘)𝑓_𝑖(𝑥_𝑘))(𝑥_𝑘−𝑥^⋆)+𝑟_𝑘. (3.10)Multiply the above equation by [(𝑉_𝑘)^𝑇𝑉_𝑘+diag(𝜆_𝑖^(𝑘)𝑓_𝑖(𝑥_𝑘))]⁻¹ and taken into account Lemma 2.7, and (3.6), (3.7), (3.8), and (3.9), we get‖𝑥_𝑘+1−𝑥^⋆‖≤𝐶(‖(𝑉_𝑘)^𝑇‖‖𝐻(𝑥_𝑘)−𝐻(𝑥^⋆)−𝑉_𝑘(𝑥_𝑘−𝑥^⋆)‖+‖diag(𝜆_𝑖^(𝑘)𝑓_𝑖(𝑥_𝑘))‖‖𝑥_𝑘−𝑥^⋆‖+𝑎‖(𝑉_𝑘)^𝑇𝐻(𝑥_𝑘)‖)≤𝐶(𝑐₁𝛿‖𝑥_𝑘−𝑥^⋆‖+𝑀‖𝑥_𝑘−𝑥^⋆‖+𝑎𝑐₁𝐿‖𝑥_𝑘−𝑥^⋆‖)=𝐶(𝑐₁𝛿+𝑀+𝑎𝑐₁𝐿)‖𝑥_𝑘−𝑥^⋆‖. (3.11)Let 𝜏=𝐶(𝑐₁𝛿+𝑀+𝑎𝑐₁𝐿), so‖𝑥_𝑘+1−𝑥^⋆‖≤𝜏‖𝑥_𝑘−𝑥^⋆‖. (3.12)Since 𝛿 can be chosen arbitrarily small, by taking 𝑥_𝑘 sufficiently close to 𝑥^⋆, there exist 𝑀>0 and 𝑎>0 such that 𝜏<1, so that the Q-linear convergence of {𝑥_𝑘} to 𝑥^⋆ follows by taking ‖𝑥₀−𝑥^⋆‖≤𝜖 for a small enough 𝜖>0. Thus we complete the proof of the theorem.

Theorem 3.2. Suppose that {𝑥_𝑘} is a sequence generated by the above method and there exist constants 𝑎>0, 𝛼_𝑘≤𝑎 for all 𝑘. ‖𝑟_𝑘‖≤𝛼_𝑘‖𝐻(𝑥_𝑘)‖,𝛼_𝑘≥0. Then the sequence {𝑥_𝑘} converges Q-linearly to 𝑥^⋆ for ‖𝑥₀−𝑥^⋆‖≤𝜖.

The proof is similar to that of Theorem 3.1, so we omit it.

Following the proof of Theorem 3.1, the following statement holds.

Remark 3.3. Theorems 3.1 and 3.2 hold with ‖𝑟_𝑘‖=0 in (3.4).

4. Numerical Test

In order to show the performance of the modified Levenberg-Marquardt method, in this section, we present numerical results and compare the Levenberg-Marquardt method and modified Levenberg-Marquardt method. The results indicate that the modified Levenberg-Marquardt algorithm works quite well in practice. All the experiments were implemented in Matlab 7.0.

Example 4.1. max{𝑓₁₁(𝑥₁,𝑥₂),𝑓₁₂(𝑥₁,𝑥₂)}=0,max{𝑓₂₁(𝑥₁,𝑥₂),𝑓₂₂(𝑥₁,𝑥₂)}=0, (4.1) where𝑓₁₁=15𝑥²₁,𝑓₁₂=𝑥²₁,𝑓₂₁=13𝑥²₁,𝑓₂₂=𝑥²₁. (4.2)From (1.1), we know𝐻(𝑥)=(𝑓₁(𝑥),𝑓₂(𝑥))^𝑇, (4.3)where 𝑓₁(𝑥)=𝑥²₁,𝑓₂(𝑥)=𝑥²₁,𝑥∈𝑅².

Our subroutine computes 𝑑_𝑘 such that (3.3) and (3.4) hold with 𝛼_𝑘=0. We also use the condition ‖𝑥_𝑘−𝑥_𝑘−1‖≤10⁻⁴ as the stopping criterion. We can see that our method is good for Example 4.1.

Results for Example 4.1 with initial point 𝑥₀=(1,1)^𝑇𝜆₁=0.01,𝜆₂=1 and computes 𝑑_𝑘 by (3.4) are listed in Table 1.

Table 1 

𝑥₀=(1,1)^𝑇𝜆₁=0.01,𝜆₂=1 and computes 𝑑_𝑘 by (3.4).

Results for Example 4.1 with initial point 𝑥₀=(1,1)^𝑇𝜎₁=𝜆₁=0.01,𝜎₂=𝜆₂=1 and computes 𝑑_𝑘 by (3.3) are listed in Tables 2 and 3.

Table 2 

𝑥₀=(1,1)^𝑇𝜎₁=0.01,𝜎₂=1 and computes 𝑑_𝑘 by (3.3).

Table 3 

𝑥₀=(1,1)^𝑇𝜎₁=0.01,𝜎₂=1 and computes 𝑑_𝑘 by (3.3).

Results for Example 4.1 with initial point 𝑥₀=(10,1)^𝑇𝜆₁=0.01,𝜆₂=1 and computes 𝑑_𝑘 by (3.4) are listed in Tables 4 and 5.

Table 4 

𝑥₀=(10,1)^𝑇𝜆₁=0.01,𝜆₂=1 and computes 𝑑_𝑘 by (3.4).

Table 5 

𝑥₀=(10,1)^𝑇𝜆₁=0.01,𝜆₂=1 and computes 𝑑_𝑘 by (3.4).

Results for Example 4.1 with initial point 𝑥₀=(10,1)^𝑇𝜎₁=𝜆₁=0.01,𝜎₂=𝜆₂=1 and computes 𝑑_𝑘 by (3.3) are listed in Tables 6 and 7.

Table 6 

𝑥₀=(10,1)^𝑇𝜆₁=0.01,𝜆₂=1 and computes 𝑑_𝑘 by (3.3).

Table 7 

𝑥₀=(10,1)^𝑇𝜎₁=0.01,𝜎₂=1 and computes 𝑑_𝑘 by (3.3).

Results are shown for Example 4.1 with initial point 𝑥₀=(100,1)^𝑇. We also use the condition ‖𝑥_𝑘−𝑥_𝑘−1‖≤10⁻⁴ as the stopping criterion and computes 𝑑_𝑘 by (3.4) we get that by 21 steps 𝐹(𝑥)=1.0𝑒−008∗(0.9560,0.9560)^𝑇. When we compute 𝑑_𝑘 by (3.3), we get that by 45 steps 𝐹(𝑥)=1.0𝑒−004∗(0.3919,0.3919)^𝑇. We can test the method with other examples and will think the global convergence of the method in another paper.

Acknowledgments

This work was supported by National Science Foundation of China (under Grant: 10671126). The Innovation Fund Project for Graduate Student of Shanghai (JWCXSL0801) and key project for Fundamental Research of STCSM (Project no. 06JC14057) and Shanghai Leading Academic Discipline Project (S30501). The authors are also very grateful to referees for valuable suggestions and comments.