A New Smoothing Nonlinear Conjugate Gradient Method for Nonsmooth Equations with Finitely Many Maximum Functions

Abstract

The nonlinear conjugate gradient method is of particular importance for solving unconstrained optimization. Finitely many maximum functions is a kind of very useful nonsmooth equations, which is very useful in the study of complementarity problems, constrained nonlinear programming problems, and many problems in engineering and mechanics. Smoothing methods for solving nonsmooth equations, complementarity problems, and stochastic complementarity problems have been studied for decades. In this paper, we present a new smoothing nonlinear conjugate gradient method for nonsmooth equations with finitely many maximum functions. The new method also guarantees that any accumulation point of the iterative points sequence, which is generated by the new method, is a Clarke stationary point of the merit function for nonsmooth equations with finitely many maximum functions.

1. Introduction

In this paper, we consider the following nonsmooth equations with finitely many maximum functions: max_𝑗∈𝐽1ℎ_1𝑗(𝑥⋮)=0,max_{𝑗∈𝐽𝑛}ℎ_𝑛𝑗(𝑥)=0, (1) where ℎ_𝑖𝑗∶𝑅^𝑛→𝑅 for 𝑖=1,…,𝑛 and 𝑗∈𝐽_𝑖,  𝑖=1,…,𝑛 are continuously differentiable functions, and 𝐽_𝑖 are finite index sets. Denote that ℎ_𝑖(𝑥)=max_{𝑗∈𝐽𝑖}ℎ_𝑖𝑗(𝑥𝐻(𝑥)=ℎ),𝑖=1,…,𝑛,₁(𝑥),…,ℎ_𝑛(𝑥)^𝑇, (2) then (1) can be reformulated as 𝐻(𝑥)=0, (3) where 𝐻∶𝑅^𝑛→𝑅^𝑛 is nonsmooth equations and 𝐽_𝑖(𝑥)={𝑗_𝑖∈𝑁∣ℎ_𝑖𝑗(𝑥)=ℎ_𝑖(𝑥)},  𝑖=1,…,𝑛. Nonsmooth equations have been studied for decades, which is proposed in the study of the optimal control, the variational inequality and complementarity problems, equilibrium problems, and engineering mechanics [1–3], because many practical problems, such as stochastic complementarity problems, variational inequality problems, KKT systems of constrained nonlinear programming problems, and many problems in equilibrium problems, can be reformulated into (1). In the past few years, there has been a growing interest in the study of (1) (such as [4, 5]). Due to its simplicity and global convergence, the iterative methods, such as nonsmooth Levenberg-Marquardt method, general Newton method, and smoothing methods for solving nonsmooth equations, have been widely studied [4–11].

In this paper, we give a new smoothing nonlinear conjugate gradient method for (1). In the following section, we will recall some definitions and some background of nonlinear conjugate gradient methods. And we also give the new smoothing nonlinear conjugate gradient method for (1), which guarantees that any accumulation point of the iterative points sequence is a Clarke stationary point of the merit function for (1). In the last section, some discussions are also given.

Notation. In the following, a quantity with a subscript 𝑘 denotes that quantity is evaluated at  𝑥_𝑘, the norm is the 2 norm, and 𝑅₊₊={𝑡∣𝑡>0}.

2. Preliminaries and New Method

In this section, firstly, we give some definitions and some backgrounds of nonlinear conjugate gradient method. Secondly, we propose the new methods for (1) and give the convergence analysis.

In the following, we give two definitions, which will be used in this paper.

Definition 1. Let 𝐹∶𝑅^𝑛→𝑅 be a locally Lipschitz function. Then, 𝐹 is almost everywhere differentiable. Denote 𝐷_𝐹 be the set of points where 𝐹 is differentiable, then the general gradient of 𝐹 at 𝑥 in the sense of Clark is 𝜕𝐹(𝑥)=convlim_{𝑥𝑘→𝑥,𝑥𝑘∈𝐷𝐹}∇𝐹𝑥_𝑘, (4) where conv denotes the convex set.

Definition 2. Let 𝑟∶𝑅^𝑛→𝑅 be a locally Lipschitz continuous function. We call 𝑟^∼∶𝑅^𝑛×𝑅₊₊→𝑅 a smoothing function of  𝑟, if  𝑟^∼(⋅,𝜇)  is continuously differentiable for any fixed 𝜇∈𝑅₊₊ and lim_𝑥→𝑥^∼,𝜇↓0𝑟𝑥,𝜇=𝑟(𝑥). (5)

In the following, we will give the new method for (1). In order to describe the method clearly, we divide this section into two parts. In Case 1, we give the new nonlinear conjugate gradient method for smooth objective function. In Case 2, we give the new smoothing nonlinear conjugate gradient method for nonsmooth objective function.

Denote that 𝑓(𝑥)=12‖𝐻(𝑥)‖², (6) where 𝐻 is defined in (3). Then (1) is equivalent to the following unconstrained minimization problem with zero optimal value min_{𝑥∈𝑅𝑛}𝑓(𝑥). (7)

Case 1. In this section, we assume that 𝑓 is a continuously differentiable function. Then (7) is a standard unconstrained optimization problem. There are many methods for solving the unconstrained optimization problem, such as Newton method, nonlinear conjugate gradient method, and quasi-Newton method [12–16]. Here, based on [13, 14], we will give a new nonlinear conjugate gradient method to solve (7). The iterates for solving (7) is given by 𝑥_𝑘+1=𝑥_𝑘+𝛼_𝑘𝑑_𝑘, (8) where 𝑑_𝑘 is the direction and 𝛼_𝑘>0 is a step size; in this paper, we use the Wolfe type line search [14]. Compute 𝛼_𝑘>0, such that 𝑓𝑥_𝑘+𝛼_𝑘𝑑_𝑘−𝑓𝑥_𝑘≤−𝜌𝛼²_𝑘𝑑_𝑘²,𝑔𝑥_𝑘+𝛼_𝑘𝑑_𝑘^𝑇𝑑_𝑘≥−2𝜎𝛼_𝑘𝑑_𝑘², (9) where 𝜌,𝜎∈(0,1),  𝜌<𝜎, and 𝑔 is the gradient of 𝑓. The direction is defined by 𝑑_𝑘=−𝑔_𝑘,if𝑘=1,−𝑔_𝑘+𝛽_𝑘𝑑_𝑘−1,if𝑘≥2, (10) where 𝛽_𝑘=𝑔^𝑇_𝑘𝑦_𝑘−1𝑑^𝑇_𝑘−1𝑧_𝑘−1𝑦−𝛿^𝑘−12𝑑^𝑇_𝑘−1𝑧_𝑘−1²𝑔^𝑇_𝑘𝑑_𝑘−1, (11)𝑧_𝑘−1=𝑦_𝑘−1+𝑡_𝑘−1𝑠_𝑘−1,  𝑡_𝑘−1=𝜖₀+max{0,−𝑠^𝑇_𝑘−1𝑦_𝑘−1/𝑠^𝑇_𝑘−1𝑠_𝑘−1},  𝜖₀>0,  𝛿>1/4,  𝑦_𝑘−1=𝑔_𝑘−𝑔_𝑘−1,  𝑠_𝑘−1=𝑥_𝑘−𝑥_𝑘−1.

Now we give the new method for (7) as follows.

Method 1. Consider the following steps.
Step  0.  Given 𝑥₀∈𝑅^𝑛,  𝑑₀=−𝑔₀, and 𝑘∶=0, if  ‖𝑔₀‖=0, then stop.
Step  1. Find 𝛼_𝑘>0 satisfying (9), and 𝑥_𝑘+1 is given by (8).
Step  2. Compute 𝑑_𝑘 by (10). Set 𝑘∶=𝑘+1,  and go to Step 1.

We also need the following assumptions.

Assumption 3. Consider the following:(i)𝐿={𝑥∈𝑅^𝑛∣𝑓(𝑥)≤𝑓(𝑥₀)} is level bounded;(ii)in the neighborhood of 𝐿, there exists 𝐿>0, such that 𝑔(𝑥)−𝑔𝑦≤𝐿𝑥−𝑦, (12) where 𝑥,𝑦∈𝑈,  𝑈  is a neighborhood of 𝐿.

In the following, we will give the global convergence analysis about Method 1. Firstly, we give some lemmas.

Lemma 4. Let {𝑥_𝑘} be generated by Method 1, then 𝑔^𝑇_𝑘𝑑_𝑘1≤−1−𝑔4𝜇^𝑘2, (13) where 𝜇>1/4.

By [13, Theorem  2.1], we have Lemma 4.

Lemma 5 (see [14]). Suppose that Assumption 3 holds, 𝛼_𝑘 is computed by (9), and we have ^∞_𝑘=1𝑔^𝑇_𝑘𝑑_𝑘²𝑑_𝑘²<+∞. (14)

By Lemmas 4 and 5, we have Lemma 6.

Lemma 6. Suppose that Assumption 3 holds, 𝛼_𝑘 is determined by (9), and we get ^∞_𝑘=1𝑔_𝑘⁴𝑑_𝑘²<+∞. (15)

Now, we give the global convergence theorem for Method 1.

Theorem 7. Suppose that Assumption 3 holds and {𝑥_𝑘} is generated by Method 1, then liminf_𝑘→∞𝑔_𝑘=0. (16)

Proof. Suppose by contradiction that there exists 𝜖>0, such that 𝑔_𝑘≥𝜖, (17) holds for 𝑘≥1. By 𝑑^𝑇_𝑘−1𝑧_𝑘−1≥𝜖₀𝑑^𝑇_𝑘−1𝑠_𝑘−1, (18) and (11), we have 𝛽_𝑘≤𝐿𝜖₀+𝜇𝐿²𝜖²₀𝑔_𝑘𝑑_𝑘−1,𝑑_𝑘≤𝑔_𝑘+𝐿𝜖₀+𝜇𝐿²𝜖²₀𝑔_𝑘. (19) Denoting that 𝑚=(1+((𝐿𝜖₀+𝜇𝐿²)/𝜖²₀)), we get ‖𝑑_𝑘‖²≤𝑚²‖𝑔_𝑘‖². Then we have ^∞_𝑘=1𝑔_𝑘⁴𝑑_𝑘²≥^∞_𝑘=1𝜖²𝑚²=+∞, (20) which contradicts (15). So we get the theorem.

Case 2. In this section, we assume that 𝑓 is a nonsmooth function. Equation (7) is the nonsmooth unstrained optimization problem. Denote that 𝑓^∼(𝑥,𝜇) is the smoothing function for 𝑓(𝑥), 𝑔^∼_𝑘=∇𝑓_𝑥^∼(𝑥_𝑘,𝜇_𝑘), then 𝛼_𝑘>0 is computed by 𝑓^∼𝑥_𝑘+𝛼_𝑘𝑑_𝑘,𝜇_𝑘≤𝑓^∼𝑥_𝑘,𝜇_𝑘−𝜌𝛼²_𝑘𝑑_𝑘²,_∼𝑔^𝑇_𝑘+1𝑑_𝑘≥−2𝜎𝛼_𝑘𝑑_𝑘², (21) where 0<𝜌<𝜎<1. The direction is defined by 𝑑_𝑘=−𝑔^∼_𝑘−,if𝑘=1,_∼𝑔_𝑘+𝛽_𝑘𝑑_𝑘−1,if𝑘≥2, (22) where 𝛽_𝑘=^∼𝑔𝑦^𝑇_𝑘^∼_𝑘−1𝑑^𝑇∼_𝑘−1𝑧_𝑘−1−𝛿^∼𝑦_𝑘−1²𝑑^𝑇∼_𝑘−1𝑧_𝑘−1^2∼𝑔^𝑇_𝑘𝑑_𝑘−1, (23)𝑧^∼_𝑘−1=𝑦^∼_𝑘−1+𝑡_𝑘−1𝑠_𝑘−1,  𝑡_𝑘−1=𝜖₀+max{0,−𝑠^𝑇∼_𝑘−1𝑦_𝑘−1/𝑠^𝑇_𝑘−1𝑠_𝑘−1},  𝜖₀>0,  𝛿>1/4, 𝑦^∼_𝑘−1=𝑔^∼_𝑘−𝑔^∼_𝑘−1,  𝑠_𝑘−1=𝑥_𝑘−𝑥_𝑘−1.
We give the following smoothing nonlinear conjugate gradient method.

Method 2. Give 𝑥₀∈𝑅^𝑛, 𝜇₀>0,  𝛾>0,  and  𝛾₁∈(0,1).
Step  1. Find 𝛼_𝑘>0 satisfying (21), and 𝑥_𝑘+1 is given by (8).
Step  2. If ‖∇𝑓_𝑥^∼(𝑥_𝑘+1,𝜇_𝑘)‖≥𝛾𝜇_𝑘, then set 𝜇_𝑘+1=𝜇_𝑘; otherwise, let  𝜇_𝑘+1≤𝛾₁𝜇_𝑘.
Step  3. Compute 𝑑_𝑘 by (22). Set 𝑘∶=𝑘+1, and  go to Step 1.

Theorem 8. Suppose that 𝑓^∼(⋅,𝜇) satisfies Assumption 3, for fixed 𝜇>0. Then the sequence {𝑥_𝑘} generated by Method 2 satisfies lim_𝑘→∞𝜇_𝑘=0,liminf_𝑘→∞∇𝑓_𝑥^∼𝑥_𝑘,𝜇_𝑘−1=0. (24)

Proof. If the set {𝑘∣𝜇_𝑘+1≤𝛾₁𝜇_𝑘} is finite, then for a fixed 𝐾, ‖∇𝑓^∼(𝑥_𝑘,𝜇_𝑘−1)‖≥𝛾𝜇_𝑘−1, for all 𝑘>𝐾. Denoteing that 𝜇=𝜇_𝑘,  𝑘>𝐾, because 𝑓^∼(⋅,𝜇) is a smooth function, the previous method reduces to Method 1, where 𝑓(𝑥)=𝑓^∼(𝑥,𝜇). By Theorem 7, we get liminf_𝑘→∞∇𝑓_𝑥^∼𝑥_𝑘,𝜇=0. (25) So we know that ‖∇𝑓^∼(𝑥_𝑘,𝜇_𝑘−1)‖≥𝛾𝜇_𝑘−1 for 𝑘>𝐾 is impossible. Then we can assume that the set {𝑘∣𝜇_𝑘+1≤𝛾₁𝜇_𝑘} is infinite, then lim_𝑘→∞𝜇_𝑘=0. (26) By the infinity of {𝑘∣𝜇_𝑘+1≤𝛾₁𝜇_𝑘}, we can assume that the set is {𝑘₀,𝑘₁,…∣𝑘₀<𝑘₁<⋯}. Then we have liminf_𝑖→∞∇𝑓^∼𝑥_𝑘𝑖+1,𝜇_𝑘𝑖≤𝛾₁lim_𝑖→∞𝜇_𝑘𝑖. (27) By lim_𝑖→∞𝜇_𝑘𝑖=0, (28) we have liminf_𝑘→∞∇𝑓_𝑥^∼𝑥_𝑘,𝜇_𝑘−1=0. (29) So we complete the proof.

Remark 9. From the result of Theorem 8, we know that for some kinds of smoothing functions [9, 10], any accumulation point 𝑥^⋆ of {𝑥_𝑘} generated by Method 2 is a Clarke stationary point of 𝑓; that is,0∈𝜕𝑓(𝑥^⋆).

3. Some Discussions

In this paper, we give a new smoothing nonlinear conjugate gradient method for (1). The new method also guarantees that any accumulation point is a Clarke stationary point of the merit function for (1).

Discussion 1. In our Methods 1 and 2, we can use any line search, which is well defined under the condition that the search directions are descent directions.

Discussion 2. There are some kinds of smoothing functions, which satisfied Assumption 3 for fixed 𝜇>0 (see [9, 10]). When the smoothing function of 𝑓 has gradient consistent property, then any accumulation point of the sequence {𝑥_𝑘} generated by Method 2 is a Clarke stationary point. Under some assumptions, we can also use the methods in [15] to solve nonsmooth equations with finitely many maximum functions (1).

Discussion 3. The new method can also be used for solving nonlinear complementarity problem (NCP). By F-B function 𝜑(𝑎,𝑏)=𝑎²+𝑏²−(𝑎+𝑏), (30) we know that NCP is equivalent to 𝜓(𝑥)=0, where 𝜓 is a continuously differentiable function, so we can use the smooth version Method 1 to solve it. Method 2 can also be used to solve the vertical complementarity problems 𝐻_𝑖(𝑥)≥0,𝑖=1,…,𝑚,_𝑚𝑖=1𝐻^𝑗_𝑖(𝑥)=0,𝑗=1,…,𝑛, (31) where 𝐻_𝑖(𝑥)∶𝑅^𝑛→𝑅^𝑛 are continuously differentiable functions.

Discussion 4. The new method can also be used for solving Hamilton-Jacobi-Bellman equations (HJB) (see [17]). The Hamilton-Jacobi-Bellman equations (HJB) is used to find 𝑥∈𝑅^𝑛, such that max_{1≤𝑖≤𝑘}𝐿^𝑖𝑥−𝑙^𝑖=0,inΘ,𝑥=0,on𝜕Θ, (32) where  Θ  is a bounded domain in 𝑅^𝑑,  𝐿^𝑖  and  𝑖=1,…,𝑘 are elliptic operators of second order. HJB arise in stochastic control problems and often are used to solve finance and control problems. By finite element method, we can obtain the following discrete HJB equation; find 𝑥∈𝑅^𝑛, such that max_{1≤𝑗≤𝑘}𝐴^𝑗𝑥−𝜁^𝑗=0, (33) where 𝐴^𝑗∈𝑅^𝑛×𝑛,  𝜁^𝑗∈𝑅^𝑛,  𝑗=1,…,𝑘.

Discussion 5. We also can consider to use the new method to solve the general variational inequality problem (see [18]). The general variational inequality problem is to compute 𝑥∈𝑅^𝑛, such that 𝑞(𝑥)∈𝑋,𝑦−𝑞(𝑥)^𝑇𝑝(𝑥)≥0,forany𝑦∈𝑋, (34) where 𝑝,𝑞∶𝑅^𝑛→𝑅^𝑛 are two continuously differentiable functions and 𝑋⊆𝑅^𝑛 is a closed convex set. Equation (34) is denoted by  GVI(𝑋,𝑝,𝑞)  in [18].  GVI(𝑋,𝑝,𝑞)  is a generalization of complementarity problems, nonlinear variational inequalities problems, and general nonlinear complementarity problems. The variational inequalities problems are to compute 𝑦−𝑥^𝑇𝑝(𝑥)≥0,forany𝑦∈𝑋, (35) where 𝑥∈𝑋. The general nonlinear complementarity problems are to compute 𝑥∈𝑅^𝑛, such that 𝑞(𝑥)≥0,𝑝(𝑥)≥0,𝑞(𝑥)^𝑇𝑝(𝑥)=0. (36) We can rewrite (34) as the following nonsmooth equation: 𝑞(𝑥)−𝑃_𝑋𝑞(𝑥)−𝑝(𝑥)=0, (37) where 𝑃_𝑋 is the projection operator onto 𝑋 under the Euclidean norm. The general nonlinear complementarity problems are widely used in solving engineering problems and economic problems; such that, under some conditions, the 𝑛-person noncooperative game problem can be reformulated as (37) (see [19]). Therefore, how to use our new method to solve the general nonlinear complementarity problems and the 𝑛-person noncooperative game problem would be an interesting topic and deserves further investigation.

Acknowledgments

This work was supported by National Science Foundation of China (11101231, 70971070), a Project of Shandong Province Higher Educational Science and Technology Program (J10LA05), and International Cooperation Program for Excellent Lecturers of 2011 by Shandong Provincial Education Department.