Existence of Local Saddle Points for a New Augmented Lagrangian Function

Abstract

We give a new class of augmented Lagrangian functions for nonlinear programming problem with both equality and inequality constraints. The close relationship between local saddle points of this new augmented Lagrangian and local optimal solutions is discussed. In particular, we show that a local saddle point is a local optimal solution and the converse is also true under rather mild conditions.

1. Introduction

Consider the nonlinear optimization problem min𝑓(𝑥),s.t.𝑔_𝑖ℎ(𝑥)≤0,𝑖=1,2,…,𝑚,_𝑗(𝑥)=0,𝑗=1,2,…,𝑙,𝑥∈𝑋, (𝑃) where 𝑓, 𝑔_𝑖∶ℝ^𝑛→ℝ for 𝑖=1,…,𝑚 and ℎ_𝑗∶ℝ^𝑛→ℝ for 𝑗=1,2,…,𝑙 are twice continuously differentiable functions and 𝑋⊆ℝ^𝑛 is a nonempty closed subset.

The classical Lagrangian function associated with (𝑃) is defined as 𝐿(𝑥,𝜆,𝜇)=𝑓(𝑥)+^𝑚_𝑖=1𝜆_𝑖𝑔_𝑖(𝑥)+^𝑙_𝑗=1𝜇_𝑗ℎ_𝑗(𝑥), (1.1) where 𝜆=(𝜆₁,𝜆₂,…,𝜆_𝑚)^𝑇∈ℝ^𝑚₊ and 𝜇=(𝜇₁,𝜇₂,…,𝜇_𝑙)^𝑇∈ℝ^𝑙.

The Lagrangian dual problem (𝐷) is presented:max𝜃(𝜆,𝜇),s.t.𝜆≥0, (𝐷) where 𝜃(𝜆,𝜇)=inf_𝑥∈𝑋𝐿(𝑥,𝜆,𝜇). (1.2)

Lagrange multiplier theory not only plays a key role in many issues of mathematical programming such as sensitivity analysis, optimality conditions, and numerical algorithms, but also has important applications, for example, in scheduling, resource allocation, engineering design, and matching problems. According to both analysis and experiments, it performs substantially better than classical methods for solving some engineering projects, especially for medium-sized or large projects.

Roughly speaking, the augmented Lagrangian method uses a sequence of iterate point of unconstrained optimization problems, which are constructed by utilizing the Lagrangian multipliers, to approximate the optimal solution of the original problem. Toward this end, we must ensure that the zero dual gap property holds between primal and dual problems. Therefore, saddle point theory received much attention, due to its equivalence with zero dual gap property. It is well known that, for convex programming problems, the zero dual gap holds by using the above classical Lagrangian function. However, the nonzero duality gap may appear for nonconvex optimization problems. The main reason is that the classical Lagrangian function is linear with respect to the Lagrangian multiplier. To overcome this drawback, various types of nonlinear Lagrangian functions and augmented Lagrangian functions have been developed in recent years. For example, Hestenes [1] and Powell [2] independently proposed augmented Lagrangian methods for solving equality constrained problems by incorporating the quadratic penalty term in the classical Lagrangian function. This was extended by Rockafellar [3] to the constrained optimization problem with both equality and inequality constraints. A convex augmented function and the corresponding augmented Lagrangian with zero duality gap property were introduced by Rockafellar and Wets in [4]. This was further extended by Huang and Yang by removing the convexity assumption imposed on the augmented functions as in [4]; see [5, 6] for the details. Wang et al. [7] proposed two classes of augmented Lagrangian functions, which are simpler than those given in [4, 5], and discussed the existence of saddle points. For other kinds of augmented Lagrangian methods refer to [8–16]; for saddle points theory and multiplier methods, refer to [17–20]. It should be noted that the sufficient conditions given in the above papers for the existence of local saddle points of augmented Lagrangian functions all require the standard second-order sufficient conditions. So, a natural question arises: whether we can exploit local saddle points under rather mild assumptions, other than the standard second-order sufficient conditions. Motivated by this, in this paper, we propose a new augmented Lagrangian function and establish the close relationship between local saddle points and local optimal solutions of the original problem. In particular, we show that this property holds under weak second-order sufficient conditions.

The paper is organized as follows. After introducing some basic notation and definitions, we mainly present sufficient conditions for the existence of a local saddle point and discuss the close relationship between a local saddle point and a local optimal solution of the original problem. Finally, an example to illustrate our result is given.

2. Notation and Definition

We first introduce some basic notation and definitions, which will be used in the sequel. Let ℝ^𝑛₊ be the nonnegative orthant. For notational simplification, let 𝐺(𝑥)=(𝑔₁(𝑥),𝑔₂(𝑥),…,𝑔_𝑚(𝑥))^𝑇,𝜆=(𝜆₁,…,𝜆_𝑚)^𝑇,𝑡=(𝑡₁,…,𝑡_𝑚)^𝑇,𝜇=(𝜇₁,…,𝜇_𝑙)^𝑇,𝑢=(𝑢₁,…,𝑢_𝑚)^𝑇. (2.1)

Definition 2.1. A pair (𝑥^∗,𝜆^∗,𝜇^∗) is said to be a global saddle point of 𝐿_𝑟(𝑥,𝜆,𝜇) for some 𝑟>0, if 𝐿_𝑟(𝑥^∗,𝜆,𝜇)≤𝐿_𝑟(𝑥^∗,𝜆^∗,𝜇^∗)≤𝐿_𝑟(𝑥,𝜆^∗,𝜇^∗) (2.2) whenever (𝑥,𝜆,𝜇)∈𝑋×ℝ^𝑚₊×ℝ^𝑙. If there exists some positive scalar 𝛿>0 such that the above inequality holds for all (𝑥,𝜆,𝜇)∈(𝑋∩𝑁(𝑥^∗,𝛿))×ℝ^𝑚₊×ℝ^𝑙, where 𝑁(𝑥^∗,𝛿)={𝑥∈ℝ^𝑛∣‖𝑥−𝑥^∗‖≤𝛿}, then (𝑥^∗,𝜆^∗,𝜇^∗) is said to be a local saddle point of 𝐿_𝑟(𝑥,𝜆,𝜇) for 𝑟>0.

Definition 2.2 (weak second-order sufficient conditions). Let 𝑥^∗ be a feasible solution. (1)Suppose that the KKT conditions hold at 𝑥^∗; that is, there exist scalars 𝜆^∗_𝑖≥0 for 𝑖=1,…,𝑚 and 𝜇^∗_𝑗 for 𝑗=1,…,𝑙 such that ∇𝑓(𝑥^∗)+^𝑚_𝑖=1𝜆^∗_𝑖∇𝑔_𝑖(𝑥^∗)+^𝑚_𝑖=1𝜇^∗_𝑗∇ℎ_𝑗(𝑥^∗𝜆)=0,_∗𝑖𝑔_𝑖(𝑥^∗)=0,𝑖=1,…,𝑚. (2.3)(2.4)(2)The Hessian matrix ∇²_𝑥𝑥𝐿(𝑥^∗,𝜆^∗,𝜇^∗)=∇²𝑓(𝑥^∗)+^𝑚_𝑖=1𝜆^∗_𝑖∇²𝑔_𝑖(𝑥^∗)+^𝑙_𝑗=1𝜇^∗_𝑗∇²ℎ_𝑗(𝑥^∗) (2.5) is positive definite on the cone 𝑀(𝑥^∗)=∇ℎ_𝑗(𝑥^∗)^𝑇𝑑=0,𝑗=1,…,𝑙,𝑑∈ℝ^𝑛,𝑑≠0∣∇𝑔_𝑖(𝑥^∗)^𝑇𝑑=0,𝑖∈𝐽(𝑥^∗),∇𝑔_𝑖(𝑥^∗)^𝑇𝑑≤0,𝑖∈𝐼(𝑥^∗)⧵𝐽(𝑥^∗), (2.6) where 𝐼(𝑥^∗)={𝑖∣𝑔_𝑖(𝑥^∗)=0,𝑖=1,…,𝑚},𝐽(𝑥^∗)={𝑖∈𝐼(𝑥^∗)∣𝜆^∗_𝑖>0,𝑖=1,…,𝑚}. (2.7)

Clearly, the above cone is included in the cone 𝑀^′(𝑥^∗)={𝑑∈ℝ^𝑛,𝑑≠0∣∇ℎ_𝑗(𝑥^∗)^𝑇𝑑=0,𝑗=1,…,𝑙;∇𝑔_𝑖(𝑥^∗)^𝑇𝑑=0,𝑖∈𝐽(𝑥^∗)}, which is involved in the second-order sufficient condition. Hence, we refer to above conditions as weak second-order sufficient conditions.

3. Existence of Local Saddle Points

For inequality constrained optimization, Sun et al. [21] introduced a class of the generalized augmented Lagrangian function 𝐿_𝑟∶ℝ^𝑛×ℝ^𝑚₊→ℝ,𝐿_𝑟(𝑥,𝜆)=𝑓(𝑥)+𝑃_𝑟(−𝐺(𝑥),𝜆), (3.1) where 𝑃_𝑟(𝑠,𝑡)∶ℝ^𝑚×ℝ^𝑚₊→ℝ is defined as 𝑃_𝑟(𝑠,𝑡)=inf_{𝑢−𝑠≤0}{−𝑢^𝑇𝑡+𝑟𝜎(𝑢)}. (3.2) The function 𝜎(𝑢) satisfies the following assumptions:(A1)𝜎(0)=0, and 𝜎(𝑢)>0 for all 𝑢≠0;(A2)for each 𝑖∈{1,2,…,𝑚}, 𝜎(𝑢) is nondecreasing on 𝑢_𝑖≥0 and nonincreasing on 𝑢_𝑖<0;(A3)𝜎(𝑢) is continuously differentiable and ∇𝜎(0)=0;(A4)𝜎(𝑢) is twice continuously differentiable in a neighborhood of 0∈ℝ^𝑚 and ℎ^𝑇∇²𝜎(0)ℎ>0 for all nonzero ℎ∈ℝ^𝑚.

We extend this function to treat the optimization problems with equality and inequality constraints. Consider a new augmented Lagrangian function 𝐿_𝑟(𝑥,𝜆,𝜇)∶ℝ^𝑛×ℝ^𝑚₊×ℝ^𝑙→ℝ𝐿_𝑟(𝑥,𝜆,𝜇)=𝑓(𝑥)+^𝑙_𝑗=1𝜇_𝑗ℎ_𝑗𝑟(𝑥)+2^𝑙_𝑗=1ℎ²_𝑗(𝑥)+𝑃_𝑟(−𝐺(𝑥),𝜆), (3.3) where 𝑃_𝑟(−𝐺(𝑥),𝜆) is defined as above and the function 𝜎(𝑢) satisfies (A1)–(A4). Several important augmented functions satisfy the above assumptions, as for example:

Example 3.1. 𝜎(𝑢)=∑^𝑚_𝑖=1𝑢²_𝑖.

Example 3.2. 𝜎(𝑢)=𝑒^‖𝑢‖2−1.

Under the weak second-order sufficient conditions (instead of the standard second-order sufficient conditions), we show that a local optimal solution is also a local saddle point of the augmented Lagrangian function.

Theorem 3.3. Let 𝑥^∗ be a local optimal solution to problem (𝑃). If the weak second-order sufficient conditions are satisfied at 𝑥^∗, then there exist 𝑟₀>0 and 𝛿>0 such that for any 𝑟≥𝑟₀, 𝐿_𝑟(𝑥^∗,𝜆,𝜇)≤𝐿_𝑟(𝑥^∗,𝜆^∗,𝜇^∗)≤𝐿_𝑟(𝑥,𝜆^∗,𝜇^∗),∀(𝑥,𝜆,𝜇)∈(𝑋∩𝑁(𝑥^∗,𝛿))×ℝ^𝑚₊×ℝ^𝑙. (3.4)

Proof. Since 𝑥^∗ is a feasible solution to problem (𝑃), then ℎ_𝑗(𝑥^∗)=0 and 𝑃_𝑟(−𝐺(𝑥^∗),𝜆)=inf_{𝑢+𝐺(𝑥∗)≤0}{−𝑢^𝑇𝜆+𝑟𝜎(𝑢)}≤0^𝑇𝜆+𝑟𝜎(0)=0,∀𝜆≥0. (3.5) It follows from (A1) and (2.4) that 𝑃_𝑟(−𝐺(𝑥^∗),𝜆^∗)=inf_{𝑢+𝐺(𝑥∗)≤0}{−𝑢^𝑇𝜆^∗+𝑟𝜎(𝑢)}≥inf_{𝑢+𝐺(𝑥∗)≤0}{−𝑢^𝑇𝜆^∗}=^𝑚_𝑖=1𝑔_𝑖(𝑥^∗)𝜆^∗_𝑖=0. (3.6) Combining the last two inequalities yields 𝑃_𝑟(−𝐺(𝑥^∗),𝜆^∗)=0. Hence, 𝐿_𝑟(𝑥^∗,𝜆,𝜇)=𝑓(𝑥^∗)+^𝑙_𝑗=1𝜇_𝑗ℎ_𝑗(𝑥^∗𝑟)+2^𝑙_𝑗=1ℎ²_𝑗(𝑥^∗)+𝑃_𝑟(−𝐺(𝑥^∗),𝜆)≤𝑓(𝑥^∗)=𝐿_𝑟(𝑥^∗,𝜆^∗,𝜇^∗). (3.7) We obtain the left inequality of (2.2) as desired.
To show the right inequality of (2.2), it suffices to prove that it holds for some 𝑟₀ and 𝛿, since 𝐿_𝑟(𝑥,𝜆,𝜇) is nondecreasing in 𝑟>0. Suppose the contrary that such 𝑟₀ and 𝛿 do not exist. Then for each positive integer 𝑘, there must exist 𝑥^𝑘∈𝑋 such that 𝑥^𝑘→𝑥^∗ and 𝐿_𝑘(𝑥^𝑘,𝜆^∗,𝜇^∗)<𝐿_𝑘(𝑥^∗,𝜆^∗,𝜇^∗)=𝑓(𝑥^∗). (3.8) Define 𝑢^𝑘_𝐺=(𝑢^𝑘_𝐺1,𝑢^𝑘_𝐺2,…,𝑢^𝑘_𝐺𝑚) as follows: 𝑢^𝑘_𝐺𝑖=0,𝑖∈{1,2,…,𝑚}⧵𝐼(𝑥^∗),0,𝑖∈𝐽^𝑘₁(𝑥^∗),−𝑔_𝑖(𝑥^𝑘),𝑖∈𝐽^𝑘₂(𝑥^∗),−𝑔_𝑖(𝑥^𝑘),𝑖∈𝐽(𝑥^∗), (3.9) where 𝐽^𝑘₁(𝑥^∗)={𝐼(𝑥^∗)⧵𝐽(𝑥^∗)}∩{𝑖∣𝑔_𝑖(𝑥^𝑘𝐽)≤0},_𝑘2(𝑥^∗)={𝐼(𝑥^∗)⧵𝐽(𝑥^∗)}∩{𝑖∣𝑔_𝑖(𝑥^𝑘)≥0}. (3.10) For 𝑥^𝑘, we have 𝑃_r(−𝐺(𝑥^𝑘),𝜆^∗)=inf_{𝑢𝑖+𝑔𝑖(𝑥𝑘)≤0,1≤𝑖≤𝑚}{_{𝑖∈𝐽(𝑥∗)}−𝑢^𝑇_𝑖𝜆^∗_𝑖+𝑘𝜎(𝑢)}=_{𝑖∈𝐽(𝑥∗)}𝑔_𝑖(𝑥^𝑘)𝜆^∗_𝑖+𝑘𝜎(𝑢^𝑘_𝐺). (3.11)
Three cases may be considered.
Case 1. When 𝑖∈{{1,2,…,𝑚}⧵𝐼(𝑥^∗)}∪𝐽^𝑘₁(𝑥^∗), take 𝑢_𝑖(𝑖∈𝐽^𝑘₂(𝑥^∗)∪𝐽(𝑥^∗)). Since 𝜆^∗_𝑖=0, then the original point is a minimizer of inf_{𝑢𝑖+𝑔𝑖(𝑥𝑘)≤0,𝑖∈{{1,2,…,𝑚}⧵𝐼(𝑥∗)}∪𝐽𝑘1(𝑥∗)}{_{𝑖∈𝐽(𝑥∗)}−𝑢^𝑇_𝑖𝜆^∗_𝑖+𝑘𝜎(𝑢)}. (3.12)Case 2. When 𝑖∈𝐽^𝑘₂(𝑥^∗), taking into account to the fact that the function ∑_{𝑖∈𝐽(𝑥∗)}−𝑢^𝑇_𝑖𝜆^∗_𝑖+𝑘𝜎(𝑢) is decreasing on 𝑢_𝑖 in (−∞,−𝑔_𝑖(𝑥^𝑘)], then 𝑢_𝑖=−𝑔_𝑖(𝑥^𝑘) is a minimizer of inf_{𝑢𝑖+𝑔𝑖(𝑥𝑘)≤0,𝑖∈𝐽𝑘2(𝑥∗)}{_{𝑖∈𝐽(𝑥∗)}−𝑢^𝑇_𝑖𝜆^∗_𝑖+𝑘𝜎(𝑢)}. (3.13)Case 3. When 𝑖∈𝐽(𝑥^∗), let ∇𝜎(𝑢)_𝑖 be the ith component of vector ∇𝜎(𝑢), for any 0≤𝑢_𝑖≤−𝑔_𝑖(𝑥^𝑘). We get from (A3) that ∇𝜎(𝑢)_𝑖≤𝜆^∗_𝑖𝑘,∀𝑘∈𝑁, (3.14) that is, −𝜆^∗_𝑖+𝑘∇𝜎(𝑢)_𝑖≤0, and this implies that ∑_{𝑖∈𝐽(𝑥∗)}−𝑢^𝑇_𝑖𝜆^∗_𝑖+𝑘𝜎(𝑢) is decreasing on 𝑢_𝑖 in (0,−𝑔_𝑖(𝑥^𝑘)]. So 𝑢_𝑖=−𝑔_𝑖(𝑥^𝑘) is a minimizer of inf_{𝑢𝑖+𝑔𝑖(𝑥𝑘)≤0,𝑖∈𝐽(𝑥∗)}{_{𝑖∈𝐽(𝑥∗)}−𝑢^𝑇_𝑖𝜆^∗_𝑖+𝑘𝜎(𝑢)}. (3.15) Therefore, 0>𝑓(𝑥^𝑘)−𝑓(𝑥^∗)+^𝑙_𝑗=1𝜇^∗_𝑗ℎ_𝑗(𝑥^𝑘𝑘)+2^𝑙_𝑗=1ℎ²_𝑗(𝑥^𝑘)+_{𝑖∈𝐽(𝑥∗)}𝑔_𝑖(𝑥^𝑘)𝜆^∗_𝑖+𝑘𝜎(𝑢^𝑘_𝐺)=𝐿(𝑥^𝑘,𝜆^∗,𝜇^∗)−𝐿(𝑥^∗,𝜆^∗,𝜇^∗𝑘)+2^𝑙_𝑗=1{ℎ_𝑗(𝑥^∗)+∇ℎ_𝑗(𝑥^∗)(𝑥^𝑘−𝑥^∗)+𝑜(‖𝑥^𝑘−𝑥^∗‖)}²+𝑘𝜎(0)+𝑘∇𝜎(0)^𝑇𝑢^𝑘_𝐺+𝑘2𝑢^𝑘_𝐺∇²𝜎(0)𝑢^𝑘_𝐺+𝑘𝑜(‖𝑢^𝑘_𝐺‖²). (3.16) Since ∇_𝑥𝐿(𝑥^∗,𝜆^∗,𝜇^∗)=0, then 𝐿(𝑥^𝑘,𝜆^∗,𝜇^∗)−𝐿(𝑥^∗,𝜆^∗,𝜇^∗1)=2(𝑥^𝑘−𝑥^∗)^𝑇∇²_𝑥𝑥𝐿(𝑥^∗,𝜆^∗,𝜇^∗)(𝑥^𝑘−𝑥^∗)+𝑜(‖𝑥^𝑘−𝑥^∗‖)². (3.17) Hence 10>2(𝑥^𝑘−𝑥^∗)^𝑇∇²_𝑥𝑥𝐿(𝑥^∗,𝜆^∗,𝜇^∗)(𝑥^𝑘−𝑥^∗)+𝑜(‖𝑥^𝑘−𝑥^∗‖)²+𝑘2^𝑙_𝑗=1{∇ℎ_𝑗(𝑥^∗)(𝑥^𝑘−𝑥^∗)+𝑜(‖𝑥^𝑘−𝑥^∗‖)}²+𝑘2𝑢^𝑘_𝐺∇²𝜎(0)𝑢^𝑘_𝐺+𝑘𝑜(‖𝑢^𝑘_𝐺‖²). (3.18) Set 𝑑_𝑘=(𝑥^𝑘−𝑥^∗)/‖𝑥^𝑘−𝑥^∗‖, which is bounded, we can assume without loss of generality that 𝑑_𝑘 converges to 𝑑 with ‖𝑑‖=1. It follows from (3.18) that 10>2𝑑^𝑇_𝑘∇²_𝑥𝑥𝐿(𝑥^∗,𝜆^∗,𝜇^∗)𝑑_𝑘+𝑜(‖𝑥^𝑘−𝑥^∗‖)²‖𝑥^𝑘−𝑥^∗‖²+𝑘2^𝑙_𝑗=1{∇ℎ_𝑗(𝑥^∗)𝑑_𝑘+𝑜(‖𝑥^𝑘−𝑥^∗‖)‖𝑥^𝑘−𝑥^∗‖}²+𝑘2(𝑢^𝑘_𝐺‖𝑥^𝑘−𝑥^∗‖)^𝑇∇²𝑢𝜎(0)(^𝑘𝐺‖𝑥^𝑘−𝑥^∗‖)+𝑘𝑜(‖𝑢^𝑘_𝐺‖²)‖𝑥^𝑘−𝑥^∗‖². (3.19) Let 𝑙 be the small eigenvalue of ∇²𝜎(0). Then 𝑙>0 by assumption. We claim that 𝑢^𝑘_𝐺/‖𝑥^𝑘−𝑥^∗‖ converges to zero. Suppose the contrary, that for 𝑥^𝑘→𝑥^∗, we have 𝑘2(𝑢^𝑘_𝐺‖𝑥^𝑘−𝑥^∗‖)^𝑇∇²𝑢𝜎(0)(^𝑘𝐺‖𝑥^𝑘−𝑥^∗‖𝑢)+𝑘𝑜(‖^𝑘𝐺𝑥^𝑘−𝑥^∗‖²)≥𝑘𝑙2‖𝑢^𝑘_𝐺𝑥^𝑘−𝑥^∗‖²𝑢+𝑘𝑜(‖^𝑘𝐺𝑥^𝑘−𝑥^∗‖²). (3.20) Taking limits in the above inequality as 𝑘→+∞, the right hand converges to +∞, which contradicts (3.19). So 𝑢^𝑘_𝐺/‖𝑥^𝑘−𝑥^∗‖→0 as claimed.Noting that (3.9) amounts to 𝑢^𝑘_𝐺𝑖=0,𝑖∈{1,2,…,𝑚}⧵𝐼(𝑥^∗),0,𝑖∈𝐽^𝑘₁(𝑥^∗),−∇𝑔_𝑖(𝑥^∗)^𝑇(𝑥^𝑘−𝑥^∗)+𝑜(‖𝑥^𝑘−𝑥^∗‖),𝑖∈𝐽^𝑘₂(𝑥^∗),−∇𝑔_𝑖(𝑥^∗)^𝑇(𝑥^𝑘−𝑥^∗)+𝑜(‖𝑥^𝑘−𝑥^∗‖),𝑖∈𝐽(𝑥^∗), (3.21) then 𝑢^𝑘_𝐺𝑖‖𝑥^𝑘−𝑥^∗‖=0,𝑖∈{1,2,…,𝑚}⧵𝐼(𝑥^∗),0,𝑖∈𝐽^𝑘₁(𝑥^∗),−∇𝑔_𝑖(𝑥^∗)^𝑇𝑑_𝑘+𝑜(𝑥^𝑘−𝑥^∗)‖𝑥^𝑘−𝑥^∗‖,𝑖∈𝐽^𝑘₂(𝑥^∗),−∇𝑔_𝑖(𝑥^∗)^𝑇𝑑_𝑘+𝑜(𝑥^𝑘−𝑥^∗)‖𝑥^𝑘−𝑥^∗‖,𝑖∈𝐽(𝑥^∗). (3.22) So for any 𝑖∈𝐽(𝑥^∗)∪𝐽^𝑘₂(𝑥^∗), taking limits in (3.19) with 𝑘 approaches to +∞, we must have 𝑑^𝑇∇²_𝑥𝑥𝐿(𝑥^∗,𝜆^∗,𝜇^∗)𝑑≤0,∇ℎ_𝑗(𝑥^∗)^𝑇𝑑=0,𝑗=1,…,𝑙,∇𝑔_𝑖(𝑥^∗)^𝑇𝑑=0,𝑖∈𝐽(𝑥^∗)∪𝐽^𝑘₂(𝑥^∗). (3.23)(3.24)(3.25) For 𝑖∈𝐼(𝑥^∗)⧵{𝐽(𝑥^∗)∪𝐽^𝑘₂(𝑥^∗)}, there is an infinite index set 𝑁₀⊆𝑁 such that 𝑔_𝑖(𝑥^𝑘)≤0 for all 𝑘∈𝑁₀. So ∇𝑔_𝑖(𝑥^∗)^𝑇𝑑=lim_𝑘→∞∇𝑔_𝑖(𝜉^𝑘_𝑖)𝑑_𝑘=lim_𝑘→∞𝑔_𝑖(𝑥^𝑘)−𝑔_𝑖(𝑥^∗)‖𝑥^𝑘−𝑥^∗‖≤0, (3.26) where 𝜉^𝑘_𝑖 lies in the line segment between 𝑥^∗ and 𝑥^𝑘. Putting (3.24)–(3.26) together implies that 𝑑∈𝑀(𝑥^∗). We get 𝑑^𝑇∇²_𝑥𝑥𝐿(𝑥^∗,𝜆^∗,𝜇^∗)𝑑>0 by Definition 2.2, which is a contradiction with (3.23). So the right inequality of (2.2) holds. The proof is complete.

The converse of Theorem 3.3 is given below.

Theorem 3.4. If (𝑥^∗,𝜆^∗,𝜇^∗) is a local saddle point of 𝐿_𝑟(𝑥,𝜆,𝜇) for some 𝑟>0, then 𝑥^∗ is a local optimal solution to the problem (𝑃).

Proof. Let (𝑥^∗,𝜆^∗,𝜇^∗) be a local saddle point of 𝐿_𝑟(𝑥,𝜆,𝜇) for some 𝑟>0. Then 𝐿_𝑟(𝑥^∗,𝜆,𝜇)≤𝐿_𝑟(𝑥^∗,𝜆^∗,𝜇^∗)≤𝐿_𝑟(𝑥,𝜆^∗,𝜇^∗) (3.27) whenever (𝑥,𝜆,𝜇)∈(𝑋∩𝑁(𝑥^∗,𝛿))×ℝ^𝑚₊×ℝ^𝑙. We first show that 𝑥^∗ is a feasible solution. If not, there must exist 𝑔_𝑖0(𝑥^∗)>0 for some 𝑖₀ or ℎ_𝑗0(𝑥^∗)≠0 for some 𝑗₀.Case 1. There exists 𝑔_𝑖0(𝑥^∗)>0 for some 𝑖₀. Note that 𝐿_𝑟(𝑥^∗,𝜆,𝜇)=𝑓(𝑥^∗)+^𝑙_𝑗=1𝜇_𝑗ℎ_𝑗(𝑥^∗𝑟)+2^𝑙_𝑗=1ℎ²_𝑗(𝑥^∗)+𝑃_𝑟(−𝐺(𝑥^∗),𝜆). (3.28) Choose 𝜆_𝑖0>0 and 𝜆_𝑖=0 for all 𝑖≠𝑖₀. Then we get from (A1) that 𝑃_𝑟(−𝐺(𝑥^∗),𝜆)=inf_{𝑢+𝐺(𝑥∗)≤0}{−𝑢^𝑇𝜆+𝑟𝜎(𝑢)}≥inf_{𝑢+𝐺(𝑥∗)≤0}{−𝑢^𝑇𝜆}≥𝑔_𝑖0(𝑥^∗)𝜆_𝑖0. (3.29) Taking the limit as 𝜆_𝑖0→+∞ yields 𝑃_𝑟(−𝐺(𝑥^∗),𝜆)→+∞, which is a contradiction with (2.2). So we have 𝑔_𝑖(𝑥^∗)≤0 for all 𝑖=1,…,𝑚.Case 2. There exists ℎ_𝑗0(𝑥^∗)≠0 for some 𝑗₀. Choose 𝜇_𝑗0 and ℎ_𝑗0(𝑥^∗) with the same signal and let 𝜇_𝑗0 approach to +∞, which is a contradiction with (2.2). So we have ℎ_𝑗(𝑥^∗)=0 for all 𝑗∈{1,…,𝑙}. Then 𝑥^∗ is a feasible solution as claimed.Since 𝑥^∗ is feasible, we have ℎ_𝑗(𝑥^∗)=0 and for 𝑢∈{𝑢∈ℝ^𝑚∣𝑢+𝐺(𝑥^∗)≤0}. In particular for 𝑢=0, we have 𝑃_𝑟(−𝐺(𝑥^∗),0)=inf_{𝑢+𝐺(𝑥∗)≤0}{−𝑢^𝑇0+𝑟𝜎(𝑢)}=𝑟𝜎(0)=0. (3.30) Substituting (3.30) into (2.2) yields 𝑃_𝑟(−𝐺(𝑥^∗),𝜆^∗)≥0. (3.31) On the other hand, for any feasible 𝑥 and 𝜆≥0, 𝑃_𝑟(−𝐺(𝑥^∗),𝜆)=inf_{𝑢+𝐺(𝑥∗)≤0}{−𝑢^𝑇𝜆+𝑟𝜎(𝑢)}≤0^𝑇𝜆+𝑟𝜎(0)=0, (3.32) which, together with (3.31) and (3.32), implies that 𝑃_𝑟(−𝐺(𝑥^∗),𝜆^∗)=0. (3.33) So 𝐿_𝑟(𝑥^∗,𝜆^∗,𝜇^∗)=𝑓(𝑥^∗). (3.34) Finally, for any feasible 𝑥∈𝑋∩𝑁(𝑥^∗,𝛿), we have 𝑓(𝑥^∗)=𝐿_𝑟(𝑥^∗,𝜆^∗,𝜇^∗)≤𝐿_𝑟(𝑥,𝜆^∗,𝜇^∗)=𝑓(𝑥)+^𝑙_𝑗=1𝜇^∗_𝑗ℎ_𝑗𝑟(𝑥)+2^𝑙_𝑗=1ℎ²_𝑗(𝑥)+𝑃_𝑟(−𝐺(𝑥),𝜆^∗)≤𝑓(𝑥), (3.35) which means that 𝑥^∗ is a local optimal solution of (𝑃).

Example 3.5. Consider the nonconvex programming problem min𝑓(𝑥)=𝑒^{𝑥21−𝑥22},s.t.𝑔₁(𝑥)=1−𝑥²₂𝑔≤0,₂(𝑥)=𝑒^−𝑥1−1≤0,ℎ(𝑥)=𝑥₁⁻²−𝑥²₂+1=0,𝑥∈𝑋=𝑅². (3.36)

The optimal solutions of the above problem are 𝑥^∗,1=(0,1)^𝑇 and 𝑥^∗,2=(0,−1)^𝑇 with objective value 𝑒⁻¹. Setting 𝜆^∗=(𝑒⁻¹,0) and 𝜇^∗=0, then we get from KKT conditions that ∇_𝑥𝐿(𝑥^∗,𝑖,𝜆^∗,𝜇^∗)=∇𝑓(𝑥^∗,𝑖)+²_𝑖=1𝜆^∗_𝑖∇𝑔_𝑖(𝑥^∗,𝑖)+𝜇^∗∇ℎ(𝑥^∗,𝑖)=0−2𝑥₂^∗,𝑖𝑒⁻¹+𝑒⁻¹02𝑥₂^∗,𝑖=0. (3.37) The Hessian matrix ∇²_𝑥𝑥𝐿(𝑥^∗,𝑖,𝜆^∗,𝜇^∗)=∇²𝑓(𝑥^∗,𝑖)+²_𝑖=1𝜆^∗_𝑖∇²𝑔_𝑖(𝑥^∗,𝑖)+𝜇^∗∇²ℎ(𝑥^∗,𝑖)=2𝑒⁻¹002𝑒⁻¹+𝑒⁻¹=00022𝑒⁻¹004𝑒⁻¹ (3.38) is positive definite. The weak second-order sufficient conditions are satisfied at (𝑥^∗,𝑖,𝜆^∗,𝜇^∗),𝑖=1,2. By Theorem 3.4, (𝑥^∗,𝑖,𝜆^∗,𝜇^∗)(𝑖=1,2) is a local saddle point for 𝐿_𝑟(𝑥,𝜆,𝜇), and hence 𝑥^∗,1 and 𝑥^∗,2 are the optimal solutions to (𝑃).

Based on the above discussion, we know that, if (𝑥^∗,𝜆^∗,𝜇^∗) is a saddle point of 𝐿_𝑟, then 𝑣𝑎𝑙(𝑃)=𝐿_𝑟(𝑥^∗,𝜆^∗,𝜇^∗)=sup_{𝜆≥0,𝜇∈ℝ𝑙}inf_{𝑥∈ℝ𝑛}𝐿_𝑟(𝑥,𝜆,𝜇), (3.39) where we denote by 𝑣𝑎𝑙(𝑃) the optimal value of problem (𝑃) and by (𝑃^′) the problem given in the right-hand side. Note that the problem (𝑃^′) has just the nonnegative constraints 𝜆≥0 (rather simple constraints). Hence, we successful convert the original nonconvex problem to a simple constrained optimization problems by using the augmented Lagrangian function. Furthermore, many efficient algorithms for solving unconstrained optimization problems can be used to solve (𝑃^′), such as gradient-type algorithms. Therefore, our results, obtained with weaker conditions, provide a new insight and theoretical foundation for the use of augmented Lagrangian functions in constrained optimization problems.

Acknowledgments

The authors would like to give their sincere thanks to the anonymous referees for their helpful suggestions and valuable comments which improved the presentation of this paper. This research was supported by the National Natural Science Foundation of China (10971118, 10701047, 10901096).