Modified Lagrangian Methods for Separable Optimization Problems

Abstract

We propose a convergence analysis of a new decomposition method to solve structured optimization problems. The proposed scheme is based on a class of modified Lagrangians combined with the allocation of resources decomposition algorithm. Under mild assumptions, we show that the method generates convergent primal-dual sequences.

1. Introduction

The interest for large-scale optimization problems [1] has grown over the past twenty years and it is likely to continue to increase during the upcoming decades; while on the one hand, the systemic approach to modeling real systems results in more and more complicated models, the increasing computing power of the microprocessors, along with the recent advancements in parallel architectures on the other hand, seems to push back the limits of practical treatment of these models. Decomposition methods (including splitting, partitioning, and parallel methods) are practical candidates to solve large problems where an internal structure allows to identify the “weakly coupled subsystems.” It should be clear that the reduction of the dimension of the original problem is not the only motivation for decomposing it into subproblems. Other important motivations are as follows:(i)partitioning heterogeneous models when it is the juxtaposition of various parts of the model which turns its numerical treatment difficult (as in mixed models with continuous and discrete variables) [2];(ii)decentralizing the decision-making such that the decomposition procedure could lead to autonomous subsystems, which are capable of computing their own part of the global solution independently without the need of a centralized decision level [2, 3];(iii)parallelizing the computation among different processors [1, 2].

Our objective in this paper is to present a family of decomposition algorithms based on proximal-like techniques, which are suitable for decentralized and parallelized computations. The algorithm can be seen as a separable version of the nonlinear rescaling principle method [4] and is closely related to three known techniques: the Partial Inverse method, proposed by Spingarn in 1985 [5, 6], the Alternate Direction Multiplier method [7, 8], proposed originally by Gabay and Mercier [9] for the splitting of variational inequalities, and the Separable Augmented Lagrangian Algorithm (SALA) developed by Hamdi [10–12].

We will use a simplified framework to present the algorithm called the 𝜑-SALA. It is tailored towards minimizing a convex separable function with separable constraints. A nice feature of the 𝜑-SALA is that it preserves separability, as each iteration splits into a proximal step on the dual function and a projection step on the subspace. On the other hand, it yields the proximal decomposition on the graph of a maximal operator that has been introduced by Mahey et al. [13] for quadratic choice of the auxiliary functions 𝜑.

The first drawback associated with the classical quadratic multiplier method (augmented lagrangian [14, 15]) and/or the nonlinear rescaling principle algorithm is that the augmented Lagrangian function and also the rescaled Lagrangian function are no long separable even when the original problem is separable. In other words, when applied to separable constraints like 𝑔(𝑥)=∑^𝑝_𝑖=1𝑔_𝑖(𝑥_𝑖), the terms ‖∑^𝑝_𝑖=1𝑔_𝑖(𝑥_𝑖)‖ or (1/𝑘)𝜑(𝑘∑^𝑝_𝑖=1𝑔_𝑖(𝑥_𝑖)) are no long separable. However, some careful reformulation of the problem (e.g., by introducing additional variables) may preserve some of the given separable structure, thus giving a chance to the augmented Lagrangian framework, to play again an important role in the development of efficient decomposition schemes.

The second drawback associated with the multiplier methods is the fact that it is only differentiable once even when the problem’s data allow for higher differentiability, disabling the application of efficient Newton type methods. In fact such a lack of continuity in the second derivative can significantly slow down the rate of convergence of these methods and thus causing algorithmic failure. One way of coping with this difficulty is to use the recently developed nonquadratic multiplier methods based on entropy-like proximal methods (see [3, 16–20]), and thereby leading to multiplier methods of which are twice continuously differentiable as opposed to the classical quadratic multiplier (if the original problem is also 𝐶²). This is an important advantage since Newton type methods can then be applied.

With respect to the first drawback, in [10, 11], Hamdi has proposed a separable augmented Lagrangian algorithm (SALA) which can be derived from the resource directive subproblems associated with the coupling constraints ((SALA) was developed for non-convex and convex separable problems.). And in this paper, we alleviate the second drawback by using the Nonlinear Re-scaling NR method, so as to get a separable augmented Lagrangian function, which is at least twice continuously differentiable.

It is worth citing here some recent works where decomposition methods related to our subject were developed (to the best of our knowledge). The most recent one is a modification of the original algorithm (SALA) [10–12] where Guèye et al. [21] replaced the scalar penalty parameter in [10] by a diagonal positive definite matrix of scaling factors. Their convergence analysis was done for only affine constraints. Auslender and Teboulle [22] proposed the entropic proximal decomposition method inducing 𝐶^∞-lagrangians for solving the structured convex minimization problems and variational inequalities based on the combination of their recent logarithmic-quadratic proximal point theory [23, 24] with the Chen-Teboulle decomposition scheme [25]. Kyono and Fukushima [26] proposed an extension of the Chen-Teboulle decomposition scheme [25] combined with the Bregman-based proximal point algorithm. Kyona and Fukushima their method was developed for solving large-scale Variational Inequalities (VIPs) (see [26] and for more developments on VIPs see [27–29] and references therein.). Hamdi and Mahey [11] proposed a stabilized version of the original (SALA) by using a primal proximal regularization that yields to better numerical stability, specially for nonconvex minimization problems. For other references on decomposition methods, one may refer to [5, 7, 11, 12, 25, 26, 30–34] and to Hamdi’s survey on decomposition methods based on augmented lagrangian functions [35] and references therein.

The remainder of this paper is organized as follows. In Section 2, we present the nonlinear rescaling principle of Polyak. In Section 3, we describe the application of the nonlinear rescaling method in conjunction with (SALA) to a general structured convex program in order to yield to the decomposition method presented in [30], which can be seen as separable augmented lagrangian method. Section 4 is dedicated to the extended convergence analysis of the proposed algorithm.

2. Nonlinear Rescaling Principle

Let 𝑓 be a convex real-valued function and let (𝑔₁(𝑥),…,𝑔_𝑝(𝑥))^𝖳 be finite concave real-valued functions on 𝑅^𝑛, and consider the convex programming problem:min_{𝑥∈ℝ𝑛}𝑓(𝑥)∶𝑔_𝑖(𝑥)≥0,𝑖=1,𝑚. (𝐶) Recently nonquadratic augmented lagrangian have received much attention in the literature; see, for example, ([3, 4, 16–20, 36] and references therein.). These methods basically rely on applying new class of proximal-like maps on the dual of (𝐶), see [20], and in turns are in fact equivalent to using the nonlinear re-scaling method, as shown in [37]. In this paper we use the nonlinear re-scaling principle to construct smooth lagrangian decomposition methods. Thus, we begin by summarizing this approach; for details see [4, 18, 37] and references therein.

The main idea of the nonlinear re-scaling principle (called here the NR method) is considering a class of strictly concave and smooth enough scalar function with particular properties, and using it to transform the constraints terms of the classical lagrangian. The NR method alternates at each step the unconstrained minimization of the classical lagrangian for the equivalent problem with the lagrange multiplier update. It allows for generating a wide class of augmented lagrangian methods (for instance, the exponential multiplier method, the modified log-barrier method, and the Log-sigmoid multiplier method [18], etc.).

Let us consider the following class of 𝐶² functions 𝜑 defined of ℝand satisfying the following properties:(P1)𝜑^′(𝑡)>0, for all 𝑡∈ℝ;(P2)𝜑^′′(𝑡)<0, for all 𝑡∈ℝ;(P3)𝜑(0)=0; (P4)𝜑^′(0)=1; (P5)𝑚⁻¹≤𝜑^′′(𝑡)<0, for all 𝑡∈ℝ and 𝜑^′′(𝑡)≤−𝑀⁻¹, for all 𝑡<0, where 𝑀,𝑚>0;(P6)(−𝜑)_∞(−1)≥0;(P7)(−𝜑)_∞(1)=0.

Note that from (P2) and (P3) the functions 𝜑 and 𝜑′ are one-to-one and 𝜑 is strictly concave. Let 𝜑∈Φ, then we have for any 𝑘>0,1𝑡≥0𝑘𝜑(𝑡𝑘)≥0. (2.1) The nonlinear re-scaling principle is based on the idea of transforming the original problem (𝐶) to an equivalent problem, namely, to one which has the same set of optimal solutions as (𝐶). To this end, let us consider here a parameterized transformed problems written as follows:min_{𝑥∈𝑅𝑛}𝑓(𝑥)∶𝑘⁻¹𝜑𝑘𝑔_𝑖(𝑥)≥0,𝑖=1,𝑚. ([Ck,𝜑]) Clearly, problem ([C_k,φ]) is also convex and has the same feasible set as (𝐶). The nonlinear re-scaling iterations are based on the classical lagrangian function denoted here by 𝑃_𝑘(𝑥,𝑢) associated to the problem ([C_k,φ]); that is,𝑃_𝑘(𝑥,𝑢)=𝑓(𝑥)−^𝑚_𝑖=1𝑘⁻¹𝑢_𝑖𝜑𝑘𝑔_𝑖(𝑥) (2.2) and can be resumed as follows.

Algorithm 2.1. Given 𝜑∈Φ, 𝑢⁰>0, 𝑘>0, generate the sequence {𝑥^𝑘,𝑢^𝑘}:
Find 𝑥^𝑘+1∈argmin_{𝑥∈𝑅𝑛}𝑃_𝑘(𝑥,𝑢^𝑘)=𝑓(𝑥)−∑^𝑚_𝑖=1𝑘⁻¹𝑢^𝑘_𝑖𝜑(𝑘𝑔_𝑖(𝑥)).
Update 𝑢_𝑖^𝑘+1=𝑢^𝑘_𝑖𝜑^′(𝑘𝑔_𝑖(𝑥^𝑘+1)), 𝑖=1,…,𝑚.

Remark 2.2. Note that the multipliers are nonnegative for all 𝑘 by (P1). Also, it is worth to mention the possibility to change the penalty parameter 𝑘 at each iteration 𝑠. In [37], the authors propose a dynamic scaling parameters update as follows: 𝑘_𝑖^𝑘+1=𝑘/𝑢_𝑖^𝑘+1. This update will be used in our decomposition scheme in the next section.
The NR algorithm allows a generation of a wide class of augmented lagrangian. Typical examples include the choices 𝜑(𝑡)=𝑒^𝑡−1 to get the exponential multiplier method, or 𝜑(𝑡)=−log(1−𝑡) to get the modified log-barrier function method. Convergence analysis of Algorithm 2.1 is given in [4, 18, 37, 38].

3. Separable Modified Lagrangian Algorithm

In this section, we recall the generalization of the separable augmented lagrangian algorithm (SALA) algorithm (see [10, 11]) called (𝜑 SALA) proposed in [30], to solve large-scale convex inequality constrained programs with separable structure. We are concerned here with block separable nonlinear constrained optimization problems:min_{𝑥∈ℝ𝑛}𝐹(𝑥)=^𝑝_𝑖=1𝑓_𝑖𝑥_𝑖∶𝑥∈Ω, (𝑆𝑃) where 𝑓_𝑖∶ℝ^𝑛𝑖→ℝare all convex functions, andΩ=𝑥∈ℝ^𝑛∶^𝑝_𝑖=1𝑔_𝑖𝑗𝑥_𝑖≥0,𝑗=1,𝑚 (3.1) is the convex set where 𝑔_𝑖𝑗 are defined from ℝ^𝑛𝑖→ℝfor 𝑗=1,𝑚, 𝑖=1,𝑝, ∑^𝑝_𝑖=1𝑛_𝑖=𝑛. Along this work, all the functions 𝑓_𝑖, 𝑔_𝑖𝑗 are 𝐶² and we assume the following.(A1) The optimal set 𝑋^∗ of (𝑆𝑃) is nonempty and bounded.(A2) The Slater’s condition holds, that is, ∃𝑥∈ℝ^𝑛∶^𝑝_𝑖=1𝑔_𝑖𝑗𝑥_𝑖>0,𝑗=1,𝑚. (3.2)

Now, to construct our decomposition algorithm, we use the 𝑚 allocation vectors 𝑦_𝑗∈𝐴={𝑧∈ℝ^𝑝∣∑^𝑝_𝑖=1𝑧_𝑖=0} to get the equivalent problem (If (𝑥^∗,𝑦^∗) is an optimal solution to (SP_y) then 𝑥^∗ is an optimal solution to (𝑆𝑃).),min^𝑝_𝑖=1𝑓_𝑖𝑥_𝑖suchthat𝑔_𝑖𝑗𝑥_𝑖+𝑦_𝑖𝑗≥0,𝑗=1,𝑚,𝑖=1,𝑝,_𝑝𝑖=1𝑦_𝑖𝑗=0𝑗=𝑥1,𝑚,_𝑖∈ℝ^𝑛𝑖,𝑖=1,𝑝, (SPy) to which we propose to apply the nonlinear re-scaling principle with partial elimination of the constraints. We mean that only the constraints 𝑔_𝑖𝑗(𝑥_𝑖)+𝑦_𝑖𝑗≥0 are replaced by the equivalent ones (1/𝜆)𝜑(𝜆(𝑔_𝑖𝑗(𝑥_𝑖)+𝑦_𝑖𝑗))≥0. Then for any 𝜆>0, the following minimization problem:min_𝑥^𝑝_𝑖=1𝑓_𝑖𝑥_𝑖1s.t𝜆𝜑𝜆𝑔_𝑖𝑗𝑥_𝑖+𝑦_𝑖𝑗≥0𝑗=1,𝑚,𝑖=1,𝑝_𝑝𝑖=1𝑦_𝑖𝑗=0𝑗=1,𝑚 ([SPy,𝜑]) is equivalent to the (SP_y) and according to Algorithm 2.1, for all 𝑢⁰∈ℝ^𝑝𝑚₊₊, 𝜆⁰=(𝜆⁰₁₁,…,𝜆⁰_𝑝𝑚) where 𝜆⁰_𝑖𝑗=𝜆(𝑢⁰_𝑖𝑗)⁻¹, we have the following iterative scheme: 𝑥^𝑘+1,𝑦^𝑘+1⟶min_{∑𝑝𝑖=1𝑦𝑖𝑗=0,𝑗=1,𝑚}Θ_𝜆𝑘𝑥,𝑦,𝑢^𝑘,𝑢^𝑘+1_𝑖𝑗=𝑢^𝑘_𝑖𝑗𝜑^′𝜆^𝑘_𝑖𝑗𝑔_𝑖𝑗𝑥_𝑖^𝑘+1+𝑦^𝑘+1_𝑖𝑗𝜆_{𝑘+1𝑖𝑗},=𝜆𝑢^𝑘+1_𝑖𝑗⁻¹,∀𝑖,𝑗, (3.3)(3.4)(3.5) where Θ_𝜆(𝑥,𝑦,𝑢) denotes the classical lagrangian for ([SPy,φ]) given byΘ_𝜆𝑥,𝑦,𝑢=^𝑝_𝑖=1𝑓_𝑖𝑥_𝑖−1𝜆^{𝑝𝑖=1𝑚𝑗=1}𝑢_𝑖𝑗𝜑𝜆𝑔_𝑖𝑗𝑥_𝑖+𝑦_𝑖𝑗, (3.6) where 𝑢=(𝑢₁₁,𝑢₂₁,…,𝑢_𝑝1,𝑢₁₂,…,𝑢_𝑝2,…,𝑢_1𝑚,…,𝑢_𝑝𝑚)∈ℝ^𝑝𝑚₊₊.

The minimization in (3.3) is done by alternating the minimization with respect to 𝑥, then followed by the one w.r.t. the allocation variable 𝑦; that is, we fix 𝑦=𝑦^𝑘 and find𝑥^𝑘+1∈Argmin_𝑥Θ_𝜆𝑘𝑥,𝑦^𝑘,𝑢^𝑘=Argmin_𝑥^𝑝_𝑖=1𝑓_𝑖𝑥_𝑖−^𝑚_𝑗=11𝜆^𝑘_𝑖𝑗𝑢^𝑘_𝑖𝑗𝜑𝜆𝑔_𝑖𝑗𝑥_𝑖+𝑦^𝑘_𝑖𝑗. (3.7) Then we can split the above minimization into 𝑝 independent sub-problems with low dimension. That is,𝑥_𝑖^𝑘+1∈Argmin_𝑥𝑖𝑓_𝑖𝑥_𝑖−^𝑚_𝑗=11𝜆^𝑘_𝑖𝑗𝑢^𝑘_𝑖𝑗𝜑𝜆^𝑘_𝑖𝑗𝑔_𝑖𝑗𝑥_𝑖+𝑦^𝑘_𝑖𝑗. (3.8) And now we fix 𝑥=𝑥^𝑘+1 to solve for 𝑦^𝑘+1𝑦^𝑘+1∈Argmin^𝑝_𝑖=1^𝑚_𝑗=1−1𝜆^𝑘_𝑖𝑗𝑢^𝑘_𝑖𝑗𝜑𝜆^𝑘_𝑖𝑗𝑔_𝑖𝑗𝑥_𝑖^𝑘+1+𝑦_𝑖𝑗∶^𝑝_𝑖=1𝑦_𝑖𝑗=0. (3.9) The following lemma gives an important link between the allocation variable and the lagrange dual variable.

Lemma 3.1. According to  (3.9), 𝑢^𝑘+1 and 𝑦^𝑘+1 are orthogonal and satisfy 𝑦^𝑘+1_𝑖𝑗=−𝑔_𝑖𝑗𝑥_𝑖^𝑘+1+𝛿_𝑗^𝑘+1,𝑖=1,𝑝,𝑗=𝑢1,𝑚,_𝑗𝑘+1=𝑢^𝑘_𝑗𝜑^{′(𝜆𝑘𝑗𝛿𝑗𝑘+1)},𝑗=1,𝑚, (3.10)(3.11) where 𝛿_𝑗^𝑘+1=𝑝⁻¹∑^𝑝_𝑖=1𝑔_𝑖𝑗(𝑥_𝑖^𝑘+1).

Proof. For any 𝑗=1,𝑚, by writing the classical lagrangian to (3.9) 𝐿^𝑘𝑧_𝑗,𝑡_𝑗=^𝑝_𝑖=1−1𝜆^𝑘_𝑖𝑗𝑢^𝑘_𝑖𝑗𝜑𝜆^𝑘_𝑖𝑗𝑔_𝑖𝑗𝑥_𝑖^𝑘+1+𝑦_𝑖𝑗+^𝑝_𝑖=1𝑦_𝑖𝑗𝑡_𝑗, (3.12) where 𝑡_𝑗∈ℝ, 𝑗=1,𝑚, and using the optimality, with (3.4), we show that 𝑡_𝑗=𝑢^𝑘+1_𝑖𝑗=𝑢^𝑘_𝑖𝑗𝜑^′𝜆^𝑘_𝑖𝑗𝑔_𝑖𝑗𝑥_𝑖^𝑘+1+𝑦^𝑘+1_𝑖𝑗, (3.13) which means that 𝑢^𝑘+1 does not depend on 𝑖; that is, 𝑢^𝑘+1_𝑖𝑗=𝑢^𝑘+1_𝑙𝑗, for all 𝑖, 𝑙=1,𝑝, and 𝑢_𝑖𝑗 can be replaced by 𝑢_𝑗 for all 𝑖=1,𝑝.
Now, according to (P1) and after straightforward calculations, we reach (3.10). Equation (3.11) is obtained directly using (3.4) and (3.10). The orthogonality of the vectors 𝑢^𝑘+1 and 𝑦^𝑘+1 is direct. 𝑢,𝑦=^𝑚_𝑗=1^𝑝_𝑖=1𝑢_𝑖𝑗𝑦_𝑖𝑗=^𝑚_𝑗=1𝑢_𝑗^𝑝_𝑖=1𝑦_𝑖𝑗=0. (3.14)

One can observe that also the penalty parameters (𝜆_1𝑗,𝜆_2𝑗,…,𝜆_𝑝𝑗) belong to the set 𝑉={(𝑎₁,…,𝑎_𝑝)∶𝑎₁=⋯=𝑎_𝑝}, and finally our algorithm (𝜑SALA) can be stated as follows.

Algorithm 3.2. We have the following steps.Step 1. Select 𝜑∈Φ, 𝑢⁰∈ℝ^𝑚𝑝₊₊where𝑢_𝑗∈𝑉, 𝑗=1,𝑚, 𝜆>0, 𝑦⁰=(𝑦₁,…,𝑦_𝑚), where 𝑦_𝑗∈𝐴, 𝑗=1,𝑚and 𝜆⁰=(𝜆₁₁,…,𝜆_𝑝𝑚) where 𝜆⁰_𝑗=𝜆(𝑢⁰_𝑗)⁻¹, 𝜖₁>0, 𝜖₂>0, 𝜖₃>0.Step 2. Determine: for any𝑖=1,𝑝𝑥_𝑖^𝑘+1∶=argmin_{𝑥𝑖∈ℝ𝑛𝑖}𝑓_𝑖𝑥_𝑖−^𝑚_𝑗=11𝜆^𝑘_𝑗𝑢^𝑘_𝑗𝜑𝜆^𝑘_𝑗𝑔_𝑖𝑗𝑥_𝑖+𝑦^𝑘_𝑖𝑗. (3.15)Step 3. If 𝑣₁≤𝜖₁, 𝑣₂≤𝜖₂, 𝑣₃≤𝜖₃ where𝑣₁=^𝑚_𝑗=1^𝑝_𝑖=1𝑢^𝑘_𝑗𝑔_𝑖𝑗𝑥_𝑖^𝑘+1+𝑦^𝑘_𝑖𝑗,𝑣₂=max_{1≤𝑗≤𝑚}−𝛿_𝑗^𝑘+1,𝑣₃=𝐹𝑥^𝑘−𝐹𝑥^𝑘+1. (3.16)Stop. Else: go to Step 4.Step 4. Update and go back to Step 2:𝑦^𝑘+1_𝑖𝑗=−𝑔_𝑖𝑗𝑥_𝑖^𝑘+1+𝛿_𝑗^𝑘+1𝑖=1,𝑝,𝑗=𝑢1,𝑚_𝑗𝑘+1=𝑢^𝑘_𝑗𝜑^′𝜆^𝑘_𝑗𝛿_𝑗^𝑘+1,𝜆_𝑗^𝑘+1=𝜆𝑢_𝑗^𝑘+1−1. (3.17)

The following proposition gives us some properties of Θ_𝜆.

Proposition 3.3. (1)  Θ_𝜆(𝑥,𝑦,𝑢) is strictly convex in 𝑥∈ℝ^𝑛 for any 𝑢∈ℝ^𝑚𝑝₊₊,𝑦∈ℝ^𝑚𝑝, 𝜆>0.
(2) For any K.K.T point (𝑥^∗,𝑦^∗,𝑢^∗) of (SP_y) One has(i)Θ_𝜆(𝑥^∗,𝑦^∗,𝑢^∗)=𝐿(𝑥^∗,𝑦^∗,𝑢^∗)=∑^𝑝_𝑖=1𝑓_𝑖(𝑥^∗_𝑖), (ii)∇_𝑥Θ_𝜆(𝑥^∗,𝑦^∗,𝑢^∗)=∇_𝑥𝐿(𝑥^∗,𝑦^∗,𝑢^∗)=0, (iii)∇²_𝑥Θ_𝜆(𝑥^∗,𝑦^∗,𝑢^∗)=∇²_𝑥𝐿(𝑥^∗,𝑦^∗,𝑢^∗)−𝜑^′′(0)∇𝐺(𝑥^∗)Λ∇𝐺(𝑥^∗)^′, where Λ=diag(𝑢^∗_𝑖𝑗)^(𝑝,𝑚)_{(𝑖,𝑗)=(1,1)}and 𝐺(𝑥)=(𝜆₁₁𝑔₁₁(𝑥₁),…,𝜆_𝑚𝑝𝑔_𝑚𝑝(𝑥_𝑝)).

Proof. (1) Θ_𝜆(𝑥,𝑦,𝑢)=∑^𝑝_𝑖=1𝑓_𝑖(𝑥_𝑖)−∑^𝑚_𝑗=1(1/𝜆_𝑖𝑗)𝑢_𝑖𝑗𝜑(𝜆_𝑖𝑗(𝑔_𝑖𝑗(𝑥_𝑖)+𝑦_𝑖𝑗)). Since 𝑔_𝑖𝑗, 𝜑, 𝑖=1,𝑝, 𝑗=1,𝑚 are concave, strictly concave (resp.) and 𝜑 is increasing. Then Θ is strictly convex in 𝑥∈ℝ^𝑛 for any 𝑢∈ℝ^𝑚𝑝₊₊, 𝑦∈ℝ^𝑚𝑝, 𝜆>0.
(2) Let (𝑥^∗,𝑦^∗,𝑢^∗) be any K.K.T point of (SP_y) then (i)we have Θ_𝜆𝑥^∗,𝑦^∗,𝑢^∗=^𝑝_𝑖=1𝑓_𝑖𝑥^∗_𝑖−^𝑚_𝑗=11𝜆_𝑖𝑗𝑢^∗_𝑖𝑗𝜑𝜆_𝑖𝑗𝑔_𝑖𝑗𝑥^∗_𝑖+𝑦^∗_𝑖𝑗,𝐿𝑥^∗,𝑦^∗,𝑢^∗=^𝑝_𝑖=1𝑓_𝑖𝑥^∗_𝑖−^𝑚_𝑗=1𝑢^∗_𝑖𝑗𝑔_𝑖𝑗𝑥^∗_𝑖+𝑦^∗_𝑖𝑗=^𝑝_𝑖=1𝑓_𝑖𝑥^∗_𝑖 (3.18) from the complementary condition. ⟹𝜑𝜆_𝑖𝑗𝑔_𝑖𝑗𝑥^∗_𝑖+𝑦^∗_𝑖𝑗=0⟹𝑢^∗_𝑖𝑗𝜑𝜆_𝑖𝑗𝑔_𝑖𝑗𝑥^∗_𝑖+𝑦^∗_𝑖𝑗=0, (3.19) then we get Θ_𝜆𝑥^∗,𝑦^∗,𝑢^∗=^𝑝_𝑖=1𝑓_𝑖𝑥^∗_𝑖. (3.20)(ii)We have ∇_𝑥Θ_𝜆𝑥^∗,𝑦^∗,𝑢^∗=^𝑝_𝑖=1∇_𝑥𝑓_𝑖𝑥^∗_𝑖−^𝑚_𝑗=1𝑢^∗_𝑖𝑗𝜑^′𝜆_𝑖𝑗𝑔_𝑖𝑗𝑥^∗_𝑖+𝑦^∗_𝑖𝑗∇_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖. (3.21) Similarly, if 𝑢^∗_𝑖𝑗≠0 then 𝑔_𝑖𝑗(𝑥^∗)+𝑦^∗_𝑖𝑗=0 therefore 𝜑′(𝜆_𝑖𝑗(𝑔_𝑖𝑗(𝑥^∗_𝑖)+𝑦^∗_𝑖𝑗))=1. This means that ∇_𝑥Θ_𝜆𝑥^∗,𝑦^∗,𝑢^∗=^𝑝_𝑖=1∇_𝑥𝑓_𝑖𝑥^∗_𝑖−^𝑚_𝑗=1𝑢^∗_𝑖𝑗∇_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖=∇_𝑥𝐿𝑥^∗,𝑦^∗,𝑢^∗=0. (3.22)(iii) From (ii) we can calculate ∇²_𝑥Θ_𝜆(𝑥^∗,𝑦^∗,𝑢^∗), ∇²_𝑥Θ_𝜆𝑥,𝑦,𝑢=^𝑝_𝑖=1∇²_𝑥𝑓_𝑖𝑥_𝑖−^𝑚_𝑗=1𝜆_𝑖𝑗𝑢_𝑖𝑗𝜑^′′𝜆_𝑖𝑗𝑔_𝑖𝑗𝑥_𝑖+𝑦_𝑖𝑗∇_𝑥𝑔_𝑖𝑗𝑥_𝑖∇_𝑥𝑔_𝑖𝑗𝑥_𝑖^′−^𝑚_𝑗=1𝑢_𝑖𝑗𝜑^′𝜆_𝑖𝑗𝑔_𝑖𝑗𝑥_𝑖+𝑦_𝑖𝑗∇²_𝑥𝑔_𝑖𝑗𝑥_𝑖. (3.23) At (𝑥^∗,𝑦^∗,𝑢^∗), we have ∇²_𝑥Θ_𝜆𝑥^∗,𝑦^∗,𝑢^∗=^𝑝_𝑖=1∇²_𝑥𝑓_𝑖𝑥^∗_𝑖−^𝑚_𝑗=1𝜆_𝑖𝑗𝑢^∗_𝑖𝑗𝜑^′′𝜆_𝑖𝑗𝑔_𝑖𝑗𝑥^∗_𝑖+𝑦^∗_𝑖𝑗∇_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖∇_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖^′+𝑢^∗_𝑖𝑗𝜑^′𝜆_𝑖𝑗𝑔_𝑖𝑗𝑥^∗_𝑖+𝑦^∗_𝑖𝑗∇²_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖=^𝑝_𝑖=1∇²_𝑥𝑓_𝑖𝑥^∗_𝑖−^𝑚_𝑗=1𝑢^∗_𝑖𝑗∇²_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖−^𝑝_𝑖=1^𝑚_𝑗=1𝜆_𝑖𝑗𝑢^∗_𝑖𝑗𝜑^′′𝜆_𝑖𝑗𝑔_𝑖𝑗𝑥^∗_𝑖+𝑦^∗_𝑖𝑗∇_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖∇_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖^′=∇²_𝑥𝐿𝑥^∗,𝑦^∗,𝑢^∗−^𝑝_𝑖=1^𝑚_𝑗=1𝜆_𝑖𝑗𝑢^∗_𝑖𝑗𝜑^′′𝜆_𝑖𝑗𝑔_𝑖𝑗𝑥^∗_𝑖+𝑦^∗_𝑖𝑗∇_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖×∇_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖^′=∇²_𝑥𝐿𝑥^∗,𝑦^∗,𝑢^∗−𝜑^′′(0)^𝑝_𝑖=1^𝑚_𝑗=1𝜆_𝑖𝑗𝑢^∗_𝑖𝑗∇_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖∇_𝑥𝑔_𝑖𝑗𝑥^∗_𝑖^′. (3.24) Let 𝐺(𝑥)=𝜆₁₁𝑔₁₁𝑥₁,…,𝜆_𝑝1𝑔_𝑝1𝑥_𝑝,𝜆₁₂𝑔₁₂𝑥₁,…,𝜆_𝑚𝑝𝑔_𝑚𝑝𝑥_𝑝 (3.25) then ∇𝐺(𝑥)=𝜆₁₁∇𝑔₁₁𝑥₁𝜆₂₁∇𝑔₂₁𝑥₂⋯∇𝜆_𝑚𝑝𝑔_𝑚𝑝𝑥_{𝑝𝑛×𝑝𝑚}. (3.26) That is, the first column is the gradient ∇𝑔₁₁(𝑥₁) and so on. Let Λ=diag(𝑢^∗_𝑖𝑗)^(𝑝,𝑚)_{(𝑖,𝑗)=(1,1)} that is, the diagonal is 𝑢^∗₁₁,𝑢^∗₂₁,…,𝑢^∗_𝑝1,𝑢^∗₁₂,…,𝑢^∗_𝑚𝑝. (3.27) Then, ∇𝐺𝑥^∗Λ=∇𝑔₁₁𝑥^∗₁𝑢^∗₁₁𝜆₁₁∇𝑔₂₁𝑥^∗₂𝑢^∗₂₁𝜆₂₁⋯∇𝑔_𝑚𝑝𝑥^∗_𝑝𝑢^∗_𝑚𝑝𝜆_{𝑚𝑝𝑛×𝑝𝑚},∇𝐺𝑥^∗Λ∇𝐺𝑥^∗′=𝜆₁₁∇𝑔₁₁𝑥^∗₁𝑢^∗₁₁⋯𝜆_𝑝𝑚∇𝑔_𝑚𝑝𝑥^∗_𝑝𝑢^∗_𝑚𝑝∇𝑔₁₁𝑥^∗₁^′⋯∇𝑔_𝑚𝑝𝑥^∗_𝑝^′=^𝑝_𝑖=1^𝑚_𝑗=1𝜆_𝑖𝑗𝑢^∗_𝑖𝑗∇𝑔_𝑖𝑗𝑥^∗_𝑖∇𝑔_𝑖𝑗𝑥^∗_𝑖^′. (3.28) Then ∇²_𝑥Θ_𝜆𝑥^∗,𝑦^∗,𝑢^∗=∇²_𝑥𝐿𝑥^∗,𝑦^∗,𝑢^∗−𝜑^′′(0)∇𝐺𝑥^∗Λ∇𝐺𝑥^∗′. (3.29)

Remark 3.4. The analysis presented in this paper differs from the short one presented in [30]. In this paper, our analysis is made possible by the strong tool of recession functions.
Assumption (A1) can be written in terms of recession functions (𝑓_∞(𝑑)=lim_𝑡→∞(𝑓(𝑥+𝑡𝑑)−𝑓(𝑥))/𝑡, for all 𝑥∈dom𝑓 for convex proper and lower semicontinuous functions. In our case the functions are convex proper and continuous.)

𝐹_∞(𝑑)≤0,−^𝑝_𝑖=1𝑔_𝑖𝑗∞(𝑑)≤0,∀𝑗=1,𝑚⟹𝑑=0. (3.30) Further, since ∑^𝑝_𝑖=1𝑔_𝑖𝑗 is not identically +∞, [39, Theorem  9.3, page  77], allows us to reformulate (A1) as follows:𝐹_∞(𝑑)>0,^𝑝_𝑖=1−𝑔_𝑖𝑗∞𝑑_𝑖>0,∀𝑗=1,𝑚,∀𝑑≠0. (3.31) The next proposition shows that the minimization subproblems are solvable.

Proposition 3.5. If 𝑋^∗ is nonempty and bounded, then for any 𝑢∈ℝ^𝑚𝑝₊₊, 𝑦∈ℝ^𝑝𝑚 and 𝜆>0ArgminΘ_𝜆𝑥,𝑦,𝑢∶𝑥∈ℝ^𝑛≠∅andbounded. (3.32)

Proof. To this goal, we need to show that (Θ_𝜆)_∞(𝑑)>0 for any 𝑑∈ℝ^𝑛,𝑑≠0. According to Proposition  2.1 in [40] and Proposition  2.6.1 in [41], we can express the recession function of Θ_𝜆 as follows Θ_𝜆∞(𝑑)=𝐹_∞(𝑑)+^𝑚_𝑗=1^𝑝_𝑖=1𝑢^𝑘_𝑗𝜆_𝑗𝜓𝜆_𝑗𝑔_𝑖𝑗𝑥_𝑖+𝑦_𝑖𝑗∞𝑑_𝑖, (3.33) where 𝜓(𝑡)=−𝜑(−𝑡).
If, we denote 𝐼₊={(𝑖,𝑗)∶(−𝑔_𝑖𝑗)_∞(𝑑_𝑖)>0}, 𝐼₋={(𝑖,𝑗)∶(−𝑔_𝑖𝑗)_∞(𝑑_𝑖)≤0}, thenΘ_𝜆∞(𝑑)=𝐹_∞(𝑑)+^𝑚_{(𝑖,𝑗)∈𝐼+}𝑢^𝑘_𝑗𝜆_𝑗𝜓_∞𝜆_𝑗−𝑔_𝑖𝑗∞𝑑_𝑖+^𝑚_{(𝑖,𝑗)∈𝐼−}𝑢^𝑘_𝑗𝜆_𝑗𝜓_∞𝜆_𝑗−𝑔_𝑖𝑗∞𝑑_𝑖 (3.34) and by using 𝜓_∞(𝑠)=𝑠𝜓_∞(1)if𝑠>0,−𝑠𝜓_∞(−1)if𝑠≤0, (3.35) we have Θ_𝜆∞(𝑑)=𝐹_∞(𝑑)+𝜓_∞(1)_{(𝑖,𝑗)∈𝐼+}𝑢^𝑘_𝑗−𝑔_𝑖𝑗∞𝑑_𝑖−𝜓_∞(−1)_{(𝑖,𝑗)∈𝐼−}𝑢^𝑘_𝑗−𝑔_𝑖𝑗∞𝑑_𝑖. (3.36) Now, since 𝜓_∞(−1)=(−𝜑)_∞(1)=0, the above relation becomes Θ_𝜆∞(𝑑)=𝐹_∞(𝑑)+𝜓_∞(1)_{(𝑖,𝑗)∈𝐼+}𝑢^𝑘_𝑗−𝑔_𝑖𝑗∞𝑑_𝑖, (3.37) and finally, using (3.31) and (P6) the proof is complete.

4. Convergence Analysis

In this section, we present the convergence analysis of the sequence (𝑥^𝑘,𝑦^𝑘,𝑢^𝑘) for a wide class of constraint transformations 𝜑∈Φ under some assumptions on the input data. To this goal, we give the following main two propositions.

Proposition 4.1. Under  𝐴₁ and 𝐴₂, the dual sequence {𝑢^𝑘} is bounded.

Proof. Let 𝐷_𝜆(𝑢^′)=min_𝑥,𝑦{Θ_𝜆(𝑥,𝑦,𝑢^′)∶∑_𝑖𝑦_𝑖𝑗=0.} and let the vector 𝜋 such that 𝜋_𝑖𝑗≤𝑔_𝑖𝑗(𝑥_𝑖)+𝑦_𝑖𝑗, for all 𝑖=1,𝑝, 𝑗=1,𝑚. Then, it is easy to see that 𝐷_𝜆𝑢^′=min_𝜋𝑃(𝜋)−^𝑚_𝑗=11𝜆_𝑗^𝑝𝑖=1𝑢^′_𝑖𝑗𝜑𝜆𝜋_𝑖𝑗, (4.1) where 𝑃 denotes the perturbation function associated to ( [SPy,φ]) and defined by 𝑃(𝜋)=min_𝑥,𝑦𝐹(𝑥)∶𝑔_𝑖𝑗𝑥_𝑖+𝑦_𝑖𝑗≥𝜋_𝑖𝑗,𝑦∈𝐴. (4.2) So, by adding and subtracting the term ∑^𝑚_𝑗=1(1/𝜆_𝑗)𝑤^′_𝑖𝑗𝜑[𝜆𝜋_𝑖𝑗], for 𝑤^′ with the same structure as 𝑢^′, and if we set 𝐵={(𝑥,𝑦)∶𝑔_𝑖𝑗(𝑥_𝑖)+𝑦_𝑖𝑗≥𝜋_𝑖𝑗,𝑦∈𝐴}, we obtain 𝐷_𝜆𝑢^′=min_𝜋min_{(𝑥,𝑦)∈𝐵}𝐹(𝑥)−_𝑖,𝑗1𝜆_𝑗𝑤^′_𝑖𝑗𝜑𝜆𝜋_𝑖𝑗+_𝑖,𝑗1𝜆_𝑗𝑤^′_𝑖𝑗𝜑𝜆𝜋_𝑖𝑗−_𝑖,𝑗1𝜆_𝑗𝑢^′_𝑖𝑗𝜑𝜆𝜋_𝑖𝑗. (4.3) That is, 𝐷_𝜆𝑢^′=min_{𝜋,(𝑥,𝑦)∈𝐵}𝐹(𝑥)−^𝑚_𝑗=11𝜆_𝑗^𝑝𝑖=1𝑤^′_𝑖𝑗𝜑𝜆𝜋_𝑖𝑗+^𝑚_𝑗=11𝜆_𝑗^𝑝𝑖=1𝑤^′_𝑖𝑗−𝑢^′_𝑖𝑗𝜑𝜆𝜋_𝑖𝑗 (4.4) which can be rewritten also as follows: 𝐷_𝜆𝑢^′=min_𝜋𝑃(𝜋)−^𝑚_𝑗=11𝜆_𝑗^𝑝𝑖=1𝑤^′_𝑖𝑗𝜑𝜆𝜋_𝑖𝑗+^𝑚_𝑗=11𝜆_𝑗^𝑝𝑖=1𝑤^′_𝑖𝑗−𝑢^′_𝑖𝑗𝜑𝜆𝜋_𝑖𝑗 (4.5) and then we have 𝐷_𝜆𝑢^′≥𝐷_𝜆𝑤^′+min_𝜋^𝑚_𝑗=11𝜆_𝑗^𝑝𝑖=1𝑤^′_𝑖𝑗−𝑢^′_𝑖𝑗𝜑𝜆𝜋_𝑖𝑗. (4.6) Using (P1), (P3), and (P4) and if we take 𝑤^′=𝑢^𝑘, 𝑢^′=𝑢^𝑘+1, we can show that when 𝑥^𝑘+1 is not feasible, 𝑢^𝑘+1≥𝑢^𝑘 and it is easy to see that the minimum in this case equals zero and we have 𝐷_𝜆(𝑢^𝑘+1)≥𝐷_𝜆(𝑢^𝑘)≥⋯≥𝐷_𝜆(𝑢⁰), for all 𝑘≥0. Thus, using the fact that 𝐷_𝜆 is concave, 𝐿^∗ is bounded, and 𝑢^𝑘+1 is in the dual level set, the sequence {𝑢^𝑘} is bounded.

Proposition 4.2. Under 𝐴₁, 𝐴₂, the primal sequence {𝑥^𝑘} is bounded.

Proof. Let (𝑥^𝑘+1,𝑦^𝑘+1)∈Argmin_𝑥{Θ_𝜆𝑘(𝑥,𝑦,𝑢^𝑘)∶∑_𝑖𝑦_𝑖𝑗=0} and fix 𝑥∈𝑋^∗ and 𝑦_𝑖𝑗=−𝑔_𝑖𝑗(𝑥_𝑖)+𝑝⁻¹∑𝑔_𝑖𝑗(𝑥_𝑖). It is clear that 𝐹𝑥^𝑘+1−_𝑗𝑖𝑢^𝑘_𝑗𝜆^𝑘_𝑗𝜑𝜆^𝑘_𝑗𝑔_𝑖𝑗𝑥_𝑖^𝑘+1+𝑦^𝑘+1_𝑖𝑗≤Θ_{𝜆𝑘(𝑥,𝑦,𝑢𝑘)}, (4.7) and using the feasibility of 𝑥, we obtain directly 𝐹𝑥^𝑘+1−_𝑗𝑖𝑢^𝑘_𝑗𝜆^𝑘_𝑗𝜑𝜆^𝑘_𝑗𝑔_𝑖𝑗𝑥_𝑖^𝑘+1+𝑦^𝑘+1_𝑖𝑗≤𝐹(𝑥). (4.8) Now, let us assume that the primal sequence is unbounded, then the sequence 𝑤_𝑘=𝑥^𝑘/‖𝑥^𝑘‖ is bounded and lim_𝑘→∞𝑤_𝑘=𝑑≠0.
Let 𝜖₁ and 𝜖₂ such that𝐹_∞(𝑑)>𝜖₁,_𝑖−𝑔_𝑖𝑗∞(𝑑)>𝜖₂, (4.9) and take 𝑘₀ such that 𝐹𝑥^𝑘+1≥𝜖₁𝑥^𝑘,∀𝑘≥𝑘₀,_𝑖−𝑔_𝑖𝑗𝑥_𝑖^𝑘+1>𝜖₂𝑥^𝑘+1,∀𝑘≥𝑘₀. (4.10)(4.11)
By dividing both sides in (4.8) by ‖𝑥^𝑘+1‖, we get𝐹𝑥^𝑘+1𝑥^𝑘+1−_𝑖,𝑗𝑢^𝑘_𝑗𝜆^𝑘_𝑗𝜑𝜆^𝑘_𝑗𝑔_𝑖𝑗𝑥_𝑖^𝑘+1+𝑦^𝑘+1_𝑖𝑗𝑥^𝑘+1≤𝐹(𝑥)𝑥^𝑘+1, (4.12) and according to (4.11) and the monotonicity of 𝜑, and by denoting 𝜈_𝑘=𝜆^𝑘_𝑗‖𝑥^𝑘+1‖/𝑝, (4.12) becomes 𝐹𝑥^𝑘+1𝑥^𝑘+1−_𝑖,𝑗𝑢^𝑘_𝑗𝜆^𝑘_𝑗𝑥^𝑘+1𝜑−𝜖₂𝜈_𝑘≤𝐹(𝑥)𝑥^𝑘+1. (4.13) Since the dual sequence is bounded, at the limit we have 𝜖₁−_𝑖,𝑗𝑢_𝑗𝑝lim_𝑘→∞𝜑−𝜖₂𝜈_𝑘𝜈_𝑘≤0. (4.14) Since lim_𝑘→∞𝜈_𝑘=+∞ and using 𝜑(𝑡)=−𝜓(−𝑡), we can rewrite (4.14) as follows: 𝜖₁+_𝑖,𝑗𝑢_𝑗𝑝lim_𝑘→∞𝜓𝜖₂𝜈_𝑘𝜈_𝑘≤0. (4.15) Since lim_𝑘→∞𝜓𝜖₂𝜈_𝑘𝜈_𝑘=lim_𝑡→∞𝜓𝜖₂𝑡𝑡=𝜓_∞𝜖₂, (4.16) and (4.15) is equivalent to 𝜖₁+_𝑗𝑢_𝑗𝜓_∞𝜖₂≤0. (4.17) If 𝜖₂≤0 then 𝜓_∞(𝜖₂)=−𝜖₂𝜓_∞(−1)=0 and then 𝜖₁≤0.
If 𝜖₂>0 then 𝜓_∞(𝜖₂)=𝜖₂𝜓_∞(1).
Now by letting 𝜖₁→𝐹_∞(𝑑) and 𝜖₂→(∑_𝑖(−𝑔_𝑖𝑗)_∞(𝑑)), we deduct that𝐹_∞(𝑑)≤0,_𝑖−𝑔_𝑖𝑗∞(𝑑)≤0, (4.18) and since 𝑋^∗ is not empty and is bounded, then 𝑑=0 which contradicts the fact that 𝑑≠0. Thus, what we assumed is false, and the primal sequence {𝑥^𝑘}_𝑘 is bounded.

Proposition 4.3. Let the sequences {𝑥^𝑘,𝑦^𝑘,𝑢^𝑘} generated by ΨSALA and assume that there exist 𝑥^∗ a primal solution to the original problem (𝑆𝑃). Then the following inequality holds: ^𝑝_𝑖=1𝑓_𝑖𝑥_𝑖^𝑘+1≤^𝑝_𝑖=1𝑓_𝑖𝑥^∗_𝑖+^𝑝_𝑖=1𝑢^𝑘+1𝜃_𝑖^𝑘+1𝑔_𝑖𝑥^∗_𝑖−𝑔_𝑖𝑥_𝑖^𝑘+1, (4.19) where 𝜃_𝑖^𝑘+1=Ψ^′𝜆𝑔_𝑖𝑥_𝑖^𝑘+1+𝑦^𝑘_𝑖∣Ψ^′𝜆𝑔_𝑖𝑥_𝑖^𝑘+1+𝑦_𝑖^𝑘+1. (4.20)

Proof. See [30].

Proposition 4.4. Let the sequences {𝑥^𝑘,𝑦^𝑘,𝑢^𝑘}generated by ΨSALA and assume that there exists a saddle point (𝑥^∗,𝑦^∗,𝑢^∗) of the Lagrangian 𝑙(𝑥,𝑦,𝑢) associated to problem (𝑆𝑃). Then the following inequality holds: ^𝑝_𝑖=1𝑓_𝑖𝑥^∗_𝑖≤^𝑝_𝑖=1𝑓_𝑖𝑥_𝑖^𝑘+1+𝑢^∗,𝛿^𝑘+1. (4.21)

Proof. See [30].

Proposition 4.5. All the three sequences {𝑥^𝑘}, {𝑢^𝑘}, and {𝑦^𝑘} generated in Algorithm 2.1 are bounded.

Theorem 4.6. Let 𝑦 and 𝑢, the respective limit point of the bounded sequences {𝑦^𝑘} and {𝑢^𝑘} generated by ΨSALA. Then one has the following properties:(i)limsup_𝑘→∞∑^𝑝_𝑖=1𝑔_𝑖(𝑥_𝑖^𝑘+1)≥0;(ii)limsup_𝑘→∞(𝑔_𝑖(𝑥_𝑖^𝑘+1)+𝑦_𝑖^𝑘+1)≥0;(iii)lim_𝑘→∞⟨𝑢^𝑘+1,𝑔_𝑖(𝑥_𝑖^𝑘+1)+𝑦_𝑖^𝑘+1⟩=0.

Proof. See [30].

Theorem 4.7. If the assumptions𝐴₁, 𝐴₂, and 𝐴₃ are satisfied, then one has the following.(1)Any limit point (𝑥,𝑢) of the sequence {(𝑥^𝑘,𝑢^𝑘)} is in the set 𝑋^∗×𝐿^∗.(2)The sequences 𝑓(𝑥^𝑘) and 𝑑(𝑢^𝑘) are convergent andlim_𝑠→∞𝑓𝑥^𝑘=𝑓𝑥^∗,lim_𝑠→∞𝑑𝑢^𝑘=𝑑𝑢^∗. (4.22)

Proof. Let 𝑢 be any limit point of the sequence {𝑢^𝑘}, then there exists a subsequence converging to 𝑢. Without restricting the generality, we can assume 𝑢^𝑘→𝑢. For 𝑗 such that 𝑢_𝑖𝑗>0, then lim_𝑠→∞𝑢^𝑘+1_𝑖𝑗𝑢^𝑘_𝑖𝑗=1, (4.23) and since 𝑢^𝑘+1_𝑖𝑗=𝑢^𝑘_𝑖𝑗𝜑′((𝜆^𝑘_𝑖𝑗/𝑝)∑^𝑝_𝑖=1𝑔_𝑖𝑗(𝑥^𝑘_𝑖)), 𝑖=1,𝑝, 𝑗=1,𝑚, then lim_𝑠→∞𝜑^′𝜆^𝑘_𝑖𝑗𝑝^𝑝𝑖=1𝑔_𝑖𝑗𝑥^𝑘_𝑖=1. (4.24) Since lim_𝑠→∞𝑢^𝑘_𝑖𝑗=𝑢_𝑖𝑗>0, we have 𝜆^𝑘_𝑖𝑗𝑢^𝑘_𝑖𝑗=𝜆>0. Therefore, 𝜆^𝑘_𝑖𝑗=(𝜆/𝑢^𝑘_𝑖𝑗)>0. Then, lim_𝑠→∞𝜆_𝑖𝑗≥𝜆𝑢_𝑖𝑗>0 (4.25) and then, lim_𝑠→∞^𝑝_𝑖=1𝑔_𝑖𝑗𝑥^𝑘_𝑖=0; (4.26) therefore, lim_𝑠→∞𝑢^𝑘+1_𝑖𝑗^𝑝_𝑖=1𝑔_𝑖𝑗𝑥_𝑖^𝑘+1=0. (4.27)
Now, we will prove that the set {∑^𝑝_𝑖=1𝑔_𝑖𝑗(𝑥^𝑘_𝑖)|𝑗=1,𝑚,𝑠∈ℕ} is bounded. Since {𝑥^𝑘} is bounded, there exists 𝛿∈ℝsuch that ‖𝑥^𝑘‖≤𝛿. Let 𝐵(0,𝛿)⊆ℝ^𝑛 be the closed ball of center zero and radius 𝛿 then𝑥^𝑘∈𝐵(0,𝛿),∀𝑘∈ℕ. (4.28) Since 𝐶_𝑗(𝑥)=∑^𝑝_𝑖=1𝑔_𝑖𝑗(𝑥_𝑖) is continuous for all 𝑗=1,𝑚, then, 𝐶_𝑗(𝐵(0,𝛿)) is closed and bounded for all 𝑗=1,𝑚. Then there exists 𝑀_𝑗≥0such that ^𝑝_𝑖=1𝑔_𝑖𝑗𝑥_𝑖≤𝑀_𝑗,∀𝑥∈𝐵(0,𝛿),∀𝑗=1,𝑚. (4.29) Let 𝜌 = max{𝑀_𝑗∶𝑗=1,𝑚}, then ^𝑝_𝑖=1𝑔_𝑖𝑗𝑥_𝑖≤𝜌,∀𝑥∈𝐵(0,𝛿),∀𝑗=1,𝑚. (4.30) Therefore, |∑^𝑝_𝑖=1𝑔_𝑖𝑗(𝑥^𝑘_𝑖)|≤𝜌, for all 𝑗=1,𝑚, for all 𝑘∈ℕ, which means {∑^𝑝_𝑖=1𝑔_𝑖𝑗(𝑥^𝑘_𝑖)∣𝑗=1,𝑚,𝑠∈ℕ} is a bounded set.
Thus, for 𝑗 such that 𝑢_𝑖𝑗=0, and the above result, we getlim_𝑠→∞𝑢^𝑘+1_𝑖𝑗^𝑝_𝑖=1𝑔_𝑖𝑗𝑥_𝑖^𝑘+1=0. (4.31) That is, lim_𝑠→∞𝑢^𝑘+1_𝑖𝑗^𝑝_𝑖=1𝑔_𝑖𝑗𝑥_𝑖^𝑘+1=0∀𝑗=1,𝑚. (4.32) Also, if 𝑥 is a limit point of {𝑥^𝑘}, keeping in mind that (see the proof given by Polyak in [38]) max{−𝑔_𝑖𝑗(𝑥^𝑘_𝑖}→0, then lim_𝑠→∞^𝑝_𝑖=1𝑔_𝑖𝑗𝑥_𝑖^𝑘+1≥0∀𝑗=1,𝑚. (4.33) On another hand, from (4.32) we get ∇_𝑥𝐿(𝑥,𝑢)(=0⟹∇𝐹𝑥)−^𝑚_𝑗=1^𝑝_𝑖=1𝑢_𝑖𝑗∇𝑔_𝑖𝑗𝑥_𝑖=0,𝑢_𝑖𝑗≥0,𝑗=1,𝑚,𝑢_𝑖𝑗^𝑝_𝑖=1𝑔_𝑖𝑗𝑥_𝑖=0,from(4.32_𝑝𝑖=1),𝑔_𝑖𝑗𝑥_𝑖≥0,from(4.33). (4.34)(4.35) Since the Lagrange of problem (𝑆𝑃) is 𝐿(𝑥,𝑢)=𝐹(𝑥)−^𝑚_𝑗=1^𝑝_𝑖=1𝑢_𝑖𝑗𝑔_𝑖𝑗𝑥_𝑖 (4.36) and it is convex, then from (4.34) 𝑥∈Argmin_{𝑥∈ℝ𝑛}𝐿(𝑥,𝑢) (4.37) and then 𝐹(𝑥)=𝐹(𝑥)−^𝑚_𝑗=1𝑢_𝑖𝑗^𝑝_𝑖=1𝑔_𝑖𝑗𝑥_𝑖=𝐿(𝑥,𝑢)≤𝐿(𝑥,𝑢),∀𝑥∈ℝ^𝑛. (4.38) Also, we know that ∑^𝑝_𝑖=1𝑔_𝑖𝑗(𝑥_𝑖)≥0, for all 𝑗=1,𝑚, and for any 𝑢_𝑖𝑗≥0∶−∑^𝑚_𝑗=1∑^𝑝_𝑖=1𝑢_𝑖𝑗𝑔_𝑖𝑗(𝑥_𝑖)≤0 which implies ⟹𝐿(𝑥,𝑢)=𝐹(𝑥)−^𝑚_𝑗=1^𝑝_𝑖=1𝑢_𝑖𝑗𝑔_𝑖𝑗𝑥_𝑖≤𝐹(𝑥)=𝐿(𝑥,𝑢) (4.39) and then 𝐿(𝑥,𝑢)≤𝐿(𝑥,𝑢)≤𝐿(𝑥,𝑢) (4.40) for all𝑥∈ℝ^𝑛, for all 𝑢≥0, and by the saddle point theorem we have (𝑥,𝑢)∈𝑋^∗×𝐿^∗. (4.41) Since {𝑑(𝑢^𝑘)} is an increasing sequence bounded above by 𝑑(𝑢^∗) then it is convergent. Let {(𝑥^𝑘𝑙,𝑢^𝑘𝑙)} be any convergent subsequence of {(𝑥^𝑘,𝑢^𝑘)}. Then from (3.4) lim_𝑙→∞𝐿𝑥^𝑘𝑙,𝑢^𝑘𝑙=𝐹𝑥^∗=𝑑𝑢^∗. (4.42) Since ∇_𝑥𝐿(𝑥^𝑘,𝑢^𝑘)=0⟹min_{𝑥∈ℝ𝑛}𝐿𝑥,𝑢^𝑘=𝐿𝑥^𝑘,𝑢^𝑘⟹𝑑𝑢^𝑘=𝐿𝑥^𝑘,𝑢^𝑘⟹lim_𝑙→∞𝐿𝑥^𝑘𝑙,𝑢^𝑘𝑙=lim_𝑙→∞𝑑𝑢^𝑘𝑙=𝑑𝑢^∗. (4.43) Then {𝑑(𝑢^𝑘)} converges to 𝑑(𝑢^∗). The remaining proof is similar to the one given in [30].

In the previous theorem, we proved the boundedness of the primal and dual sequences, but for the sequence of allocation vectors {𝑦^𝑘} we can get the boundedness directly by proving that the set𝑔_𝑖𝑗𝑥^𝑘_𝑖∣𝑗=1,𝑚,𝑖=1,𝑝,𝑘∈ℕ (4.44) is bounded. (The same way as we proved that the set {∑^𝑝_𝑖=1𝑔_𝑖𝑗(𝑥^𝑘_𝑖)∣𝑗=1,𝑚,𝑘∈ℕ} is bounded.).