Well-Posedness and Primal-Dual Analysis of Some Convex Separable Optimization Problems

Abstract

We focus on some convex separable optimization problems, considered by the author in previous papers, for which problems, necessary and sufficient conditions or sufficient conditions have been proved, and convergent algorithms of polynomial computational complexity have been proposed for solving these problems. The concepts of well-posedness of optimization problems in the sense of Tychonov, Hadamard, and in a generalized sense, as well as calmness in the sense of Clarke, are discussed. It is shown that the convex separable optimization problems under consideration are calm in the sense of Clarke. The concept of stability of the set of saddle points of the Lagrangian in the sense of Gol'shtein is also discussed, and it is shown that this set is not stable for the “classical” Lagrangian. However, it turns out that despite this instability, due to the specificity of the approach, suggested by the author for solving problems under consideration, it is not necessary to use modified Lagrangians but only the “classical” Lagrangians. Also, a primal-dual analysis for problems under consideration in view of methods for solving them is presented.

1. Introduction

1.1. Statement of Problems under Consideration: Preliminary Results

In this paper, we study well-posedness and present primal-dual analysis of some convex separable optimization problems, considered by the author in previous papers. For the sake of convenience, in this subsection we recall main results of earlier papers that are used in this study.

In paper [1], the following convex separable optimization problem was considered:

(𝐶)min𝑐(𝑥)=_𝑗∈𝐽𝑐_𝑗𝐱_𝑗,subjectto_𝑗∈𝐽𝑑_𝑗𝑥_𝑗𝑎≤𝛼,_𝑗≤𝑥_𝑗≤𝑏_𝑗,𝑗∈𝐽, (1)(2)(3) where 𝑐_𝑗(𝑥_𝑗) are twice differentiable strictly convex functions, and 𝑑_𝑗(𝑥_𝑗) are twice differentiable convex functions, defined on the open convex sets 𝑋_𝑗 in  ℝ,𝑗∈𝐽, respectively, 𝑑^′_𝑗(𝑥_𝑗)>0 for every 𝑗∈𝐽,  𝐱=(𝑥_𝑗)_𝑗∈𝐽, and 𝐽≡{1,…,𝑛}.

Assumptions for problem (𝐶) are as follows.(I.1)𝑎_𝑗≤𝑏_𝑗 for all 𝑗∈𝐽. If 𝑎_𝑘=𝑏_𝑘 for some 𝑘∈𝐽, then the value 𝑥_𝑘∶=𝑎_𝑘=𝑏_𝑘 is determined a priori.(II.1)∑_𝑗∈𝐽𝑑_𝑗(𝑎_𝑗)≤𝛼. Otherwise, the constraints (2) and (3) are inconsistent and the feasible set 𝑋, defined by (2)-(3), is empty. In addition to this assumption, we suppose that 𝛼≤∑_𝑗∈𝐽𝑑_𝑗(𝑏_𝑗) in some cases which are specified below.(III.1) (Slater’s constraint qualification) There exists a point 𝐱=(𝑥₁,…,𝑥_𝑛)∈𝑋 such that ∑_𝑗∈𝐽𝑑_𝑗(𝑥_𝑗)<𝛼.

Under these assumptions, the following characterization theorem (necessary and sufficient condition) for problem (𝐶) was proved in [1].

Denote by ℎ^≤_𝑗,ℎ⁼_𝑗,ℎ^≥_𝑗 the values of 𝑥_𝑗, for which 𝑐^′_𝑗(𝑥_𝑗)=0 for the three problems under consideration in this paper, respectively.

Theorem 1 (characterization of the optimal solution to problem (𝐶)). Under the above assumptions, a feasible solution 𝐱^∗=(𝑥^∗_𝑗)_𝑗∈𝐽∈𝑋 is an optimal solution to problem (𝐶) if and only if there exists a 𝜆∈ℝ¹₊ such that 𝑥^∗_𝑗=𝑎_𝑗,𝑗∈𝐽=^𝜆_𝑎^def𝑐𝑗∈𝐽∶𝜆≥−^′𝑗𝑎_𝑗𝑑^′_𝑗𝑎_𝑗𝑥^∗_𝑗=𝑏_𝑗,𝑗∈𝐽=^𝜆_𝑏^def𝑐𝑗∈𝐽∶𝜆≤−^′𝑗𝑏_𝑗𝑑^′_𝑗𝑏_𝑗𝑥^∗_𝑗∶𝜆𝑑^′_𝑗𝑥^∗_𝑗=−𝑐^′_𝑗𝑥^∗_𝑗,𝑗∈𝐽=^𝜆def𝑐𝑗∈𝐽∶−^′𝑗𝑏_𝑗𝑑^′_𝑗𝑏_𝑗𝑐<𝜆<−^′𝑗𝑎_𝑗𝑑^′_𝑗𝑎_𝑗. (4)(5)(6)

The following polynomial algorithm for solving problem (𝐶) with strictly convex differentiable functions 𝑐_𝑗(𝑥_𝑗),𝑗∈𝐽, was suggested in [1].

Algorithm 2. (0) Initialization: 𝐽∶={1,…,𝑛},  𝑘∶=0,  𝛼⁽⁰⁾∶=𝛼,  𝑛⁽⁰⁾∶=𝑛,𝐽⁽⁰⁾∶=𝐽,   𝐽^𝜆_𝑎∶=∅,   𝐽^𝜆_𝑏∶=∅, initialize ℎ^≤_𝑗, 𝑗∈𝐽. If ∑_𝑗∈𝐽𝑑_𝑗(𝑎_𝑗)≤𝛼 go to Step (1), else go to Step (9). (1) Construct the sets 𝐽⁰_𝑎,  𝐽⁰_𝑏,  𝐽⁰ by using (4), (5), (6) with 𝜆=0.  Calculate 𝛿(0)∶=_{𝑗∈𝐽0𝑎}𝑑_𝑗𝑎_𝑗+_{𝑗∈𝐽0𝑏}𝑑_𝑗𝑏_𝑗+_𝑗∈𝐽0𝑑_𝑗ℎ^≤_𝑗−𝛼. (7) If 𝛿(0)≤0 then 𝜆∶=0, go to Step (8)  else if 𝛿(0)>0, then  if 𝛼≤∑_𝑗∈𝐽𝑑_𝑗(𝑏_𝑗) go to Step (2)  else if 𝛼>∑_𝑗∈𝐽𝑑_𝑗(𝑏_𝑗) go to Step (9)   (there does not exist 𝜆^∗>0 such that 𝛿(𝜆^∗)=0). (2)𝐽^𝜆(𝑘)∶=𝐽^(𝑘). Calculate 𝜆^(𝑘) by using the explicit expression of 𝜆, calculated from the equality ∑_𝑗∈𝐽^𝜆(𝑘)𝑑_𝑗(𝑥_𝑗)=𝛼^(𝑘), where 𝑥_𝑗,𝑗∈𝐽^𝜆(𝑘), are given by (6). Go to Step (3). (3) Construct the sets 𝐽_𝑎^𝜆(𝑘),  𝐽_𝑏^𝜆(𝑘),  𝐽^𝜆(𝑘) through (4), (5), (6) (with 𝑗∈𝐽^(𝑘) instead of 𝑗∈𝐽) and find their cardinal numbers|𝐽_𝑎^𝜆(𝑘)|,  |𝐽_𝑏^𝜆(𝑘)|,  |𝐽^𝜆(𝑘)|, respectively. Go to Step (4). (4) Calculate 𝛿𝜆^(𝑘)∶=_{𝑗∈𝐽𝑎𝜆(𝑘)}𝑑_𝑗𝑎_𝑗+_{𝑗∈𝐽𝑏𝜆(𝑘)}𝑑_𝑗𝑏_𝑗+_{𝑗∈𝐽𝜆(𝑘)}𝑑_𝑗𝑥^∗_𝑗−𝛼^(𝑘), (8) where 𝑥^∗_𝑗,𝑗∈𝐽^𝜆(𝑘), are calculated from (6) with 𝜆=𝜆^(𝑘). Go to Step (5). (5) If 𝛿(𝜆^(𝑘))=0 or 𝐽^𝜆(𝑘)=∅ then 𝜆∶=𝜆^(𝑘),  𝐽^𝜆_𝑎∶=𝐽^𝜆_𝑎∪𝐽_𝑎^𝜆(𝑘),𝐽^𝜆_𝑏∶=𝐽^𝜆_𝑏∪𝐽_𝑏^𝜆(𝑘),  𝐽^𝜆∶=𝐽^𝜆(𝑘), go to Step (8)  else if 𝛿(𝜆^(𝑘))>0 go to Step (6)  else if 𝛿(𝜆^(𝑘))<0 go to Step (7). (6)𝑥^∗_𝑗∶=𝑎_𝑗 for 𝑗∈𝐽_𝑎^𝜆(𝑘),  𝛼^(𝑘+1)∶=𝛼^(𝑘)−∑_{𝑗∈𝐽𝑎}^𝜆(𝑘)𝑑_𝑗(𝑎_𝑗),  𝐽^(𝑘+1)∶=𝐽^(𝑘)⧵𝐽_𝑎^𝜆(𝑘), 𝑛^(𝑘+1)∶=𝑛^(𝑘)−|𝐽_𝑎^𝜆(𝑘)|,  𝐽^𝜆_𝑎∶=𝐽^𝜆_𝑎∪𝐽_𝑎^𝜆(𝑘),  𝑘∶=𝑘+1. Go to Step (2). (7)𝑥^∗_𝑗∶=𝑏_𝑗 for 𝑗∈𝐽_𝑏^𝜆(𝑘),  𝛼^(𝑘+1)∶=𝛼^(𝑘)−∑_{𝑗∈𝐽𝑏}^𝜆(𝑘)𝑑_𝑗(𝑏_𝑗),  𝐽^(𝑘+1)∶=𝐽^(𝑘)⧵𝐽_𝑏^𝜆(𝑘), 𝑛^(𝑘+1)∶=𝑛^(𝑘)−|𝐽_𝑏^𝜆(𝑘)|,  𝐽^𝜆_𝑏∶=𝐽^𝜆_𝑏∪𝐽_𝑏^𝜆(𝑘),  𝑘∶=𝑘+1. Go to Step (2). (8)𝑥^∗_𝑗∶=𝑎_𝑗 for 𝑗∈𝐽^𝜆_𝑎; 𝑥^∗_𝑗∶=𝑏_𝑗 for 𝑗∈𝐽^𝜆_𝑏; assign 𝑥^∗_𝑗 the value calculated from (6) for 𝑗∈𝐽^𝜆. Go to Step (10). (9) Problem (𝐶) has no optimal solution because 𝑋=∅ or there does not exist𝜆>0 satisfying Theorem 1. (10) End.

It is proved in [1] that this algorithm is convergent.

Theorem 3 (convergence of Algorithm 2). Let {𝜆^(𝑘)} be the sequence generated by Algorithm 2. Then, (i)if 𝛿(𝜆^(𝑘))>0, then 𝜆^(𝑘)≤𝜆^(𝑘+1), (ii)if 𝛿(𝜆^(𝑘))<0, then 𝜆^(𝑘)≥𝜆^(𝑘+1).

In paper [2], the following two convex separable optimization problems were considered:

(𝐶⁼)min𝑐(𝐱)=_𝑗∈𝐽𝑐_𝑗𝑥_𝑗subjectto_𝑗∈𝐽𝑑_𝑗𝑥_𝑗𝑎=𝛼,_𝑗≤𝑥_𝑗≤𝑏_𝑗,𝑗∈𝐽, (9)(10)(11) and

(𝐶^≥)min𝑐(𝐱)=_𝑗∈𝐽𝑐_𝑗𝑥_𝑗subjectto_𝑗∈𝐽𝑑_𝑗𝑥_𝑗𝑎≥𝛼_𝑗≤𝑥_𝑗≤𝑏_𝑗,𝑗∈𝐽, (12)(13)(14) where for both problems, 𝑐_𝑗(𝑥_𝑗) are twice differentiable convex functions, defined on the open convex sets 𝑋_𝑗 in ℝ, 𝑗∈𝐽, respectively, 𝑑_𝑗>0, for every 𝑗∈𝐽, 𝐱=(𝑥_𝑗)_𝑗∈𝐽, and 𝐽={1,…,𝑛}.

Assumptions for problem (𝐶⁼) are as follows.(I.2)𝑎_𝑗≤𝑏_𝑗 for each 𝑗∈𝐽.(II.2)∑_𝑗∈𝐽𝑑_𝑗𝑎_𝑗≤𝛼≤∑_𝑗∈𝐽𝑑_𝑗𝑏_𝑗. Otherwise the constraints (10), (11) are inconsistent and 𝑋^𝐿=∅, where 𝑋^𝐿 is defined by (10)-(11).

Under these assumptions, the following characterization theorem (necessary and sufficient condition) for problem (𝐶⁼) is proved in [2].

Theorem 4 (characterization of the optimal solution to problem (𝐶⁼)). A feasible solution 𝐱^∗=(𝑥^∗_𝑗)_𝑗∈𝐽∈𝑋^𝐿 is an optimal solution to problem (𝐶⁼) if and only if there exists a 𝜆∈ℝ¹ such that 𝑥^∗_𝑗=𝑎_𝑗,𝑗∈𝐽=^𝜆_𝑎^def𝑐𝑗∈𝐽∶𝜆≥−^′𝑗𝑎_𝑗𝑑_𝑗𝑥_∗𝑗,=𝑏_𝑗,𝑗∈𝐽=^𝜆_𝑏^def𝑐𝑗∈𝐽∶𝜆≤−^′𝑗𝑏_𝑗𝑑_𝑗𝑥_∗𝑗,∶𝜆𝑑_𝑗=−𝑐^′_𝑗𝑥^∗_𝑗,𝑗∈𝐽=^𝜆def𝑐𝑗∈𝐽∶−^′𝑗𝑏_𝑗𝑑_𝑗𝑐<𝜆<−^′𝑗𝑎_𝑗𝑑_𝑗. (15)(16)(17)

Assumptions for problem (𝐶^≥) are as follows.(I.3)𝑎_𝑗≤𝑏_𝑗 for all 𝑗∈𝐽.(II.3)𝛼≤∑_𝑗∈𝐽𝑑_𝑗𝑏_𝑗. Otherwise the constraints (13), (14) are inconsistent and 𝑋^≥=∅, where 𝑋^≥ is defined by (13)-(14).

Under these assumptions, the following theorem (sufficient condition) for problem (𝐶^≥) is proved in [2].

Theorem 5 (sufficient condition for optimal solution to problem (𝐶^≥)). Let𝑥^∗_𝑗, 𝑗∈𝐽be components of the optimal solution to problem 𝐶⁼. Then:(i) If 𝜆=−𝑐^′_𝑗(𝑥^∗_𝑗)/𝑑_𝑗≤0, then 𝑥^∗_𝑗,𝑗∈𝐽, solve problem (𝐶^≥) as well. (ii) If 𝜆=−𝑐^′_𝑗(𝑥^∗_𝑗)/𝑑_𝑗>0, then 𝑥^≥_𝑗,𝑗∈𝐽 defined as follows:  𝑥^≥_𝑗=𝑏_𝑗,𝑗∈𝐽^𝜆_𝑏,  𝑥^≥_𝑗=min{𝑏_𝑗,ℎ^≥_𝑗},𝑗∈𝐽^𝜆,  𝑥^≥_𝑗=min{𝑏_𝑗,ℎ^≥_𝑗}for all 𝑗∈𝐽^𝜆_𝑎 such that 𝑐^′_𝑗(𝑎_𝑗)<0, 𝑥^≥_𝑗=𝑎_𝑗  for all  𝑗∈𝐽^𝜆_𝑎, such that 𝑐^′_𝑗(𝑎_𝑗)≥0
solve problem (𝐶^≥).

The following polynomial algorithm for solving problem (𝐶⁼) with strictly convex differentiable functions 𝑐_𝑗(𝑥_𝑗) was suggested in [2].

Algorithm 6. (1) Initialization: 𝐽∶={1,…,𝑛},  𝑘∶=0,  𝛼⁽⁰⁾∶=𝛼,  𝑛⁽⁰⁾∶=𝑛,𝐽⁽⁰⁾∶=𝐽,  𝐽^𝜆_𝑎∶=∅,  𝐽^𝜆_𝑏∶=∅, initialize ℎ⁼_𝑗,𝑗∈𝐽. If ∑_𝑗∈𝐽𝑑_𝑗𝑎_𝑗≤𝛼≤∑_𝑗∈𝐽𝑑_𝑗𝑏_𝑗 go to Step (2), else go to Step (9). (2) 𝐽^𝜆(𝑘)∶=𝐽^(𝑘). Calculate 𝜆^(𝑘) by using the explicit expression of 𝜆, calculated from the equality constraint ∑_𝑗∈𝐽^𝜆(𝑘)𝑑_𝑗𝑥_𝑗=𝛼^(𝑘), where 𝑥_𝑗,𝑗∈𝐽^𝜆(𝑘), are given by (17). Go to Step (3). (3) Construct the sets 𝐽_𝑎^𝜆(𝑘),𝐽_𝑏^𝜆(𝑘),𝐽^𝜆(𝑘) through (15), (16), (17) (with 𝐽^(𝑘) instead of 𝐽) and find their cardinalities |𝐽_𝑎^𝜆(𝑘)|,|𝐽_𝑏^𝜆(𝑘)|,|𝐽^𝜆(𝑘)|, respectively. Go to Step (4). (4) Calculate 𝛿𝜆^(𝑘)∶=_{𝑗∈𝐽𝑎𝜆(𝑘)}𝑑_𝑗𝑎_𝑗+_{𝑗∈𝐽𝑏𝜆(𝑘)}𝑑_𝑗𝑏_𝑗+_{𝑗∈𝐽𝜆(𝑘)}𝑑_𝑗𝑥^∗_𝑗−𝛼^(𝑘), (18) where 𝑥^∗_𝑗,𝑗∈𝐽^𝜆(𝑘), are calculated from (17) with 𝜆=𝜆^(𝑘). Go to Step (5). (5) If 𝛿(𝜆^(𝑘))=0 or 𝐽^𝜆(𝑘)=∅ then 𝜆∶=𝜆^(𝑘),  𝐽^𝜆_𝑎∶=𝐽^𝜆_𝑎∪𝐽_𝑎^𝜆(𝑘),𝐽^𝜆_𝑏∶=𝐽^𝜆_𝑏∪𝐽_𝑏^𝜆(𝑘),  𝐽^𝜆∶=𝐽^𝜆(𝑘), go to Step (8)  else if 𝛿(𝜆^(𝑘))>0 go to Step (6)  else if 𝛿(𝜆^(𝑘))<0 go to Step (7). (6)𝑥^∗_𝑗∶=𝑎_𝑗 for 𝑗∈𝐽_𝑎^𝜆(𝑘),  𝛼^(𝑘+1)∶=𝛼^(𝑘)−∑_{𝑗∈𝐽𝑎}^𝜆(𝑘)𝑑_𝑗𝑎_𝑗,  𝐽^(𝑘+1)∶=𝐽^(𝑘)⧵𝐽_𝑎^𝜆(𝑘), 𝑛^(𝑘+1)∶=𝑛^(𝑘)−|𝐽_𝑎^𝜆(𝑘)|,  𝐽^𝜆_𝑎∶=𝐽^𝜆_𝑎∪𝐽_𝑎^𝜆(𝑘),  𝑘∶=𝑘+1. Go to Step (2). (7)𝑥^∗_𝑗∶=𝑏_𝑗 for 𝑗∈𝐽_𝑏^𝜆(𝑘),  𝛼^(𝑘+1)∶=𝛼^(𝑘)−∑_{𝑗∈𝐽𝑏}^𝜆(𝑘)𝑑_𝑗𝑏_𝑗,  𝐽^(𝑘+1)∶=𝐽^(𝑘)⧵𝐽_𝑏^𝜆(𝑘), 𝑛^(𝑘+1)∶=𝑛^(𝑘)−|𝐽_𝑏^𝜆(𝑘)|,  𝐽^𝜆_𝑏∶=𝐽^𝜆_𝑏∪𝐽_𝑏^𝜆(𝑘),  𝑘∶=𝑘+1. Go to Step (2). (8)𝑥^∗_𝑗∶=𝑎_𝑗 for 𝑗∈𝐽^𝜆_𝑎;𝑥^∗_𝑗∶=𝑏_𝑗 for 𝑗∈𝐽^𝜆_𝑏; assign 𝑥^∗_𝑗 the value, calculated from (17) for 𝑗∈𝐽^𝜆. Go to Step (10). (9) The problem has no optimal solution because 𝑋^𝐿=∅.(10) End.

It is proved in [2] that this algorithm is convergent.

Theorem 7 (convergence of Algorithm 6). Let {𝜆^(𝑘)} be the sequence generated by Algorithm 6. Then,(i)if 𝛿(𝜆^(𝑘))>0, then 𝜆^(𝑘)≤𝜆^(𝑘+1),(ii)if 𝛿(𝜆^(𝑘))<0, then 𝜆^(𝑘)≥𝜆^(𝑘+1).

The following algorithm for solving problem (𝐶^≥) with strictly convex differentiable functions 𝑐_𝑗(𝑥_𝑗) is suggested in [2].

Algorithm 8. (1) Initialization: 𝐽∶={1,…,𝑛},  𝑘∶=0,  𝛼⁽⁰⁾∶=𝛼,  𝑛⁽⁰⁾∶=𝑛,𝐽⁽⁰⁾∶=𝐽,  𝐽^𝜆_𝑎∶=∅,  𝐽^𝜆_𝑏∶=∅,   initialize ℎ^≥_𝑗,𝑗∈𝐽. If ∑_𝑗∈𝐽𝑑_𝑗𝑎_𝑗≤𝛼≤∑_𝑗∈𝐽𝑑_𝑗𝑏_𝑗 then go to Step (2), else go to Step (9).  Steps (2)–(7) are the same as Steps (2)–(7) of Algorithm 6, respectively. (8) If 𝜆≤0 then 𝑥^≥_𝑗∶=𝑎_𝑗 for 𝑗∈𝐽^𝜆_𝑎;  𝑥^≥_𝑗∶=𝑏_𝑗 for 𝑗∈𝐽^𝜆_𝑏; assign 𝑥^≥_𝑗 the value, calculated through (17) for 𝑗∈𝐽^𝜆, go to Step (10)  else if 𝜆>0 then  𝑥^≥_𝑗∶=𝑏_𝑗 for 𝑗∈𝐽^𝜆_𝑏, 𝑥^≥_𝑗∶=min{𝑏_𝑗,ℎ^≥_𝑗} for 𝑗∈𝐽^𝜆, if 𝑗∈𝐽^𝜆_𝑎 and 𝑐^′_𝑗(𝑎_𝑗)<0 then 𝑥^≥_𝑗∶=min{𝑏_𝑗,ℎ^≥_𝑗} else if 𝑗∈𝐽^𝜆_𝑎 and 𝑐^′_𝑗(𝑎_𝑗)≥0 then 𝑥^≥_𝑗∶=𝑎_𝑗,  go to Step (10). (9) Problem (𝐶^≥) has no optimal solution because 𝑋^≥=∅ or there do not exist 𝑥^∗_𝑗∈[𝑎_𝑗,𝑏_𝑗],𝑗∈𝐽, such that ∑_𝑗∈𝐽𝑑_𝑗𝑥^∗_𝑗=𝛼. (10) End.

Since Algorithm 8 is based on Theorem 5 and Algorithm 6, and since the “iterative” Steps (2)–(7) of Algorithms 6 and 8 are the same, then the “convergence” of Algorithm 8 follows from Theorem 7 as well.

1.2. Organization of the Paper

The rest of the paper is organized as follows. In Section 2, the concepts of well-posedness of optimization problems in the sense of Tychonov, Hadamard, and in a generalized sense, as well as calmness in the sense of Clarke, are discussed. It is shown in Section 2.3 that the convex separable optimization problems under consideration are calm in the sense of Clarke. In Section 3, the concept of stability of the set of saddle points of the Lagrangian in the sense of Gol’shtein is also discussed and it is shown that this set is not stable for the “classical” Lagrangian. However, it is explained that despite this instability, due to the specificity of the approach, suggested by the author in previous papers for solving problems under consideration, it is not necessary to use modified Lagrangians but only the “classical” Lagrangians. In Section 4, primal-dual analysis of the problems under consideration in view of methods for solving them is presented. Main results of well-posedness and primal-dual analysis are included in Section 2.3 and in Sections 3 and 4.

2. Well-Posedness of Optimization Problems

Questions of existence of solutions and how they depend on problem’s parameters are usually important for many problems of mathematics, not only in optimization. The term well-posedness refers to the existence and uniqueness of a solution and its continuous behavior with respect to data perturbations, which is referred to as stability. In general, a problem is said to be stable if 𝜀(𝛿)⟶0when𝛿⟶0, (19) where 𝛿 is a given tolerance of the problem’s data, 𝜀(𝛿) is the accuracy with which the solution can be determined, and 𝜀(𝛿) is a continuous function of 𝛿. Besides these conditions, accompanying robustness properties in the convergence of sequence of approximate solutions are also required.

Problems which are not well-posed are called ill-posed, or, sometimes, improperly posed.

2.1. Tychonov and Hadamard Well-Posedness: Well-Posedness in the Generalized Sense

Recall that 𝑓 is a proper function if 𝑓(𝐱)<∞ for at least one 𝐱∈ℝ^𝑛 and 𝑓(𝐱)>−∞ for all 𝐱∈ℝ^𝑛, or, in other words, if dom𝑓=^def𝐱∈ℝ^𝑛∶𝑓(𝐱)<∞ (20) is a nonempty set on which 𝑓(𝐱)>−∞, where dom 𝑓 is the effective domain of 𝑓. Otherwise, 𝑓 is improper.

Definition 9. Let 𝑋 be a space with either a topology or a convergence structure associated and let 𝑓∶𝑋→ℝ≡ℝ∪{+∞} be a proper extended real-valued function. Consider the problem min𝑓(𝐱)subjectto𝐱∈𝑋. (21) The problem (21) is Tychonov well-posed if and only if 𝑓 has a unique global minimum point on 𝑋 towards which every minimizing sequence converges.

An equivalent definition is as follows: problem (21) is Tychonov well-posed if and only if there exists a unique 𝐱₀∈𝑋 such that 𝑓(𝐱₀)≤𝑓(𝐱) for all 𝐱∈𝑋 and 𝑓𝐱_𝑛⟶𝑓𝐱₀implies𝐱_𝑛⟶𝐱₀. (22)

There are two ways to cope with ill-posedness.

The first one is to change the statement of the problem.

The second one is the so-called Tychonov regularization. A parametric functional is constructed such that if it approaches 0, the solution of the “regularized” problem converges to the exact solution of the original problem.

Consider the problem min_{𝐱∈𝑋⊂ℝ𝑛}𝑓(𝐱)=𝑓𝐱^∗. (23)

Associate the following problem with (23): min_𝐱∈𝑋𝑓(𝐱)+𝑛_𝑘(𝐱)=𝑧_𝑘𝐱^∗𝑛_𝑘(𝐱), (24) where 𝑛_𝑘(𝐱) is perturbation in the input data and 𝐱^∗(𝑛_𝑘(𝐱)) is an optimal solution to the perturbed problem.

Let _ℝ𝑛𝑛²_𝑘(𝐱)𝑑𝐱≤𝑐²_𝑘<∞. (25)

If 𝑧_𝑘𝐱^∗𝑛_𝑘(𝐱)−𝑓𝐱^∗⟶0, (26) when 𝑐_𝑘→0, then problem (23) is stable with respect to perturbation 𝑛_𝑘(𝐱).

A parametric function 𝐹(𝐱,Δ,𝑓(𝐱)) with a parameter Δ is called a regularizing function for problem (23) with respect to perturbation 𝑛_𝑘(𝐱) if the following conditions are satisfied.

(1) 𝐹(𝐱,Δ,𝑓(𝐱)) is defined for all 𝐱∈𝑋 and Δ>0.

(2) If 𝐱^∗(𝑛_𝑘(𝐱),Δ) is an optimal solution to problem min_𝐱∈𝑋𝐹𝐱,Δ,𝑧_𝑘(𝐱)=𝐹𝐱^∗𝑛_𝑘(𝐱),Δ,Δ,𝑧_𝑘(𝐱), (27) then there exists a function Δ_𝑘=Δ_𝑘(𝑐_𝑘) such that 𝐹𝐱^∗𝑛_𝑘(𝐱),Δ_𝑘,Δ_𝑘,𝑧_𝑘(𝐱)⟶𝑓𝐱^∗, (28) when 𝑐_𝑘→0.

Following Tychonov, an ill-posed problem is said to be regularizable if there exists at least one regularizing function for it.

The concept of Tychonov well-posedness can be extended to problems without the uniqueness of the optimal solution.

Definition 10. Let 𝑋 be a space with either a topology or a convergence structure associated, and 𝑓∶𝑋→ℝ≡ℝ∪{+∞} be a proper real-valued function. Problem (21) is said to be well-posed in the generalized sense if and only if argmin_𝐱∈𝑋𝑓(𝐱)≠∅ and every sequence {𝐮_𝑛}⊂𝑋 such that 𝑓(𝐮_𝑛)→inf{𝑓(𝐱)∶𝐱∈𝑋} has some subsequence {𝐯_𝑛}→𝐮 with 𝐮∈ arg min_𝐱∈𝑋𝑓(𝐱).

Problem (21) is Tychonov well-posed if and only if it is well-posed in the generalized sense and arg min_𝐱∈𝑋𝑓(𝐱) is a singleton.

Hadamard well-posedness is primarily connected with problems of mathematical physics (boundary value problems for partial differential equations) and can be extended to mathematical programming problems. We do not discuss this topic here.

As recent studies in the calculus of variations, optimal control, and numerical methods of optimization show, uniqueness and continuity are often too restrictive to be adopted as the standards of well-posedness. It turns out that practical concepts concerning well-posedness are some forms of semicontinuity in the problem’s data and solution mapping, along with potential multivaluedness in this mapping.

2.2. Calmness in the Sense of Clarke

Let 𝑋 be a Banach space.

Definition 11. Let 𝑌 be a subset of 𝑋. A function 𝑓∶𝑌→ℝ is said to satisfy a Lipschitz condition on 𝑌 provided that, for some nonnegative scalar 𝐾, the following inequality holds true: 𝑓(𝐲)−𝑓𝐲^′≤𝐾𝐲−𝐲^′, (29) for all points 𝐲,𝐲^′∈𝑌; this is also referred to as a Lipschitz condition of rank 𝐾. We say that 𝑓 is Lipschitz (of rank 𝐾) near 𝐱 if for some 𝜀>0, 𝑓 satisfies a Lipschitz condition (of rank 𝐾) on the set 𝐱+𝜀𝐵 (i.e., within an 𝜀-neighborhood of 𝐱), where 𝐵 is the open unit ball around 𝟎.
A function 𝑓, which satisfies a Lipschitz condition, sometimes is said to be Lipschitz continuous.

Consider the following general mathematical programming problem:

(𝑃)min𝑓(𝐱)subjectto𝑔_𝑖(𝐱)ℎ≤0,𝑖=1,…,𝑡,_𝑗(𝐱)=0,𝑗=1,…,𝑚,𝐱∈𝐶,𝐶⊂𝑋, (30) where 𝑔_𝑖,ℎ_𝑗 are real-valued functions on 𝑋.

Let 𝐠 and 𝐡 be the functions 𝐠=[𝑔₁,…,𝑔_𝑡]∶𝑋→ℝ^𝑡,  𝐡=[ℎ₁,…,ℎ_𝑚]∶𝑋→ℝ^𝑚.

Let (𝑃) be imbedded in a parametrized family 𝑃(𝐩,𝐪) of mathematical programs, where 𝐩∈ℝ^𝑡,  𝐪∈ℝ^𝑚:

𝑃(𝐩,𝐪)(𝐱)(𝐱)𝐡(𝐱)min𝑓subjectto𝐠+𝐩≤𝟎+𝐪=𝟎𝐱∈𝐶. (31)

Denote by 𝐴 the feasible region of problem 𝑃(𝐩,𝐪).

Definition 12 (Clarke [3]). The value function 𝑉∶ℝ^𝑡×ℝ^𝑚→ℝ∪{±∞} is defined via 𝑉(𝐩,𝐪)=inf{𝑃(𝐩,𝐪)} (i.e., the value of the problem 𝑃(𝐩,𝐪)). If there are no feasible points for 𝑃(𝐩,𝐪), then the infimum is over the empty set and 𝑉(𝐩,𝐪) is assigned the value +∞.

Definition 13 (Clarke [3]). Let 𝐱 solve (𝑃). The problem (𝑃) is calm at 𝐱 provided that there exist positive 𝜀 and 𝑀 such that for all (𝐩,𝐪)∈𝜀𝐵, for all 𝐱^′∈𝐱+𝜀𝐵 which are feasible for 𝑃(𝐩,𝐪), one has 𝑓𝐱^′−𝑓(𝐱)+𝑀‖(𝐩,𝐪)‖≥0, (32) where 𝐵 is the open unit ball in 𝑋 and ‖(𝐩,𝐪)‖ is the Euclidean norm of (𝐩,𝐪).

Let 𝑈 be an open convex subset of 𝑋.

Theorem 14 (Roberts and Varberg [4], Clarke [3]; Lipschitz condition from boundedness of a convex function). Let 𝑓 be a convex function, bounded above on a neighborhood of some point of 𝑈. Then, for any 𝐱 in 𝑈, 𝑓 is Lipschitz near 𝐱.

Recall that limit superior of a bounded sequence {𝑥_𝑛} in ℝ, denoted limsup{𝑥_𝑛} or lim{𝑥_𝑛}, equals the infimum of all numbers 𝑞∈ℝ for which at most a finite number of elements of {𝑥_𝑛} (strictly) exceed 𝑞. Similarly, limit inferior of {𝑥_𝑛} is given by liminf{𝑥_𝑛}≡lim{𝑥_𝑛}≡sup{𝑞∶ at most a finite number of elements of {𝑥_𝑛} are (strictly) less than 𝑞}.

A bounded sequence always has a unique limit superior and limit inferior.

Theorem 15 (Clarke [3], Calmness). Let 𝑉(𝟎,𝟎) be finite and suppose that liminf_{()→()𝐩,𝐪𝟎,𝟎}𝑉(𝐩,𝐪)−𝑉(𝟎,𝟎)‖(𝐩,𝐪)‖>−∞ (33)(this is true in particular if 𝑉 is Lipschitz near (𝟎,𝟎)). Then, for any solution 𝐱 to (𝑃), problem (𝑃) is calm at 𝐱.

Sometimes problem (𝑃) is said to be calm provided 𝑉 satisfies the hypothesis of Theorem 15.

Slater-Type Conditions.  Suppose that (𝑃) has no equality constraints (i.e., 𝑚=0), that the functions 𝑔_𝑖,  𝑖=1,…,𝑡, are convex, and that 𝐶 is a convex set. Recall that Slater’s condition (Slater’s constraint qualification), then is: there exists a point 𝐱 in 𝐶 such that 𝑔_𝑖(𝐱)<0,  𝑖=1,…,𝑡 (𝐱 is called a strictly feasible point).

For 𝐩∈ℝ^𝑡, let 𝑉(𝐩) be the infimum in the problem 𝑃(𝐩) in which the constraints 𝑔_𝑖(𝐱)≤0 of problem (𝑃) are replaced by 𝑔_𝑖(𝐱)+𝑝_𝑖≤0.

Theorem 16 (Clarke [3]; Lipschitz property of the value function from Slater’s condition). If 𝐶 is bounded and 𝑓 is Lipschitz on 𝐶, then Slater’s condition (i.e., the existence of a strictly feasible point) implies that 𝑉 is Lipschitz near 𝟎.

Theorems 15 and 16 mean that Slater’s constraint qualification implies calmness of problem 𝑃(𝐩) in this case.

Theorem 17 (Clarke [3]; Calmness of a problem subject to inequality constraints). Let 𝑃 incorporate only inequality constraints 𝑔_𝑖(𝐱)≤0 and the abstract constraint 𝐱∈𝐶 and suppose that the value function 𝑉(𝐩) is finite for 𝐩 near 𝟎. Then, for almost all 𝐩 in a neighborhood of 𝟎, the problem 𝑃(𝐩) is calm.

Remark 18. In the case of problem (𝑃), in which equality constraints exist, it is a consequence of Ekeland’s theorem that 𝑃(𝐩,𝐪) is calm for all (𝐩,𝐪) in a dense subset of any open set upon which 𝑉 is bounded and lower semicontinuous.

Consider the following way of perturbing problem (𝑃):

𝑃(𝜶)min𝑓(𝐱,𝜶)∶𝐠(𝐱,𝜶)≤𝟎,𝐡(𝐱,𝜶)=𝟎,(𝐱,𝜶)∈𝐷, (34) where 𝜶is a vector of 𝑘 real components. The value function 𝑉 then would be a function of 𝜶∶𝑉(𝜶)=inf𝑃(𝜶).

This is a special case of problem 𝑃(𝐩,𝐪) with 𝑘=𝑡+𝑚,  𝜶=(𝐩,𝐪),  𝑓(𝐱,𝜶)=𝑓(𝐱),  𝐠(𝐱,𝜶)=𝐠(𝐱)+𝐩,𝐡(𝐱,𝜶)=𝐡(𝐱)+𝐪,𝐷=𝐶×ℝ^𝑡+𝑚. At least when the dependence of 𝑓,𝐠, and 𝐡 on 𝜶 is locally Lipschitz, we can consider problem 𝑃(𝐩,𝐪) with 𝑘=𝑡+𝑚,  𝜶=(𝐩,𝐪),𝑓(𝐱,𝜶)=𝑓(𝐱),𝐠(𝐱,𝜶)=𝐠(𝐱)+𝐩,𝐡(𝐱,𝜶)=𝐡(𝐱)+𝐪, and 𝐷=𝐶×ℝ^𝑡+𝑚 rather than problem 𝑃(𝜶). Hence, the methods and results, considered above, can be applied to perturbed family 𝑃(𝜶) as well.

Constraint qualifications (regularity conditions) can be classified into two categories: on the one hand, Mangasarian-Fromowitz and Slater-type conditions and their extensions, and, on the other hand, constraint qualifications called calmness. It turns out that calmness is the weakest of these conditions, since it is implied by all the others (see, e.g., Theorem 16).

2.3. Well-Posedness of Problems (𝐶), (𝐶⁼), and (𝐶^≥)

2.3.1. Existence of Solutions

The question of existence of solutions to problems (𝐶), (𝐶⁼), and (𝐶^≥) has been discussed in Theorems 1, 4, and 5, respectively. Steps (0), (1), and (9) of Algorithm 2 and Steps (1) and (9) of Algorithms 6 and 8, respectively, refer to these results.

2.3.2. Uniqueness of Solution

The question of uniqueness of the optimal solution to problems under consideration is also important.

If 𝑐(𝐱)≡∑_𝑗∈𝐽𝑐_𝑗(𝑥_𝑗) defined by (1) (by (9), (12), resp.) is a strictly convex function, then problem (𝐶) (problem (𝐶⁼) or problem (𝐶^≥), resp.) has a unique optimal solution in the feasible region 𝑋  (𝑋^𝐿,𝑋^≥, resp.) in case problem (𝐶) (problem (𝐶⁼) or problem (𝐶^≥), resp.) has feasible solutions; that is, 𝑥^∗_𝑗,𝑗∈𝐽^𝜆, are uniquely determined from (6) [(17)] in the interval [𝑎_𝑗,𝑏_𝑗] in this case. If the parameters 𝑎_𝑗,𝑏_𝑗, and so forth of particular problems of the form (𝐶)  ((𝐶⁼) and (𝐶^≥), resp.) are generated in intervals where the functions 𝑐_𝑗(𝑥_𝑗) are strictly convex, then problem (𝐶)  (problem (𝐶⁼) or problem (𝐶^≥), resp.), if it has feasible solutions, has a unique optimal solution.

In the general case, if functions 𝑐_𝑗(𝑥_𝑗) are convex but not necessarily strictly convex, then, as it is known, a convex programming problem has more than one optimal solution and the set of optimal solutions to such a problem is convex. Further, the optimal value of the objective function is the same for all optimal solutions to problem (𝐶) (problem (𝐶⁼) or problem (𝐶^≥), resp.) if it has more than one optimal solution. If, for example, (6) ((17), resp.) is a linear equation of 𝑥^∗_𝑗, then 𝑥^∗_𝑗,𝑗∈𝐽^𝜆, are also uniquely determined from (6) (from (17), resp.).

2.3.3. Calmness of the Problems (of the Optimal Solutions)

Let (𝐶(𝐩)),(𝐶⁼(𝐩,𝑞)), and (𝐶^≥(𝐩)) be the parametrized families of mathematical programs associated with problems (𝐶),(𝐶⁼), and (𝐶^≥), respectively.

Feasible regions of problems (𝐶) and (𝐶^≥) are nonempty by the assumption; this is satisfied when ∑_𝑗∈𝐽𝑑_𝑗(𝑎_𝑗)≤𝛼 and ∑_𝑗∈𝐽𝑑_𝑗𝑏_𝑗≥𝛼, respectively. Without loss of generality, feasible regions

𝑋(𝐩): _𝑗∈𝐽𝑑_𝑗𝑥_𝑗+𝑝₀𝑎≤𝛼_𝑗≤𝑥_𝑗+𝑝_𝑗≤𝑏_𝑗,𝑗∈𝐽, (35) and 𝑋^≥(𝐩): _𝑗∈𝐽𝑑_𝑗𝑥_𝑗+𝑝₀𝑎≥𝛼_𝑗≤𝑥_𝑗+𝑝_𝑗≤𝑏_𝑗,𝑗∈𝐽, (36) of problems (𝐶(𝐩)) and (𝐶^≥(𝐩)), respectively, are also nonempty in a neighborhood of 𝐩=𝟎.

Since the value function 𝑉(𝐩), associated with problems (𝐶(𝐩)) and (𝐶^≥(𝐩)), is finite near 𝟎 (according to Definition 12 and the assumption that the corresponding feasible set is nonempty) then both problems are calm according to Theorem 17.

An alternative proof of calmness of problem (𝐶) is the following.

The objective function 𝑐(𝐱) of problem (𝐶) (1)–(3) is convex (and, therefore, Lipschitz in accordance with Theorem 14), and Slater’s constraint qualification is satisfied by the assumption. From Theorem 16, it follows that the value function 𝑉(𝐩) is Lipschitz, and problem (𝐶) is calm at any solution 𝐱^∗ of problem (𝐶) according to Theorem 15.

Consider the parametrized family (𝐶⁼(𝐩,𝑞)) in which problem (𝐶⁼) is imbedded as follows:

(𝐶⁼(𝐩,𝑞))min𝑐(𝐱)=_𝑗∈𝐽𝑐_𝑗𝑥_𝑗subjectto𝐱∈𝑋^𝐿𝐩,𝑞, (37) where 𝑋^𝐿(𝐩,𝑞) is defined as follows: _𝑗∈𝐽𝑑_𝑗𝑥_𝑗𝑎+𝑞=𝛼_𝑗≤𝑥_𝑗+𝑝_𝑗≤𝑏_𝑗,𝑗∈𝐽. (38)

As it has been pointed out, 𝑋^𝐿≠∅ if _𝑗∈𝐽𝑑_𝑗𝑎_𝑗≤𝛼≤_𝑗∈𝐽𝑑_𝑗𝑏_𝑗, (39) whereas 𝑋^𝐿(𝐩,𝑞)≠∅ if _𝑗∈𝐽𝑑_𝑗𝑎_𝑗−𝑝_𝑗≤𝛼−𝑞≤_𝑗∈𝐽𝑑_𝑗𝑏_𝑗−𝑝_𝑗. (40)

Without loss of generality, assume that there exists a (𝐩,𝑞) such that 𝑋^𝐿(𝐩,𝑞)≠∅. This is satisfied, for example, when ∑_𝑗∈𝐽𝑑_𝑗𝑝_𝑗=𝑞 in addition to the requirement 𝑋^𝐿≠∅. Then the value function 𝑉(𝐩,𝑞), associated with (𝐶⁼(𝐩,𝑞)), is finite by Definition 12.

Theorem 19 (Convexity of the infimum of a convex function subject to linear equality constraints). Let 𝑓 be a convex function and 𝑆 be a convex set in ℝ^𝑛. Then, function ℎ(𝐲)=^definf_𝐱𝑓(𝐱)∶𝐴𝐱=𝐲,𝐴_𝑚×𝑛,𝐱∈𝑆,𝐲∈ℝ^𝑚 (41) is convex.

Proof. Let 𝐱=𝜆𝐱₁+(1−𝜆)𝐱₂,𝜆∈[0,1],𝐱₁,𝐱₂∈𝑆,𝐲₁,𝐲₂∈ℝ^𝑚. Therefore 𝐱∈𝑆 as a convex combination of elements of the convex set 𝑆. Then, ℎ𝜆𝐲₁+(1−𝜆)𝐲₂=inf_𝐱𝑓(𝐱)∶𝐴𝐱=𝜆𝐲₁+(1−𝜆)𝐲₂=inf_𝐱1,𝐱2𝑓𝜆𝐱₁+(1−𝜆)𝐱₂∶𝐴𝜆𝐱₁+(1−𝜆)𝐱₂=𝜆𝐲₁+(1−𝜆)𝐲₂≤inf_𝐱1,𝐱2𝑓𝜆𝐱₁+(1−𝜆)𝐱₂∶𝐴𝐱₁=𝐲₁,𝐴𝐱₂=𝐲₂≤inf_𝐱1,𝐱2𝜆𝑓𝐱₁+(1−𝜆)𝑓𝐱₂∶𝐴𝐱₁=𝐲₁,𝐴𝐱₂=𝐲₂=inf_𝐱1𝜆𝑓𝐱₁∶𝐴𝐱₁=𝐲₁+inf_𝐱2(1−𝜆)𝑓𝐱₂∶𝐴𝐱₂=𝐲₂=𝜆ℎ𝐲₁+(1−𝜆)ℎ𝐲₂. (42) We have used that 𝑓 is a convex function, the property that 𝐱₁,𝐱₂∈𝑆∶𝐴𝐱₁=𝐲₁,𝐴𝐱₂=𝐲₂⊂𝐱₁,𝐱₂∈𝑆∶𝜆𝐴𝐱₁+(1−𝜆)𝐴𝐱₂=𝜆𝐲₁+(1−𝜆)𝐲₂ (43) and the fact that 𝑋⊂𝑌 implies inf_𝐱∈𝑌𝑓(𝐱)≤inf_𝐱∈𝑋𝑓(𝐱).
Therefore, ℎ(𝐲) is a convex function by definition.

For problem (𝐶⁼(𝐩,𝑞)), matrix 𝐴 of Theorem 19 consists of a single row, that is, 𝑚=1, and convex set 𝑆 is the 𝑛-dimensional parallelepiped 𝑆=𝐱∈ℝ^𝑛∶𝑎_𝑗≤𝑥_𝑗≤𝑏_𝑗,𝑗∈𝐽. (44) The value function associated with problem (𝐶⁼(𝐩,𝑞)) is 𝑉𝐩,𝑞=inf_{𝐱∈𝑋𝐿(𝐩,𝑞)}𝑓(𝐱). (45)

From Theorem 19 and the assumption that 𝑋^𝐿(𝐩,𝑞)≠∅, it follows that 𝑉(𝐩,𝑞) is convex and finite, respectively, and from Theorem 14 it follows that it is Lipschitz. Then, problem (𝐶⁼(𝐩,𝑞)) is calm according to Theorem 15.

In the general case, if the mathematical program is not convex and equality constraints exist, we can use the approach of the Remark 18.

3. On the Stability of the Set of Saddle Points of the Lagrangian

3.1. The Concept of Stability of Saddle Points of the Lagrangian

Besides well-posedness of the optimization problems, stability of methods for solving these problems is also important.

Let Φ(𝐱,𝐲) be a convex function of 𝐱∈𝑋 and a concave function of 𝐲∈𝑌, where 𝑋 and 𝑌 are convex and closed sets.

Recall the definition of a saddle point.

A point (𝐱,𝐲) is said to be a saddle point of function Φ(𝐱,𝐲),  𝐱∈𝑋,  𝐲∈𝑌, if the following inequalities hold: Φ(𝐱,𝐲)≤Φ(𝐱,𝐲)≤Φ(𝐱,𝐲) (46) for all 𝐱∈𝑋,𝐲∈𝑌, that is, if Φ(𝐱,𝐲)=min_𝐱∈𝑋max_𝐲∈𝑌Φ(𝐱,𝐲)=max_𝐲∈𝑌min_𝐱∈𝑋Φ(𝐱,𝐲). (47)

This means that Φ(𝐱,𝐲) attains at the saddle point (𝐱,𝐲) its maximum with respect to 𝐲 for fixed 𝐱 and Φ(𝐱,𝐲) attains at (𝐱,𝐲) its minimum with respect to 𝐱 for fixed 𝐲.

Set 𝜒(𝐱)=max_𝐲∈𝑌Φ(𝐱,𝐲),𝜓(𝐲)=min_𝐱∈𝑋Φ(𝐱,𝐲). (48)

Denote by 𝑋^∗ and 𝑌^∗ the sets of optimal solutions to the optimization problems min_𝐱∈𝑋𝜒(𝐱),max_𝐲∈𝑌𝜓(𝐲), (49) respectively, that is, 𝑋=^∗def𝐱∶𝜒(𝐱)=min_𝐱∈𝑋max_𝐲∈𝑌Φ(≡𝐱,𝐲)𝐱∶max_𝐲∈𝑌Φ()(𝐲,𝑌𝐱,𝐲=Φ𝐱,)_∗def=𝐲∶𝜓(𝐲)=max_𝐲∈𝑌min_𝐱∈𝑋Φ(≡𝐱,𝐲)𝐲∶min_𝐱∈𝑋Φ(𝐱,𝐲)=Φ(𝐱,𝐲). (50) Let 𝑋^∗,𝑌^∗ be bounded sets. Then, 𝜒𝐱^∗=𝜓𝐲^∗,𝐱^∗∈𝑋^∗,𝐲^∗∈𝑌^∗, (51)

that is, 𝜒𝐱=^∗defmax_𝐲∈𝑌Φ𝐱^∗,𝐲=𝜓^∗_def𝐲=min_𝐱∈𝑋Φ𝐱,𝐲^∗=Φ(𝐱,𝐲). (52) This means that 𝑋^∗×𝑌^∗ is the set of saddle points of Φ(𝐱,𝐲) and Φ(𝐱,𝐲)=Φ𝐱^∗,𝐲^∗. (53) Consider the sets 𝑋=_𝐲∗^def𝐱∶Φ𝐱,𝐲^∗=Φ𝐱^∗,𝐲^∗,𝑌=_𝐱∗^def𝐲∶Φ𝐱^∗,𝐲=Φ𝐱^∗,𝐲^∗, (54) that is, 𝑋_𝐲∗ and 𝑌_𝐱∗ denote the sets of arguments of Φ(𝐱,𝐲) with 𝐲=𝐲^∗ and 𝐱=𝐱^∗, respectively, for which the value of Φ(𝐱,𝐲) is equal to its value at the saddle point.

In the general case, 𝑋^∗⊂𝑋_𝐲∗,  𝑌^∗⊂𝑌_𝐱∗; that is, the sets 𝑋_𝐲∗,  𝑌_𝐱∗ contain sets 𝑋^∗,  𝑌^∗, respectively.

Definition 20. If 𝑋^∗=𝑋_𝐲∗ and 𝑌^∗=𝑌_𝐱∗, then the set of saddle points of Φ(𝐱,𝐲) is said to be stable.

If the set of saddle points of Φ(𝐱,𝐲) is stable, then from lim_𝑘→∞Φ𝐱^(𝑘),𝐲^∗(𝑘)=Φ𝐱^∗,𝐲^∗,𝐱^(𝑘)∈𝑋,𝐲^∗(𝑘)∈𝑌 (55) it follows that 𝛿𝐱^(𝑘),𝑋^∗⟶0,𝑘⟶∞, (56) and from lim_𝑘→∞Φ𝐱^∗(𝑘),𝐲^(𝑘)=Φ𝐱^∗,𝐲^∗,𝐱^∗(𝑘)∈𝑋,𝐲^(𝑘)∈𝑌 (57) it follows that 𝛿𝐲^(𝑘),𝑌^∗⟶0,𝑘⟶∞, (58) where 𝛿(𝐱,𝑋)=^defmin_𝐳∈𝑋‖𝐱−𝐳‖ is the distance from 𝐱 to the set 𝑋. The implications written above mean that convergence of Φ(𝐱,𝐲^∗) to Φ(𝐱^∗,𝐲^∗) with respect to 𝐱^(𝑘) and convergence of Φ(𝐱^∗,𝐲) to Φ(𝐱^∗,𝐲^∗) with respect to 𝐲^(𝑘) implies convergence of sequence ({𝐱^(𝑘)},{𝐲^(𝑘)}) to the set 𝑋^∗×𝑌^∗ of saddle points of Φ(𝐱,𝐲).

The concept of stability, introduced by Definition 20, is important for constructing iterative gradient algorithms for finding saddle points of the Lagrangian associated with an optimization problem.

The set of saddle points of the Lagrangian associated with the problem min_𝐱∈𝑆𝑓(𝐱),𝑆=𝐱∈ℝ^𝑛∶𝜑_𝑖(𝐱)≤0,𝑖=1,…,𝑚,𝐱∈𝑋 (59)(60)is not stable according to Definition 20. Concerning the dual variables this can be proved as follows.

Let 𝑟th constraint of (60) be satisfied as an equality at 𝐱^∗, that is, 𝜑_𝑟𝐱^∗=0 (61) for some 𝑟,  1≤𝑟≤𝑚. Then, the Lagrangian 𝐿(𝐱^∗,𝝀) of problem (59)-(60) does not depend on 𝜆_𝑟 and therefore 𝐿(𝐱^∗,𝝀^∗)=𝐿(𝐱^∗,𝝀) is satisfied for every 𝜆_𝑟. Hence, it is impossible to determine 𝜆_𝑟 by using the relation 𝐿(𝐱^∗,𝝀^∗)=𝐿(𝐱^∗,𝝀).

In order to avoid this difficulty, so-called modified Lagrangians are used instead of the “classical” Lagrangian. Modified Lagrangians are usually nonlinear functions of 𝝀 and the set of their saddle points is stable and it coincides, under some assumptions, with the set of saddle points of the “classical” Lagrangian for the same problem. This is important to ensure convergence of iterative gradient algorithms (see, e.g., Gol’shtein [5]).

3.2. About the Stability of the Set of Saddle Points for the Approach Considered in this Paper

Consider problem (𝐶) (problem (𝐶⁼) and problem (𝐶^≥), resp.). Obviously, the Lagrange multiplier 𝜆, associated with the constraint (2) ((10) and (13), resp.), is not involved in the equality 𝐿𝐱^∗,𝐮^∗,𝐯^∗,𝜆^∗=𝐿𝐱^∗,𝐮,𝐯,𝜆 (62) when 𝛿(𝜆^∗)=0, that is, when ∑_𝑗∈𝐽𝑑_𝑗(𝑥_𝑗)=𝛼. For problem (𝐶),  𝜆^∗(>0) is either determined uniquely from 𝛿(𝜆^∗)=0 when 𝛿(0)>0 or we set 𝜆^∗∶=0 when 𝛿(0)≤0 (Algorithm 2). Although the set of saddle points of the Lagrangian 𝐿(𝐱,𝐮,𝐯,𝜆), associated with problem (𝐶) (problem (𝐶⁼) and problem (𝐶^≥), resp.) is not stable in the sense of Gol’shtein, the specificity of the approach suggested (the algorithms are not of gradient type and 𝜆^∗ is determined uniquely in all cases for the three problems under consideration) overcomes this “weakness” of the classical Lagrangian. That is why it is not necessary to use modified Lagrangians for problems (𝐶), (𝐶⁼), and (𝐶^≥).

On the one hand, we need a closed form expression of 𝜆 at Step (2) of the algorithms suggested. However, it is this feature of the algorithms that allows us to use classical Lagrangians instead of modified Lagrangians in the approach suggested. Moreover, the method for finding 𝜆, and, therefore, for finding 𝑥^∗_𝑗,𝑗∈𝐽, in the corresponding problem (𝐶), (𝐶⁼), and (𝐶^≥), is exact although it is an iterative method.

As it usually happens, the disadvantage in one aspect turns out to be an advantage in another aspect and vice versa.

All conclusions in this section have been drawn under the assumption that the objective function 𝑐(𝐱) and the constraint function(s) 𝑑_𝑖(𝐱) of the three problems under consideration (𝐶), (𝐶⁼), (𝐶^≥) are nondegenerate, that is, 𝐜^′(𝐱^∗)≢𝟎,  𝐝^′_𝑖(𝐱^∗)≢𝟎; otherwise, the application of the Karush-Kuhn-Tucker theorem with differentiability is void of meaning.

Some optimality criteria for degenerate mathematical programs are given, for example, in the book of Karmanov [6].

4. Primal-Dual Analysis

Some of the main characteristics of the approach, suggested for solving problems (𝐶), (𝐶⁼), and (𝐶^≥), are following.

Since the method, proposed for problem (𝐶), uses values of the first derivatives of functions 𝑐_𝑗(𝑥_𝑗),𝑗∈𝐽, we can consider it as a first-order method. Also, this method is a saddle point method or, more precisely, a dual variables saddle point method because it is based on convergence with respect to the Lagrange multiplier (dual variable) 𝜆 associated with the single constraint (2).

At Step (2) of Algorithm 2, we use the expression of 𝜆^(𝑘), calculated from the equality 𝛿(𝜆^(𝑘))=0, where 𝑥^∗_𝑗 are determined from (6), 𝑗∈𝐽^𝜆(𝑘)=𝐽^(𝑘). As it was proved, under the assumptions for problem (𝐶), we can always determine 𝜆=𝜆(𝐱^∗) from 𝛿(𝜆)=0 as an implicit function of 𝐱^∗. For example, when 𝑑_𝑗(𝑥_𝑗),𝑗∈𝐽, are linear functions, the explicit expression of 𝜆 is always available for Algorithm 2. There are also many other examples of functions, for which it is possible to obtain closed form expressions of 𝜆, and therefore, the suggested approach is applicable and gives good results.

Analogous commentary is valid for the method suggested for solving problem (𝐶⁼).

When the (optimal) Lagrange multiplier 𝜆^∗ associated with (2) is known, then problem (𝐶) (1)–(3) can be replaced by the following separable convex optimization problem min_𝑗∈𝐽𝑐_𝑗𝑥_𝑗+𝜆^∗𝑑_𝑗𝑥_𝑗−𝜆^∗𝛼subjectto𝐱∈𝐴=^def𝐱∈ℝ^𝑛∶𝑎_𝑗≤𝑥_𝑗≤𝑏_𝑗,𝑗∈𝐽. (63)

The problem, dual to problem (𝐶), is maxΨ(𝜆)subjectto𝜆∈ℝ¹₊, (64) where Ψ(𝜆)=min_{𝐱∈𝐴𝑗∈𝐽}𝑐_𝑗𝑥_𝑗+𝜆𝑑_𝑗𝑥_𝑗−𝜆𝛼. (65)

Problem (𝐶⁼) can be considered similarly; 𝑑_𝑗(𝑥_𝑗)=𝑑_𝑗𝑥_𝑗 and 𝜆∈ℝ¹ for it.

Thus, using the Lagrangian duality and Theorem 1 for problem (𝐶) (Theorem 4 for problem (𝐶⁼)) we have replaced the multivariate problem (𝐶) (problem (𝐶⁼)) of 𝐱∈ℝ^𝑛 by the single-variable optimization problem for finding 𝜆∈ℝ¹₊ (𝜆∈ℝ¹, resp.).

Since Algorithm 8 is based on Theorem 5 and Algorithm 6, and since the “iterative” Steps (2)–(7) of Algorithms 6 and 8 are the same, then primal-dual analysis for problem (𝐶^≥) is similar to that for problems (𝐶) and (𝐶⁼).

5. Bibliographical Notes

The definition of Tychonov well-posedness is given by Tychonov in [7]. Tychonov and Hadamard well-posedness and well-posedness in the generalized sense are considered, for example, in the work of Cavazzuti and Morgan [8], in the book of Dontchev and Zolezzi [9], in works of Hadamard [10, 11], and so forth. Other questions regarding stability, ill-posed problems, and Tychonov regularization can be found in [12–19], and so forth. Sometimes Tychonov is written as Tykhonov, Tikhonov or Tychonoff in references. Well-posedness is also discussed in the book of Rockafellar and Wets [20].

Calmness in the sense of Clarke is proposed and studied in the works of Clarke [3, 21, 22]. Concerning Theorem 14, see also the paper of Roberts and Varberg [4].

Stability of the set of saddle points of Lagrangians is considered in the papers of Gol’shtein [5], Gol’shtein and Tret’iakov [23], and so forth.

Various convex separable optimization problems are considered and convergent polynomial algorithms for solving them are proposed in Stefanov [1, 2, 24–29].