A Decomposition Algorithm for Convex Nondifferentiable Minimization with Errors

Abstract

A decomposition algorithm based on proximal bundle-type method with inexact data is presented for minimizing an unconstrained nonsmooth convex function 𝑓. At each iteration, only the approximate evaluation of 𝑓 and its approximate subgradients are required which make the algorithm easier to implement. It is shown that every cluster of the sequence of iterates generated by the proposed algorithm is an exact solution of the unconstrained minimization problem. Numerical tests emphasize the theoretical findings.

1. Introduction

Consider minimizing the following problem:min𝑓(𝑥)∣𝑥∈𝑅^𝑛, (1.1) where 𝑓∶𝑅^𝑛→𝑅 is a nondifferentiable convex function. It is well known that many practical problems can be formulated as (1.1), for example the problem of catastrophe, ruin, vitality, data mining, and finance. A classical conceptual algorithm for solving (1.1) is the proximal point method, based on the Moreau-Yosida regulation of 𝑓 [1, 2]. Implementable forms of the method can be obtained by means of a bundle technique, alternating serious steps with sequences of null steps [3, 4].

More recently, new conceptual schemes for solving (1.1) have been developed by using an approach that is somewhat different from Moreau-Yosida regularization. This is the 𝒱𝒰-theory introduced in [5]; see also [6, 7]. The idea is to decompose 𝑅^𝑛 into two orthogonal subspace 𝒱 and 𝒰 at a point 𝑥 that the nonsmoothness of 𝑓 is concentrated essentially on 𝒱, and the smoothness of 𝑓 appears on 𝒰-subspace. More precisely, for a given 𝑔∈𝜕𝑓(𝑥), where 𝜕𝑓(𝑥) denotes the subdifferential of 𝑓 at 𝑥 in the sense of convex analysis, then 𝑅^𝑛 can be decomposed as direct sum of two orthogonal subspaces, that is, 𝑅^𝑛=𝒰⊕𝒱, where 𝒱=lin(𝜕𝑓(𝑥)−𝑔) and 𝒰=𝒱^⟂. They define the 𝒰-Lagrangian, an approximation of the original function, which can be used to create a second-order expansion of 𝑓 along certain manifolds.

Mifflin and Sagastizábal design a 𝒱𝒰-algorithm for convex function in [8]. This algorithm brings the iterate to the primal track with the help of bundle subroutine. Then the 𝒰-Newton step is performed to gain superlinear decrease of the distance to solution. In order to implement this algorithm, [9] gives an algorithm which only uses the subgradients. However, this algorithm is conceptual in the sense that it needs to compute the exact function values of the objective function, which is difficult to evaluate them. For example, consider the situation of Lagrangian relaxation. The primal problem ismax𝑞(𝜉)∣𝜉∈𝑃,ℎ(𝜉)=0, (1.2) where 𝑃 is a compact subset of 𝑅^𝑚 and 𝑞∶𝑅^𝑚→𝑅, ℎ∶𝑅^𝑚→𝑅^𝑛. Lagrangian relaxation of the equality constraints in the problem leads to the following problemmin_{𝑥∈𝑅𝑛}𝐶(𝑥), (1.3) where 𝐶(𝑥)=max_𝜉∈𝑃𝑞(𝜉)+𝑥,ℎ(𝜉) (1.4) is the dual function. Trying to solve problem (1.2) by means of solving its dual problem (1.3) makes sense in many situations. In this case, evaluating the function value 𝐶(𝑥) and a subgradient 𝑔(𝑥)∈𝜕𝐶(𝑥) requires solving the optimization problem (1.4) exactly. Actually, in some cases, computing exact values of 𝐶(𝑥) is unnecessary and inefficient. For this reason, some modifications of bundle methods in [9] were needed.

The paper is organized as follows. In the next section we present the approximate 𝒰-Lagrangian based on the approximate subgradient. Then we design a conceptual Algorithm 2.6 which can deal with the approximate subgradients and approximate function values. Section 3 breaks into 3 parts. In the first part, we propose the approximate primal-dual track. The proximal bundle-type subroutine with inexact data is introduce in the second part. The third part of Section 3 is devoted to establishing an implemental Algorithm 3.5 which substitutes the approximate V-step in Algorithm 2.6 with proximal bundle subroutine. Numerical testing of the resulting Algorithm 3.5 is reported in the final section.

2. A Conceptual Approximate Decomposition Algorithm

2.1. Approximate 𝒰-Lagrangian and Its Properties

In some cases, computing exact values of the objective function and exact subgradient is unnecessary and inefficient. For this reason, some modification of the 𝒰-Lagrangian will be proposed in this section. We assume that 𝑔 satisfies 𝑓(𝑥+𝑢⊕𝑣)≥𝑓(𝑥)+𝑔,𝑢⊕𝑣,where𝑓(𝑥)∈𝑓(𝑥)−𝜀,𝑓(𝑥),𝜀>0. (2.1) Introducing this 𝑔 to [5], one can restate the definition of 𝒰-Lagrangian and its properties as follows.

Definition 2.1. Assume (2.1). The approximate 𝒰-Lagrangian of 𝑓, denoted by 𝐿_𝑔(𝑢), is defined as follows. 𝐿_𝑔(𝑢)=min_𝑣∈𝒱𝑓(𝑥+𝑢⊕𝑣)−𝑔_𝒱,𝑣_𝒱, (2.2) and 𝑊_𝑔(𝑢) is defined by 𝑊_𝑔(𝑢)=Argmin_𝑣∈𝒱𝑓(𝑥+𝑢⊕𝑣)−𝑔_𝒱,𝑣_𝒱. (2.3)

Theorem 2.2. Assume (2.1). Then the following assertions are true:(i)the function 𝐿_𝑔 defined in (2.2) is convex and finite everywhere;(ii)𝑊_𝑔(𝑢)={𝑤∣∃𝑔∈𝜕𝑓(𝑥+𝑢⊕𝑤)∶𝑔_𝒱=𝑔_𝒱};(iii)0∈𝑊_𝑔(0) and 𝐿_𝑔(0)=𝑓(𝑥).

Theorem 2.3. Assume (2.1) and 𝑊_𝑔(𝑢)≠∅. Then one has that 𝜕𝐿_𝑔(𝑢)=𝑢^∗∣𝑢^∗⊕𝑔_𝒱∈𝜕𝑓(𝑥+𝑢⊕𝑤). (2.4)

Remark 2.4. Assume (1.2). If 𝜀=0, then the approximate 𝒰-Lagrangian in this paper is exactly the 𝒰-Lagrangian in [5].

2.2. Approximate Decomposition Algorithm Frame

In order to give an approximate decomposition algorithm frame, we restate the definition of Hessian matrix in [5] as follows.

Definition 2.5. Assume that 𝑓 is finite, 𝑥 is fixed, and 𝑔 satisfies (2.1). We say that 𝑓 has at 𝑥 a 𝒰-Hessian 𝐻_𝒰𝑓(𝑥) associated with 𝑔 if 𝐿_𝑔(𝑢) has a generalized Hessian at 0, setting 𝐻_𝒰𝑓(𝑥)=𝐻𝐿_𝑔(0). (2.5)

Assume (2.1), we investigate an approximate decomposition algorithm frame based on the definition of the approximate 𝒰-Lagrangian.

Algorithm 2.6. Step 0. Initiation
𝑥₀,𝜂>0,0<𝜏<1,𝜀⁰>0,𝑔⁰ satisfies for all 𝑥, 𝑓(𝑥)≥𝑓(𝑥₀)+⟨𝑔⁰,𝑥−𝑥₀⟩, where 𝑓(𝑥₀)∈[𝑓(𝑥₀)−𝜀⁰,𝑓(𝑥₀)] and 𝑘=0.
Step 1. Stop if ‖𝑔^𝑘‖≤𝜂.
Step 2. Approximate 𝒱-step.
Compute an optimal solution 𝛿𝑣∈𝒱 satisfying 𝛿𝑣∈Argmin_{𝛿𝑣∈𝒱}𝑓𝑥_𝑘+0⊕𝛿𝑣. (2.6) Set 𝑥′=𝑥_𝑘+0⊕𝛿𝑣.
Step 3. 𝒰-step. Make a Newton step in 𝑥^′.
Compute that 𝑔^𝑘 satisfies for all 𝑥, 𝑓(𝑥)≥𝑓(𝑥^′)+⟨𝑔^𝑘,𝑥−𝑥^′⟩, where 𝑓(𝑥^′)∈[𝑓(𝑥^′)−𝜀^𝑘,𝑓(𝑥^′)], such that 𝑔^𝑘_𝒱=0,𝑔^𝑘_𝒰∈𝜕𝐿_𝑔𝑘𝑥^′−𝑥_𝒰. (2.7) Compute the solution 𝛿𝑢∈𝒰 satisfying 𝑔^𝑘_𝒰+𝐻_𝒰𝑓(𝑥)𝛿𝑢=0. (2.8) Set 𝑥_𝑘+1=𝑥^′+𝛿𝑢⊕0=𝑥_𝑘+𝛿𝑢⊕𝛿𝑣 and 𝜀^𝑘+1=𝜏𝜀^𝑘.
Step 4. Update-set
Compute that 𝑔^𝑘+1 satisfies for all 𝑥, 𝑓(𝑥)≥𝑓(𝑥_𝑘+1)+⟨𝑔^𝑘+1,𝑥−𝑥_𝑘+1⟩, where 𝑓(𝑥_𝑘+1)∈[𝑓(𝑥_𝑘+1)−𝜀^𝑘+1,𝑓(𝑥_𝑘+1)]. Set 𝑘=𝑘+1 and go to Step 1.

Theorem 2.7. Assume (2.1) and 𝑓 has a positive definite 𝒰-Hessian at 𝑥, a minimizer of 𝑓. Then the iterate points {𝑥_𝑘} constructed by Algorithm 2.6 satisfy 𝑥_𝑘+1−𝑥=𝑜𝑥_𝑘−𝑥. (2.9)

Remark 2.8. If {𝜀^𝑘}^∞_𝑘=0≡0, this algorithm is the same as Algorithm 4.5 in [5]. However, it only uses the approximate objective function values which make the algorithm easier to implement.

3. Approximate Decomposition Algorithm

Since the Algorithm 2.6 in Section 2 relies on knowing the subspaces 𝒰 and 𝒱 and converges only locally, it needs significant modification. In [10], Mifflin and Sagastizábal show that a proximal point sequence follows primal track near a minimizer. This opens the way for defining a 𝒱𝒰-algorithm where 𝒱-steps are replaced by proximal steps. In addition, the proximal step can be estimated with a bundle technique which also can approximate the unknown 𝒰 and 𝒱 subspaces as a computational byproduct. Therefore, they establish Algorithm 6 in [8] by combing the bundle subroutine with the 𝒱𝒰-space decomposition method. However, this algorithm needs the exact function values and exact subgradients, which is expensive to compute. Therefore, the study of using approximate values instead of the exact ones is deserving.

3.1. Approximate Primal-Dual Track

Given a positive scalar parameter 𝜇, the proximal point function depending on 𝑓 is defined by𝑝_𝜇(𝑥)∶=argmin_{𝑝∈𝑅𝑛}𝑓𝑝+12𝜇𝑝−𝑥²,𝑥∈𝑅^𝑛, (3.1) where ‖⋅‖ stands for the Euclidean norm. It has the property: 𝑔_𝜇(𝑥)∶=𝜇(𝑥−𝑝_𝜇(𝑥))∈𝜕𝑓(𝑝_𝜇(𝑥)).

Similarly to the definition of primal track, we define the approximate primal track.

Definition 3.1. For any 𝜀>0, 𝜇=𝜇(𝑥)>0, we say that Θ(𝑢)=𝑥+𝑢⊕𝑣(𝑢) is an approximate primal track leading to 𝑥, a minimizer of 𝑓, if for all 𝑢∈𝑅^dim𝒰 small enough, it satisfies the following:(i)Θ(𝑢_𝜇(𝑥))=𝑥+𝑢_𝜇(𝑥)⊕𝑣(𝑢_𝜇(𝑥)), where 𝑢_𝜇(𝑥)=(𝑝_𝜇(𝑥)−𝑥)_𝒰;(ii)𝑣∶𝑅^dim𝒰→𝑅^dim𝒱 is a 𝐶²-function satisfying 𝑉𝑣∈𝑊_𝑔(𝑢) for all 𝑔 satisfies (2.1);(iii)the Jacobian 𝐽Θ(𝑢) is a basis matrix for 𝒱(Θ(𝑢))^⟂;(iv)the particular 𝒰-Lagrangian 𝐿₀(𝑢) is a 𝐶²-function.

Accordingly, we have the approximate dual track denoted by Γ(𝑢) corresponding to the approximate primal track. More precisely, Γ(𝑢)=argmin𝑔²∶𝑓(𝑥)≥𝑓(Θ(𝑢))+𝑔,𝑥−Θ(𝑢). (3.2)

In fact, if 𝜀=0, the approximate primal-dual track is exactly the primal-dual track shown in [8].

The next lemma addresses that making an approximate 𝒱-step in Algorithm 2.6 essentially amounts to finding a corresponding approximate primal track point.

Lemma 3.2. Let Θ(𝑢_𝜇(𝑥)) be an approximate primal track leading to 𝑥, a minimizer of 𝑓, and let 𝐻∶=∇²𝐿₀(0). Then for all 𝑢 sufficiently small Θ(𝑢_𝜇(𝑥)) is the unique minimizer of 𝑓 on the affine set Θ(𝑢_𝜇(𝑥))+𝒱(Θ(𝑢_𝜇(𝑥))).

Proof. Since 𝐽Θ(𝑢) is a basis for 𝒱(Θ(𝑢))^⟂, Theorem  3.4 in [10] with ℬ_𝒰(𝑢)=𝐽Θ(𝑢) gives the result.

3.2. The Proximal Bundle-Type Subroutine with Inexact Data

Throughout this section, we make the following assumption: at each given point 𝑥∈𝑅^𝑛, and for 𝜀≥0, we can find some 𝑓(𝑥)∈𝑅 and 𝑔(𝑥)∈𝑅^𝑛 satisfying 𝑓(𝑥)∈𝑓(𝑥)(𝑥,𝑓(𝜁)(𝑥)+𝑔(𝑥)−−𝜀,𝑓)≥𝑓,𝜁−𝑥𝜀,∀𝜁∈𝑅ⁿ, (3.3) where 𝜀=𝑓(𝑥)−𝑓(𝑥). At the same time, it can be ensured that 𝑓_𝑗⟶𝑓_𝑘if𝑦^𝑗⟶𝑥_𝑘, (3.4) where 𝑓_𝑗∈[𝑓(𝑦^𝑗)−𝜀^𝑘,𝑓(𝑦^𝑗)] and 𝑓_𝑘∈[𝑓(𝑥_𝑘)−𝜀^𝑘,𝑓(𝑥_𝑘)] for given 𝜀^𝑘>0. The condition (3.3) means that 𝑔(𝑥)∈𝜕_𝜀𝑓(𝑥). This setting is realistic in many applications; see [11].

The bundle method includes two phases. (i) The first phase makes use of the information (𝑥_𝑘,𝑓_𝑘,𝑔^𝑘),(𝑥_𝑘−1,𝑓_𝑘−1,𝑔^𝑘−1),…,(𝑥₁,𝑓₁,𝑔¹) in bundles to establish a polyhedral approximation of 𝑓 at the actual iterate 𝑥_𝑘. (ii) Due to the kinky structure of 𝑓, the model is possibly not precise for approximation 𝑓. Then, more information around the actual iterate 𝑥_𝑘 is mobilized to obtain a more reliable model. Feature (i) leads to the following approximation of 𝑓 at 𝑥_𝑘. Let 𝐼_𝑘 denote the index set at the 𝑘th iteration with each 𝑗∈𝐼_𝑘 representing (𝑦^𝑗,𝑓_𝑗,𝑔^𝑗), where 𝑓_𝑗 and 𝑔^𝑗 satisfy𝑓_𝑗∈𝑓𝑦^𝑗−𝜀^𝑘,𝑓𝑦^𝑗,𝑓(𝑥)≥𝑓_𝑗+𝑔^𝑗,𝑥−𝑦^𝑗,∀𝑥∈𝑅^𝑛, (3.5) for given 𝜀^𝑘>0. From the choices of 𝑓_𝑗 and 𝑔^𝑗, we have that, for all 𝑥∈𝑅^𝑛 and for all 𝑗∈𝐼_𝑘,𝑓(𝑥)≥𝑓_𝑗+𝑔^𝑗,𝑥−𝑦^𝑗=𝑓_𝑘+𝑔^𝑗𝑇𝑥−𝑥_𝑘+𝜀^𝑘−𝛼_𝑘,𝑗, (3.6) where𝛼_𝑘,𝑗=𝛼𝑥_𝑘,𝑦^𝑗=𝑓_𝑘−𝑓_𝑗−𝑔^𝑗𝑇𝑥_𝑘−𝑦^𝑗+𝜀^𝑘. (3.7)

On the basis of the above observation, we attempt to explore the possibility of utilizing the approximate subgradient and approximate function values instead of the exact ones. We approximate 𝑓 at 𝑥_𝑘 from below by a piecewise linear convex function 𝜑 of the form:𝜑(𝑧)∶=𝑓_𝑘+max_{𝑗∈𝐼𝑘}𝑔^𝑗𝑇𝑧−𝑥_𝑘−𝛼_𝑘,𝑗+𝜀^𝑘. (3.8)

Since (3.8) becomes more and more crude if an approximation of 𝑓 is farther away from 𝑥_𝑘, we add the proximal term (1/2)𝜇‖𝑝−𝑥_𝑘‖²,𝜇>0, to it. To approximate an proximal point, we solve the first quadratic programming subproblem 1min𝑟+2𝜇𝑝−𝑥_𝑘²s.t.𝑟≥𝑓_𝑘+𝑔^𝑗𝑇𝑝−𝑥_𝑘−𝛼_𝑘,𝑗+𝜀^𝑘,𝑗∈𝐼_𝑘. (3.9) Its corresponding dual problem is1min2𝜇_{𝑗∈𝐼𝑘}𝜆_𝑗𝑔^𝑗2+_{𝑗∈𝐼𝑘}𝜆_𝑗𝛼_𝑘,𝑗s.t._{𝑗∈𝐼𝑘}𝜆_𝑗=1,𝜆_𝑗≥0,𝑗∈𝐼_𝑘. (3.10) Let (𝑟,̂𝑝) and 𝜆=(𝜆₁,…,𝜆_|𝐼𝑘|) denote the optimal solution of (3.9) and (3.10), then it is easily seen that𝑟̂=𝜑𝑝,𝑝=𝑥_𝑘−1𝜇𝑔,where𝑔∶=_{𝑗∈𝐼𝑘}𝜆_𝑗𝑔^𝑗. (3.11) In addition, 𝜆_𝑗=0 for all 𝑗∈𝐼_𝑘 such that 𝑟>𝑓_𝑘+𝑔^𝑗𝑇(𝑝−𝑥_𝑘)−𝛼_𝑘,𝑗+𝜀^𝑘 and𝜑𝑝=𝑓_𝑘+_{𝑗∈𝐼𝑘}𝜆_𝑗𝑔^𝑗𝑇𝑝−𝑥_𝑘−𝛼_𝑘,𝑗+𝜀^𝑘=𝑓_𝑘−_{𝑗∈𝐼𝑘}𝜆_𝑗𝛼_𝑘,𝑗+𝜀^𝑘−1𝜇𝑔². (3.12) The vector 𝑝 is an estimate of an approximate proximal point. Hence, it approximates an approximate primal track point when the latter exists. To proceed further we let 𝑦^𝑗+∶=𝑝 and compute 𝑓(𝑝), 𝑒̂∶=𝑓(𝑝)−𝜑(𝑝)=𝑓(𝑝)−𝑟̂, and 𝑔^𝑗+ satisfying 𝑓(𝑧)≥𝑓(𝑝)+⟨𝑔^𝑗+,𝑧−𝑝⟩,forall𝑧∈𝑅^𝑛.

An approximate dual path point, denoted by 𝑠̂, is constructed by solving a second quadratic problem, which depends on a new index set𝐼_𝑘̂∶=𝑗∈𝐼_𝑘∶𝑟̂=𝑓_𝑘+𝑔^𝑗𝑇𝑝−𝑥_𝑘−𝛼_𝑘,𝑗+𝜀^𝑘∪𝑗₊. (3.13) The second quadratic programming problem is1min𝑟+2𝑝−𝑥_𝑘²s.t.𝑟≥𝑔^𝑗𝑇𝑝−𝑥_𝑘,𝑗∈𝐼_𝑘̂. (3.14) It has a dual problem1min2_𝑗∈𝐼_𝑘̂𝜆_𝑗𝑔^𝑗2s.t._𝑗∈𝐼_𝑘̂𝜆_𝑗=1,𝜆_𝑗≥0,𝑗∈𝐼_𝑘̂. (3.15) Similar to (3.11), the respective solutions, denoted by (𝑟,𝑝) and 𝜆, satisfy𝑟=𝜑𝑝,𝑝−𝑥_𝑘=−𝑠,̂where𝑠̂=_𝑗∈𝐼_𝑘̂𝜆_𝑗𝑔^𝑗. (3.16)

Given 𝜎∈(0,1/2], the proximal bundle subprocedure is terminated and 𝑝 is declared to be an approximation of 𝑝_𝜇(𝑥_𝑘) if𝜎𝑒̂≤𝜇‖𝑠‖̂². (3.17) Otherwise, 𝐼_𝑘 above is replaced by 𝐼_𝑘̂, and new iterate data are computed by solving updated subproblems (3.9) and (3.14). This update, appending (𝛼_𝑘,𝑗+,𝑔^𝑗+) to active data at (3.9), ensures convergence to a minimizing point 𝑥 in case of nontermination.

Remark 3.3. From the talking above, the following results are true:(i)𝑠̂=argmin{‖𝑠‖²∶𝑠∈co{𝑔^𝑗∶𝑗∈𝐼_𝑘̂}}; (ii)since 𝑝_𝜇(𝑥) is an approximate primal track point Θ(𝑢_𝜇(𝑥)) approximated by 𝑝 and co{𝑔^𝑗∶𝑗∈𝐼_𝑘̂} approximates co{𝑔∶𝑓(𝑥)≥𝑓(Θ(𝑢))+⟨𝑔,𝑥−Θ(𝑢)⟩}, from (3.2) the corresponding Γ(𝑢_𝜇(𝑥)) is estimated by 𝑠̂;(iii)we can obtain the 𝑈 by means of the following iteration.Let 𝐼^act_𝑘̂∶=𝑗∈𝐼_𝑘̂∶𝑟=𝑔^𝑗𝑇𝑝−𝑥. (3.18) Then, from (3.16), 𝑟=−𝑔^𝑗𝑇𝑠,̂𝑗∈𝐼^act_𝑘̂, so 𝑔^𝑗−𝑔^𝑙𝑇𝑠̂=0, (3.19) for all such 𝑗 and for a fixed 𝑙∈𝐼^act_𝑘̂. Define a full column rank matrix 𝑉 by choosing the largest number of indices 𝑗 satisfying (3.19) such that the corresponding vectors 𝑔^𝑗−𝑔^𝑙 are linearly independent and by letting these vectors be the columns of 𝑉. Then let 𝑈 be a matrix whose columns form an orthonormal basis for the null space of 𝑉^𝑇 with 𝑈=𝐼 if 𝑉 is vacuous.

Theorem 3.4. At the 𝑘th iteration, the above proximal bundle subprocedure satisfies the following:(i)𝑔^𝑗∈𝜕_𝜀+𝑒𝑓̂(𝑝),𝑗∈𝐼_𝑘̂, where 𝜀=𝑓(𝑝)−𝑓(𝑝) and 𝑓(𝑝)∈[𝑓(𝑝)−𝜀^𝑘,𝑓(𝑝)];(ii)𝑠̂∈𝜕_𝜀+𝑒𝑓̂(𝑝) and 𝑔∈𝜕_𝜀+𝑒𝑓̂(𝑝);(iii)𝜇‖𝑝−𝑝_𝜇(𝑥_𝑘)‖²≤𝜀+𝑒̂;(iv)‖𝑠‖̂≤‖𝑔‖, where 𝑔=𝜇(𝑥_𝑘−𝑝);(v)for any parameter 𝑚∈(0,1), (3.17) implies𝑓𝑝−𝑓_𝑘𝑚≤−2𝜇𝑔²+𝜀^𝑘. (3.20)

Proof. (i) Since 𝑔^𝑗+ satisfies 𝑓(𝑧)≥𝑓(𝑝)+⟨𝑔^𝑗+,𝑧−𝑝⟩ and 𝑒̂=𝑓(𝑝)−𝜑(𝑝)≥0, 𝑔^𝑗+ satisfies 𝑓(𝑧)≥𝑓𝑝+𝑔^𝑗+𝑝𝑝+𝑔,𝑧−−𝑒̂=𝑓^𝑗+,𝑧−𝑝−𝑒̂−𝜀, (3.21) where 𝜀=𝑓(𝑝)−𝑓(𝑝), so the result of item (i) holds for 𝑗=𝑗₊. From the definition of 𝑝,𝑟̂, and 𝐼_𝑘̂ we have that for all 𝑗≠𝑗₊ in 𝐼_𝑘̂𝜑𝑝=𝑟̂=𝑓_𝑘+𝑔^𝑗𝑇𝑝−𝑥_𝑘−𝛼_𝑘,𝑗+𝜀^𝑘, (3.22) so for all such 𝑗, 𝑒̂=𝑓𝑝−𝜑𝑝=𝑓𝑝−𝑓_𝑘−𝑔^𝑗𝑇𝑝−𝑥_𝑘+𝛼_𝑘,𝑗−𝜀^𝑘. (3.23) In addition, 𝑓(𝑧)≥𝑓_𝑘+𝑔^𝑗𝑇𝑧−𝑥_𝑘−𝛼_𝑘,𝑗+𝜀^𝑘,𝑧∈𝑅^𝑛. (3.24) Adding 0=𝑒̂−𝑒̂ to this inequality gives 𝑓(𝑧)≥𝑓_𝑘+𝑔^𝑗𝑇𝑧−𝑥_𝑘+𝜀^𝑘−𝛼_𝑘,𝑗=𝑓𝑝+𝑒̂−𝑒̂+𝑔^𝑗𝑇𝑝𝑝𝑧−−𝑒̂=𝑓+𝑔^𝑗𝑇𝑧−𝑝−𝑒̂−𝜀,∀𝑧∈𝑅^𝑛, (3.25) which means that 𝑔^𝑗∈𝜕_𝜀+𝑒𝑓̂(𝑝) for 𝑗₊≠𝑗∈𝐼_𝑘̂.
(ii) Multiplying each inequality in (3.25) by its corresponding multiplier 𝜆_𝑗≥0 and summing these inequalities, we have I_𝑘̂_𝑗∈𝜆_𝑗𝑓(𝑧)≥_𝑗∈𝐼_𝑘̂𝜆_𝑗𝑓𝑝+_𝑗∈𝐼_𝑘̂𝜆_𝑗𝑔^𝑗𝑇𝑧−𝑝−_𝑗∈𝐼_𝑘̂𝜆_𝑗𝑒̂−_𝑗∈𝐼_𝑘̂𝜆_𝑗𝜀,∀𝑧∈𝑅^𝑛. (3.26) Using the definition of 𝑠̂ from (3.16) and the fact that ∑_𝑗∈̂𝐼_𝑘𝜆_𝑗=1 gives 𝑓(𝑧)≥𝑓𝑝+𝑠^𝑇̂𝑧−𝑝−𝑒̂−𝜀,∀𝑧∈𝑅^𝑛, (3.27) which means that 𝑠̂∈𝜕_𝜀+𝑒𝑓̂(𝑝). In a similar manner, this time using the multipliers 𝜆_𝑗 that solve dual problem (3.10) and define 𝑔 in (3.11), together with 𝜆_𝑗+∶=0, obtains the result.
(iii) Since 𝑔_𝜇∈𝜕𝑓(𝑝_𝜇(𝑥_𝑘)), we have 𝑓𝑝_𝜇𝑥_𝑘≤𝑓𝑝−𝑔_𝜇𝑥_𝑘^𝑇𝑝−𝑝_𝜇𝑥_𝑘. (3.28) From (ii): 𝑔∈𝜕_𝜀+𝑒𝑓̂(𝑝), we get 𝑓𝑝_𝜇𝑥_𝑘≥𝑓𝑝+𝑔^𝑇𝑝_𝜇𝑥_𝑘−𝑝−𝜀−𝑒.̂ (3.29) Therefore, 𝑓𝑝+𝑔^𝑇𝑝_𝜇𝑥_𝑘−𝑝−𝜀−𝑒̂≤𝑓𝑝_𝜇𝑥_𝑘≤𝑓𝑝−𝑔_𝜇𝑥_𝑘^𝑇𝑝−𝑝_𝜇𝑥_𝑘, (3.30) that is, (𝑔−𝑔_𝜇(𝑥_𝑘))^𝑇(𝑝_𝜇(𝑥_𝑘)−𝑝)−𝜀−𝑒̂≤0. Then, since the expression for 𝑔 from (3.11) written in the form 𝑔=−𝜇𝑝−𝑥_𝑘, (3.31) combined with 𝑔_𝜇(𝑥_𝑘)=𝜇(𝑥_𝑘−𝑝_𝜇(𝑥_𝑘)) implies that 𝑔−𝑔_𝜇(𝑥_𝑘)=𝜇(𝑝_𝜇(𝑥_𝑘)−𝑝), we obtain item (iii).
(iv) From (3.10), (3.11), (3.31), and the definition of 𝐼_𝑘̂, we have that 𝜇(𝑥_𝑘−𝑝)=𝑔 is in the convex hull of {𝑔^𝑗,𝑗∈𝐼_𝑘̂}. We obtain the result by virtue of the minimum norm property of 𝑠̂.
(v) Since 𝜎≤1/2 and 𝑚∈(0,1), we have 𝜎≤1−(𝑚/2). Thus if (3.17) holds then 𝑒̂≤[(1−(𝑚/2))/𝜇]‖𝑠‖̂². Together with the definition of 𝑒̂, (3.12) and the nonnegativity of 𝜆_𝑗𝛼_𝑗 gives 𝑓𝑝−𝑓_𝑘=𝑒̂+𝜑𝑝−𝑓_𝑘=𝑒̂−_{𝑗∈𝐼𝑘}𝜆_𝑗𝛼_𝑘,𝑗−1𝜇𝑔²+𝜀^𝑘1≤𝑒̂−𝜇𝑔²+𝜀^𝑘≤(1−(𝑚/2))𝜇‖𝑠‖̂²−1𝜇𝑔²+𝜀^𝑘. (3.32) Finally, combing this inequality with item (iv) gives (3.20).

3.3. Approximate Decomposition Algorithm and Convergence Analysis

Substituting the approximate 𝒱-step in Algorithm 2.6 with proximal bundle-type subroutine, we present an approximate decomposition algorithm as follows. Afterwards a detailed convergence analysis is given. The main statement comprises the fact that each cluster point of the sequence of iterates generated by the algorithm is an optimal solution.

Algorithm 3.5. Choose a starting point 𝑝₀∈𝑅^𝑛 and positive parameters 𝜂,𝜀⁰,𝜇,𝜏, and 𝑚 with 𝜏<1, 𝑚<1.
Step 0. Compute 𝑔⁰ satisfying 𝑓(𝑧)≥𝑓(𝑝₀)+⟨𝑔⁰,𝑧−𝑝₀⟩, where 𝑓(𝑝₀)∈[𝑓(𝑝₀)−𝜀⁰,𝑓(𝑝₀)]. Let 𝑈₀ be a matrix with orthonormal 𝑛-dimensional columns estimating an optimal 𝒰-basis. Set 𝑠₀=𝑔⁰ and 𝑘∶=0.
Step 1. Stop if ‖𝑠_𝑘‖²≤𝜂.
Step 2. Choose an 𝑛_𝑘×𝑛_𝑘 positive definite matrix 𝐻_𝑘, where 𝑛_𝑘 is the number of columns of 𝑈_𝑘.
Step 3. Compute an approximate 𝒰-Newton step by solving the linear system 𝐻_𝑘Δ𝑢_𝑘=−𝑈^𝑇_𝑘𝑠_𝑘. (3.33) Set 𝑥^′_𝑘+1∶=𝑝_𝑘+𝑈_𝑘Δ𝑢_𝑘.
Step 4. Choose 𝜇_𝑘+1>𝜇,𝜎_𝑘+1∈(0,1/2], initialize 𝐼_𝑘, and run the bundle subprocedure with 𝑥=𝑥^′_𝑘+1. Compute recursively, and set (𝑒^′_𝑘+1,𝑝^′_𝑘+1,𝑠^′_𝑘+1,𝑈^′_𝑘+1)∶=(𝑒,̂𝑝,𝑠,̂𝑈).
Step 5. If 𝑓𝑝^′_𝑘+1−𝑓𝑝_𝑘𝑚≤−2𝜇_𝑘+1𝑠^′_𝑘+1²+𝜀^𝑘+1, (3.34) then set 𝑥_𝑘+1,𝑒_𝑘+1,𝑝_𝑘+1,𝑠_𝑘+1,𝑈_𝑘+1,𝜀^𝑘+1∶=𝑥^′_𝑘+1,𝑒^′_𝑘+1,𝑝^′_𝑘+1,𝑠^′_𝑘+1,𝑈^′_𝑘+1,𝜏𝜀^𝑘. (3.35) Otherwise, execute a line search 𝑥_𝑘+1∶=argmin𝑓𝑝_𝑘,𝑓𝑝^′_𝑘+1, (3.36) reinitialize 𝐼_𝑘, and rerun the bundle subroutine with 𝑥=𝑥_𝑘+1, to find new values for (𝑒,̂𝑝,𝑠,̂𝑈), then set (𝑒_𝑘+1,𝑝_𝑘+1,𝑠_𝑘+1,𝑈_k+1,𝜀^𝑘+1)=(𝑒,̂𝑝,𝑠,̂𝑈,𝜏𝜀^𝑘).
Step 6. Replace 𝑘 by 𝑘+1 and go to Step 1.

Remark 3.6. In this algorithm, {𝜀^𝑘}^∞_𝑘=0↓0. If {𝜀^𝑘}^∞_𝑘=0≡0, this algorithm is the same as Algorithm 6 in [8]. However, this algorithm uses proximal bundle-type subroutine which can deal with the approximate subgradients and the approximate function values.

Theorem 3.7. One of the following two cases is true:(i)if the proximal bundle procedure in Algorithm 3.5 does not terminate, that is, if (3.17) never hold, then the sequence of 𝑝-values converges to 𝑝_𝜇(𝑥) and 𝑝_𝜇(𝑥) is a minimizer of 𝑓;(ii)if the procedure terminates with 𝑠̂=0, then the corresponding 𝑝 equals 𝑝_𝜇(𝑥) and is a minimizer of 𝑓.

Proof. By ([12], Prop. 4.3), if this procedure does not terminate then it generates an infinite sequence of 𝑒̂-values and 𝜀-value converging to zero. Since (3.17) does not hold, the sequence of ‖𝑠‖̂-values also converges to 0. Thus, item (iii) in Theorem 3.4 implies that {𝑝}→𝑝_𝜇(𝑥). And Theorem 3.4 (ii) gives 𝑓(𝑧)≥𝑓𝑝+𝑠^𝑇̂𝑧−𝑝−(𝑒̂+𝜀)̂,𝑧∈𝑅^𝑛. (3.37) By the continuity of 𝑓, this becomes 𝑓(𝑧)≥𝑓𝑝_𝜇(𝑥),𝑧∈𝑅^𝑛. (3.38) The termination case with 𝑠̂=0 follows in a similar manner, since (3.17) implies 𝑒̂=0 in this case.

The next theorem establishes the convergence of Algorithm 3.5, and the proof of which is similar to Theorem  9 in [8].

Theorem 3.8. Suppose that the sequence {𝜇_𝑘} in Algorithm 3.5 is bounded above by 𝜇. Then the following hold:(i)the sequence {𝑓(𝑝_𝑘)} is decreasing and either {𝑓(𝑝_𝑘)}→−∞ or {‖𝑠_𝑘‖} and {𝑒_𝑘} both converge to 0;(ii)if 𝑓 is bounded from below, then any accumulation point of {𝑝_𝑘} is a minimizers of 𝑓.

Proof. In this paper, the inequalities of (3.15), (3.16), and (3.17) in [8] become 𝑓𝑝_𝑘+1−𝑓𝑝_𝑘𝑚≤−2𝜇_𝑘+1𝑠_𝑘+1²,𝑓(𝑧)≥𝑓𝑝_𝑘+𝑠^𝑇_𝑘𝑧−𝑝_𝑘−𝜀^𝑘_𝑝−𝑒_𝑘,∀𝑧∈𝑅^𝑛,where𝜀^𝑘_𝑝=𝑓𝑝_𝑘−𝑓𝑝_𝑘𝑒_𝑘,≤𝜎_𝑘𝜇_𝑘𝑠_𝑘², (3.39)(3.40)(3.41) since ‖𝑠_𝑘‖≠0, (3.39) implies that {𝑓(𝑝_𝑘)} is decreasing. Suppose {𝑓(𝑝_𝑘)}↛−∞. Then summing (3.39) over 𝑘 and using the fact that 𝑚/2𝜇_𝑘≥𝑚/2𝜇 for all 𝑘 implies that {‖𝑠_𝑘‖}→0. Then (3.41) with 𝜎_𝑘≤1/2 and 𝜇_𝑘≥𝜇>0 implies that {𝑒_𝑘}→0, which establishes (i).
Now suppose 𝑓 is bounded from below and 𝑝 is any accumulation point of {𝑝_𝑘}. Then, because {‖𝑠_𝑘‖}, {𝑒_𝑘}, and {𝜀^𝑘_𝑝} converge to 0, (3.40) together with the continuity of 𝑓 implies that 𝑓(𝑝)≤𝑓(𝑧) for all 𝑧∈𝑅^𝑛 and (ii) is proved.

4. An Illustrated Numerical Example

We test some examples in this section to validate the effectiveness of the proposed algorithm. The platform is Matlab R2009a, Intel Pentium(R) Dual CPU T2390 1.87 GHz. All test examples are of the form𝑓=max_𝑖∈𝐼𝑓_𝑖, (4.1) where 𝐼 is finite and each 𝑓_𝑖 is 𝐶² on 𝑅^𝑛.

For our runs we used the following examples:(i)F2d: the objective function is given in [8], defined for 𝑥∈𝑅^𝑛 by(𝑥)1F2d∶=max2𝑥²₁+𝑥²₂−𝑥₂,𝑥₂; (4.2)(ii)F3d-𝑈𝑣: four functions of three variables, where 𝑣=3,2,1,0 denotes the corresponding dimension of the 𝒰-subspace. Given 𝑒∶=(0,1,1)^𝑇 and four parameter vectors 𝛽^𝑣∈𝑅⁴, for 𝑥∈𝑅³(𝑥)1F3d-𝑈𝑣∶=max2𝑥²₁+𝑥²₂+0.1𝑥²₃−𝑒^𝑇𝑥−𝛽^𝑣₁,𝑥²₁−3𝑥₁−𝛽^𝑣₂,𝑥₂−𝛽^𝑣₃,𝑥₂−𝛽^𝑣₄, (4.3) where 𝛽³∶=(−5.5,10,11,20),𝛽²∶=(−5,10,0,10),𝛽¹∶=(0,10,0,0), and 𝛽⁰∶=(0.5,−2,0,0).

In Table 1, we show some relevant data for the problems, including the dimensions of 𝒱 and 𝒰, the (known) optimal values and solutions, and the starting points.

Table 1 

Problem data.

We calculate an 𝜀-subgradient at 𝑥 by using the method in [13]: 𝑔_𝜀(𝑥)=𝜆𝑔(𝑥)+(1−𝜆)𝑔(𝑥₁), where 𝑔(𝑥) is a subgradient at 𝑥 and 𝑔(𝑥₁) is a subgradient at a point 𝑥₁ such that 0<𝛼₀(𝑥,𝑥₁)=𝑓(𝑥)−𝑓(𝑥₁)−𝑔(𝑥₁)^𝑇(𝑥−𝑥₁)≤𝜀. Here 𝑥₁∈𝐵_𝑟(𝑥)={𝑧∈𝑅^𝑛∣‖𝑧−𝑥‖≤𝑟} and 𝜆∈[0,1] are randomly chosen. The approximate function value 𝑓(𝑥) is randomly taken out from the interval [𝑓(𝑥)−𝜀,𝑓(𝑥)]. The radius 𝑟 is adjusted iteratively in the following way: If we find the linearization error 𝛼₀(𝑥,𝑥₁)>𝜀 then 𝑟 is reduced by a multiple smaller than one. On the other hand, if 𝛼₀(𝑥,𝑥₁) is significantly smaller than 𝜀, then 𝑟 is increased by a multiple greater than one. When 𝑠_𝑘=𝑠̂ in the algorithm, then 𝒰-Hessian at 𝑥 is computed in the following form: 𝐻_𝑘=𝑈^𝑇_𝑘(∑_𝑗∈ℬ𝜆_𝑗𝐻^𝑗)𝑈_𝑘, where 𝐻^𝑗=∇²𝑓_𝑖𝑗(𝑦^𝑗), 𝑖_𝑗 is an active index such that 𝑓_𝑖𝑗(𝑦^𝑗)=𝑓(𝑦^𝑗),𝜆_𝑗 correspond to 𝑠̂ via (3.16).

The parameters have values 𝜂=1.0×10⁻⁴, 𝜀⁰=1.0×10⁻⁴, 𝑚=1.0×10⁻¹, 𝜏=1.0×10⁻¹, and 𝑈₀ equal to the 𝑛×𝑛 identity matrix. As for 𝜎_𝑘,𝜇_𝑘, one can refer to [8].

Table 2 shows the results of Algorithm 3.5 for these examples, compared with Algorithm  6 in [8]. Number of 𝑓/𝑔 denotes the number of evaluation of the function and subgradient (𝜀-subgradient) in Algorithm 6 and Algorithm 3.5. 𝑥 is the calculated solution, |𝑓𝑥−𝑓𝑥^∗| stands for the difference between the function values at 𝑥 and 𝑥^∗.

Table 2 

Numerical results of UV-decomposition algorithm with inexact data.

It is shown in Table 2 that we obtain quite accurate solutions by Algorithm 3.5 with inexact data costing a slightly more evaluation number than that with exact data. One noticeable exceptional occurs in the example F3d-𝑈1; it seems that the decomposition algorithm is sensible with exact data, but is more stable when applying inexact data (function values and subgradients). This favorable results demonstrate that it is suitable to use approximate decomposition algorithm to solve (1.1) numerically.

Acknowledgment

The research is supported by the National Natural Science Foundation of China under Project no. 11171049 and no. 11171138.