Strong Convergence of a Projected Gradient Method

Abstract

The projected-gradient method is a powerful tool for solving constrained convex optimization problems and has extensively been studied. In the present paper, a projected-gradient method is presented for solving the minimization problem, and the strong convergence analysis of the suggested gradient projection method is given.

1. Introduction

In the present paper, our main purpose is to solve the following minimization problem:min_𝑥∈𝐶𝑓(𝑥),(1.1) where 𝐶 is a nonempty closed and convex subset of a real Hilbert space 𝐻, 𝑓∶𝐻→𝑅 is a real-valued convex function.

Now it is well known that the projected-gradient method is a powerful tool for solving the above minimization problem and has extensively been studied. See, for instance, [1–8]. The classic algorithm is the following form of the projected-gradient method:𝑥_𝑛+1=𝑃_𝐶𝑥_𝑛−𝛾∇𝑓𝑥_𝑛,𝑛≥0,(1.2) where 𝛾>0 is an any constant, 𝑃_𝐶 is the nearest point projection from 𝐻 onto 𝐶, and ∇𝑓 denotes the gradient of 𝑓.

It is known [1] that if 𝑓 has a Lipschitz continuous and strongly monotone gradient, then the sequence {𝑥_𝑛} generated by (1.2) can be strongly convergent to a minimizer of 𝑓 in 𝐶. If the gradient of 𝑓 is only assumed to be Lipschitz continuous, then {𝑥_𝑛} can only be weakly convergent if 𝐻 is infinite dimensional. An interesting problem is how to appropriately modify the projected gradient algorithm so as to have strong convergence? For this purpose, recently, Xu [9] introduced the following algorithm:𝑥_𝑛+1=𝜃_𝑛ℎ𝑥_𝑛+1−𝜃_𝑛𝑃_𝐶𝑥_𝑛−𝛾_𝑛∇𝑓𝑥_𝑛,𝑛≥0.(1.3) Under some additional assumptions, Xu [9] proved that the sequence {𝑥_𝑛} converges strongly to a minimizer of (1.1). At the same time, Xu [9] also suggested a regularized method:𝑥_𝑛+1=𝑃_𝐶𝐼−𝛾_𝑛∇𝑓+𝛼_𝑛𝐼𝑥_𝑛,𝑛≥0.(1.4) Consequently, Yao et al. [10] proved the strong convergence of the regularized method (1.4) under some weaker conditions.

Motivated by the above works, in this paper we will further construct a new projected gradient method for solving the minimization problem (1.1). It should be pointed out that our method also has strong convergence under some mild conditions.

2. Preliminaries

Let 𝐶 be a nonempty closed convex subset of a real Hilbert space 𝐻. A bounded linear operator 𝐵 is said to be strongly positive on 𝐻 if there exists a constant 𝛼>0 such that⟨𝐵𝑥,𝑥⟩≥𝛼‖𝑥‖²,∀𝑥∈𝐻.(2.1) A mapping 𝑇∶𝐶→𝐶 is called nonexpansive if‖𝑇𝑥−𝑇𝑦‖≤‖𝑥−𝑦‖,∀𝑥,𝑦∈𝐶.(2.2) A mapping 𝑇∶𝐶→𝐶 is said to be an averaged mapping, if and only if it can be written as the average of the identity 𝐼 and a nonexpansive mapping; that is,𝑇=(1−𝛼)𝐼+𝛼𝑅,(2.3) where 𝛼∈(0,1) is a constant and 𝑅∶𝐶→𝐶 is a nonexpansive mappings. In this case, we call 𝑇 is 𝛼-averaged.

A mapping 𝑇∶𝐶→𝐶 is said to be 𝜈-inverse strongly monotone (𝜈-ism), if and only if⟨𝑥−𝑦,𝑇𝑥−𝑇𝑦⟩≥𝜈‖𝑇𝑥−𝑇𝑦‖²,𝑥,𝑦∈𝐶.(2.4) The following proposition is well known, which is useful for the next section.

Proposition 2.1 (See [9]). (1) The composite of finitely many averaged mappings is averaged. That is, if each of the mappings {𝑇_𝑖}^𝑁𝑖=1 is averaged, then so is the composite 𝑇₁,…,𝑇_𝑁. In particular, if 𝑇₁ is 𝛼₁-averaged and 𝑇₂ is 𝛼₂-averaged, where 𝛼₁,𝛼₂∈(0,1), then the composite 𝑇₁𝑇₂ is 𝛼-averaged, where 𝛼=𝛼₁𝛼₂−𝛼₁𝛼₂.
(2) T is 𝜈-ism, then for 𝛾>0, 𝛾𝑇 is (𝜈/𝛾)-ism.

Recall that the (nearest point or metric) projection from 𝐻 onto 𝐶, denoted by 𝑃_𝐶, assigns, to each 𝑥∈𝐻, the unique point 𝑃_𝐶(𝑥)∈𝐶 with the property ‖‖𝑥−𝑃_𝐶(𝑥)‖‖=inf{‖𝑥−𝑦‖∶𝑦∈𝐶}.(2.5) We use 𝑆 to denote the solution set of (1.1). Assume that (1.1) is consistent, that is, 𝑆≠∅. If 𝑓 is Frechet differentiable, then 𝑥^∗∈𝐶 solves (1.1) if and only if 𝑥^∗∈𝐶 satisfies the following optimality condition:∇𝑓𝑥^∗,𝑥−𝑥^∗≥0,∀𝑥∈𝐶,(2.6) where ∇𝑓 denotes the gradient of 𝑓. Observe that (2.6) can be rewritten as the following VI𝑥^∗−𝑥^∗−∇𝑓𝑥^∗,𝑥−𝑥^∗≥0,∀𝑥∈𝐶.(2.7) (Note that the VI has been extensively studied in the literature, see, for instance [11–25].) This shows that the minimization (1.1) is equivalent to the fixed point problem𝑃_𝐶𝑥^∗−𝛾∇𝑓𝑥^∗=𝑥^∗,(2.8) where 𝛾>0 is an any constant. This relationship is very important for constructing our method.

Next we adopt the following notation:(i)𝑥_𝑛→𝑥 means that 𝑥_𝑛 converges strongly to 𝑥;(ii)𝑥_𝑛⇀𝑥 means that 𝑥_𝑛 converges weakly to 𝑥;(iii)Fix(𝑇)∶={𝑥∶𝑇𝑥=𝑥} is the fixed points set of 𝑇.

Lemma 2.2 (See [26]). Let {𝑥_𝑛} and {𝑦_𝑛} be bounded sequences in a Banach space 𝑋 and let {𝛽_𝑛} be a sequence in [0,1] with 0<liminf_𝑛→∞𝛽_𝑛≤limsup_𝑛→∞𝛽_𝑛<1.(2.9) Suppose 𝑥_𝑛+1=1−𝛽_𝑛𝑦_𝑛+𝛽_𝑛𝑥_𝑛(2.10) for all 𝑛≥0 and limsup_𝑛→∞‖‖𝑦_𝑛+1−𝑦_𝑛‖‖−‖‖𝑥_𝑛+1−𝑥_𝑛‖‖≤0.(2.11) Then, lim_𝑛→∞‖𝑦_𝑛−𝑥_𝑛‖=0.

Lemma 2.3 (See [27] (demiclosedness principle)). Let 𝐶 be a closed and convex subset of a Hilbert space 𝐻 and let 𝑇∶𝐶→𝐶 be a nonexpansive mapping with Fix(𝑇)≠∅. If {𝑥_𝑛} is a sequence in 𝐶 weakly converging to 𝑥 and if {(𝐼−𝑇)𝑥_𝑛} converges strongly to 𝑦, then (𝐼−𝑇)𝑥=𝑦.(2.12) In particular, if 𝑦=0, then 𝑥∈Fix(𝑇).

Lemma 2.4 (See [28]). Assume {𝑎_𝑛} is a sequence of nonnegative real numbers such that 𝑎_𝑛+1≤1−𝛾_𝑛𝑎_𝑛+𝛿_𝑛,(2.13) where {𝛾_𝑛} is a sequence in (0,1) and {𝛿_𝑛} is a sequence such that(1)∑^∞𝑛=1𝛾_𝑛=∞;(2)limsup_𝑛→∞𝛿_𝑛/𝛾_𝑛≤0 or ∑^∞𝑛=1|𝛿_𝑛|<∞.Then, lim_𝑛→∞𝑎_𝑛=0.

3. Main Results

Let 𝐶 be a closed convex subset of a real Hilbert space 𝐻. Let 𝑓∶𝐶→𝑅 be a real-valued Frechet differentiable convex function with the gradient ∇𝑓. Let 𝐴∶𝐶→𝐻 be a 𝜌-contraction. Let 𝐵∶𝐻→𝐻 be a self-adjoint, strongly positive bounded linear operator with coefficient 𝛼>0. First, we present our algorithm for solving (1.1). Throughout, we assume 𝑆≠∅.

Algorithm 3.1. For given 𝑥₀∈𝐶, compute the sequence {𝑥_𝑛} iteratively by 𝑥_𝑛+1=𝑃_𝐶𝐼+(𝜎𝐴−𝐵)𝜃_𝑛𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛,𝑛≥0,(3.1) where 𝜎>0,𝛾>0 are two constants and the real number sequence {𝜃_n}⊂[0,1].

Remark 3.2. In (3.1), we use two projections. Now, it is well-known that the advantage of projections, which makes them successful in real-word applications, is computational.

Next, we show the convergence analysis of this Algorithm 3.1.

Theorem 3.3. Assume that the gradient ∇𝑓 is 𝐿-Lipschitzian and 𝜎𝜌<𝛼. Let {𝑥_𝑛} be a sequence generated by (3.1), where 𝛾∈(0,2/𝐿) is a constant and the sequence {𝜃_𝑛} satisfies the conditions: (i) lim_𝑛→∞𝜃_𝑛=0 and (ii) ∑^∞𝑛=0𝜃_𝑛=∞. Then {𝑥_𝑛} converges to a minimizer 𝑥̃ of (1.1) which solves the following variational inequality: 𝑥̃∈S𝑠𝑢𝑐ℎ𝑡ℎ𝑎𝑡⟨𝜎A(𝑥̃)−B(𝑥̃),𝑥−𝑥̃⟩≤0,∀𝑥∈𝑆.(3.2)

By Algorithm 3.1 involved in the projection, we will use the properties of the metric projection for proving Theorem 3.3. For convenience, we list the properties of the projection as follows.

Proposition 3.4. It is well known that the metric projection 𝑃_𝐶 of 𝐻 onto 𝐶 has the following basic properties:(i)‖𝑃_𝐶(𝑥)−𝑃_𝐶(𝑦)‖≤‖𝑥−𝑦‖, for all 𝑥,𝑦∈𝐻;(ii)⟨𝑥−𝑦,𝑃_𝐶(𝑥)−𝑃_𝐶(𝑦)⟩≥‖𝑃_𝐶(𝑥)−𝑃_𝐶(𝑦)‖², for every 𝑥,𝑦∈𝐻;(iii)⟨𝑥−𝑃_𝐶(𝑥),𝑦−𝑃_𝐶(𝑥)⟩≤0, for all 𝑥∈𝐻, 𝑦∈𝐶.

The Proof of Theorem 3.3
Let 𝑥^∗∈𝑆. First, from (2.8), we note that 𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥^∗=𝑥^∗. By (3.1), we have ‖‖𝑥_𝑛+1−𝑥^∗‖‖=≤‖‖‖‖𝑃𝑃_𝐶𝐶𝐼+(𝜎𝐴−𝐵)𝜃_𝑛𝑃_𝐶(𝐼−𝛾∇f)𝑥_𝑛−𝑥^∗‖‖+‖‖𝑃_𝐶𝐼+(𝜎𝐴−𝐵)𝜃_𝑛𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛−𝑃_𝐶𝐼+(𝜎𝐴−𝐵)𝜃_𝑛𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥^∗‖‖≤1−(𝛼𝐼−+𝜎(𝜌𝜎)𝐴𝜃_𝑛−𝐵)𝜃_𝑛𝑥^∗−𝑥^∗‖‖=1−(𝛼−𝜎𝜌)𝜃_𝑛‖‖𝑥_𝑛−𝑥^∗‖‖+𝜃_𝑛‖‖𝜎𝐴𝑥^∗−𝐵𝑥^∗‖‖‖‖𝑥_𝑛−𝑥^∗‖‖+(𝛼−𝜎𝜌)𝜃_𝑛‖‖𝜎𝐴𝑥^∗−𝐵𝑥^∗‖‖≤max‖‖𝑥_𝑛𝛼−𝜎𝜌−𝑥^∗‖‖,‖‖𝜎𝐴𝑥^∗−𝐵𝑥^∗‖‖𝛼−𝜎𝜌.(3.3) Thus, by induction, we obtain ‖‖𝑥_𝑛−𝑥^∗‖‖≤max‖‖𝑥₀−𝑥^∗‖‖,‖‖𝜎𝐴𝑥^∗−𝐵𝑥^∗‖‖𝛼−𝜎𝜌.(3.4)
Note that the Lipschitz condition implies that the gradient ∇𝑓 is (1/𝐿)-inverse strongly monotone (ism), which then implies that 𝛾∇𝑓 is (1/𝛾𝐿)-ism. So, 𝐼−𝛾∇𝑓 is (𝛾𝐿/2)-averaged. Now since the projection 𝑃_𝐶 is (1/2)-averaged, we see that 𝑃_𝐶(𝐼−𝛾∇𝑓) is ((2+𝛾𝐿)/4)-averaged. Hence we have that 𝑃_𝐶=12𝐼+12𝑅𝑃^𝐶(𝐼−𝛾∇𝑓)=2−𝛾𝐿4𝐼+2+4𝛾𝐿𝑇=(1−𝛽)𝐼+𝛽𝑇,(3.5) where 𝑅,𝑇 are nonexpansive and 𝛽=(2+𝛾𝐿)/4∈(0,1). Then we can rewrite (3.1) as 𝑥_𝑛+1=12𝐼+12𝑅𝐼+(𝜎𝐴−𝐵)𝜃^𝑛(1−𝛽)𝑥_𝑛=1−𝛽+𝛽𝑇𝑥_𝑛2𝑥^𝑛+𝛽2𝑇𝑥^𝑛+𝜃^𝑛2(𝜎𝐴−𝐵)+𝑅2𝐼+(𝜎𝐴−𝐵)𝜃^𝑛(1−𝛽)𝑥_𝑛=1−𝛽+𝛽𝑇𝑥_𝑛2𝑥^𝑛+1+𝛽2𝑦^𝑛,(3.6) where 𝑦_𝑛=21+𝛽𝜃2^𝑛(𝜎𝐴−𝐵)+𝑅2𝐼+(𝜎𝐴−𝐵)𝜃^𝑛(1−𝛽)𝑥_𝑛+𝛽𝑇𝑥_𝑛+𝛽1+𝛽𝑇𝑥^𝑛.(3.7) Set 𝑧_𝑛=(1−𝛽)𝑥_𝑛+𝛽𝑇𝑥_𝑛 for all 𝑛. Since {𝑥_𝑛} is bounded, we deduce {𝐴(𝑥_𝑛)}, {𝐵(𝑥_𝑛)}, and {𝑇𝑥_𝑛} are all bounded. Hence, there exists a constant 𝑀>0 such that sup_𝑛‖‖(𝜎𝐴−𝐵)𝑧_𝑛‖‖≤𝑀.(3.8) Thus, ‖‖𝑦_𝑛+1−𝑦_𝑛‖‖≤21+𝛽‖‖‖𝜃2^𝑛+1(𝜎𝐴−𝐵)𝑧^𝑛+1−𝜃^𝑛2(𝜎𝐴−𝐵)𝑧^𝑛‖‖‖+𝛽1+𝛽‖‖𝑇𝑥^𝑛+1+1−𝑇𝑥_𝑛‖‖1+𝛽‖‖𝑅𝜃^𝑛+1≤1𝜎𝐴+𝐼−𝜃_𝑛+1𝐵𝑧_𝑛+1−𝑅𝐼+(𝜎𝐴−𝐵)𝜃_𝑛𝑧_𝑛‖‖1+𝛽𝜃^𝑛+1+𝜃_𝑛𝑀+𝛽1+𝛽‖‖𝑥^𝑛+1−𝑥_𝑛‖‖+11+𝛽‖‖𝑧^𝑛+1+1−𝑧_𝑛‖‖1+𝛽‖‖𝜃^𝑛+1≤2(𝜎𝐴−𝐵)𝑧_𝑛+1−𝜃_𝑛(𝜎𝐴−𝐵)𝑧_𝑛‖‖1+𝛽𝜃^𝑛+1+𝜃_𝑛𝑀+‖‖𝑥_𝑛+1−𝑥_𝑛‖‖.(3.9) It follows that limsup_𝑛→∞‖‖𝑦_𝑛+1−𝑦_𝑛‖‖−‖‖𝑥_𝑛+1−𝑥_𝑛‖‖≤0.(3.10) This together with Lemma 2.2 implies that lim_𝑛→∞‖‖𝑦_𝑛−𝑥_𝑛‖‖=0.(3.11) So, lim_𝑛→∞‖‖𝑥_𝑛+1−𝑥_𝑛‖‖=lim_𝑛→∞1+𝛽2‖‖𝑦^𝑛−𝑥_𝑛‖‖=0.(3.12) Since ‖‖𝑥_𝑛−𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛‖‖≤=‖‖‖‖𝑥𝑥_𝑛𝑛−𝑥_𝑛+1‖‖+‖‖𝑥_𝑛+1−𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛‖‖≤‖‖𝑥_𝑛−𝑥_𝑛+1‖‖+‖‖𝑃_𝐶𝐼+(𝜎𝐴−𝐵)𝜃_𝑛𝑃_𝐶(𝐼−𝛾∇𝑓)−𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛‖‖−𝑥_𝑛+1‖‖+𝜃_𝑛‖‖(𝜎𝐴−𝐵)𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛‖‖,(3.13) we deduce lim_𝑛→∞‖‖𝑥_𝑛−𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛‖‖=0.(3.14) Next we prove limsup_𝑘→∞𝜎𝐴𝑥^∗−𝐵𝑥^∗,𝑥_𝑛−𝑥^∗≤0,(3.15) where 𝑥^∗ is the unique solution of VI (3.2).
Indeed, we can choose a subsequence {𝑥_𝑛𝑖} of {𝑥_𝑛} such that limsup_𝑛→∞𝜎𝐴𝑥^∗−𝐵𝑥^∗,𝑥_𝑛−𝑥^∗=lim_𝑖→∞𝜎𝐴𝑥^∗−𝐵𝑥^∗,𝑥_𝑛𝑖−𝑥^∗.(3.16) Since {𝑥_𝑛𝑖} is bounded, there exists a subsequence of {𝑥_𝑛𝑖} which converges weakly to a point 𝑥̃. Without loss of generality, we may assume that {𝑥_𝑛𝑖} converges weakly to 𝑥̃. Since 𝛾∈(0,2/𝐿), 𝑃_𝐶(𝐼−𝛾∇𝑓) is nonexpansive. Thus, from (3.14) and Lemma 2.3, we have 𝑥_𝑛𝑖⇀𝑥̃∈Fix(𝑃_𝐶(𝐼−𝛾∇𝑓))=𝑆. Therefore, limsup_𝑛→∞𝜎𝐴𝑥^∗−𝐵𝑥^∗,𝑥_𝑛−𝑥^∗==li𝜎m_𝑖→∞𝐴𝜎𝑥_∗𝐴𝑥^∗−𝐵𝑥^∗,𝑥_𝑛𝑖−𝑥^∗−𝐵𝑥^∗,𝑥̃−𝑥^∗≤0.(3.17) Finally, we show 𝑥_𝑛→𝑥̃. By using the property of the projection 𝑃_𝐶, we have ‖‖𝑥_𝑛+1−𝑥̃‖‖²=‖‖𝑃_𝐶𝐼+(𝜎𝐴−𝐵)𝜃_𝑛𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛−𝑃_𝐶(𝑥̃)‖‖²≤=𝐼𝐼++((𝜎𝜎𝐴𝐴−−𝐵𝐵))𝜃𝜃_𝑛𝑛𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛−𝑥̃,𝑥_𝑛+1−𝑥̃+𝜃_𝑛𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛−𝑥̃,𝑥_𝑛+1−𝑥̃≤‖‖𝐼+(𝜎𝜎𝐴𝐴(𝑥−̃)𝐵−)𝐵𝜃_𝑛(𝑥̃),𝑥_𝑛+1−𝑥̃+𝜃_𝑛‖‖‖‖𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛−𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥̃‖‖‖‖𝑥_𝑛+1−𝑥̃‖‖≤1−(𝜎𝛼𝐴−(𝑥𝜎̃)𝜌−)𝜃𝐵_𝑛(𝑥̃),𝑥_𝑛+1−𝑥̃≤1−(𝛼−𝜎𝜌)𝜃_𝑛‖‖𝑥_𝑛−𝑥̃‖‖‖‖𝑥_𝑛+1−𝑥̃‖‖+𝜃_𝑛𝜎𝐴(𝑥̃)−𝐵(𝑥̃),𝑥_𝑛+1−𝑥̃2‖‖𝑥^𝑛−𝑥̃‖‖²+12‖‖𝑥^𝑛+1−𝑥̃‖‖²+𝜃_𝑛𝜎𝐴(𝑥̃)−𝐵(𝑥̃),𝑥_𝑛+1−𝑥̃.(3.18) It follows that ‖‖𝑥_𝑛+1−𝑥̃‖‖²=≤11−−((𝛼𝛼−−𝜎𝜎𝜌𝜌))𝜃𝜃_𝑛𝑛‖‖𝑥_𝑛−𝑥̃‖‖²+2𝜃_𝑛𝜎𝐴(𝑥̃)−𝐵(𝑥̃),𝑥_𝑛+1−𝑥̃‖‖𝑥_𝑛−𝑥̃‖‖²+(𝛼−𝜎𝜌)𝜃_𝑛𝛼−2𝜎𝜌𝜎𝐴(𝑥̃)−𝐵(𝑥̃),𝑥^𝑛+1−𝑥̃.(3.19) It is obvious that limsup_𝑛→∞((2/(𝛼−𝜎𝜌))⟨𝜎𝐴(𝑥̃)−𝐵(𝑥̃),𝑥_𝑛+1−𝑥̃⟩≤0). Then we can apply Lemma 2.4 to the last inequality to conclude that 𝑥_𝑛→𝑥̃. The proof is completed.

In (3.1), if we take 𝐴=0 and 𝐵=𝐼, then (3.1) reduces to the following.

Algorithm 3.5. For given 𝑥₀∈𝐶, compute the sequence {𝑥_𝑛} iteratively by 𝑥_𝑛+1=𝑃_𝐶1−𝜃_𝑛𝑃_𝐶(𝐼−𝛾∇𝑓)𝑥_𝑛,𝑛≥0,(3.20) where 𝜎>0,𝛾>0 are two constants and the real number sequence {𝜃_𝑛}⊂[0,1].

From Theorem 3.3, we have the following result.

Theorem 3.6. Assume that the gradient ∇𝑓 is 𝐿-Lipschitzian and 𝜎𝜌<𝛼. Let {𝑥_𝑛} be a sequence generated by (3.20), where 𝛾∈(0,2/𝐿) is a constant and the sequences {𝜃_𝑛} satisfies the conditions: (i) lim_𝑛→∞𝜃_𝑛=0 and (ii) ∑^∞𝑛=0𝜃_𝑛=∞. Then {𝑥_𝑛} converges to a minimizer 𝑥̃ of (1.1) which is the minimum norm element in 𝑆.

Proof. As a consequence of Theorem 3.3, we obtain that the sequence {𝑥_𝑛} generated by (3.20) converges strongly to 𝑥̃ which satisfies 𝑥̃∈𝑆suchthat⟨−𝑥̃,𝑥−𝑥̃⟩≤0,∀𝑥∈𝑆.(3.21) This implies ‖𝑥̃‖²≤⟨𝑥,𝑥̃⟩≤‖𝑥‖‖𝑥̃‖,∀𝑥∈𝑆.(3.22) Thus, ‖𝑥̃‖≤‖𝑥‖,∀𝑥∈𝑆.(3.23) That is, 𝑥̃ is the minimum norm element in 𝑆. This completes the proof.

Acknowledgment

Y. Yao was supported in part by Colleges and Universities Science and Technology Development Foundation (20091003) of Tianjin, NSFC 11071279 and NSFC 71161001-G0105.