An Improved Line Search Filter Method for the System of Nonlinear Equations

Abstract

An improved line search filter algorithm for the system of nonlinear equations is presented. We divide the equations into two groups, one contains the equations that are treated as equality constraints and the square of other equations is regarded as objective function. Two groups of equations are updated at every iteration in the works by Nie (2004, 2006, and 2006), by Nie et al. (2008), and by Gu (2011), while we just update them at the iterations when it is needed indeed. As a consequence, the scale of the calculation is decreased in a certain degree. Under some suitable conditions the global convergence can be induced. In the end, numerical experiments show that the method in this paper is effective.

1. Introduction

Many applied problems are reduced to solve the system of nonlinear equations, which is one of the most basic problems in mathematics. This task has applications in many scientific fields such as physics, chemistry, and economics. More formally, the problem to be solved is stated as follows 𝑐_𝑖(𝑥)=0,𝑖=1,2,…,𝑚, (1.1) where each 𝑐_𝑖∶𝑅^𝑛→𝑅(𝑖=1,2,…,𝑚) is a smooth function.

A well-known method for solving nonlinear equations is the Newton method, an iterative scheme which is locally quadratical convergent only if the initial iteration is sufficiently close to the solution. To improve the global properties, some important algorithms [1] for nonlinear equations proceed by minimizing a least square problem: minℎ(𝑥)=𝑐(𝑥)^𝑇𝑐(𝑥), (1.2) which can be also handled by the Newton method, while Powell [2] gives a counterexample to show a dissatisfactory fact that the iterates generated by the above least square problem may converge to a nonstationary point of ℎ(𝑥).

Traditionally, a penalty or augmented Lagrange function is always used as a merit function to test the acceptability of the iterates. However, as we all know, there are several difficulties associated with the use of penalty function, and in particular the choice of the penalty parameter. Too low a choice may result in an optimal solution that cannot be obtained, on the other hand, too large a choice damps out the effect of the objective function. Hence, filter method has been first introduced for constrained nonlinear optimization problems in a sequential quadratic programming (SQP) trust-region algorithm by Fletcher and Leyffer [3], offering an alternative to merit functions, as a tool to guarantee global convergence in algorithms for nonlinear programming (NLP). The computational results are very promising in [3]. Furthermore, Fletcher et al. [4, 5] give the global convergence of the trust-region filter-SQP method, then Ulbrich [6] gets its superlinear local convergence. Consequently, filter method has been actually applied in many optimization techniques [7–13], for instance the pattern search method [7], the SLP method [8], the interior method [9], the bundle approaches [10, 11], and so on. Also combined with the trust-region search technique, Gould et al. extended the filter method to the system of nonlinear equations (1.1) and nonlinear least squares in [14], and to the unconstrained optimization problem with multidimensional filter technique in [15]. In addition, Wächter and Biegler [16, 17] presented line search filter methods for nonlinear equality constrained programming and the global and local convergence were given.

Recently, some other ways were given to attack the problem (1.1) (see [18–22]). There are two common features in these papers, one is the filter approach that is utilized, and the other is that at every iteration the system of nonlinear equations is transformed into a constrained nonlinear programming problem and the equations are divided into two groups, some equations are treated as constraints and the others act as the objective function. For instance combined with the filter line search technique [16, 17], the system of nonlinear equations in [21] at the kth iteration 𝑥_𝑘 becomes the following optimization problem with equality constraints: min_{𝑖∈𝑆𝑘1}𝑐²_𝑖(𝑥)s.t.𝑐_𝑗(𝑥)=0,𝑗∈𝑆^𝑘₂, (1.3) where the sets 𝑆^𝑘₁ and 𝑆^𝑘₂ are defined as 𝑆^𝑘₁={𝑖_𝑗∣𝑗≤𝑛₀} and 𝑆^𝑘₂={𝑖_𝑗∣𝑗≥𝑛₀+1} for some positive constant 𝑛₀>0 such that 𝑐²_𝑖1(𝑥_𝑘)≥𝑐²_𝑖2(𝑥_𝑘)≥⋯≥𝑐²_𝑖𝑛(𝑥_𝑘).

Motivated by the ideas and methods above, we propose an improved line search filter method for the system of nonlinear equations. We also divide the equations into two groups, one contains the equations that are treated as equality constraints and the square of other equations is regarded as objective function. Two groups of equations are updated at every iteration in those works [18–22], while we just update them at the iterations when it is needed indeed. Specifically, using similar transformed optimization problem (1.3), the difference between our method with [21] is that the sets 𝑆^𝑘₁ and 𝑆^𝑘₂ in (1.3) need not be updated at every iteration. As a consequence, the scale of the calculation is decreased in a certain degree. In our algorithm two groups of equations cannot be changed after a f-type iteration, thus in the case that |𝒜|<∞, the two groups are fixed after finite number of iterations. In addition, the filter should not be updated after a f-type iteration, so naturally the global convergence is discussed, respectively, according to whether the number of the updated filter is infinite or not. And it is shown that every limit point of the sequence of iterates generated by the algorithm is the 𝜖 solution to (1.1) or a local infeasible point when |𝒜|<∞. Furthermore, the global convergent property is induced under mild assumptions. In the end, numerical experiments show that the method in this paper is effective.

The paper is outlined as follows. In Section 2, the line search filter method is developed and its key ingredients are described. In Section 3 we prove that under suitable conditions, the method is well defined and globally convergent. Finally, some numerical results are given in Section 4.

2. Description of the Algorithm

To solve the system of nonlinear Equation (1.1), we also transform it into the optimization problem (1.3) where equations are divided into two groups as the one in [21], then let 𝑚_𝑘(𝑥)=‖𝑐_𝑆^𝑘₁(𝑥)‖²₂=∑_𝑖∈𝑆^𝑘₁𝑐²_𝑖(𝑥) and 𝜃_𝑘(𝑥)=‖𝑐_𝑆^𝑘₂(𝑥)‖²₂=∑_𝑖∈𝑆^𝑘₂𝑐²_𝑖(𝑥). The linearization of the KKT condition of (1.3) at the kth iteration 𝑥_𝑘 is as follows: 𝐵_𝑘𝐴^𝑘_𝑆𝑘2𝐴^𝑘_𝑆𝑘2^𝑇0𝑠_𝑘𝜆⁺_𝑘𝑔𝑥=−^𝑘𝑐_𝑆^𝑘₂𝑥_𝑘, (2.1) where 𝐵_𝑘 is the Hessian or approximate Hessian matrix of 𝐿(𝑥,𝜆)=𝑚_𝑘(𝑥)+𝜆^𝑇𝑐_𝑆^𝑘₂(𝑥), 𝐴^𝑘_𝑆𝑘2=∇𝑐_𝑆^𝑘₂(𝑥_𝑘) and 𝑔(𝑥_𝑘)=∇𝑚_𝑘(𝑥_𝑘).

After a search direction 𝑠_𝑘 has been computed, a step size 𝛼_𝑘,𝑙∈(0,1] is determined in order to obtain the trial iteration 𝑥_𝑘𝛼_𝑘,𝑙=𝑥_𝑘+𝛼_𝑘,𝑙𝑠_𝑘. (2.2) More precisely, for fixed constants 𝛾_𝑚,𝛾_𝜃∈(0,1), we say that a trial step size 𝛼_𝑘,𝑙 provides sufficient reduction with respect to the current 𝑥_𝑘 if 𝜃_𝑘𝑥_𝑘𝛼_𝑘,𝑙≤1−𝛾_𝜃𝜃_𝑘𝑥_𝑘or𝑚_𝑘𝑥_𝑘𝛼_𝑘,𝑙≤𝑚_𝑘𝑥_𝑘−𝛾_𝑚𝜃_𝑘𝑥_𝑘. (2.3)

For the sake of a simplified notation, we define the filter in this paper not as a list but as a set ℱ_𝑘⊆[0,∞]×[0,∞] containing all (𝜃,𝑚)-pairs which are prohibited in iteration 𝑘. We say that a trial point 𝑥_𝑘(𝛼_𝑘,𝑙) is acceptable to the filter if its (𝜃,𝑚)-pair does not lie in the taboo region, that is, if 𝜃𝑥_𝑘𝛼_𝑘,𝑙,𝑚𝑥_𝑘𝛼_𝑘,𝑙∉ℱ_𝑘. (2.4) At the beginning of the optimization, the filter is initialized to be empty: ℱ₀=∅. Throughout the optimization the filter is then augmented in some iterations after the new iterate 𝑥_𝑘+1 has been accepted. For this, the following updating formula is used: ℱ_𝑘+1=ℱ_𝑘∪(𝜃,𝑚)∈𝑅²∶𝜃≥1−𝛾_𝜃𝜃_𝑘𝑥_𝑘,𝑚≥𝑚_𝑘𝑥_𝑘−𝛾_𝑚𝜃_𝑘𝑥_𝑘. (2.5)

Similar to the traditional strategy of the filter method, to avoid obtaining a feasible point but not an optimal solution, we consider the following f-type switching condition: 𝑢_𝑘𝛼_𝑘,𝑙<0,−𝑢_𝑘𝛼_𝑘,𝑙>𝛿𝜃_𝑘𝑥_𝑘^𝑠𝜃, (2.6) where 𝑢_𝑘(𝛼_𝑘,𝑙)=𝛼_𝑘,𝑙𝑔^𝑇_𝑘𝑠_𝑘, 𝛿>0 and 𝑠_𝜃∈(0,1).

When Condition (2.6) holds, the step 𝑠_𝑘 is a descent direction for current objective function. Then, instead of insisting on (2.3), the Armijo-type reduction condition is employed as follows: 𝑚_𝑘𝑥_𝑘𝛼_𝑘,𝑙≤𝑚_𝑘𝑥_𝑘+𝜏₃𝑢_𝑘𝛼_𝑘,𝑙, (2.7) where 𝜏₃∈(0,1/2) is a fixed constant.

If (2.6) and (2.7) hold for the accepted trial step size, we may call it an f-type point, and accordingly this iteration is called an f-type iteration. An f-type point should be accepted as 𝑥_𝑘+1 with no updating of the filter, that is, ℱ_𝑘+1=ℱ_𝑘. (2.8) While if a trial point 𝑥_𝑘(𝛼_𝑘,𝑙) does not satisfy the switching condition (2.6) but satisfies (2.3), we call it an h-type point (or accordingly an h-type iteration). An h-type point should be accepted as 𝑥_𝑘+1 with updating of the filter.

In the situation, where no admissible step size can be found, the method switches to a feasibility restoration stage, whose purpose is to find a new iterate that satisfies (2.3) and is also acceptable to the current filter by trying to decrease the constraint violation. In order to detect the situation where no admissible step size can be found and the restoration phase has to be invoked, we define that 𝛼_𝑘^min=min𝛾_𝜃,𝛾_𝑚𝜃_𝑘𝑥_𝑘^𝑠𝜃−𝑔^𝑇_𝑘𝑠_𝑘,if𝑔^𝑇_𝑘𝑠_𝑘𝛾<0,_𝜃,otherwise. (2.9)

We are now ready to formally state the overall algorithm for solving the the system of nonlinear Equation (1.1).

Algorithm 2.1. We have the following steps.
Step 1. Initialization: Choose an initial point 𝑥₀∈𝑅^𝑛, 0<𝜌₁<𝜌₂<1 and 𝜖>0. Compute 𝑔₀, 𝑐_𝑖(𝑥₀), 𝑆⁰₁, 𝑆⁰₂ and 𝐴_𝑘 for 𝑖∈𝑆⁰₂. Set 𝑘=0 and ℱ₀=∅.
Step 2. If ‖𝑐(𝑥_𝑘)‖≤𝜖 then stop. Otherwise compute (2.1) to obtain 𝑠_𝑘. If there exists no solution to (2.1), go to Step 6. If ‖𝑠_𝑘‖≤𝜖 then stop.
Step 3. Let 𝛼_𝑘,𝑙=1 and 𝑙=0. Compute 𝛼_𝑘^min by (2.9).
Step 3.1. If 𝛼_𝑘,𝑙<𝛼_𝑘^min, go to Step 6. Otherwise compute 𝑥_𝑘(𝛼_𝑘,𝑙)=𝑥_𝑘+𝛼_𝑘,𝑙𝑠_𝑘. If 𝑥_𝑘(𝛼_𝑘,𝑙)∈ℱ_𝑘, go to Step 3.2. Step 3.1.1. Case 1. The switching condition (2.6) holds. If the reduction condition (2.7) holds, set 𝑥_𝑘+1=𝑥_𝑘(𝛼_𝑘,𝑙), ℱ_𝑘+1=ℱ_𝑘, 𝑆₁^𝑘+1=𝑆^𝑘₁, 𝑆₂^𝑘+1=𝑆^𝑘₂ and go to Step 5. Otherwise, go to Step 3.2.  Step 3.1.2. Case 2. The switching condition (2.6) is not satisfied. If (2.3) holds, set 𝑥_𝑘+1=𝑥_𝑘(𝛼_𝑘,𝑙), augment the filter using (2.5) and go to Step 4. Otherwise, go to Step 3.2.
Step 3.2. Choose 𝛼_𝑘,𝑙+1∈[𝜌₁𝛼_𝑘,𝑙,𝜌₂𝛼_𝑘,𝑙]. Set 𝑙=𝑙+1 and go to Step 3.1.
Step 4. Compute 𝑆₁^𝑘+1 and 𝑆₂^𝑘+1 by (1.3). If (𝜃_𝑘+1(𝑥_𝑘+1),𝑚_𝑘+1(𝑥_𝑘+1))∈ℱ_𝑘+1, set 𝑆₁^𝑘+1=𝑆^𝑘₁ and 𝑆₂^𝑘+1=𝑆^𝑘₂.
Step 5. Compute 𝑔_𝑘+1, 𝐵_𝑘+1 and 𝐴_𝑘+1. Go to Step 2 with 𝑘 replaced by 𝑘+1.
Step 6. (Feasibility Restoration Stage) Find 𝑥^𝑟_𝑘=𝑥_𝑘+𝛼^𝑟_𝑘𝑠^𝑟_𝑘 such that 𝑥^𝑟_𝑘 is accepted by the filter and the infeasibility 𝜃 is reduced. Go to Step 2.

3. Global Convergence of Algorithm

In the reminder of this paper we denote the set of indices of those iterations in which the filter has been augmented by 𝒜⊆ℕ. Let us now state the assumptions necessary for the global convergence analysis.

Assumption 3.1. The sequence {𝑥^𝑘} generated by Algorithm 2.1 is nonempty and bounded.

Assumption 3.2. The functions 𝑐_𝑖(𝑥), 𝑗=1,2,…,𝑚 are all twice continuously differentiable on an open set containing 𝑋.

Assumption 3.3. There exist two constants 𝑏≥𝑎>0 such that the matrices sequence {𝐵_𝑘} satisfy 𝑎‖𝑠‖²≤𝑠^𝑇𝐵_𝑘𝑠≤𝑏‖𝑠‖² for all 𝑘 and 𝑠∈𝑅^𝑛.

Assumption 3.4. (𝐴^𝑘_𝑠𝑘2)^𝑇 has full column rank and ‖𝑠_𝑘‖≤𝛾_𝑠 for all 𝑘 with a positive constant 𝛾_𝑠.

By Assumption 3.1 we have Σ^𝑛_𝑖=1𝑐²_𝑖(𝑥_𝑘)≤𝑀^max. Since 0≤𝑚_𝑘(𝑥_𝑘)≤𝑚_𝑘(𝑥_𝑘)+𝜃_𝑘(𝑥_𝑘)=Σ^𝑛_𝑖=1𝑐²_𝑖(𝑥_𝑘) and 0≤𝜃_𝑘(𝑥_𝑘)≤𝑚_𝑘(𝑥_𝑘)+𝜃_𝑘(𝑥_𝑘)=Σ^𝑛_𝑖=1𝑐²_𝑖(𝑥_𝑘), then (𝜃,𝑚) associated with the filter is restricted to ℬ=0,𝑀^max×0,𝑀^max. (3.1)

Theorem 3.5. If Algorithm 2.1 terminates at Step 2, then an 𝜖 solution to (1.1) is achieved or a local infeasibility point is obtained.

Proof. The proof of this theorem can be found in [21].

Next we can assume that our algorithm does not terminate finitely and an infinite sequence of points is generated.

Lemma 3.6. Under the above assumptions, there exists a solution to (2.1) with exact (or inexact) line search which satisfies the following descent conditions, 𝜃_𝑘𝑥_𝑘+𝛼𝑠_𝑘−(1−2𝛼)𝜃_𝑘𝑥_𝑘≤𝜏₁𝛼²𝑠_𝑘²,𝑚_𝑘𝑥_𝑘+𝛼𝑠_𝑘−𝑚_𝑘𝑥_𝑘−𝑢_𝑘(𝛼)≤𝜏₂𝛼²𝑠_𝑘², (3.2)(3.3) where 𝛼∈(0,1), 𝜏₁, and 𝜏₂ are all positive constants independent of 𝑘.

Proof. By virtue of Taylor expansion of 𝑐²_𝑖(𝑥_𝑘+𝛼𝑠_𝑘) with 𝑖∈𝑆₂, we obtain 𝑐²_𝑖𝑥_𝑘+𝛼𝑠_𝑘−𝑐²_𝑖𝑥_𝑘−2𝛼𝑐_𝑖𝑥_𝑘∇𝑐_𝑖𝑥_𝑘^𝑇𝑠_𝑘=𝑐²_𝑖𝑥_𝑘+𝛼𝑠_𝑘−𝑐²_𝑖𝑥_𝑘−2𝑐_𝑖𝑥_𝑘∇𝑐_𝑖𝑥_𝑘^𝑇𝛼𝑠_𝑘=12𝛼𝑠_𝑘^𝑇2𝑐_𝑖𝑥_𝑘+𝜁𝛼𝑠_𝑘∇𝑐²_𝑖𝑥_𝑘+𝜁𝛼𝑠_𝑘+2∇𝑐_𝑖𝑥_𝑘+𝜁𝛼𝑠_𝑘∇𝑐_𝑖𝑥_𝑘+𝜁𝛼𝑠_𝑘^𝑇𝛼𝑠_𝑘=𝛼²𝑠^𝑇_𝑘𝑐_𝑖𝑥_𝑘+𝜁𝛼𝑠_𝑘∇𝑐²_𝑖𝑥_𝑘+𝜁𝛼𝑠_𝑘+∇𝑐_𝑖𝑥_𝑘+𝜁𝛼𝑠_𝑘∇𝑐_𝑖𝑥_𝑘+𝜁𝛼𝑠_𝑘^𝑇𝑠_𝑘≤1𝑚𝜏₁𝛼²𝑠_𝑘², (3.4) where the last inequality follows the above assumptions and 𝜁∈[0,1]. Also, from (2.1) we immediately get that 𝑐_𝑖(𝑥_𝑘)+∇𝑐_𝑖(𝑥_𝑘)^𝑇𝑠_𝑘=0, that is, −2𝛼𝑐²_𝑖(𝑥_𝑘)−2𝛼𝑐_𝑖(𝑥_𝑘)∇𝑐_𝑖(𝑥_𝑘)^𝑇𝑠_𝑘=0. With |𝑆₂|≤𝑚, thereby, 𝜃_𝑘𝑥_𝑘+𝛼𝑠_𝑘−(1−2𝛼)𝜃_𝑘𝑥_𝑘=_𝑖∈𝑆2𝑐²_𝑖𝑥_𝑘+𝛼𝑠_𝑘−(1−2𝛼)𝑐²_𝑖𝑥_𝑘≤_𝑖∈𝑆2𝑐²_𝑖𝑥_𝑘+𝛼𝑠_𝑘−(1−2𝛼)𝑐²_𝑖𝑥_𝑘=_𝑖∈𝑆2𝑐²_𝑖𝑥_𝑘+𝛼𝑠_𝑘−𝑐²_𝑖𝑥_𝑘−2𝛼𝑐_𝑖𝑥_𝑘∇𝑐_𝑖𝑥_𝑘^𝑇𝑠_𝑘1≤𝑚⋅𝑚𝜏₁𝛼²𝑠_𝑘²≤𝜏₁𝛼²𝑠_𝑘², (3.5) then (3.2) consequently holds.
According to Taylor expansion of Σ_𝑖∈𝑆1(𝑐²_𝑖(𝑥_𝑘+𝛼𝑠_𝑘)) (i.e., 𝑚_𝑘(𝑥_𝑘+𝛼𝑠_𝑘)), we then have _𝑖∈𝑆1𝑐²_𝑖𝑥_𝑘+𝛼𝑠_𝑘−_𝑖∈𝑆1𝑐²_𝑖𝑥_𝑘−𝛼𝑔^𝑇_𝑘𝑠_𝑘=12𝛼²𝑠_𝑘^𝑇∇²_𝑖∈𝑆1𝑐²_𝑖𝑥_𝑘+𝜌𝛼𝑠_𝑘𝑠_𝑘≤𝜏₂𝛼²𝑠_𝑘², (3.6) where the last inequality follows the first two assumptions and 𝜌∈[0,1]. That is to say, 𝑚_𝑘𝑥_𝑘+𝛼𝑠_𝑘−𝑚_𝑘𝑥_𝑘−𝑢_𝑘(𝛼)≤𝜏₂𝛼²𝑠_𝑘², (3.7) which is just (3.3).

Theorem 3.7. If there are only finite or infinite number of iterates entering the filter, then lim_𝑘→∞𝜃𝑥_𝑘=0. (3.8)

Proof. Consider the following.
Case 1 (if |𝒜|=∞). The proof is by contraction. Suppose that there exists an infinite subsequence {𝑘_𝑖} of 𝒜 such that 𝜃_𝑘𝑖𝑥_𝑘𝑖≥𝜀, (3.9) for some 𝜀>0. At each iteration 𝑘_𝑖, (𝜃_𝑘𝑖(𝑥_𝑘𝑖),𝑚_𝑘𝑖(𝑥_𝑘𝑖)) is added to the filter which means that no other (𝜃,𝑚) can be added to the filter at a later stage within the area 𝜃_𝑘𝑖𝑥_𝑘𝑖−𝛾_𝜃𝜃_𝑘𝑖𝑥_𝑘𝑖,𝜃_𝑘𝑖𝑥_𝑘𝑖×𝑚_𝑘𝑖𝑥_𝑘𝑖−𝛾_𝑚𝜃_𝑘𝑖𝑥_𝑘𝑖,𝑚_𝑘𝑖, (3.10) and the area of each of these squares is at least 𝛾_𝜃𝛾_𝑚𝜀².
Thus the ℬ is completely covered by at most a finite number of such areas in contraction to the infinite subsequence {𝑘_𝑖} satisfying (3.9). This means that (3.8) is true.
Case 2 (if |𝒜|<∞). From |𝒜|<∞, we know the filter updates in a finite number, then there exists 𝐾∈ℕ, for 𝑘>𝐾 the filter does not update. As h-type iteration and restoration algorithm all need the updating of the filter, then for 𝑘>𝐾 our algorithm only executes the f-type iterations.
Because 𝑆₁ and 𝑆₂ do not change in the f-type iteration, then for 𝑘≥𝐾+1 we have 𝜃_𝑘(𝑥)=𝜃_𝐾(𝑥) and 𝑚_𝑘(𝑥)=𝑚_𝐾(𝑥). Thus we obtain ^∞_𝑘=𝐾+1𝑚_𝑘𝑥_𝑘−𝑚_𝑘𝑥_𝑘+1=^∞_𝑘=𝐾+1𝑚_𝐾𝑥_𝑘−𝑚_𝐾𝑥_𝑘+1. (3.11)
From the switching condition (2.6) we know that 𝜏₃𝑢_𝑘𝛼_𝑘,𝑙=𝜏₃𝛼_𝑘,𝑙𝑔^𝑇_𝑘𝑠_𝑘<−𝜏₃𝛿𝜃_𝑘𝑥_𝑘^𝛾𝜃, (3.12) together with the reduction condition (2.7) we get 𝑚_𝑘𝑥_𝑘𝛼_𝑘,𝑙≤𝑚_𝑘𝑥_𝑘+𝜏₃𝑢_𝑘𝛼_𝑘,𝑙≤𝑚_𝑘𝑥_𝑘−𝜏₃𝛿𝜃_𝑘𝑥_𝑘^𝛾𝜃, (3.13) and since 𝑥_𝑘+1=𝑥_𝑘(𝛼_𝑘,𝑙) for 𝑘≥𝐾+1, this reduces to 𝑚_𝑘𝑥_𝑘+1=𝑚_𝑘𝑥_𝑘𝛼_𝑘,𝑙≤𝑚_𝑘𝑥_𝑘−𝜏₃𝛿𝜃_𝑘𝑥_𝑘^𝛾𝜃. (3.14)
Suppose to the contrary that there exists an infinite subsequence {𝑘_𝑖} of 𝒜 such that 𝜃_𝑘𝑖𝑥_𝑘𝑖≥𝜀, (3.15) for some 𝜀>0. Choose an arbitrary subsequence {𝑘_𝑖} of {𝑘_𝑖} satisfying 𝑘_𝑖>𝐾, we find that +∞>^∞_𝑘=𝐾+1𝑚_𝐾𝑥_𝑘−𝑚_𝐾𝑥_𝑘+1≥^∞_𝑖=1𝑚_𝐾𝑥_𝑘𝑖−𝑚_𝐾𝑥_𝑘𝑖+1≥^∞_𝑖=1𝜏₃𝛿𝜃_𝑘𝑖𝑥_𝑘𝑖^𝛾𝜃→+∞, (3.16) which is a contraction.

Lemma 3.8. Let {𝑥_𝑘𝑖} be a subsequence of iterates for which (2.6) hold and have the same 𝑆₁ and 𝑆₂. Then there exists a 𝛼∈(0,1] such that 𝑚_𝑘𝑖𝑥_𝑘𝑖+𝛼𝑠_𝑘𝑖≤𝑚_𝑘𝑖𝑥_𝑘𝑖+𝛼𝜏₃𝑔^𝑇_𝑘𝑖𝑠_𝑘𝑖. (3.17)

Proof. Because {𝑥_𝑘𝑖} have the same 𝑆₁ and 𝑆₂, 𝑚_𝑘𝑖(𝑥) are fixed. And 𝑑_𝑘𝑖 is a decent direction due to (2.6). Hence there exists a 𝛼∈(0,1] satisfying (3.17).

Lemma 3.9. If there exists 𝐾∈ℕ such that for 𝑘>𝐾 it holds ‖𝑠_𝑘‖≥(1/2)𝜀₁, where 𝜀₁>0, then there exist 𝐾₁∈ℕ and a constant 𝜀₀>0, for 𝑘>𝐾₁ such that 𝑔^𝑇_𝑘𝑠_𝑘≤−𝜀₀. (3.18)

Proof. From (2.1) we obtain 𝑔_𝑘+𝐴^𝑘_𝑆2𝜆⁺_𝑘+𝐵_𝑘𝑠_𝑘=0 and 𝑐^𝑘_𝑆2+(𝐴^𝑘_𝑆2)^𝑇𝑠_𝑘=0, so 𝑔^𝑇_𝑘𝑠_𝑘=−𝑠^𝑇_𝑘𝐵_𝑘𝑠_𝑘−𝜆⁺_𝑘^𝑇𝑐^𝑘_𝑆2≤−𝑎𝑠_𝑘²−𝜆⁺_𝑘^𝑇𝑐^𝑘_𝑆2≤−𝑎𝑠_𝑘²+𝑐₁𝑐^𝑘_𝑆2. (3.19) As lim_𝑘→∞𝜃(𝑥_𝑘)=0 implies lim_𝑘→∞‖𝑐^𝑘_𝑆2‖=0, then there exists 𝐾₁(∈𝑁)≥𝐾, for 𝑘>𝐾₁ such that ‖𝑐^𝑘_𝑆2‖≤(𝑎𝜀²₁)/(8𝑐₁). Thus 𝑔^𝑇_𝑘𝑠_𝑘≤−𝑎𝑠_𝑘²+𝑐₁𝑐^𝑘_𝑆21≤−𝑎2𝜀₁²+𝑐₁𝑎𝜀²₁8𝑐₁=−𝜀₀, (3.20) with 𝜀₀=(𝑎𝜀²₁)/(8𝑐₁).

Theorem 3.10. Suppose that {𝑥_𝑘} is an infinite sequence generated by Algorithm 2.1 and |𝒜|<∞, one has lim_𝑘→∞𝑐^𝑘_𝑆𝑘2+𝑠_𝑘=0. (3.21) Namely, every limit point is the 𝜖 solution to (1.1) or a local infeasible point. If the gradients of 𝑐_𝑖(𝑥_𝑘) are linear independent for all 𝑘 and 𝑖=1,2,…,𝑚, then the solution to (1.1) is obtained.

Proof. As was noted in the proof of the case 2 in Theorem 3.7, from |𝒜|<∞, the algorithm only executes the f-type iterations. Now the proof is by contraction. If the result is false, there should have been a constant 𝜀₁>0 and a subsequence {𝑥_𝑘𝑖} of {𝑥_𝑘} such that 𝑐^𝑘𝑖_{𝑆𝑘𝑖2}+𝑠_𝑘𝑖>𝜀₁, (3.22) for all 𝑘_𝑖. Without loss of generality, we can suppose there are only f-type iterations for 𝑘≥𝑘₁. Moreover, there exist the following results for sufficiently large 𝑘_𝑖.
As lim_𝑘→∞𝜃(𝑥_𝑘)=0, it is apparent that ‖𝑐^𝑘𝑖_{𝑆𝑘𝑖2}‖≤(1/2)𝜀₁ hold for large enough 𝑘_𝑖. It is reasonable that we have ‖𝑠_𝑘𝑖‖≥(1/2)𝜀₁ from (3.22), so similar to the Lemma 3.9, we can get 𝑔^𝑇_𝑘𝑖𝑠_𝑘𝑖≤−𝜀₀ for large enough 𝑘_𝑖. Without loss of generality, suppose 𝑔^𝑇_𝑘𝑖𝑠_𝑘𝑖≤−𝜀₀ for all 𝑘_𝑖.
Because 𝑆₁ and 𝑆₂ do not change in the f-type iteration, then for all 𝑘_𝑖 we have 𝜃_𝑘𝑖(𝑥)=𝜃_𝑘1(𝑥) and 𝑚_𝑘𝑖(𝑥)=𝑚_𝑘1(𝑥). Since the f-type point satisfies the reduction condition (2.7), that is, 𝑚_𝑘(𝑥_𝑘)−𝑚_𝑘(𝑥_𝑘+1)>−𝜏₃𝛼𝑔^𝑇_𝑘𝑠_𝑘, then it holds that ^∞_𝑖=1𝑚_𝑘𝑖𝑥_𝑘𝑖−𝑚_𝑘𝑖𝑥_𝑘𝑖+1=^∞_𝑖=1𝑚_𝑘1𝑥_𝑘𝑖−𝑚_𝑘1𝑥_𝑘𝑖+1>^∞_𝑖=1−𝜏₃𝛼𝑔^𝑇_𝑘𝑖𝑠_𝑘𝑖. (3.23) By 𝑚_𝑘(𝑥_𝑘) is decreasing for 𝑘≥𝑘₁ and Lemma 3.8, we get +∞>^∞_𝑘=𝑘1𝑚_𝑘𝑥_𝑘−𝑚_𝑘𝑥_𝑘+1>^∞_𝑖=1𝑚_𝑘𝑖𝑥_𝑘𝑖−𝑚_𝑘𝑖𝑥_𝑘𝑖+1>^∞_𝑖=1−𝛼𝜏₃𝑔^𝑇_𝑘𝑖𝑠_𝑘𝑖>^∞_𝑖=1𝛼𝜏₃𝜀₀→+∞, (3.24) which is a contraction.
If the gradients of 𝑐_𝑖(𝑥_𝑘) are linear independent for all 𝑘 and 𝑖=1,2,…,𝑚, then the solution to (1.1) is obtained by virtue of the result in [23]. Thus, the result is established.

Lemma 3.11. If 𝑔^𝑇_𝑘𝑠_𝑘≤−𝜀₀ for a positive constant 𝜀₀ independent of 𝑘, then there exists a constant 𝛼>0, for all 𝑘 and 𝛼≤𝛼 such that 𝑚_𝑘𝑥_𝑘+𝛼𝑠_𝑘−𝑚_𝑘𝑥_𝑘≤𝜏₃𝑢_𝑘(𝛼). (3.25)

Proof. Let 𝛼=((1−𝜏₃)𝜀₀)/𝜏₂𝛾²_𝑠. From (3.3), ‖𝑠_𝑘‖≤𝛾_𝑠 and 𝛼≤𝛼, we obtain 𝑚_𝑘𝑥_𝑘+𝛼𝑠_𝑘−𝑚_𝑘𝑥_𝑘−𝑢_𝑘(𝛼)≤𝜏₂𝛼²𝑠_𝑘²≤𝜏₂𝛼𝛼𝛾²_𝑠=1−𝜏₃𝛼𝜀₀≤−1−𝜏₃𝛼𝑔^𝑇_𝑘𝑠_𝑘=−1−𝜏₃𝑢_𝑘(𝛼), (3.26) which shows that (3.25) is true.

Lemma 3.12. Under all existing assumptions, if 𝑢_𝑘(𝛼)≤−𝛼𝜀₀ for a positive constant 𝜀₀ independent of 𝑘 and 𝛼≥𝛼^min_𝑘,𝑙 for all 𝛼∈(0,1] with (𝜃_𝑘(𝑥_𝑘),𝑚_𝑘(𝑥_𝑘))∉ℱ_𝑘, then there exist 𝛾₁, 𝛾₂>0 such that (𝜃_𝑘(𝑥_𝑘+𝛼𝑠_𝑘),𝑚_𝑘(𝑥_𝑘+𝛼𝑠_𝑘))∉ℱ_𝑘 for all 𝑘 and 𝛼≤min{𝛾₁,𝛾₂𝜃_𝑘(𝑥_𝑘)}.

Proof. Choose 𝛾₁=𝜀₀/𝜏₂𝛾²_𝑠, then 𝛼≤𝛾₁ implies that −𝛼𝜀₀+𝜏₂𝛼²𝛾²_𝑠≤0. So from (3.3), we obtain 𝑚_𝑘𝑥_𝑘+𝛼𝑠_𝑘≤𝑚_𝑘𝑥_𝑘+𝑢_𝑘(𝛼)+𝜏₂𝛼²𝑠_𝑘²≤𝑚_𝑘𝑥_𝑘−𝛼𝜀₀+𝜏₂𝛼²𝛾²_𝑠≤𝑚_𝑘𝑥_𝑘. (3.27)
And choose 𝛾₂=2/𝜏₁𝛾²_𝑠, then 𝛼≤𝛾₂𝜃_𝑘(𝑥_𝑘) implies that −2𝛼𝜃_𝑘(𝑥_𝑘)+𝜏₁𝛼²𝛾²_𝑠≤0. It follows from (3.2) that 𝜃_𝑘𝑥_𝑘+𝛼𝑠_𝑘≤𝜃_𝑘𝑥_𝑘−2𝛼𝜃_𝑘𝑥_𝑘+𝜏₂𝛼²𝑠_𝑘²≤𝜃_𝑘𝑥_𝑘−2𝛼𝜃_𝑘𝑥_𝑘+𝜏₁𝛼²𝛾²_𝑠≤𝜃_𝑘𝑥_𝑘. (3.28)
We further point out a fact according to the definition of filter. If (𝜃,𝑚)∉ℱ_𝑘 and 𝜃≤𝜃, 𝑚≤𝑚, we note that (𝜃,𝑚)∉ℱ_𝑘. So from (𝜃_𝑘(𝑥_𝑘),𝑚_𝑘(𝑥_𝑘))∉ℱ_𝑘, 𝑚_𝑘(𝑥_𝑘+𝛼𝑠_𝑘)≤𝑚_𝑘(𝑥_𝑘) and 𝜃_𝑘(𝑥_𝑘+𝛼𝑠_𝑘)≤𝜃_𝑘(𝑥_𝑘), we get (𝜃_𝑘(𝑥_𝑘+𝛼𝑠_𝑘),𝑚_𝑘(𝑥_𝑘+𝛼𝑠_𝑘))∉ℱ_𝑘.

Lemma 3.13. If there exists 𝐾∈ℕ such that for 𝑘>𝐾 it holds ‖𝑠_𝑘‖≥(1/2)𝜀₁ for a constant 𝜀₁>0, then there exists 𝐾∈ℕ so that for all 𝑘>𝐾 the f-type iterate is always satisfied.

Proof. By Lemma 3.9, there exists 𝐾₁∈ℕ and a constant 𝜀₀>0, for 𝑘>𝐾₁ such that 𝑔^𝑇_𝑘𝑠_𝑘≤−𝜀₀. From the definition of 𝛼_𝑘^min, when 𝑔^𝑇_𝑘𝑠_𝑘≤0, we have 𝛼_𝑘^min=min{𝛾_𝜃,(𝛿[𝜃_𝑘(𝑥_𝑘)]^𝑠𝜃)/(−𝑔^𝑇_𝑘𝑠_𝑘)}. As lim_𝑘→∞𝜃(𝑥_𝑘)=0, we have lim_𝑘→∞(𝛿[𝜃_𝑘(𝑥_𝑘)]^𝑠𝜃)/𝜀₀=0. So there exists 𝐾₂ (>𝐾₁) ∈ℕ such that for 𝑘>𝐾₂ it holds 𝛿𝜃_𝑘𝑥_𝑘^𝑠𝜃−𝑔^𝑇_𝑘𝑠_𝑘<𝛿𝜃_𝑘𝑥_𝑘^𝑠𝜃𝜀₀<𝛾_𝜃, (3.29) which implies for 𝑘>𝐾₁ we have 𝛼_𝑘^min=min{𝛾_𝜃,(𝛿[𝜃_𝑘(𝑥_𝑘)]^𝑠𝜃)/(−𝑔^𝑇_𝑘𝑠_𝑘)}=(𝛾_𝑚[𝜃_𝑘(𝑥_𝑘)]^𝑠𝜃)/(−𝑔^𝑇_𝑘𝑠_𝑘).
Because lim_𝑘→∞𝜃(𝑥_𝑘)=0, there exists 𝐾 (>𝐾₂) ∈ℕ such that for 𝑘>𝐾 it follows 𝜃_𝑘𝑥_𝑘<min𝛼𝛾₂,𝛾₁𝛾₂,𝜌₁𝛾₂𝜀₀𝛿^{1/(𝑠𝜃−1)}, (3.30) which shows that 𝛾₂𝜃_𝑘𝑥_𝑘≤min𝛼,𝛾₁,𝛾₂𝜃_𝑘𝑥_𝑘,𝛿𝜃_𝑘𝑥_𝑘^𝑠𝜃𝜀₀<𝜌₁𝛾₂𝜃_𝑘𝑥_𝑘. (3.31) Moreover, 𝐾>𝐾₂>𝐾₁ implies 𝑔^𝑇_𝑘𝑠_𝑘≤−𝜀₀ for all 𝑘>𝐾.
Let 𝛽_𝑘=𝛾₂𝜃_𝑘(𝑥_𝑘), by Lemmas 3.11 and 3.12, for 𝛼≤𝛽_𝑘 we obtain 𝑚_𝑘𝑥_𝑘+𝛼𝑠_𝑘−𝑚_𝑘𝑥_𝑘≤𝜏₃𝑢_𝑘(𝛼),𝜃_𝑘𝑥_𝑘+𝛼𝑠_𝑘,𝑚_𝑘𝑥_𝑘+𝛼𝑠_𝑘∉ℱ_𝑘. (3.32) If we now denote with 𝛼_𝑘,𝐿 the first trial size satisfying (3.32), the backtracking line search procedure then implies that for 𝛼≥𝛼_𝑘,𝐿𝛼≥𝜌₁𝛽_𝑘=𝜌₁𝛾₂𝜃_𝑘𝑥_𝑘>𝛿𝜃_𝑘𝑥_𝑘^𝑠𝜃𝜀₀, (3.33) from which it follows that 𝛿𝜃𝛼>^𝑘𝑥_𝑘^𝑠𝜃𝜀₀>𝛿𝜃_𝑘𝑥_𝑘^𝑠𝜃−𝑔^𝑇_𝑘𝑠_𝑘>min𝛾_𝜃,𝛿𝜃_𝑘𝑥_𝑘^𝑠𝜃𝜀₀=𝛼_𝑘^min,−𝑢_𝑘(𝛼)>𝛼𝜀₀>𝛿𝜃_𝑘𝑥_𝑘^𝑠𝜃. (3.34) This means that 𝛼_𝑘,𝐿 and all previous trial step sizes are f-step sizes and larger than 𝛼_𝑘^min. Therefore, 𝛼_𝑘,𝐿 is the accepted step size 𝛼_𝑘 indeed. Since the trial point 𝑥_𝑘(𝛼_𝑘,𝐿)∉ℱ_𝑘 satisfies the switching condition (2.6) and the reduction condition (2.7), the iteration is a f-type iteration for each 𝑘>𝐾. The claim is true.

Theorem 3.14. Suppose that {𝑥_𝑘} is an infinite sequence generated by Algorithm 2.1 and |𝒜|=∞, then lim_𝑘→∞inf𝑐^𝑘_𝑆𝑘2+𝑠_𝑘=0. (3.35) Namely, it has an accumulation which is the 𝜖 solution to (1.1) or a local infeasible point. If the gradients of 𝑐_𝑖(𝑥_𝑘) are linear independent for all 𝑘 and 𝑖=1,2,…,𝑚, then the solution to (1.1) is obtained.

Proof. Suppose by contraction that there should have been a constant 𝜀₁>0 such that 𝑐^𝑘_𝑆𝑘2+𝑠_𝑘>𝜀₁, (3.36) for all 𝑘. Furthermore, there exists the following results for sufficiently large 𝑘.
As lim_𝑘→∞𝜃(𝑥_𝑘)=0, it is apparent that ‖𝑐^𝑘_𝑆𝑘2‖≤(1/2)𝜀₁ holds for large enough 𝑘. It is reasonable that we have ‖𝑠_𝑘‖≥(1/2)𝜀₁ for large enough 𝑘 from (3.36). Therefore there exists 𝐾∈ℕ such that for 𝑘>𝐾 it holds ‖𝑠_𝑘‖≥(1/2)𝜀₁ for a constant 𝜀₁>0, then by Lemma 3.13, we know that there exists 𝐾∈ℕ such that for all 𝑘>𝐾 the f-type iterate is always satisfied, which is a contraction to |𝒜|=∞.
Similar to Theorem 3.10, if the gradients of 𝑐_𝑖(𝑥_𝑘) are linear independent for all 𝑘 and 𝑖=1,2,…,𝑚, then the solution to (1.1) is also obtained. The whole proof is completed.

4. Numerical Examples

In this section, we develop the implementation of Algorithm 2.1 in order to observe its performance on some illustrative examples. In the whole process, the program is coded in MatLab with exact line search. The first example is from [2], which converges to a nonstationary point if the least squares approach is employed. The second example comes from [24], which Newton method fails to solve, and the final problem is given from [25]. In the tables, the notations NIT, NOF, and NOG mean the number of iterates, number of functions and number of gradients, respectively.

Example 4.1. Find a solution of the nonlinear equations system: 𝑥10𝑥𝑥+0.1+2𝑦²=00. (4.1)
The unique solution is (𝑥^∗,𝑦^∗)=(0,0). It has been proved in [2] that, under initial point (𝑥₀,𝑦₀)=(3,1), the iterates converge to the point 𝑧=(1.8016,0.0000), which is not a stationary point. Utilizing our algorithm, a sequence of points converging to (𝑥^∗,𝑦^∗) is obtained. We assume the error tolerance 𝜖 in this paper is always 1.0𝑒−5. The detailed numerical results for Example 4.1 are listed in Table 1.

Table 1 

Numerical results of Example 4.1.

Example 4.2. Find a solution of the nonlinear equations system as follows: 𝑥+3𝑦²()𝑦=0𝑥−1.00. (4.2)
The only solution of Example 4.2 is (𝑥^∗,𝑦^∗)=(0,0). Define the line Γ={(1,𝑦)∶𝑦∈ℝ}. If the starting point (𝑥₀,𝑦₀)∈Γ, the Newton method [24] is confined to Γ. We choose two starting points which are belong to Γ in the experiments and then the (𝑥^∗,𝑦^∗) is obtained. The numerical results of Example 4.2 are given in Table 2.

Table 2 

Numerical results of Example 4.2.

Example 4.3. Consider the following system of nonlinear equations: 𝑓₁(𝑥)=𝑥²₁+𝑥₁𝑥₂+2𝑥²₂−𝑥₁−𝑥₂𝑓−2,₂(𝑥)=2𝑥²₁+𝑥₁𝑥₂+3𝑥²₂−𝑥₁−𝑥₂−4. (4.3)
There are three solutions of above example, (1,1)^𝑇, (−1,1)^𝑇, (1,−1)^𝑇. The numerical results of Example 4.3 are given in Table 3.

Table 3 

Numerical results of Example 4.3.

Example 4.4. Consider the system of nonlinear equations: 𝑓_𝑖(𝑥)=−(𝑁+1)+2𝑥_𝑖+^𝑁_{𝑗=1,𝑗≠𝑖}𝑥_𝑗𝑓,𝑖=1,2,…,𝑁−1,_𝑁(𝑥)=−1+^𝑁_𝑗=1𝑥_𝑗, (4.4) with the with the initial point 𝑥_𝑖⁽⁰⁾=0.5, 𝑖=1,2,…,𝑁. The solution to Example 4.4 is 𝑥^∗=(1,1,…,1)^𝑇. The numerical results of Example 4.4 are given in Table 4.
All results summarized show that our proposed algorithm is practical and effective.

Table 4 

Numerical results of Example 4.4.

Acknowledgments

The authors would like to thank the anonymous referees for the careful reading and helpful comments, which led to an improved version of the original paper. The authors gratefully acknowledge the National Natural Science Foundation of China (no. 11126060) and the Science and Technology Program of Shanghai Maritime University (no. 20120060).