Abstract
Taking a new choice of the LM parameter with , we give a new modified Levenberg–Marquardt method. Under the error bound condition which is weaker than the nonsingular of Jacobian , we present that the new Levenberg–Marquardt method has at least superlinear convergence when and quadratic convergence when , respectively, which indicates that our new Levenberg–Marquardt method is performed for the systems of nonlinear equations. Also, numerical experiments indicate its efficiency on a set of systems of nonlinear equations.
1. Introduction
Nonlinear systems of equations are of interest to engineers, physicists, mathematicians, and other scientists because the modulizations of many nonlinear problems arising in different fields of science are made using a system of nonlinear equations. Consider the following systems of nonlinear equationswhere is a continuously differentiable function, and is a vector. We denote by the set of solutions for (1), which is assumed to be nonempty. Let be the Euclidean norm, here we use 2-norm, both for vectors and matrices.
There are many classical methods for solving the systems of nonlinear equations (1), including, to name a few of them, the Gauss–Newton method, the inexact Newton method, Broyden’s method, and the Levenberg–Marquardt method [1]. In this paper, we are mainly concerned with the Levenberg–Marquardt method (hereinafter referred to as the LM method) which was proposed by Levenberg [2] and Marquardt [3]. At every iteration, the trial step of the LM method is obtained by computing equations as follows:where , is the Jacobian at , denotes the identity matrix, and is the LM parameter. If is nonsingular and Lipschitz continuous for the case , the LM method is shown to have quadratic convergence provided that the initial point is sufficiently close to the solution of (1) and the LM parameter is updated by an adaptive rule.
As we all know, the LM method is widely used to solve nonlinear least-squares problems besides solving systems of nonlinear equations. Taking the LM parameter with , Behling et al. [4] propose a modified LM (BMLM) method for solving the nonzero-residue nonlinear least-squares problems. And numerical experiments show that the BMLM method is efficient. But it might be low efficient for systems of nonlinear equations.
Example 1. The Powell singular function is given byLet the initial point is suggested by Moré et al. [5]. Set the stopping criterion as . We test it by using the LM method proposed by Behling et al. in [4] and choose three choices of . The results are listed in Table 1.
From Table 1, we note that the BMLM method is not perfect for the Powell singular function because there are more iteration steps, especially . So, the BMLM method is not always performed for the systems of nonlinear equations.
From equation (2), we observe that the LM parameter helps to overcome problematic cases where is singular or nearly singular. Increasing decreases step size, and vice versa. Hence, if a step is unacceptable, should be increased until a smaller, acceptable step is found. If a step is accepted, we want to increase step size by decreasing , in order to proceed more quickly in the correct descent direction, speeding up convergence rate. Thus, the LM parameter plays a key role in the performance of the LM method. By choosing a suitable parameter , the LM method acts like the gradient descent method whenever the current iteration is far from a solution and behaves similar to the Gauss–Newton method if the current iteration is close to . For this reason, several choices of the LM parameter have been given, for instance, [6], with [7], [8], [9], with [10], and with , [11], to name only a few. For more choices, readers can refer to [12–16].
For some applications, the condition of the nonsingularity of Jacobian is too strong, and some efforts have been made by using weaker conditions. For instance, a local error bound condition is proposed by Yamashita and Fukushima [6] which is weaker than nonsingularity, and the Hölderian local error bound condition is given by Vui [17] and Ngai [18] which is a generalization of the local error bound condition. Under these conditions, the LM method converges at least linearly and solves many nonlinear problems, such as the nonlinear least-squares problems [4], the systems of nonlinear equations [19–21], the linear complementarity problem [22], the system identification [23, 24], and other problems [25–28].
In [4], to establish the local convergence, Behling et al. propose an error-bound condition upon the gradient of the nonlinear least-squares function as follows:where is a positive constant, with is a neighbourhood of , and is the distance from to the set . And the LM method is proved to have at least linearly convergence order. The error-bound condition (4) does not require the full rank of the Jacobian in a neighborhood of a stationary point. It is also weaker than the nonsingularity of Jacobian, and it can be derived from the local error bound condition ([13], Lemma 5.4).
In this paper, to solve the systems of nonlinear equations (1), we propose a new modified LM method by extending the LM parameter given by Behling et al. aswhere , that is, we extend the exponent of their LM parameter from to . Using the trust region technique, under the error bound condition (4) which is the same as the condition used by Behling et al., the convergence of our new LM method will also be investigated. Since the convergence criterion of the LM method is , we always make sure that is sufficiently small. It means that the LM method will be the Gauss–Newton method when is sufficiently large.
The outline of this paper is organized as follows. In the next section, we present the proposed LM algorithm and analyze the local convergence under the error-bound condition (4). The global convergence is discussed in Section 3. Numerical experiments are tested using the proposed LM algorithm in Section 4. Finally, we draw a conclusion in Section 5.
2. The Levenberg–Marquardt Method and Its Local Convergence
In this section, to solve the system of the nonlinear equations (1), we consider the LM method with the LM parameter as (5). By using trust region technology, we investigate its convergence rate near a solution.
Takeas the merit function for (1). The actual reduction and predicted reduction of at the -th iteration can be defined asrespectively, where is obtained by (2). It follows from the result given by Powell [29] that, for all ,
By trust region technique, to make sure whether accept the step and how to update the LM parameter , we need the following ratio:
Firstly, we give some assumptions which will be needed throughout this paper.
Assumption 1. (a) is Lipschitz continuous on , i.e., there exists a positive constant that makes(b) provides a local error bound (w.r.t. ) on , i.e., there exists a positive constant such that From Assumption 1 and the mean value inequality, we derive the following lemma without proof.
Lemma 1. Let Assumption 1 holds, then we getwhere constants , and .
Denote by which satisfies
Lemma 2. Let the sequence is generated by Algorithm 1 and Assumption 1 holds. If , there exists a positive constant that
Proof. From (15), we havewhich indicates that . From (5), (11), and Assumption 1(b), we haveDefineIt follows from (2) that is a stationary point of , and also, is a minimizer of because of the convexity of . Thus, from (12) and (18), we haveIt means thatwhere .
Lemma 3. Under Assumption 1, for all sufficiently large ,(a)There exists a constant such that(b)There exists a constant such that
Proof. We first prove (a). It follows from Assumption 1, Lemma 2, (8), and (12) thatThis, together with (9), (12), (13), and , givesTherefore, . Hence, we have that there exists a constant such that holds for all large in view of Algorithm 1.
Now, we prove (b). It follows from (14) and (13) thatSince and , we obtain (b) with .
Theorem 1. Let the sequence is generated by Algorithm 1 and Assumption 1 holds. Then, the sequence converges to a solution of (1) with convergence rate .
Proof. It follows from Lemma 2 and (15) thatFrom (12) and (14), we obtainAlsowhich yieldsFor every , together with (27), we getSince and , together with Assumption 1(b), Lemma 3(b), (2), (27), (30), and (31), we obtainwhere . This implies that converges to solution set with convergence rate .
It is clear thatIt follows from (32) and Lemma 2 thatholds for every sufficiently large . Therefore, deduce from (32), we havewhich indicates the sequence converges to a solution of (1) with convergence rate .
Remark 1. Theorem 1 shows that our new Levenberg–Marquardt algorithm converges to a solution of (1) superlinearly when , while quadratically when . Thus, Algorithm 1 has at least superlinearly convergence rate, which is consistent with the conclusion obtained in [8, 9, 11].
3. Global Convergence
We study the global convergence of Algorithm 1 in this section.
Theorem 2. Let Assumption 1(a) holds. Then, Algorithm 1 terminates in finite iterations or satisfies
Proof. We show it by contradiction. Assume the result is not true, then there exists a positive constant such thatDefine the index set of successful iterations asNow, we deduce the contradiction in two cases.
Case 1. is infinite. From (8), (14), and (37), since is nonincreasing and bounded below, we haveHence,By assumption, we know that for all . Thus,It follows from (2), (15) and is nonincreasing thatMoreover, from (8) and (13), we haveHence, . From Algorithm 1, we have that there exists a constant such thatholds for all large . This contradicts to (40).
Case 2. is finite. From the definition of , together with Algorithm 1, there exists a such thatAccording to the update rule of that . We also have by the same discussion as (43), which contradicts the above inequality.
4. Numerical Experiments
The effectiveness of Algorithm 1 is verified by some numerical experiments in this section. The first experiment is Powell singular function which is computed in Section 1, and the second experiment is a nonzero-residue NLS problem which is computed by Behling et al. in ([4], Example 5.1), while the finals are some systems of nonlinear equation proposed by Moré et al. in [5]. The LM method (BMLM) proposed by Behling et al. in [4] is computed for comparison on Powell singular function and nonzero-residueleast-squares problem.
Example 2. Recompute Example 1 by using Algorithm 1 with the same conditions of Example 1. Since the range of the exponent of has been extended from to , we also give the results about the exponent more than . The results for and are listed in Tables 2 and 3, respectively.
As can be shown from Tables 1–3, the performance of Algorithm 1 is better than that of BMLM because of fewer iterations and improved accuracy.
Example 3. Consider the nonzero-residue NLS problem [4]where .
The minimizer set consists of points verifying so that . The Jacobian is not full rank since the rank of the Jacobian will be one except at the origin while being zero at the origin. Set be the starting point and be the termination criterion. The results are shown in Table 4.
As illustrated in Table 4, Algorithm 1 achieves the required accuracy, not only faster than BMLM but also more accurate than BMLM. Besides, the points generated by Algorithm 1 are closer to the solution set than those of BMLM.
Example 4. Let us consider systems of nonlinear equations created by modifying the singular problems proposed by Moré et al. [5] and have the same form as in [30].where is its root, function is the standard nonsingular function, and matrix has full column rank with . It is easy to check that and have .
We chose the rank of to be by usingand by usingWe set the parameters as , , , , , and in Algorithm 1. The algorithm is terminated when or the number of iterates exceeds . To consider the global convergence of Algorithm 1, we test three starting points , , and , where is suggested by Moré et al. in [5]. The notations listed in tables have the following meanings:(i)NF: the function calculations amount(ii)NJ: the Jacobian calculations amount(iii)NJ NF: the total calculations since the Jacobian calculations are usually times of the function calculations(iv)“–”: the algorithm failed to find the solution in iterationsIn accordance with the range of defined in (5), we take five choices, such as , , , and , associated with for Algorithm 1. The results for the first set of problems of rank with low dimensional and high dimensional are listed in Tables 5 and 6, respectively, and the second set of rank with low dimensional and high dimensional in Tables 7 and 8, respectively.
The results in Tables 5–8 show that Algorithm 1 is effective for systems of nonlinear equations. Algorithm 1 is perfect when the initial point is close enough to the solution just as Theorem 1 requires. However, when the initial points are and , which mean that the initial points are not close to solution , Algorithm 1 is not perfect because the number of iteration increases rapidly, especially the exponent of the LM parameter is more than . Also, we can see that Algorithm 1 saves the Jacobian calculations to find a solution to some nonlinear equations since NJ is less than NF as well as the total calculations decreased.
5. Conclusion
For systems of nonlinear (1), we discuss the Levenberg–Marquardt method with a modified LM parameter which extended from the LM method proposed by Behling et al. Under an error-bound condition, the local and global convergences have been analyzed by using the trust region technique. The convergence theory shows that our new LM method achieves a solution of nonlinear equations superlinearly when and quadratically when . Numerical experiments indicate that the new LM method is outperformed than BMLM on nonlinear least-squares problems, and the new LM method is efficient on a set of systems of nonlinear equations.
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This research was supported by the NFSC Grant 11901061, the Natural Science Foundation of Anhui Province Grant 2108085MF204, and the Natural Science Foundation of the Anhui Higher Education Institutions Grant KJ2020ZD008. Part of this work was completed during the authors’ visit to the University of Texas at Arlington. They were very grateful to Prof. Ren-Cang Li for his hospitality during their visit.