A New Hybrid Conjugate Gradient Projection Method for Solving Nonlinear Monotone Equations
In this study, we propose a new modified hybrid conjugate gradient projection method with a new scale parameter for solving large-scale nonlinear monotone equations. The proposed method includes two major features: projection techniques and sufficient descent property independent of line search technique. Global convergence of the proposed method is proved under some suitable assumptions. Finally, numerical results illustrating the robustness of the suggested strategy and its comparisons are shown.
The following system of constrained nonlinear monotone equations is considered:where is a nonempty closed convex set and is continuous and monotone function. The monotone property of means that
In many contemporary domains, some problems can be turned into nonlinear monotonic equation problems, such as variational inequality problems , image restoration problems , signal reconstruction problems , financial forecasting problems [4, 5], and optimal power flow control problems in power . It is very necessary to solve nonlinear monotone equation.
At present, there are many ways to solve problem (1), such as the Newtons method , Levenberg–Marquardt method , and various variants of the method. Although iterative approaches are recognized for their simplicity and good convergence, each iteration of these methods requires a substantial amount of space to compute and store the Jacobian or comparable to the Jacobian, which is not conducive to solving large-scale nonlinear monotone equations. In order to efficiently solve problem (1) and avoid solving a linear system of equations at each iteration, the emergence of the derivative-free method is necessary [9–11].
In recent years, a number of researchers have proposed the conjugate gradient method in conjunction with the projection method  for solving large-scale monotone nonlinear equations. Sabi’u et al.  proposed the Hager–Zhang conjugate gradient method for nonlinear monotone equations using singular value analysis and proposed two adaptive parameter choices: the first by minimizing the Frobenius condition number of the search direction matrix; the other is achieved by minimizing the difference between the maximum and singular values, combined with the projection technique, using some appropriate assumptions to demonstrate that the method satisfies the global convergence, see [14, 15]. Yin et al.  proposed a hybrid three-term conjugate gradient projection method, whose search direction is close to the search direction generated by the memoryless BFGS method and possesses a descending characteristic independent of the line search technique as well as a trust region characteristic. Using the adaptive line search technique, the global convergence of the method is established under certain mild conditions, and numerical experiments demonstrate that the method inherits the beneficial properties of the three-term conjugate gradient method and the hybrid conjugate gradient method [17, 18]. Zhou et al.  proposed a novel hybrid PRPFR conjugate gradient method with sufficient descent and trust region properties. The method can be considered as a convex combination of the PRP method and the FR method, while using the hyperplane projection technique. In accordance with the acceleration step size, global convergence is attained with the help of some suitable assumptions. Experiments with numbers demonstrate that the PRPFR method is more competitive for solving nonlinear equations and image restoration problems [20, 21].
In this paper, we propose a new hybrid conjugate gradient projection method to solve (1). Interestingly, the new search direction has better theoretical properties, that is, it automatically satisfies the sufficient descent condition and the trust region property. The rest of the paper is organized as follows. In Section 2, we detail the motivation for this paper and propose a new algorithm. In Section 3, we demonstrate that the search direction has sufficient descent property and trust region property, and we obtain global convergence under a few moderate conditions. In Section 4, we solve six large-scale nonlinear equations numerically to demonstrate the effectiveness of the proposed method. In Section 5, we draw the conclusion.
2. Motivation and Algorithm
This section quickly reviews the conjugate gradient method’s general formula, followed by a new hybrid conjugate gradient approach based on adaptive line search. Finally, we combine the suggested method with the projection technique to solve unconstrained optimization problems.
Due to its ease of use and storage, the conjugate gradient approach has been effectively applied to the following unconstrained optimization problems:where is a continuously differentiable function.
Generally, these methods generate a sequence of iterates recurrently bywhere (it is the step size calculated by performing some suitably precise or imprecise line search) and is the search direction defined by
The term is a scalar known as the conjugate gradient parameter. Different choices for the conjugate gradient update lead to different conjugate gradient methods. There are some famous conjugate gradient methods such as the FR method , the PRP method , the DY method , the LS method , the HS method , the CD method , and so on. The parameters we mentioned are as follows:where and the symbol stands for the Euclidean norm.
An important class of conjugate gradient methods, hybrid conjugate gradient methods, has been proposed by many scholars, in order to obtain a conjugate gradient method with better performance than the classical one. For example, Djordjević  combined the LS method and FR method by convex combination form to get conjugate gradient parameters:using the good practical performance of the LS method and the strong convergence of the FR method.
Soon after, Zhou et al.  gave a variant of the PRP method and FR method:where . Similarly, define a new parameter
This technique is a convex mixture of two types of conjugate gradient algorithms, and the search direction very easily satisfied sufficient descent, besides conforming to Newton direction under appropriate conditions. The global convergence of the proposed method can be established when a strong Wolfe line search is performed. Shockingly, this hybrid method not only performs better than the classical conjugate gradient method but also outperforms some complex conjugate gradient methods in many problems.
Inspired by above, a new convex combination is proposed as follows:where , and the choice of the parameter satisfies the conjugate condition in each iteration: .
Multiplying from the right both sides of the transposed equation by , we getand with some mathematical calculation, we obtain
If is outside the interval [0, 1], to maintain the convex combination in (11), it can be fixed by
For line search, the key is to obtain the step size at the lowest cost. Therefore, the most widely used line search proposed by Solodov and Svaiter  was used, for which the step size is computed as such that
Another commonly used technique is the line search technique proposed by Zhang and Zhou , and the step size satisfied the following inequality:
Recently, an adaptive line search approach that takes into consideration a disturbance component was presented by Liu et al.  and Guo and Wan , i.e., the step size satisfied the following inequality:where and . If , then ; otherwise, if , then .
To establish the algorithm, we introduce the definition of projection operator. Let be a nonempty closed convex set of ; then,
Also, it satisfies the nonexpansive property
We now go into great depth about our algorithm, based on the abovementioned preliminaries.
3. Convergence Analysis
In order to show the global convergence of Algorithm 1, the following assumptions need to be established: (H 3.1) The solution set of problem (1) is nonempty. (H 3.2) The mapping is Lipschitz continuous on , i.e., there exists a constant such that
The following lemma demonstrates that the direction of the search satisfies sufficient descent condition and possesses trust region property independent on any line search.
Lemma 1. For all , if the search direction is generated by Algorithm 1, then the following two properties can be satisfied:
Proof. For , it is easy to know that and (23) and (24) hold.
For , it can be broken down into three cases for discussion:(i)Firstly, if , then . If , then , and if , then . It follows that . Multiplying both sides of (5) by , from (11) and (12), we get According to (5), it is easy to obtain that(ii)Secondly, if , then . As in the case of the first case, we get(iii)Finally, for there exist , in which , and we can simply rewrite formula (5): Then, we can get the relationship among , and : From (26) and (28), we have Obviously, the descending property can be obtained by combining the step relationship (30) with (25) and (27), and we getThe proof is completed.
The following lemma shows that Algorithm 1 can get a step size and stop within finite steps, which shows that this line search is reasonable.
Lemma 2. If assumptions (H3.1) and (H3.2) hold, then Algorithm 1 stops with a finite number of iterations.
Proof. Suppose that the assumptions are invalid or that Algorithm 1 is not terminated. In this case, there exists a constant such thatSuppose there exists , so that line search (19) does not hold. Then, for all , let , and we havewhere . It can be obtained by assumption (H3.2) and formula (23) thatUsing (24), we getThis is contradictory to the definition of . Thus, line search (19) can attain a positive step size in finite steps and we complete the proof.
The following outcomes are required in order to establish that Algorithm 1 has achieved global convergence.
Proof. First, according to the definition of and line search, we haveFrom (2) and , the following relation holds:Then, combining the relation above with equations in Algorithm 1 and (21), we come to the conclusion that(40) shows the relation , which means that the sequence is decreasing and bounded. Thus, the sequence is convergent, and the sequence is bounded. Combined with assumption (H3.2), we haveTherefore, has an upper bound. Through Lemma 1 and the continuity of , it is obtained that is bounded; furthermore, sequence is bounded. This shows that there is a constant such thatLet , where .
This together with (40) impliesSince , the above formula implies that .
On the basis of the lemma above, we show that the algorithm is globally convergent under certain conditions.
Theorem 1. If assumptions (H3.1) and (H3.2) hold, let sequence and be generated by Algorithm 1. Then, we have
Proof. It can be proved by contradiction. Assuming that formula (44) does not hold, then there is a constant such that . According to , where , and formula (23), we getMoreover,Combining the above formula with Lemma 3, it is easy to show thatWe know that the sequences and are bounded by Lemma 3 and formula (24), so there is a cluster point and an infinite index set such that , and there is a clustering point and an infinite index set such that . There exists a in line search (19) such thatTaking the limit of both sides of the above equation, for all , it turns out thatTaking the limit of both sides of (19), we getObviously, the above two formulas are contradictory. Therefore, the assumption does not hold, that is, .
4. Numerical Results
In this section, in order to ensure the effectiveness of the proposed algorithm (Algorithm 1), specific numerical experiments are given below. We denote the proposed Algorithm 1 by WF, and compare with some existing algorithms. The conjugate gradient method in [35, 36, 37, 38] are denoted by JKL, EMDY, PDY, and HG, respectively. These are derivative-free methods, where JKL and EMDY methods are the state-of-the-art methods. Among them, the JKL method, PDY method, and HG method use the line search of formula (17), and the EMDY method uses the line search of formula (18). All codes used were written in Matlab R2015a and run on PC with 4 GB of RAM and Windows 10 operating system.
For different methods, we choose the optimal parameters: WF: . JKL: . EMDY: . PDY: . HG: .
We choose five methods whose termination condition is or .
The test questions are as follows: Problem 1 : set , for and . Problem 2 : set nonsmooth function , for and . Problem 3 : set logarithmic function for and . Problem 4 : set Problem 5 : set where and . Problem 6 : set , for and .
Problems 1–6 use six initial points: . In addition, choose , , and 30000 as the dimension of the problem.
In addition, we compare the efficacy of the five different approaches by utilizing the performance characteristics that Dolan and Moré  have provided. Assuming we have solvers and test problems, compare the performance of solving on the problem with the best performance of any solver on the problem. This method is used to compare and measure how well solver set works on test set . The comparison between different solvers is based on
In order to obtain the performance curve of each solver , is defined as the probability of solver , that is, the performance ratio of is within the factor of the best possible ratio. Its performance is superior to that of the following distribution functions:where size is the number of items in set and . In the event that the solver is unable to find a solution to the problem , we will adjust the ratio to a value that is sufficiently high.
Tables 1–6 display all numerical findings derived from the five approaches. In the tables, “Dim” represents the dimension of the problem, “Initial” represents the initial point, “—” indicates that the method failed to converge inside the iteration termination condition, “NI” refers to the number of iterations, “NF” refers to the number of function evaluation, “CPU” denotes the CPU time, and “” refers to the final value of when the program is stopped. In problem 3, the MEDY method outperforms the WF method at some initial points; however, in problem 6, the MEDY method cannot find the corresponding solution for initial point . In problem 4, the PDY method is superior to the WF method in the number of iterations. It can also be observed that in most of the remaining problems, the FW method has better performance than other methods in terms of the number of iterations, the number of function evaluation and CPU time.
As shown in Figures 1, compared with JKL, EMDY, HG, and PDY methods, the proposed method achieves about 70%, 55%, and 43% wins in terms of the number of iterations, function evaluation, and CPU time, respectively. These demonstrate that when compared to the JKL and MEDY methods, the FW method is more robust in terms of the number of iterations, the number of function evaluation, and CPU time.
In this paper, we suggested a way to solve nonlinear monotone equations without using derivatives. This method was a combination of the hybrid conjugate gradient method and the hyperplane projection method. An adaptive technique was used to get the search direction . We showed that the proposed method had global convergence under appropriate conditions. This technique worked well for solving large-scale monotone equations since it required a very small amount of memory. Preliminary numerical findings demonstrated the viability of our strategy. In the future, more experiments will be conducted to prove the performance of the proposed algorithm and we will try to extend the algorithm to practical large-scale nonlinear problems.
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
This study was supported by the Key Program of University Natural Science Research Fund of Anhui Province (no. KJ2021A0451) and Anhui Provincial Natural Science Foundation (no. 2008085MA01).
Y. B. Zhao and D. Li, “Monotonicity of fixed point and normal mappings associated with variational inequality and its application,” SIAM Journal on Optimization, vol. 11, no. 4, pp. 962–973, 2001.View at: Google Scholar
B. Zhang, Z. Zhu, and S. Li, “A modified spectral conjugate gradient projection algorithm for total variation image restoration,” Applied Mathematics Letters, vol. 27, pp. 26–35, 2014.View at: Google Scholar
A. J. Wood, B. F. Wollenberg, and G. B. Shebl, Power Generation, Operation, and Control, John Wiley & Sons, Hoboken, NJ, USA, 2013.
M. V. Solodov and B. F. Svaiter, “A globally convergent inexact Newton method for systems of monotone equations,” Reformulation: Nonsmooth, Piecewise Smooth, Semismooth and Smoothing Methods, Springer, Boston, MA, USA, pp. 355–369, 1998.View at: Google Scholar
J. Sabi’u, A. Shah, M. Y. Waziri, and K. Ahmed, “Modified Hager-Zhang conjugate gradient methods via singular value analysis for solving monotone nonlinear equations with convex constraint,” International Journal of Computational Methods, vol. 18, no. 4, Article ID 2050043, 2021.View at: Publisher Site | Google Scholar
M. R. Hestenes and E. Steifel, “Method of conjugate gradients for solving linear equations,” Journal of Research of the National Bureau of Standards, vol. 49, pp. 409–436, 1952.View at: Google Scholar
R. Fletcher, Practical Methods of Optimization Vol 1: Unconstrained Optimization, Wiley and Sons, New York, NY, USA, 1987.
F. N. Al-Namat and G. M. Al-Naemi, “Global convergence property with inexact line search for a new hybrid conjugate gradient method,” Open Access Library Journal, vol. 7, Article ID e6048, 2020.View at: Google Scholar
A. Kambheera, A. H. Ibrahim, A. B. Muhammad, A. B. Abubakar, and B. A. Hassan, “Modified Dai-Yuan conjugate gradient method with sufficient descent property for nonlinear equations,” Thai Journal of Mathematics, pp. 145–167, 2022.View at: Google Scholar