Abstract

This paper considers the classical separable nonlinear least squares problem. Such problems can be expressed as a linear combination of nonlinear functions, and both linear and nonlinear parameters are to be estimated. Among the existing results, ill-conditioned problems are less often considered. Hence, this paper focuses on an algorithm for ill-conditioned problems. In the proposed linear parameter estimation process, the sensitivity of the model to disturbance is reduced using Tikhonov regularisation. The Levenberg–Marquardt algorithm is used to estimate the nonlinear parameters. The Jacobian matrix required by LM is calculated by the Golub and Pereyra, Kaufman, and Ruano methods. Combining the nonlinear and linear parameter estimation methods, three estimation models are obtained and the feasibility and stability of the model estimation are demonstrated. The model is validated by simulation data and real data. The experimental results also illustrate the feasibility and stability of the model.

1. Introduction

The separable nonlinear least squares problem is a special type of nonlinear least squares problem. It was proposed by Golub and Pereyra for estimating the parameters of the formula for atomic physical ion half-life [1]. In the separable nonlinear least squares model, the estimated model can be expressed as a linear combination of nonlinear problems.

To utilise the special structure of the separable nonlinear least squares problem, in 1973, Golub and Pereyra proposed the variable projection (VP) algorithm [1], which eliminates linear parameters and simplifies the problem into a set of nonlinear parameter estimation problems. This method also reduces the computational complexity. Compared with the corresponding algorithm for a direct solution, the VP algorithm has fewer iteration steps and less initial point guessing. Moreover, when the original problem is ill posed, the degree of ill-posedness can be reduced [25]. As research has progressed, the mathematical models of many problems in actual engineering can be expressed as separable nonlinear least squares, for instance, in inverse problems and problems in signal processing, medical and biological imaging, neural networks, communication, electrical and electronic engineering, and differential equation dynamic systems [615].

In 1980, Ruhe and Wedin analysed the separation and nonseparation of parameters and concluded that the parameter separation approach was simpler and more effective [16]. In 1990, Shen and Ypma used the variable projection method to transform a separable nonlinear least squares problem into a case containing only nonlinear parameters, which improved the computational efficiency of the function [17]. In 2003, Golub and Pereyra summarised the development and application of separable nonlinear least squares over the past 30 years [18]. In recent years, many studies and application of the separable nonlinear least squares problem have obtained various results. Chen studied the nonlinear four-parameter sine wave model and calculated it using the variable projection method. Experiments show that the convergence speed has a strong relationship with the frequency parameter [19]. Chung and Nagy applied separable nonlinear least squares to large-scale ill-posed problems and improved the distortion problem in image processing [20].

There are a large number of research results for separable nonlinear least squares, but there are relatively few studies on potentially ill-conditioned problems with parameters. When a singular value becomes very small, the least squares model may be ill-conditioned. Regularisation is a commonly used method to solve ill-conditioned problems that causes regression coefficients to have smaller variance values, thus solving potentially ill-posed problems [2128]. The Tikhonov regularisation (TR) method [2123], truncated singular value method [24, 25], kernel function-based regularisation method [26, 27], and norm regularisation method [28] are often used to solve ill-posed problems. When estimating nonlinear parameters, iterative search methods such as the Gauss–Newton method, steepest gradient method, and LM method are commonly used [2934]. In these methods, the Jacobian matrix has a strong influence on the computational efficiency of the algorithm.

This paper proposes a hybrid VP-based algorithm that combines the LM and TR methods and evaluates the effect of different Jacobian matrix algorithms on its accuracy and efficiency. The TR method is used to regularise the linear parameter estimation, and the LM algorithm is used to estimate the nonlinear parameters in a separable nonlinear least squares problem. The Jacobian matrix is calculated using three methods: Golub and Pereyra (GP), Kaufman (KAU), and Ruano (RJF). The algorithm proposed in this paper combines the advantages of the LM algorithm with those of a regularisation method and can improve the estimation efficiency. The model was validated using two examples: exponential fitting and the determination of waveform parameters for airborne radar sounding data.

2. VP Model and Its Parameter Estimation Method

2.1. VP Model

The separable nonlinear least squares model can be expressed in the following form:where are nonlinear functions; , are observation data; is the relevant variable of ; and and are the linear and nonlinear parameters, respectively, to be estimated.

The optimal parameters and can be obtained by the following nonlinear function:

The above formula can be expressed using matrices as follows:where the column vector of matrix is a nonlinear function , the element of vector is , and is the Euclidean norm. For given nonlinear parameters , the linear parameters can be obtained by solving the following nonlinear least squares problem:where is the pseudoinverse of . If is computable, then (4) can be solved. Hence, the premise of (4) is that matrix is nonsingular, but this premise is not true in some cases.

2.2. Determination of the Linear Parameters

Without loss of generality, the power system can be expressed aswhere , , and .

We can then perform singular value decomposition (SVD) on , which can be expressed aswhere and are the unitary matrices composed of column vectors. , , . In addition, is the singular value of . Using SVD to solve (5), we obtain

To reduce the sensitivity of the SVD method to disturbances, a filter factor can be added. The SVD form with the added filter factor is

Different filtering factors will result in different regularisation methods, and the TR method is a widely applied method [35, 36]. The filtering factor obtained by the TR method can be expressed as , and (8) can be expressed as

2.3. Determination of the Nonlinear Parameters

For matrix , the Jacobian matrix is the first derivative of :

Assume

The matrix can be expressed as

The Jacobian matrix can be calculated in the following three ways:(1)The calculation method given by Golub and Pereyra is as follows:where is the Fréchet derivative of the map. In addition, is the symmetric generalised inverse of that satisfies and (2)Kaufman proposed the following simplified calculation method for the Jacobian matrix after studying (13):(3)Ruano proposed an even simpler calculation method for the Jacobian matrix based on Kaufman’s method:

Ruano et al. [37] proved that (15) is valid and can be obtained from Kaufman’s Jacobian matrix (14). It can be proved that the same gradient vector can be obtained by the above three Jacobian matrices [38].

The following equation can be used to iterate nonlinear parameters:where is a scalar step size that ensures that the objective function is decreasing and is the search direction. In the LM algorithm, can be determined by the following equation [39]:where is the smallest nonnegative integer satisfying (17), , , and .

can be determined by the following equation:where is the damping factor and affects both the gradient and . If tends to infinity, tends to the fastest direction. If tends to zero, tends to the Gauss–Newton direction. There are many methods of selecting damping factor [40]. After obtaining , the search direction can be obtained by solving (17). Step parameter can be obtained by a linear search algorithm with mixed quadratic interpolation and cubic interpolation [4143].

Combining the above methods for the linear and nonlinear parameters, three algorithms can be obtained. The LM algorithm is used for the nonlinear parameters, and the three methods of GP, KAU, and RJF are used to calculate the Jacobian matrix in this algorithm. For the linear parameters, the TR method is used. The formula is as follows:

Moreover, the specific algorithm is as follows:Step 1: give the initial value of the nonlinear parameter, and predefine the maximum number of iteration steps and error Step 2: calculate the initial value of the linear parameter using the TR method of (5) and (9)Step 3: alternately update the nonlinear parameters and linear parametersStep 4: repeat the third step until either the maximum number of iteration steps is reached or

3. Numerical Experiments

The experimental environment used in this study was MATLAB 2016b, running on a PC equipped with a 2.30 GHz CPU, 4 GB of memory, and a Windows 10 operating system.

3.1. Index-Fitting Model

Two experiments were used to evaluate the three algorithms proposed in this paper. The first of these experiments was derived from the observation of the decay of radioactive materials. The model describes the sum of the exponential terms of two unknown attenuation factors and is written as follows:where and are the linear and nonlinear unknown parameters to be estimated, respectively. Moreover, there are 21 and corresponding observations , . We minimise the following functions according to the principle of least squares:

According to the composition of the function matrix in the second part, the nonlinear function part can be expressed as a matrix form as . After SVD, for , the estimated values of the linear parameters are obtained from (9). For the nonlinear parameters, the objective function is first constructed by (12), the Jacobian matrix is then obtained by the three different methods, and the LM algorithm is substituted into (16) to determine the parameters. When iterating, we set the initial value to , the maximum number of iteration steps to 100, and . Performing SVD on yields singular values of 1.8480 and 0.3586.

Figure 1 shows the observations and their fitting curves.

The curves of the observation value fit the results of the three methods very well. The curves obtained by the three methods are basically the same, demonstrating that all three methods are feasible and the results are reliable.

Figure 2 shows the change in parameters and with respect to the number of iterations.

As Figure 2 shows, for , the change in parameter curves for the two methods VPGP+TR and VPRJF+TR are basically the same, and the specified accuracy can be achieved after nine iterations. Visually, it is clear that after five iterations, the solution is very close to the final result. In contrast, the VPKAU+TR method reached the maximum number of iterations (100). However, after 30 iterations, even though the result changes a little, it is close to the final result.

For , the change in parameter curves for the two methods VPGP+TR and VPRJF+TR are also basically the same, and the specified accuracy is also achieved after nine iterations. The graph shows that, after four iterations, the solution is very close to the final calculation result. The VPKAU+TR method again reached the maximum number of calculations, but the solution is close to the final result after 20 iterations.

Table 1 lists the results of the parameters obtained by the three methods.

The table shows that, as for the curves in Figure 1, the results obtained by the three methods are basically the same.

Table 2 gives the runtime (in seconds), number of iterations, residual sum, and root mean square error (RMSE) for the three methods.

The quantitative indicators (residual sums and RMSE) are equal and very small, indicating that the three algorithms achieve the same calculation accuracy, which is relatively high. The difference between them is only the number of iterations and runtime. The VPGP+TR method takes the longest to calculate because this method uses the most complicated way to calculate the Jacobian matrix. In contrast, VPRJF+TR is a simple method for calculating the Jacobian matrix. However, to meet the accuracy requirements, the number of iterations is also large, so the calculation time is also long. The VPKAU+TR method somewhat simplifies the calculation of the Jacobian matrix, and to some extent, the accuracy of the calculation is guaranteed, so the runtime is the shortest. For this simulation experiment, the VPKAU+TR method performs the best of all methods.

3.2. Waveform Parameter Calculation for Airborne Laser Radar Sounding Data

The second experiment used laser radar sounding waveform data. The experimental data were flight experimental data obtained from a flight over the sea near Sanya City, Hainan Province, China.

The waveform sample data are stored as an array, and the amplitude of each sample point is recorded separately. The sample information is presented as a ‘waveform’ form on a two-dimensional plane. The onboard laser system records discrete echoes that simulate the “waveform” of the raw sampled data using the most appropriate functional model. To improve the sounding accuracy, it is highly desirable to choose a high-precision theoretical model.

This experiment has a total of 320 waveform data, and the function model is as follows:

Selecting reasonable data can effectively reduce the influence of noise. In this study, we selected some of the data that have no gross errors. We extracted 42 data from the waveform data to determine the parameters. When iterating, we set the initial value to , the maximum number of iteration steps to 100, and the error to . SVD for yielded singular values of 6.5941 and 1.9585.

Table 3 lists the calculation results of the model parameters.

The results given in the table show that the results obtained by the three methods are basically the same, indicating that all three methods are feasible approaches. The result obtained by VPRJF+TR is slightly different from those of the other two methods because this method simplifies the calculation of the Jacobian matrix the most, resulting in a gap.

Figure 3 shows the fitting curves of the model.

In Figure 3, all three methods fit the curve well, which also intuitively illustrates the feasibility of these three methods.

Figure 4 shows the change in parameters and with respect to the number of iterations.

The results in Figure 4 show that for parameters and , when error , the parameter change is basically stable after five iterations. This shows that the convergence of these three methods is fast and accurate.

Table 4 lists the runtime (in seconds), number of iterations, and root mean square error (RMSE).

For , number of iterations and RMSE (i.e., the convergence speed and accuracy) of these three methods are basically the same. With respect to runtime, the VPGP+TR method again takes the longest because of its calculation of the Jacobian matrix. Hence, the same number of iterations takes longer. Similarly, the Jacobian matrix of the VPRJF+TR method is the simplest and requires the shortest time to calculate. For the VPKAU+TR method, the computational complexity is between those of the other two methods, and it takes moderate amount of time to calculate.

4. Conclusion

For the separable nonlinear least squares problem, this paper establishes a solution model consisting of an LM algorithm combined with TR. For linear parameter estimation, the sensitivity of the model to disturbance is reduced by the TR method. For the nonlinear parameter estimation, the LM algorithm is adopted and the Jacobian matrix is calculated by the Golub and Pereyra, Kaufman, and Ruano methods. Combining nonlinear and linear parameter estimation methods, three estimation models are obtained and the stability and accuracy of model estimation are demonstrated. The model was verified by simulation data in the experiment in Section 3.1 and real data in the experiment in Section 3.2. The experimental results also illustrate the feasibility of the model.

In future research, we will apply the Jacobian matrix decomposition to the VP algorithm. This decomposition can simplify the model and improve the efficiency of the algorithm.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

This research was supported by the National Natural Science Foundation of China (Grant no. 41774002) and the scientific research fund project for the introduction of talents of Shandong University of Science and Technology (Grant no. 2015RCJJ022).