Flexible Least Squares Algorithm for Switching Models

Ni, Yunxia; Lv, Lixing; Ji, Yuejiang

doi:https://doi.org/10.1155/2022/2605570

Complexity

On this page

Abstract Introduction Examples Conclusions Data Availability Conflicts of Interest Acknowledgments References Copyright Related Articles

Research Article | Open Access

Volume 2022 | Article ID 2605570 | https://doi.org/10.1155/2022/2605570

Flexible Least Squares Algorithm for Switching Models

Yunxia Ni,¹Lixing Lv,¹and Yuejiang Ji¹

Academic Editor: Qingling Wang

Received10 Jul 2022

Revised29 Aug 2022

Accepted15 Sept 2022

Published26 Oct 2022

Abstract

The self-organizing model and expectation-maximization method are two traditional identification methods for switching models. They interactively update the parameters and model identities based on offline algorithms. In this paper, we propose a flexible recursive least squares algorithm which constructs the cost function based on two kinds of errors: the neighboring two-parameter estimation errors and the output estimation errors. Such an algorithm has several advantages over the two traditional identification algorithms: it (1) can estimate the parameters of all the sub-models without prior knowledge of the model identities; (2) has less computational efforts; and (3) can update the parameters with newly arrived data. The convergence properties and simulation examples are provided to illustrate the efficiency of the algorithm.

1. Introduction

Least squares (LS) algorithm is the most widely used method in parameter estimation [1–3]. It defines a cost function which is composed of the errors between the true outputs and the predicted outputs. Then, the estimations can be obtained by solving the derivative function of the cost function. The LS algorithm has fast convergence rates but with the cost of heavy computational efforts [4, 5]. In addition, the LS algorithm needs to compute the inverse of a matrix. If the matrix has a high order or is ill-conditioned, the LS algorithm is inefficient [6–9].

To reduce the computational efforts and to avoid the matrix inversion, the recursive least squares (RLS) algorithm is a good choice. The basic idea of the RLS algorithm is to update the parameter estimations using the newly arrived data, that is, the cost function of the RLS algorithm is composed of only one set of data rather than all the collected data [10–12]. Therefore, the RLS algorithm has less computational efforts, and it does not require calculating the matrix inverse. However, the RLS algorithm has slow convergence rates when compared with the LS algorithm [13, 14]. With the aim to increase the convergence rates, many modified RLS algorithms are developed, e.g., the multi-innovation RLS algorithm [15, 16] and the hierarchical RLS algorithm [17, 18].

Although the RLS algorithm and its modified counterparts can identify systems with less computational efforts and fast convergence rates, they have the assumption that the considered model is a single model. If the system is described by a switching model, those algorithms are inefficient. Switching models are widely used in engineering practices [19, 20]. Such models have several modes with different dynamical properties, and the modes are associated with various operating conditions [21, 22]. The difficulty in switching system identification is that the times of the operating points (model identities) may be unknown. To identify the switching models, one should first determine the operating points/model identities.

The self-organizing model (SOM)-based method and the expectation-maximization (EM) method are two classical identification algorithms which are usually used for switching models [23–26]. The SOM method introduces several cost functions which correspond to each sub-model in each iteration, and the smallest cost function is associated with the true model in this sampling instant [23]. The EM algorithm regards the model identities as hidden variables and updates these identities in the EM-E step; once the identity estimations are obtained, the parameter estimations are computed in the EM-M step. These two steps run interactively until these two kinds of estimations converge to their true values [27, 28]. The SOM method should compute several cost functions in each iteration, and the EM algorithm needs to compute the model identities first. In addition, both these algorithms are offline algorithms. That is, they have heavy computational efforts and cannot update the parameters based on newly arrived data.

The flexible least squares (FLS) algorithm, first developed by Kalaba and Tesfatsion [29], is used for time-varying system identification. Its cost function contains two parts: one is the error of the two parameters in the two neighboring instants, and the other is the error between the true outputs and predicted outputs [30–32]. Due to the first error, the parameter estimations can catch the varying parameters. Inspired by the FLS algorithm, we develop a novel FLS algorithm for switching models. This algorithm is termed as flexible recursive least squares (FRLS) algorithm. Compared with the SOM, EM, and FLS algorithms, this algorithm has the following advantages: (1) the FRLS algorithm is an online algorithm, and thus it can update the parameters with newly arrived data; (2) the FRLS algorithm has less computational efforts; and (3) the FRLS algorithm can estimate the parameters of all the sub-models without prior knowledge of the model identities.

The remainder of the paper is organized as follows. Section 2 explains the switching model and traditional identification algorithms. Section 3 proposes the offline FLS algorithm and online FLS algorithm. Section 4 provides several simulation examples. Finally, Section 5 summarizes the paper and gives some future directions.

2. Problem Statement

Let us define some notations first: means an identity matrix of the appropriate sizes; the superscript stands for the matrix transpose; the norm of a matrix is defined as ; means the maximum eigenvalue of matrix ; and the norm of a vector is defined as .

2.1. Switching Model

Consider the following switching model:where is the output of the -th model; is the information vector of the -th model, which is composed of the input and output data before the sampling instant ; is the parameter vector of the -th model; is a Gaussian white noise and satisfies ; and is the number of the sub-models.

In the sampling instant , there is no knowledge of the identity of the model. We aim to estimate the parameter vectors based on the collected data.

Collect sets of input and output data, and define the following cost function:where is the model identity in the sampling instant . For example, in the sampling instant , the true model is the -th model, and then the true values of the identities of all the sub-models are , , . To estimate the parameters, the following assumptions are introduced.

Assumption 1. The number of the collected data is larger than the number of the unknown parameters, that is,In addition, assume that the number of data of the -th sub-model is ; then,

Assumption 2. For the switching model proposed in (1), all the input data are taken as persistent excited.

Assumption 3. All the sub-models have the same information vector but different parameter vectors, that is, the switching model can be written by

Remark 1. Assumptions 1 and 2 can ensure that the information matrices of all the sub-models are nonsingular [6]. Assumption 3 can also be easily obtained [33]. For example, for a switching model with unknown structures, we can use the kernel method to describe the model, and all the sub-models approximated by using the kernel method can have the same structure.

2.2. Traditional Identification Algorithms

Rewrite the cost function of the switching model as follows:

Assume that the parameter estimations and identity estimations in iteration are and .

Both the SOM and EM algorithms estimate the parameters through two steps:(1)Estimate the model identity estimations , based on the parameter estimations .(2)Update the parameter estimations based on the model identity estimations .

The difference between the SOM and EM algorithms is in the first step. In the SOM algorithm, the model identity estimate is 1 or 0. For example, in iteration of the sampling instant , let

Then, and the other identity estimations .

On the other hand, in the EM algorithm, let

Then, the identity estimate can be computed by

Remark 2. Both the SOM and EM algorithms are offline algorithms; if the order of the system is large, their computational efforts are heavy. In addition, they cannot update the parameters with newly arrived data [23, 34].

3. Flexible Recursive Least Squares Algorithm

The SOM and EM algorithms update the parameters through two steps, and these two steps are related to each other. If one kind of estimations has poor estimation accuracy, the other may be also poor or divergent. In this section, we use the FLS algorithm for the switching models, which can estimate the parameters without prior knowledge of the model identities.

3.1. Offline FLS Algorithm

Define

Then, the switching model can be written as

Let

Unlike the SOM and EM algorithms, the cost function of the offline FLS algorithm is written by

Using the FLS algorithm to update the parameters yields

Remark 3. From equation (5), we can get that the offline FLS (O-FLS) algorithm can estimate the parameters in only one iteration and does not require the model identify estimation. However, it increases the order of the information matrix intensively. For example, the order of the information matrix is in the O-FLS algorithm, while in the EM and SOM methods, the order is .

Remark 4. The offline FLS (O-FLS) algorithm requires a high-order matrix inverse calculation, e.g., . When the high-order matrix is singular or ill-conditioned, computing its inverse is impossible.

3.2. Flexible Recursive Least Squares Algorithm

To reduce the computational efforts and to avoid a high-order matrix inversion, this section proposes an online FLS algorithm which is termed as flexible recursive least squares (FRLS) algorithm.

Assume that the parameter vector in the sampling instant is . Define the following cost function:

In the sampling instant , all the parameter estimations before have been obtained, and thus (15) is simplified as

Taking the derivative of with respect to yields

Next, we use the recursive method to obtain the relationships between and .

(17) is transformed into

Then, the FRLS algorithm can be summarized as follows:

Remark 5. Compared with the O-FLS algorithm, the FRLS algorithm performs a low-order matrix inversion rather than a high-order matrix inversion. Therefore, the FRLS algorithm has less computational efforts than the O-FLS algorithm (Algorithm1).
Then, the steps of the FRLS algorithm are listed as follows.

(i)	Initialise, is a vector whose
(ii)	entries all equal to 1, and assign a positive constant
(iii)	repeat
(iv)	for, do
(v)	Collect the input-output data
(vi)	Form
(vii)	Compute
(viii)	Compute
(ix)	Update
(x)	end
(xi)	until convergence

In the FRLS algorithm, there exists a dense matrix inversion, which leads to heavy computational efforts. To further reduce the computational efforts, the following lemma is introduced.

Lemma 1. For the matrices , , and , if the matrix is nonsingular, the following equality holds:

Proof. For the matrix , we haveIn addition, we can obtainThen, the proof is completed.
According to Lemma 1, the matrix is simplified as

Remark 6. Based on equation (15), in each sampling instant, a dense matrix inversion is transformed into vector multiplication. Therefore, the computational efforts are reduced.

3.3. Convergence Properties of the Two Kinds of FLS Algorithms

The convergence properties of the O-FLS and FRLS algorithms are given in this section which can help the researchers follow these two algorithms.

3.3.1. Convergence Property of the O-FLS Algorithm

Theorem 1. For the switching model proposed in (1), the parameter estimations updated by the O-FLS algorithm are expressed by (5). Then, are biased.

Proof. Rewrite the O-FLS algorithm as follows:Substituting into the above equation yieldswhere is Gaussian white and independent on , and the above equation can be written bySince , the matrix is singular, and the matrix cannot be a zero matrix. Therefore, the O-FLS algorithm is a biased algorithm.

Remark 7. A small can get more accurate parameter estimations. However, a small may lead to slow convergence rates between the two neighboring sub-models. Therefore, we should assign different values for . For example, in the fixed interval, a small is better, while near the switching points, a larger one is better.

3.3.2. Convergence Property of the FRLS Algorithm

Theorem 2. For the switching model proposed in (1), the parameter estimations updated by the FRLS algorithm are expressed by (10)–(12). Then, the sequence is convergent.

Proof. The FRLS algorithm is written byAssume that the data from belong to model 1, and subtracting the true value on both sides of the above equation yieldsFor the reason thatwe haveTherefore, the FRLS algorithm is convergent.

Remark 8. The FRLS algorithm has the assumption that the identities of the data are unchanging in a fixed interval. If the identities are changing continually, the FRLS algorithm is divergent.

4. Examples

4.1. Example 1

Consider the following switching model:

Let

In simulation, we collect 500 sets of input and output data, where the data from belong to model 1, and those from belong to model 2.

Use the FRLS algorithm for this switching model. The parameter estimations are shown in Figures 1 and 2. The predicted outputs and the true outputs, and their errors are shown in Figure 3. In addition, apply the EM and SOM algorithms for the switching model, where the initial identities for each sub-model are and . The estimation errors and elapsed times of the three algorithms are shown Table 1.

From this simulation, we can get the following findings:(1)The parameter estimations using the FRLS algorithm can asymptotically converge to the true values (see Figures 1 and 2).(2)The predicted outputs using the FRLS algorithm can catch the true outputs (see Figure 3).(3)The number of the data in a fixed interval must be larger than the number of the unknown parameters.(4)All the FRLS, EM, and SOM algorithms are effective for the switching model, but the FRLS algorithm has the smallest elapsed times, that is, the FRLS algorithm has the least computational efforts among these three algorithms, as shown in Table 1.

4.2. Example 2: A Switching Open Channel System

In this section, we consider an open channel system, which is shown in Figure 4. The radius of the channel is , the length of the channel is , is the discharge at the upstream end, is the discharge at the downstream end, and the slope is . To ensure the discharge to flow in a fixed speed, we should control . The relationship between and can be expressed by a linear model. In simulation, two slopes and degrees are assigned to the open channel system. These two slopes lead to two different dynamics which should be described by two models: [14]:

We collect 2000 sets of input-output data using Matlab software, where the sequence is generated by

The data from belong to model 1, and those from belong to model 2.

Apply the FRLS algorithm for the considered switching model. The parameter estimations and their estimation errors , are shown in Figure 5 and Tables 2 and 3.

Furthermore, we use the traditional EM and SOM algorithms for the switching open channel system . The parameter estimations and their estimation errors are shown in Figures 6 and 7. The elapsed times of these three algorithms are shown Table 4.

This example shows that (1) all the FRLS, EM, and SOM algorithms are convergent, as shown in Figures 5–7; (2) the FRLS algorithm has the smallest elapsed times among these three algorithms, and this is shown in Table 4, that is, the FRLS algorithm has the least computational efforts among these three algorithms.

5. Conclusions

An online FLS algorithm, termed as flexible recursive least squares (FRLS) algorithm, is proposed for switching models in this study. Its cost function is composed of the errors between the two neighboring parameter estimations and the errors between the true outputs and the predicted outputs. With the help of the two neighboring parameter estimation errors, the operating points of the switching models can be determined, and the parameters of each sub-model can also been obtained. Compared with the SOM and EM algorithms, the FRLS algorithm can estimate the parameter estimations without prior knowledge of the model identities. In addition, the FRLS algorithm is an online algorithm, which has less computational efforts and can update the parameters with newly arrived data.

Although the FRLS algorithm has several advantages over the traditional identification algorithms, several challenging issues about the FRLS algorithm need to be considered in future. For example, if the sub-models switch continually, how to apply the FRLS algorithm to the switching model? How to choose a suitable to make the FRLS algorithm converge quickly to the true values? These topics remain as open problems.

Data Availability

All data generated or analyzed during this study are included in this article.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

This study was supported by the Natural Science Foundation of Jiangsu Province (no. BK20131109).

References

C. P. Yu, L. Ljung, A. Wills, and M. Verhaegen, “Constrained subspace method for the identification of structured state-space models (COSMOS),” IEEE Transactions on Automatic Control, vol. 65, no. 10, pp. 4201–4214, 2020.
View at: Publisher Site | Google Scholar
B. Q. Mu, E. W. Bai, W. X. Zheng, and Q. M. Zhu, “A globally consistent nonlinear least squares estimator for identification of nonlinear rational systems,” Automatica, vol. 77, pp. 322–335, 2017.
View at: Publisher Site | Google Scholar
J. Ding, F. Ding, X. P. Liu, and G. Liu, “Hierarchical least squares identification for linear SISO systems with dual-rate sampled-data,” IEEE Transactions on Automatic Control, vol. 56, no. 11, pp. 2677–2683, 2011.
View at: Publisher Site | Google Scholar
F. Ding, H. Ma, J. Pan, and E. F. Yang, “Hierarchical gradient and least squares-based iterative algorithms for input nonlinear output-error systems using the key term separation,” Journal of the Franklin Institute, vol. 358, no. 9, pp. 5113–5135, 2021.
View at: Publisher Site | Google Scholar
J. M. Li and F. Ding, “Identification methods of nonlinear systems based on the kernel functions,” Nonlinear Dynamics, vol. 104, no. 3, pp. 2537–2552, 2021.
View at: Publisher Site | Google Scholar
T. Soderstrom and P. Stoica, Systen Identification, Prentice-Hall, Hoboken, New Jersey, USA, 1989.
J. Chen, J. Ma, M. Gan, and Q. Zhu, “Multi-direction gradient iterative algorithm: a unified framework for gradient iterative and least squares algorithms,” IEEE Transactions on Automatic Control, p. 1, 2021.
View at: Publisher Site | Google Scholar
J. Chen, B. Huang, M. Gan, and C. P. Chen, “A novel reduced-order algorithm for rational models based on Arnoldi process and Krylov subspace,” Automatica, vol. 129, Article ID 109663, 2021.
View at: Publisher Site | Google Scholar
G. Y. Chen, M. Gan, C. L. P. Chen, and H. X. Li, “A regularized variable projection algorithm for separable nonlinear least-squares problems,” IEEE Transactions on Automatic Control, vol. 64, no. 2, pp. 526–537, 2019.
View at: Google Scholar
M. Jiao, D. Q. Wang, Y. Yang, and F. Liu, “More intelligent and robust estimation of battery state-of-charge with an improved regularized extreme learning machine,” Engineering Applications of Artificial Intelligence, vol. 104, no. 2, Article ID 104407, 2021.
View at: Publisher Site | Google Scholar
Q. L. Liu, F. Y. Chen, F. Ding, A. Alsaedi, and T. Hayat, “Recursive least squares estimation methods for a class of nonlinear systems based on non-uniform sampling,” International Journal of Adaptive Control and Signal Processing, vol. 35, no. 8, pp. 1612–1632, 2021.
View at: Publisher Site | Google Scholar
M. Jiao and D. Q. Wang, “The Savitzky-Golay filter based bidirectional long short-term memory network for SOC estimation,” International Journal of Energy Research, vol. 45, no. 13, pp. 19467–19480, 2021.
View at: Publisher Site | Google Scholar
L. Xu, F. Y. Chen, F. Ding, A. Alsaedi, and T. Hayat, “Hierarchical recursive signal modeling for multi-frequency signals based on discrete measured data,” International Journal of Adaptive Control and Signal Processing, vol. 35, no. 5, pp. 676–693, 2021.
View at: Publisher Site | Google Scholar
L. Bako, K. Boukharouba, E. Duviella, and S. Lecoeuche, “A recursive identification algorithm for switched linear/affine models,” Nonlinear Analysis: Hybrid Systems, vol. 5, no. 2, pp. 242–253, 2011.
View at: Publisher Site | Google Scholar
S. J. Fan, L. Xu, F. Ding, A. Alsaedi, and T. Hayat, “Correlation analysis-based stochastic gradient and least squares identification methods for errors-in-variables systems using the multi-innovation,” International Journal of Control, Automation and Systems, vol. 19, no. 1, pp. 289–300, 2021.
View at: Publisher Site | Google Scholar
H. F. Xia, Y. Ji, L. Xu, A. Alsaedi, and T. Hayat, “Maximum likelihood-based gradient estimation for multivariable nonlinear systems using the multi-innovation identification theory,” International Journal of Robust and Nonlinear Control, vol. 30, no. 14, pp. 5446–5463, 2020.
View at: Publisher Site | Google Scholar
M. Gan, H. T. Zhu, G. Y. Chen, and C. L. P. Chen, “Weighted generalized cross validation based regularization for broad learning system,” IEEE Transactions on Cybernetics, vol. 52, no. 5, pp. 4064–4072, 2022.
View at: Publisher Site | Google Scholar
Y. H. Zhou, X. Zhang, and F. Ding, “Hierarchical estimation approach for RBF-AR models with regression weights based on the increasing data length,” IEEE Transactions on Circuits and Systems-II: Express Briefs, vol. 68, no. 12, pp. 3597–3601, 2021.
View at: Publisher Site | Google Scholar
H. Y. Yan and Y. G. Zhu, “Bang-bang control model for uncertain switched systems,” Applied Mathematical Modelling, vol. 39, no. 10-11, pp. 2994–3002, 2015.
View at: Publisher Site | Google Scholar
F. Y. Wu and J. Lian, “Stabilization of constrained switched systems via multiple Lyapunov R-functions,” Systems & Control Letters, vol. 139, p. 104686, 2020.
View at: Publisher Site | Google Scholar
F. Bianchi, V. Breschi, D. Piga, and L. Piroddi, “Model structure selection for switched NARX system identification: a randomized approach,” Automatica, vol. 125, Article ID 109415, p. 109415, 2021.
View at: Publisher Site | Google Scholar
F. Lauer and G. Bloch, Hybrid System Identification, Springer, Berlin, Germany, 2019.
J. E. Saxen, H. Saxen, and H. T. Toivonen, “Identification of switching linear systems using self-organizing models with application to silicon prediction in hot metal,” Applied Soft Computing, vol. 47, pp. 271–280, 2016.
View at: Publisher Site | Google Scholar
J. Chen, J. X. Ma, Y. J. Liu, and F. Ding, “Identification methods for time-delay systems based on the redundant rules,” Signal Processing, vol. 137, pp. 192–198, 2017.
View at: Publisher Site | Google Scholar
Y. J. Lu, B. Huang, and S. Khatibisepehr, “A variational Bayesian approach to robust identification of switched ARX models,” IEEE Transactions on Cybernetics, vol. 46, no. 12, pp. 3195–3208, 2016.
View at: Publisher Site | Google Scholar
X. P. Liu and X. Q. Yang, “Identification of nonlinear state-space systems with skewed measurement noises,” IEEE Transactions on Circuits and Systems I: Regular Papers, pp. 1–9, 2022.
View at: Publisher Site | Google Scholar
J. Chen, B. Huang, F. Ding, and Y. Gu, “Variational Bayesian approach for ARX systems with missing observations and varying time-delays,” Automatica, vol. 94, pp. 194–204, 2018.
View at: Publisher Site | Google Scholar
D. Q. Wang, S. Zhang, M. Gan, and J. Qiu, “A novel EM identification method for Hammerstein systems with missing output data,” IEEE Transactions on Industrial Informatics, vol. 16, no. 4, pp. 2500–2508, 2020.
View at: Publisher Site | Google Scholar
R. Kalaba and L. Tesfatsion, “Time-varying linear regression via flexible least squares,” Computers & Mathematics with Applications, vol. 17, no. 8-9, pp. 1215–1245, 1989.
View at: Publisher Site | Google Scholar
R. Kalaba and L. Tesfatsion, “Flexible least squares for approximately linear systems,” IEEE Transactions on. Systems, Man and Cybernetics, vol. 20, no. 5, pp. 978–989, 1990.
View at: Publisher Site | Google Scholar
G. Montana, K. Triantafyllopoulos, and T. Tsagaris, “Flexible least squares for temporal data mining and statistical arbitrage,” Expert Systems with Applications, vol. 36, no. 2, pp. 2819–2830, 2009.
View at: Publisher Site | Google Scholar
J. Chen, L. X. Guo, M. F. Hu, M. Gan, and Q. M. Zhu, “Augmented flexible least squares algorithm for time-varying parameter systems,” International Journal of Robust and Nonlinear Control, vol. 32, no. 6, pp. 3549–3567, 2021.
View at: Publisher Site | Google Scholar
T. S. Chen, M. S. Andersen, L. Ljung, A. Chiuso, and G. Pillonetto, “System identification via sparse multiple kernel-based regularization using sequential convex optimization techniques,” IEEE Transactions on Automatic Control, vol. 59, no. 11, pp. 2933–2945, 2014.
View at: Publisher Site | Google Scholar
N. Sammaknejad, Y. Zhao, and B. Huang, “A review of the expectation maximization algorithm in data-driven process identification,” Journal of Process Control, vol. 73, pp. 123–136, 2019.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2022 Yunxia Ni et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

232

Downloads

376

Citations