Abstract
The selforganizing model and expectationmaximization method are two traditional identification methods for switching models. They interactively update the parameters and model identities based on offline algorithms. In this paper, we propose a flexible recursive least squares algorithm which constructs the cost function based on two kinds of errors: the neighboring twoparameter estimation errors and the output estimation errors. Such an algorithm has several advantages over the two traditional identification algorithms: it (1) can estimate the parameters of all the submodels without prior knowledge of the model identities; (2) has less computational efforts; and (3) can update the parameters with newly arrived data. The convergence properties and simulation examples are provided to illustrate the efficiency of the algorithm.
1. Introduction
Least squares (LS) algorithm is the most widely used method in parameter estimation [1–3]. It defines a cost function which is composed of the errors between the true outputs and the predicted outputs. Then, the estimations can be obtained by solving the derivative function of the cost function. The LS algorithm has fast convergence rates but with the cost of heavy computational efforts [4, 5]. In addition, the LS algorithm needs to compute the inverse of a matrix. If the matrix has a high order or is illconditioned, the LS algorithm is inefficient [6–9].
To reduce the computational efforts and to avoid the matrix inversion, the recursive least squares (RLS) algorithm is a good choice. The basic idea of the RLS algorithm is to update the parameter estimations using the newly arrived data, that is, the cost function of the RLS algorithm is composed of only one set of data rather than all the collected data [10–12]. Therefore, the RLS algorithm has less computational efforts, and it does not require calculating the matrix inverse. However, the RLS algorithm has slow convergence rates when compared with the LS algorithm [13, 14]. With the aim to increase the convergence rates, many modified RLS algorithms are developed, e.g., the multiinnovation RLS algorithm [15, 16] and the hierarchical RLS algorithm [17, 18].
Although the RLS algorithm and its modified counterparts can identify systems with less computational efforts and fast convergence rates, they have the assumption that the considered model is a single model. If the system is described by a switching model, those algorithms are inefficient. Switching models are widely used in engineering practices [19, 20]. Such models have several modes with different dynamical properties, and the modes are associated with various operating conditions [21, 22]. The difficulty in switching system identification is that the times of the operating points (model identities) may be unknown. To identify the switching models, one should first determine the operating points/model identities.
The selforganizing model (SOM)based method and the expectationmaximization (EM) method are two classical identification algorithms which are usually used for switching models [23–26]. The SOM method introduces several cost functions which correspond to each submodel in each iteration, and the smallest cost function is associated with the true model in this sampling instant [23]. The EM algorithm regards the model identities as hidden variables and updates these identities in the EME step; once the identity estimations are obtained, the parameter estimations are computed in the EMM step. These two steps run interactively until these two kinds of estimations converge to their true values [27, 28]. The SOM method should compute several cost functions in each iteration, and the EM algorithm needs to compute the model identities first. In addition, both these algorithms are offline algorithms. That is, they have heavy computational efforts and cannot update the parameters based on newly arrived data.
The flexible least squares (FLS) algorithm, first developed by Kalaba and Tesfatsion [29], is used for timevarying system identification. Its cost function contains two parts: one is the error of the two parameters in the two neighboring instants, and the other is the error between the true outputs and predicted outputs [30–32]. Due to the first error, the parameter estimations can catch the varying parameters. Inspired by the FLS algorithm, we develop a novel FLS algorithm for switching models. This algorithm is termed as flexible recursive least squares (FRLS) algorithm. Compared with the SOM, EM, and FLS algorithms, this algorithm has the following advantages: (1) the FRLS algorithm is an online algorithm, and thus it can update the parameters with newly arrived data; (2) the FRLS algorithm has less computational efforts; and (3) the FRLS algorithm can estimate the parameters of all the submodels without prior knowledge of the model identities.
The remainder of the paper is organized as follows. Section 2 explains the switching model and traditional identification algorithms. Section 3 proposes the offline FLS algorithm and online FLS algorithm. Section 4 provides several simulation examples. Finally, Section 5 summarizes the paper and gives some future directions.
2. Problem Statement
Let us define some notations first: means an identity matrix of the appropriate sizes; the superscript stands for the matrix transpose; the norm of a matrix is defined as ; means the maximum eigenvalue of matrix ; and the norm of a vector is defined as .
2.1. Switching Model
Consider the following switching model:where is the output of the th model; is the information vector of the th model, which is composed of the input and output data before the sampling instant ; is the parameter vector of the th model; is a Gaussian white noise and satisfies ; and is the number of the submodels.
In the sampling instant , there is no knowledge of the identity of the model. We aim to estimate the parameter vectors based on the collected data.
Collect sets of input and output data, and define the following cost function:where is the model identity in the sampling instant . For example, in the sampling instant , the true model is the th model, and then the true values of the identities of all the submodels are , , . To estimate the parameters, the following assumptions are introduced.
Assumption 1. The number of the collected data is larger than the number of the unknown parameters, that is,In addition, assume that the number of data of the th submodel is ; then,
Assumption 2. For the switching model proposed in (1), all the input data are taken as persistent excited.
Assumption 3. All the submodels have the same information vector but different parameter vectors, that is, the switching model can be written by
Remark 1. Assumptions 1 and 2 can ensure that the information matrices of all the submodels are nonsingular [6]. Assumption 3 can also be easily obtained [33]. For example, for a switching model with unknown structures, we can use the kernel method to describe the model, and all the submodels approximated by using the kernel method can have the same structure.
2.2. Traditional Identification Algorithms
Rewrite the cost function of the switching model as follows:
Assume that the parameter estimations and identity estimations in iteration are and .
Both the SOM and EM algorithms estimate the parameters through two steps:(1)Estimate the model identity estimations , based on the parameter estimations .(2)Update the parameter estimations based on the model identity estimations .
The difference between the SOM and EM algorithms is in the first step. In the SOM algorithm, the model identity estimate is 1 or 0. For example, in iteration of the sampling instant , let
Then, and the other identity estimations .
On the other hand, in the EM algorithm, let
Then, the identity estimate can be computed by
Remark 2. Both the SOM and EM algorithms are offline algorithms; if the order of the system is large, their computational efforts are heavy. In addition, they cannot update the parameters with newly arrived data [23, 34].
3. Flexible Recursive Least Squares Algorithm
The SOM and EM algorithms update the parameters through two steps, and these two steps are related to each other. If one kind of estimations has poor estimation accuracy, the other may be also poor or divergent. In this section, we use the FLS algorithm for the switching models, which can estimate the parameters without prior knowledge of the model identities.
3.1. Offline FLS Algorithm
Define
Then, the switching model can be written as
Let
Unlike the SOM and EM algorithms, the cost function of the offline FLS algorithm is written by
Using the FLS algorithm to update the parameters yields
Remark 3. From equation (5), we can get that the offline FLS (OFLS) algorithm can estimate the parameters in only one iteration and does not require the model identify estimation. However, it increases the order of the information matrix intensively. For example, the order of the information matrix is in the OFLS algorithm, while in the EM and SOM methods, the order is .
Remark 4. The offline FLS (OFLS) algorithm requires a highorder matrix inverse calculation, e.g., . When the highorder matrix is singular or illconditioned, computing its inverse is impossible.
3.2. Flexible Recursive Least Squares Algorithm
To reduce the computational efforts and to avoid a highorder matrix inversion, this section proposes an online FLS algorithm which is termed as flexible recursive least squares (FRLS) algorithm.
Assume that the parameter vector in the sampling instant is . Define the following cost function:
In the sampling instant , all the parameter estimations before have been obtained, and thus (15) is simplified as
Taking the derivative of with respect to yields
Next, we use the recursive method to obtain the relationships between and .
(17) is transformed into
Then, the FRLS algorithm can be summarized as follows:
Remark 5. Compared with the OFLS algorithm, the FRLS algorithm performs a loworder matrix inversion rather than a highorder matrix inversion. Therefore, the FRLS algorithm has less computational efforts than the OFLS algorithm (Algorithm1).
Then, the steps of the FRLS algorithm are listed as follows.

Lemma 1. For the matrices , , and , if the matrix is nonsingular, the following equality holds:
Proof. For the matrix , we haveIn addition, we can obtainThen, the proof is completed.
According to Lemma 1, the matrix is simplified as
Remark 6. Based on equation (15), in each sampling instant, a dense matrix inversion is transformed into vector multiplication. Therefore, the computational efforts are reduced.
3.3. Convergence Properties of the Two Kinds of FLS Algorithms
The convergence properties of the OFLS and FRLS algorithms are given in this section which can help the researchers follow these two algorithms.
3.3.1. Convergence Property of the OFLS Algorithm
Theorem 1. For the switching model proposed in (1), the parameter estimations updated by the OFLS algorithm are expressed by (5). Then, are biased.
Proof. Rewrite the OFLS algorithm as follows:Substituting into the above equation yieldswhere is Gaussian white and independent on , and the above equation can be written bySince , the matrix is singular, and the matrix cannot be a zero matrix. Therefore, the OFLS algorithm is a biased algorithm.
Remark 7. A small can get more accurate parameter estimations. However, a small may lead to slow convergence rates between the two neighboring submodels. Therefore, we should assign different values for . For example, in the fixed interval, a small is better, while near the switching points, a larger one is better.
3.3.2. Convergence Property of the FRLS Algorithm
Theorem 2. For the switching model proposed in (1), the parameter estimations updated by the FRLS algorithm are expressed by (10)–(12). Then, the sequence is convergent.
Proof. The FRLS algorithm is written byAssume that the data from belong to model 1, and subtracting the true value on both sides of the above equation yieldsFor the reason thatwe haveTherefore, the FRLS algorithm is convergent.
Remark 8. The FRLS algorithm has the assumption that the identities of the data are unchanging in a fixed interval. If the identities are changing continually, the FRLS algorithm is divergent.
4. Examples
4.1. Example 1
Consider the following switching model:
Let
In simulation, we collect 500 sets of input and output data, where the data from belong to model 1, and those from belong to model 2.
Use the FRLS algorithm for this switching model. The parameter estimations are shown in Figures 1 and 2. The predicted outputs and the true outputs, and their errors are shown in Figure 3. In addition, apply the EM and SOM algorithms for the switching model, where the initial identities for each submodel are and . The estimation errors and elapsed times of the three algorithms are shown Table 1.
From this simulation, we can get the following findings:(1)The parameter estimations using the FRLS algorithm can asymptotically converge to the true values (see Figures 1 and 2).(2)The predicted outputs using the FRLS algorithm can catch the true outputs (see Figure 3).(3)The number of the data in a fixed interval must be larger than the number of the unknown parameters.(4)All the FRLS, EM, and SOM algorithms are effective for the switching model, but the FRLS algorithm has the smallest elapsed times, that is, the FRLS algorithm has the least computational efforts among these three algorithms, as shown in Table 1.
4.2. Example 2: A Switching Open Channel System
In this section, we consider an open channel system, which is shown in Figure 4. The radius of the channel is , the length of the channel is , is the discharge at the upstream end, is the discharge at the downstream end, and the slope is . To ensure the discharge to flow in a fixed speed, we should control . The relationship between and can be expressed by a linear model. In simulation, two slopes and degrees are assigned to the open channel system. These two slopes lead to two different dynamics which should be described by two models: [14]:
We collect 2000 sets of inputoutput data using Matlab software, where the sequence is generated by
The data from belong to model 1, and those from belong to model 2.
Apply the FRLS algorithm for the considered switching model. The parameter estimations and their estimation errors , are shown in Figure 5 and Tables 2 and 3.
Furthermore, we use the traditional EM and SOM algorithms for the switching open channel system . The parameter estimations and their estimation errors are shown in Figures 6 and 7. The elapsed times of these three algorithms are shown Table 4.
This example shows that (1) all the FRLS, EM, and SOM algorithms are convergent, as shown in Figures 5–7; (2) the FRLS algorithm has the smallest elapsed times among these three algorithms, and this is shown in Table 4, that is, the FRLS algorithm has the least computational efforts among these three algorithms.
5. Conclusions
An online FLS algorithm, termed as flexible recursive least squares (FRLS) algorithm, is proposed for switching models in this study. Its cost function is composed of the errors between the two neighboring parameter estimations and the errors between the true outputs and the predicted outputs. With the help of the two neighboring parameter estimation errors, the operating points of the switching models can be determined, and the parameters of each submodel can also been obtained. Compared with the SOM and EM algorithms, the FRLS algorithm can estimate the parameter estimations without prior knowledge of the model identities. In addition, the FRLS algorithm is an online algorithm, which has less computational efforts and can update the parameters with newly arrived data.
Although the FRLS algorithm has several advantages over the traditional identification algorithms, several challenging issues about the FRLS algorithm need to be considered in future. For example, if the submodels switch continually, how to apply the FRLS algorithm to the switching model? How to choose a suitable to make the FRLS algorithm converge quickly to the true values? These topics remain as open problems.
Data Availability
All data generated or analyzed during this study are included in this article.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This study was supported by the Natural Science Foundation of Jiangsu Province (no. BK20131109).