Linear Estimation of Stationary Autoregressive Processes

Abstract

Consider a sequence of an 𝑚th-order Autoregressive (AR) stationary discrete-time process and assume that at least 𝑚−1 consecutive neighboring samples of an unknown sample are available. It is not important that the neighbors are from one side or are from both the left and right sides. In this paper, we find explicit solutions for the optimal linear estimation of the unknown sample in terms of the neighbors. We write the estimation errors as the linear combination of innovation noises. We also calculate the corresponding mean square errors (MSE). To the best of our knowledge, there is no explicit solution for this problem. The known solutions are the implicit ones through orthogonality equations. Also, there are no explicit solutions when fewer than 𝑚−1 samples are available. The order of the process (𝑚) and the feedback coefficients are assumed to be known.

1. Introduction

Estimation has many applications in different areas including compression and equalization [1, 2]. The linear estimation is more common due to its mathematical simplicity. The optimal linear estimation of a random variable 𝑥 in terms of 𝑦₁, 𝑦₂, and 𝑦_𝑛 is the following linear combination𝑥̂≜𝐸𝑥∣𝑦₁,𝑦₂,…,𝑦_𝑛=^𝑛_𝑖=1𝐴_𝑖𝑦_𝑖,(1) where the coefficients 𝐴_𝑖 must be chosen to minimize the MSE 𝐸{(𝑥−𝑥̂)²} and 𝐸{⋅} stands for the expected value. To minimize the MSE, we must choose 𝐴_𝑖’s to satisfy the orthogonality principle as follows: 𝐸(𝑥−𝑥̂)𝑦_𝑖=0,𝑖=1,2,…,𝑛.(2) We also write the above condition as𝑥−𝑥̂⟂𝑦_𝑖,𝑖=1,2,…,𝑛.(3) Therefore, in the optimal linear estimation, we search for the coefficients such that the error is orthogonal to the data.

A common model for many signals including image, speech, and biological signals is the AR model [1, 3–5]. This model has applications in different areas including detection [6, 7], traffic modeling [8], channel modeling [9], and forecasting [10]. An AR process is the output of an all-pole causal filter whose input is a white sequence called innovation noise [11]. We introduce another model for the process using an all-pole anticausal filter as well. The optimal linear estimation of an AR process is accomplished through the recursive solution of Yule-Walker (YW) equations using Levinson-Durbin algorithm [12]. This solution is recursive and implicit. As we will see in some cases the equation coefficients do not form a Toeplitz matrix and we cannot enjoy the complexity reduction advantage of Levinson algorithm.

To the best of our knowledge, there is no explicit solution for YW equations. Most of the focus of researchers is on model parameters estimation from observations. When researchers arrive at YW equations, they stop, since they consider the solution as known through Levinson recursion. Broersen in his method for autocorrelation function estimation form observations points to YW equations and mainly concentrates on bias reduction in estimation using finite set of observations [13, 14]. He does not attempt to find the solution for YW equations. Fattah et al. try to estimate the autocorrelation function of an ARMA model from noisy data; they again refer to YW equation set and its solution using matrix inversion and no explicit solution is proposed [15]. Xia and Kamel propose an optimization method to estimate AR model parameters from noisy data [16]. Noise is not necessarily Gaussian. The method finds a minimum for a cost function and exploits a neural-network algorithm. Again, the explicit solution of the orthogonality equations is not the goal of the paper. Hsiao proposes an algorithm to estimate the parameters of a time-varying AR system [17]. He considers the feedback coefficients of a time-varying AR process as random variables. The proposed algorithm maximizes a posteriori probabilities conditioned on the data. The recursive algorithm is compared to Monte Carlo simulation in terms of accuracy and complexity. In this paper, the aim is parameter estimation from data and not the analytic solution of orthogonality equations. In [18], a sequence of Gaussian AR vector is considered. As the sequence elements are vectors rather than scalars, the AR model is defined by matrix feedback coefficients rather than scalar feedback coefficients. The estimation here is more complex, and some independence conditions are assumed. The method is based on convex optimization, and no exact answer can be provided. Mahmoudi and Karimi propose an LS-based method to estimate AR parameters from noisy data [19]. The method exploites YW equations, but this method also does not provide the explicit solution to the equations. Another LS-based estimation method can be seen in [20].

As mentioned above, we could not see the final solution to YW orthogonality equations in the literature. In this work we have derived explicit solutions for orthogonality equations for different cases. Consider a stationary 𝑚th order AR process. The order and the feedback coefficients of the process are assumed to be known, and the model parameter estimation is out of the scope of this paper. The main goal of this paper is finding the solution for the orthogonality equations. We will find the the optimal linear estimation of a sample in terms of the neighbors where at least 𝑚−1 consecutive neighbors are available. The consecutive neighbors include the situations where all the 𝑚−1 or more neighbors are in one side, or some of them are left neighbors and the others are right neighbors. We will show that no more that 𝑚 consecutive neighbors in each side are needed. Our approach is to find orthogonal estimation errors that are linear combinations of data. We use the well-known causal LTI AR model as well as our anticausal model to form orthogonal errors. The errors are formed as a linear combination of causal and anticausal innovation (process) noises. Beginning from suitable errors that are both orthogonal to the data and are linear combination of data, we arrive at linear estimations. We seek LTI system approach rather than trying to directly solve the orthogonality equations. The results of this paper for different cases can be important in situations where the equation matrices are ill-posed and the matrix inversion and other recursive algorithms become unstable.

This paper is organized as follows. In Section 2, the causal model is reviewed and the anticausal model is introduced. In Section 3, we review the forward prediction problem. We state the problem symmetries in Section 4. We see how we can use the similarities between two problems to exploit the solution of one problem to find the solution of the other problem. In Section 5, we extract a number of relations for cross-correlation functions that will be used later. We find the interpolation formulae when infinite data are available in Section 6. We find the prediction and interpolation with finite data in Sections 7 and 8, respectively. In Section 9, we present a detailed example to show that our relations and the matrix solution of the orthogonality principle result in the same coefficients. Finally, we conclude the work in Section 10.

2. Causal and Anticausal Models

A discrete-time stationary AR process 𝑠_𝑛 of order 𝑚 is modeled as follows.𝑠_𝑛+𝑎₁𝑠_𝑛−1+𝑎₂𝑠_𝑛−2+⋯+𝑎_𝑚𝑠_𝑛−𝑚=𝐼_𝑛,𝑛∈ℤ.(4) The above equation is meant for a causal LTI system. 𝐼_𝑛, the input of the system, is called the innovation noise and is a stationary white sequence with the zero expected value, that is, 𝐸{𝐼_𝑛𝐼_𝑘}=𝜎²𝛿[𝑛−𝑘] and 𝐸{𝐼_𝑛}=0, where 𝜎 is a positive constant. 𝛿[0]=1 and 𝛿[𝑖]=0 elsewhere. The system is causal. Therefore 𝑠_𝑛, the output of the system in the time index 𝑛, is a linear combination of the inputs in the time index 𝑛 and before. So, we can write𝑠_𝑛=ℎ₀𝐼_𝑛+ℎ₁𝐼_𝑛−1+ℎ₂𝐼_𝑛−2+⋯=^∞_𝑖=0ℎ_𝑖𝐼_𝑛−𝑖.(5) In the above equation, ℎ_𝑛 is the impulse response of the system. Assuming the causal system model, we have ℎ_𝑛=0 for 𝑛<0. Paying attention to the whiteness of the sequence {𝐼_𝑛} and from (5) we get the following result. 𝐼_𝑛+𝑘⟂𝑠_𝑛,𝑘>0,𝑛∈ℤ.(6)

Figure 1 is the causal model of the AR process. 𝐻(𝑧) is the Z-transform of ℎ_𝑛, which is defined as𝐻(𝑧)=^∞_𝑘_=−∞ℎ_𝑘𝑧^−𝑘.(7) For the system defined by (4), we have𝐻(𝑧)=1𝐴(𝑧)=1+𝑎₁𝑧⁻¹+1⋯+𝑎_𝑚𝑧^−𝑚.(8)

Figure 1 

The causal model.

Assuming a stable causal system, we conclude that the roots of 𝐴(𝑧)=0 must be inside the unit circle |𝑧|=1. The power spectral density function (PSDF) of a process is the 𝑍-transform of its autocorrelation function. The PSDF of 𝑠_𝑛 is [11]𝑆_𝑠(𝑧)=𝑆_𝐼(𝑧)𝐻(𝑧)𝐻𝑧⁻¹=𝜎²𝐻(𝑧)𝐻𝑧⁻¹.(9) In the above equation 𝑆_𝑠(𝑧) is the PSDF of 𝑠_𝑛 and 𝑆_𝐼(𝑧) is the PSDF of 𝐼_𝑛.

We now present the anticausal model. If we apply the sequence 𝑠_𝑛 to an LTI system with the transfer function 𝐻⁻¹(𝑧⁻¹), we get another innovation noise called 𝐼^𝑛. Figure 2 demonstrates the generation of the new innovation noise. To see the whiteness of the sequence 𝐼^𝑛, note that the PSDF of 𝐼^𝑛 by using Figure 2 and (9) is as follows.𝑆_𝐼^′(𝑧)==𝑆𝜎_𝑠2(𝑧)𝐻⁻¹𝑧⁻¹𝐻⁻¹(𝑧)𝐻(𝑧)𝐻𝑧⁻¹𝐻⁻¹𝑧⁻¹𝐻⁻¹(𝑧)=𝜎².(10)

Figure 2 

Generation of another innovation noise.

Equivalently we can apply 𝐼^𝑛 to the inverse system with the transfer function 𝐻^(𝑧)=𝐻(𝑧⁻¹) to get 𝑠_𝑛. The generation of 𝑠_𝑛 from 𝐼^𝑛 is depicted in Figure 3.

Figure 3 

Anticausal model.

We have𝐻^(𝑧)=𝐻𝑧⁻¹=1𝐴𝑧⁻¹.(11) Therefore ℎ^𝑛=ℎ_−𝑛. Noting that ℎ_𝑛=0 for 𝑛<0, we see that ℎ^𝑛=0 for 𝑛>0. Also note that the roots of 𝐴(𝑧⁻¹)=0 are outside the unit circle, as we had the roots of 𝐴(𝑧)=0 inside the unit circle. Regarding these points, we know that the system with the transfer function 𝐻^(𝑧) is stable and anticausal. We have𝐻^(𝑧)=11+𝑎₁𝑧+𝑎₂𝑧²+⋯+𝑎_𝑚𝑧^𝑚.(12)

Using the above equation and Figure 3, we get𝑠_𝑛+𝑎₁𝑠_𝑛+1+𝑎₂𝑠_𝑛+2+⋯+𝑎_𝑚𝑠_𝑛+𝑚=𝐼^𝑛,𝑛∈ℤ.(13)

Also, note that𝑠_𝑛=^∞_𝑖=_−∞=ℎ₀ℎ^𝑖𝐼^{𝑛−𝑖}=ℎ^0𝐼^𝑛+ℎ^−1𝐼^𝑛+1+ℎ^−2𝐼^𝑛+2+⋯𝐼^𝑛+ℎ₁𝐼^𝑛+1+⋯=^∞_𝑖=0ℎ_𝑖𝐼^𝑛+𝑖.(14)

From (14) and Figure 3, we see that 𝑠_𝑛 is a linear combination of 𝐼^𝑛 and the inputs after that. The whiteness of the sequence {𝐼^𝑛} gives then𝐼^{𝑛−𝑘}⟂𝑠_𝑛,𝑛∈ℤ,𝑘>0.(15)

3. Forward Prediction

Forward prediction can be accomplished by using the whitening filter [11]. The data are whitened, and we use the equivalent white data to achieve the prediction. As an example, consider the 1-step forward prediction of 𝑠_𝑛. It is seen that 𝑠_𝑛 is estimated as𝑠_𝑛̂=𝐸𝑠_𝑛∣𝑠_𝑛−𝑘,𝑘>0=−^𝑚_𝑘=1𝑎_𝑘𝑠_𝑛−𝑘=−𝑎₁𝑠_𝑛−1−𝑎₂𝑠_𝑛−2−⋯−𝑎_𝑚𝑠_𝑛−𝑚.(16) It can bee seen from (4) that the error 𝑠_𝑛−𝑠_𝑛̂ is equal to 𝐼_𝑛 and therefore, from (6), it is orthogonal to 𝑠_𝑛−𝑘 for 𝑘>0. It proves the optimality of (16).

The 2-step prediction can be done as [11]𝑠_𝑛̂=𝐸𝑠_𝑛∣𝑠_𝑛−𝑘,𝑘≥2=−𝑎₁𝑠̂_𝑛−1−^𝑚_𝑖=2𝑎_𝑖𝑠_𝑛−𝑖.(17) In the above equation, 𝑠̂_𝑛−1 is the prediction of 𝑠_𝑛−1 from its previous data (1-step prediction) and is obtained by replacing 𝑛 by 𝑛−1 in (16).𝑠̂_𝑛−1=𝐸𝑠_𝑛−1∣𝑠_𝑛−𝑘,𝑘≥2=−^𝑚_𝑘=1𝑎_𝑘𝑠_{𝑛−𝑘−1}=−𝑎₁𝑠_𝑛−2−𝑎₂𝑠_𝑛−3−⋯−𝑎_𝑚𝑠_{𝑛−𝑚−1}.(18) From (17), (18), and (4), the estimation error is 𝑒_𝑛=𝑠_𝑛+^𝑚_𝑖=2𝑎_𝑖𝑠_𝑛−𝑖−𝑎_1𝑚_𝑘=1𝑎_𝑘𝑠_{𝑛−𝑘−1}=𝐼_𝑛−𝑎₁𝑠_𝑛−1−𝑎_1𝑚_𝑘=1𝑎_𝑘𝑠_{𝑛−𝑘−1}=𝐼_𝑛−𝑎₁𝐼_𝑛−1.(19)

From (6), it is clear that 𝐼_𝑛 and 𝐼_𝑛−1 are orthogonal to 𝑠_𝑛−𝑘 for 𝑘≥2. It proves the optimality of (17).

The higher-order predictions can be obtained in the same manner. As the final example of this section, consider the 3-step forward prediction that is accomplished as follows. 𝑠_𝑛̂==−𝐸𝑎₁𝑠_𝑛∣𝑠_𝑛−𝑘,𝑘≥3𝑠̂_𝑛−1−𝑎₂𝑠̂_𝑛−2−^𝑚_𝑘=3𝑎_𝑘𝑠_𝑛−𝑘.(20)

In the above equation, 𝑠̂_𝑛−1 and 𝑠̂_𝑛−2 are the 2-step and 1-step predictions of 𝑠_𝑛−1 and 𝑠_𝑛−2, respectively, and are obtained from (17) and (16). The error is𝑒_𝑛=𝑠_𝑛+𝑎₁𝑠̂_𝑛−1+𝑎₂𝑠̂_𝑛−2+^𝑚_𝑖=3𝑎_𝑖𝑠_𝑛−𝑖=𝐼_𝑛−𝑎₁𝑠_𝑛−1−𝑎₂𝑠_𝑛−2+𝑎₁𝑠̂_𝑛−1+𝑎₂𝑠̂_𝑛−2==𝐼𝐼_𝑛𝑛−𝑎₁𝑠_𝑛−1−𝑠̂_𝑛−1−𝑎₂𝑠_𝑛−2−𝑠̂_𝑛−2−𝑎₁𝐼_𝑛−1−𝑎₁𝐼_𝑛−2−𝑎₂𝐼_𝑛−2.(21)

4. The Problem Symmetries

Consider the following linear interpolation of 𝑠_𝑛 from the data around it:𝑠_𝑛̂==𝑎𝐸_−𝑘1𝑠_𝑛∣𝑠_𝑛−𝑘1,𝑠_{𝑛−𝑘1+1},…,𝑠_𝑛−1,𝑠_𝑛+1,…,𝑠_𝑛+𝑘2𝑠_𝑛−𝑘1+𝑎^−𝑘₁₊₁𝑠_{𝑛−𝑘1+1}+⋯+𝑎^𝑘₂𝑠_𝑛+𝑘2.(22) The orthogonality principle gives𝐸=𝑠_𝑛0−,𝑎𝑖=^−𝑘₁𝑠−𝑘^𝑛−𝑘₁₁−𝑎^−𝑘₁₊₁𝑠_{𝑛−𝑘1+1}−⋯−𝑎^𝑘₂𝑠_𝑛+𝑘2𝑠^𝑛+𝑖,−𝑘₁+1,…,𝑘₂,𝑖≠0.(23) The above equations become𝑅_𝑠𝑖+𝑘₁𝑎^−𝑘₁+𝑅_𝑠𝑖+𝑘₁−1𝑎^−𝑘₁₊₁+⋯+𝑅_𝑠𝑖−𝑘₂𝑎^𝑘₂=𝑅_𝑠[𝑖],𝑖=−𝑘₁,−𝑘₁+1,…,𝑘₂,𝑖≠0.(24) In the above equations, 𝑅_𝑠[𝑖]=𝐸{𝑠_𝑛𝑠_𝑛−𝑖}.

Now, consider the following estimation.𝑠_𝑛̂==𝑎𝐸_𝑘1𝑠_𝑛∣𝑠_𝑛−𝑘2,𝑠_{𝑛−𝑘2+1},…,𝑠_𝑛−1,𝑠_𝑛+1,…,𝑠_𝑛+𝑘1𝑠_𝑛+𝑘1+𝑎^𝑘₁₋₁𝑠_{𝑛+𝑘1−1}+⋯+𝑎^{−𝑘}₂𝑠_𝑛−𝑘2.(25) The orthogonality of error to the data gives𝐸=𝑠_𝑛0−,𝑎𝑖^𝑘₁=𝑠𝑘^𝑛+𝑘₁₁−𝑎^𝑘₁₋₁𝑠_{𝑛+𝑘1−1}−⋯−𝑎^{−𝑘}₂𝑠_𝑛−𝑘2𝑠^𝑛+𝑖,𝑘₁−1,…,−𝑘₂,𝑖≠0.(26) They become𝑅_𝑠𝑖−𝑘₁𝑎^𝑘₁+𝑅_𝑠𝑖−𝑘₁+1𝑎^𝑘₁₋₁+⋯+𝑅_𝑠𝑖+𝑘₂𝑎^𝑘₂=𝑅_𝑠[𝑖],𝑖=𝑘₁,𝑘₁−1,…,−𝑘₂,𝑖≠0.(27) Regarding that the 𝑅_𝑠[⋅] is an even function, we notice that the set of equations (24) and the set of equations (27) are exactly the same. Therefore,𝑎^−𝑘₁=𝑎^𝑘₁,𝑎^−𝑘₁₊₁=𝑎^𝑘₁₋₁,…,𝑎^𝑘₂=𝑎^{−𝑘}₂.(28)

As an example, consider the following backward prediction.𝑠_𝑛̂=𝐸𝑠_𝑛∣𝑠_𝑛+𝑘,𝑘>0.(29) Using (16) and the symmetry, we get𝑠_𝑛̂=𝐸𝑠_𝑛∣𝑠_𝑛+𝑘,𝑘>0=−^𝑚_𝑘=1𝑎_𝑘𝑠_𝑛+𝑘=−𝑎₁𝑠_𝑛+1−𝑎₂𝑠_𝑛+2−⋯−𝑎_𝑚𝑠_𝑛+𝑚.(30) The validity of the solution can also be confirmed as from (13), it is seen that the estimation error is𝑠_𝑛+𝑎₁𝑠_𝑛+1+𝑎₂𝑠_𝑛+2+⋯+𝑎_𝑚𝑠_𝑛+𝑚=𝐼^𝑛.(31) Using (15), it is clear that the error is orthogonal to the data. It proves the optimality of (30).

5. Cross-Correlation Functions

In this section, we derive a number of properties for the cross-correlations between innovation noises and the AR process. We will exploit these properties to prove our solutions.

We define 𝑅_𝑠𝐼[𝑘]=𝐸{𝑠_𝑛𝐼_𝑛−𝑘} and 𝑅_𝐼^′_𝑠[𝑘]=𝐸{𝐼^𝑛𝑠_𝑛−𝑘}. The first simple property follows from (6) and (15) as follows.𝑅_𝑠𝐼[𝑘]=𝑅_𝐼^′_𝑠[𝑘]=0,𝑘<0.(32) Now, consider Figure 1. In this figure 𝐼_𝑛 is the input and 𝑠_𝑛 is the output. The impulse response of system is ℎ_𝑛≜ℎ[𝑛]. Therefore, we have [11]𝑅_𝑠𝐼[𝑘]=𝑅_𝐼[𝑘]∗ℎ[𝑘]=𝜎²𝛿[𝑘]∗ℎ[𝑘]=𝜎²ℎ[𝑘].(33) In this equation, 𝑅_𝐼[𝑘]=𝐸{𝐼_𝑛𝐼_𝑛−𝑘} and the “∗” operator is the discrete convolution. Taking the 𝑍-transform from both sides of (33) and using (8), we get𝑆_𝑠𝐼(𝑧)=𝜎²𝐻(𝑧)=𝜎²1+𝑎₁𝑧⁻¹+⋯+𝑎_𝑚𝑧^−𝑚.(34) Or equivalently𝑆_𝑠𝐼(𝑧)1+𝑎₁𝑧⁻¹+⋯+𝑎_𝑚𝑧^−𝑚=𝜎².(35) Taking inverse 𝑍-transform from this equation, we have𝑅_𝑠𝐼[𝑘]+𝑎₁𝑅_𝑠𝐼[𝑘−1]+⋯+𝑎_𝑚𝑅_𝑠𝐼[𝑘−𝑚]=𝜎²𝛿[𝑘].(36) The right side of (36) is zero for 𝑘≠0.

Referring to Figure 3, we have [11]𝑅_𝐼^′_𝑠[𝑘]==𝑅𝜎_𝐼^′₂[𝑘]∗ℎ^[−𝑘]=𝜎²𝛿[𝑘]∗ℎ^[−𝑘]ℎ^[−𝑘]=𝜎²ℎ[𝑘].(37) Again, we conclude that𝑅_𝐼^′_𝑠[𝑘]+𝑎₁𝑅_𝐼^′_𝑠[𝑘−1]+⋯+𝑎_𝑚𝑅_𝐼^′_𝑠[𝑘−𝑚]=𝜎²𝛿[𝑘].(38)

6. Interpolation Using an Infinite Set of Data

In this section, we assume that infinite number of data are available. However, we will see that only a finite number of data are sufficient.

6.1. Infinite Data on the Left Side

We want to obtain the following estimation.𝑠_𝑛̂=𝐸𝑠_𝑛∣𝑠_𝑛+𝑖,𝑖≤𝑘₁,𝑖≠0.(39)

𝑘₁ is a positive integer constant not greater than 𝑚. There are 𝑘₁ data available on the right side of 𝑠_𝑛 and infinite data on the left side. Define 𝑎₀≜1. We are going to prove the following.𝑠_𝑛̂==𝐸−𝑠_𝑛1∣𝑠_𝑛+𝑖,𝑖≤𝑘₁,𝑖≠0∑^𝑘1_𝑘=0𝑎^2𝑘⎛⎜⎜⎝^𝑘1_𝑘=1⎛⎜⎜⎝^𝑘1−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛+𝑘+^𝑚−𝑘1_𝑘=1⎛⎜⎜⎝^𝑘1_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛−𝑘+^𝑚_𝑘=𝑚_−𝑘1+1^𝑚−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘𝑠^𝑛−𝑘⎞⎟⎟⎠.(40) Observe from (40) that although there are infinite data on the left side of 𝑠_𝑛, only 𝑚 data 𝑠_𝑛−1 to 𝑠_𝑛−𝑚 participate in the estimation. Indeed, (40) is the optimal linear estimation solution for 𝑠_𝑛̂=𝐸{𝑠_𝑛∣𝑠_𝑛+𝑖,−𝑘₂≤𝑖≤𝑘₁,𝑖≠0}, where 𝑘₂ can be any integer greater than or equal to 𝑚.

To prove the optimality of (40), we must show that the estimation error is orthogonal to the data. Firstly the estimation error can be calculated by inserting 𝑠_𝑛̂ from (40) in 𝑒_𝑛=𝑠_𝑛−𝑠_𝑛̂. Secondly, by extending the innovation noises using (4) we can confirm that𝑒_𝑛=𝑠_𝑛−𝑠_𝑛̂=1∑^𝑘1_𝑘=0𝑎^2𝑘𝐼_𝑛+𝑎₁𝐼_𝑛+1+⋯+𝑎_𝑘1𝐼_𝑛+𝑘1.(41) Indeed, we have obtained (40) from (41). The motivation is that the estimation error has to possess two essential conditions: (1) it must be orthogonal to the data and (2) it must be only a linear combination of the data and the variable to be estimated. It remains to prove that the right side of (41) is orthogonal to the data.

Using (6), it is quite clear that 𝐼_𝑛 to 𝐼_𝑛+𝑘1 are orthogonal to 𝑠_𝑛−𝑘 for 𝑘>0, and so is 𝑒_𝑛 in (41). Further, we have𝐸=𝑠𝑅_{𝑛+𝑖𝑠𝐼}𝐼_𝑛+𝑎₁𝐼_𝑛+1+⋯+𝑎_𝑘1𝐼_𝑛+𝑘1=𝑅_𝑠𝐼[𝑖]+𝑎₁𝑅_𝑠𝐼[𝑖−1]+⋯+𝑎_𝑘1𝑅_𝑠𝐼𝑖−𝑘₁[𝑖]+𝑎₁𝑅_𝑠𝐼[𝑖−1]+⋯+𝑎_𝑖𝑅_𝑠𝐼[0],1≤𝑖≤𝑘₁.(42) The last equation of (42) is justified as we have 𝑅_𝑠𝐼[𝑘]=0 for 𝑘<0 from (32). Using (32), (36), (41), and (42) it is seen that𝐸𝐼_𝑛+𝑎₁𝐼_𝑛+1+⋯+𝑎_𝑘1𝐼_𝑛+𝑘1𝑠_𝑛+𝑖=0,1≤𝑖≤𝑘₁.(43) This completes the proof.

The MSE is 𝐸𝑒^2𝑛=1∑^𝑘1𝑘=0𝑎^2𝑘²=1⋅𝐸𝐼_𝑛+𝑎₁𝐼_𝑛+1+⋯+𝑎_𝑘1𝐼_𝑛+𝑘1²∑^𝑘1𝑘=0𝑎^2𝑘²=⋅𝐸1𝐼^2𝑛+𝑎²¹𝐸𝐼^2𝑛+1+⋯+𝑎^2𝑘₁𝐸𝐼^2𝑛+𝑘₁∑^𝑘1𝑘=0𝑎^2𝑘²⋅𝜎²+𝑎²¹𝜎²+⋯+𝑎^2𝑘₁𝜎².(44)

Therefore,𝐸𝑒^2𝑛=𝜎²∑^𝑘1_𝑘=0𝑎^2𝑘.(45)

6.2. Infinite Data on the Right Side

By symmetry, and replacing 𝑠_𝑛−𝑘 by 𝑠_𝑛+𝑘 in (40), the following estimation is derived.𝑠_𝑛̂==−𝐸𝑠_𝑛1∣𝑠_𝑛−𝑖,𝑖≤𝑘₁,𝑖≠0∑^𝑘1_𝑘=0𝑎^2𝑘⎛⎜⎜⎝^𝑘1_𝑘=1⎛⎜⎜⎝^𝑘1−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛−𝑘+^𝑚−𝑘1_𝑘=1⎛⎜⎜⎝^𝑘1_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛+𝑘+^𝑚_𝑘=𝑚_−𝑘1+1^𝑚−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘𝑠^𝑛+𝑘⎞⎟⎟⎠.(46)

Again, only 𝑚 data 𝑠_𝑛+1 to 𝑠_𝑛+𝑚 on the right side of 𝑠_𝑛 participate in the interpolation, and the data after them are not needed. Therefore, (46) is the solution for all the optimal linear interpolations 𝑠_𝑛̂=𝐸{𝑠_𝑛∣𝑠_𝑛−𝑖,−𝑘₂≤𝑖≤𝑘₁,𝑖≠0}, where 𝑘₂ can be any integer greater than or equal to 𝑚.

The validity of (46) can also be proved as follows. The error is calculated as 𝑒_𝑛=𝑠_𝑛−𝑠_𝑛̂, where 𝑠_𝑛̂ is from (46). By extending the innovation noises from (13), it can be verified that𝑒_𝑛=𝑠_𝑛−𝑠_𝑛̂=1∑^𝑘1_𝑘=0𝑎^2𝑘𝐼^𝑛+𝑎₁𝐼^𝑛−1+⋯+𝑎_𝑘1𝐼^{𝑛−𝑘}₁.(47) Using (15), it is quite clear that 𝐼^{𝑛−𝑘}₁ to 𝐼^𝑛 are orthogonal to 𝑠_𝑛+𝑘 for 𝑘>0, and so is 𝑒_𝑛 in (47). Further, we have𝐸=𝑠𝑅^𝑛−𝑖_𝐼′𝑠𝐼^𝑛+𝑎₁𝐼^𝑛−1+⋯+𝑎_𝑘1𝐼^{𝑛−𝑘}₁=𝑅_𝐼′𝑠[𝑖]+𝑎₁𝑅_𝐼^′_𝑠[𝑖−1]+⋯+𝑎_𝑘1𝑅_𝐼^′_𝑠𝑖−𝑘₁[𝑖]+𝑎₁𝑅_𝐼^′_𝑠[𝑖−1]+⋯+𝑎_𝑖𝑅_𝐼^′_𝑠[0],1≤𝑖≤𝑘₁.(48) The last equation of (48) is justified as we have 𝑅_𝐼^′_𝑠[𝑘]=0 for 𝑘<0 from (32). Using (32), (38), (47), and (48), it is seen that 𝐸{𝑒_𝑛𝑠_𝑛−𝑖}=0 for 1≤𝑖≤𝑘₁. This completes the proof.

The MSE is the same as in (45).

6.3. Infinite Data on Both Sides

Now, we want to estimate 𝑠_𝑛 from all the data around it. We will see that only 𝑚 data on each side are needed and as is expected, the data with the same distance from 𝑠_𝑛 participate with the same weight. We have𝑠_𝑛̂==−𝐸𝑠_𝑛1∣𝑠_𝑛−𝑖,𝑖≠0∑^𝑚𝑘=0𝑎^2𝑘⋅^𝑚_𝑘=1^𝑚−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘𝑠^𝑛−𝑘+𝑠_𝑛+𝑘.(49) This estimation can also be obtained by letting 𝑘₁=𝑚 in (40) or (46). Again, note that (49) is the optimal solution for the problems 𝑠_𝑛̂=𝐸{𝑠_𝑛∣𝑠_𝑛−𝑖,𝑖≠0,−𝑘₁≤𝑖≤𝑘₂}, where 𝑘₁ and 𝑘₂ can be any integer greater than or equal to 𝑚.

The validity of (49) can also be proved as follows. The error is calculated as 𝑒_𝑛=𝑠_𝑛−𝑠_𝑛̂, where 𝑠_𝑛̂ is from (49). By extending the innovation noises from (4), it can be verified that𝑒_𝑛=𝑠_𝑛−𝑠_𝑛̂=1∑^𝑚𝑘=0𝑎^2𝑘𝐼_𝑛+𝑎₁𝐼_𝑛+1+⋯+𝑎_𝑚𝐼_𝑛+𝑚.(50) Using (6) it is quite clear that 𝐼_𝑛 to 𝐼_𝑛+𝑚 are orthogonal to 𝑠_𝑛−𝑘 for 𝑘>0, and so is 𝑒_𝑛 in (50). Further, we have𝐸=𝑠𝑅_{𝑛+𝑖𝑠𝐼}𝐼_𝑛+𝑎₁𝐼_𝑛+1+⋯+𝑎_𝑚𝐼_𝑛+𝑚[𝑖]+𝑎₁𝑅_𝑠𝐼[𝑖−1]+⋯+𝑎_𝑚𝑅_𝑠𝐼[𝑖−𝑚],𝑖>0.(51) Using (32), (36), (50), and (51), it is seen that 𝐸{𝑒_𝑛𝑠_𝑛+𝑖}=0 for 𝑖>0. This completes the proof.

The MSE is𝐸𝑒^2𝑛=1∑^𝑚𝑘=0𝑎^2𝑘²=𝜎₂⋅𝐸𝐼_𝑛+𝑎₁𝐼_𝑛+1+⋯+𝑎_𝑚𝐼_𝑛+𝑚²∑^𝑚𝑘=0𝑎^2𝑘.(52)

7. Prediction with Finite Data

Assume that only 𝑚−1 consecutive data 𝑠_𝑛−1 to 𝑠_{𝑛−𝑚+1} are available. We want to prove the following.𝑠_𝑛̂==−𝐸𝑠1_𝑛∣𝑠_𝑛−𝑘,1≤𝑘≤𝑚−11−𝑎^{2𝑚𝑚−1}_𝑘=1𝑎_𝑘−𝑎_𝑚𝑎_𝑚−𝑘𝑠_𝑛−𝑘.(53)

The above estimation can be obtained as follows. Since 𝑠_𝑛−𝑚 is not available we can estimate it from data 𝑠_𝑛−1 to 𝑠_{𝑛−𝑚+1}. The estimated value can be now used to predict 𝑠_𝑛 using (16).𝑠_𝑛̂==−𝐸𝑠_𝑛𝑚−1∣𝑠_𝑛−𝑘,1≤𝑘≤𝑚−1_𝑘=1𝑎_𝑘𝑠_𝑛−𝑘−𝑎_𝑚𝑠̂_𝑛−𝑚=−𝑎₁𝑠_𝑛−1−𝑎₂𝑠_𝑛−2−⋯−𝑎_𝑚−1𝑠_{𝑛−𝑚+1}−𝑎_𝑚𝑠̂_𝑛−𝑚.(54) On the other hand, 𝑠_𝑛−𝑚 can be backward predicted using (30) as𝑠̂_𝑛−𝑚==𝐸−𝑎₁𝑠_𝑛−𝑚∣𝑠_𝑛−𝑘,1≤𝑘≤𝑚−1𝑠_{𝑛−𝑚+1}−𝑎₂𝑠_{𝑛−𝑚+2}−⋯−𝑎_𝑚−1𝑠_𝑛−1−𝑎_𝑚𝑠_𝑛̂.(55) Now we have two equations (54) and (55) with two unknowns 𝑠_𝑛̂ and 𝑠̂_𝑛−𝑚. Solving these equations, we get (53). The optimality of (53) can also be proved by seeing that the estimation error is equal to𝑒_𝑛=𝐼^𝑛−𝑎_𝑚𝐼^{𝑛−𝑚}1−𝑎^2𝑚.(56) To derive the above equation, we has used (4) and (13). It is easily seen from (6) and (15) that 𝐼_𝑛 and 𝐼^{𝑛−𝑚} are orthogonal to data 𝑠_𝑛−1 to 𝑠_{𝑛−𝑚+1}. This proves the optimality of (53). To calculate the MSE, we note that𝐸𝑒^2𝑛=𝐸𝑒_𝑛𝑠_𝑛−𝑠_𝑛̂=𝐸𝑒_𝑛𝑠_𝑛.(57) The last equation is justified, as the error is orthogonal to the data and to the estimation which is a linear combination of the data. Inserting (56) in (57), we get𝐸𝑒_𝑛𝑠_𝑛=11−𝑎^2𝑚=1⋅𝐸𝐼_𝑛−𝑎_𝑚𝐼^{𝑛−𝑚}𝑠_𝑛1−𝑎^2𝑚⋅𝑅_𝑠𝐼[0]−𝑎_𝑚𝑅_𝐼^′_𝑠[−𝑚].(58) Finally, using (58), (32), and (36), we have𝐸𝑒^2𝑛=𝜎²1−𝑎^2𝑚.(59)

Higher-order predictions with 𝑚−1 data can be obtained from (53). As an example, we have𝑠_𝑛̂==−𝐸𝑎₁𝑠_𝑛∣𝑠_𝑛−𝑘,2≤𝑘≤𝑚𝑠̂_𝑛−1−^𝑚_𝑘=2𝑎_𝑘𝑠_𝑛−𝑘,(60) where 𝑠̂_𝑛−1 is derived by replacing 𝑛 by 𝑛−1 in (53).

We could not derive a simple general form for the estimation with less than 𝑚−1 data.

8. Interpolation with Finite Data

We now derive the linear interpolation with less than 𝑚 data on each side. More clearly we allege𝑠_𝑛̂==−𝐸𝑠_𝑛∣𝑠_𝑛+𝑘1,−𝑘₂≤𝑘≤𝑘₁,𝑘≠0∑^𝑘1_𝑘=0𝑎^2𝑘−∑^{𝑚𝑘=𝑘}₂₊₁𝑎^2𝑘×⎛⎜⎜⎝^𝑘1_𝑘=_𝑚−𝑘2⎛⎜⎜⎝^𝑘1−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛+𝑘+^{𝑚−𝑘2−1}_𝑘=1⎛⎜⎜⎝^𝑘1−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘−^𝑚−𝑘_𝑝=_𝑘2+1𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛+𝑘+^{𝑚−𝑘1−1}_𝑘=1⎛⎜⎜⎝^𝑘1_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘−^𝑚−𝑘_𝑝=𝑘_2−𝑘+1𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛−𝑘+^𝑘2_𝑘=_𝑚−𝑘1⎛⎜⎜⎝^𝑘2−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛−𝑘⎞⎟⎟⎠.(61) In (61) we must have 𝑘₁+𝑘₂≥𝑚−1 and 𝑘₁≤𝑘₂≤𝑚−1. It means that the distance between 𝑠_𝑛 and the farthest data on the right side is less than the distance between 𝑠_𝑛 and the farthest data on the left side. The optimality of (61) can be seen as we can verify that from (61), (4), and (13) the estimation error is𝑒_𝑛=1∑^𝑘1_𝑘=0𝑎^2𝑘−∑^{𝑚𝑘=𝑘}₂₊₁𝑎^2𝑘⋅𝐼_𝑛+𝑎₁𝐼_𝑛+1+⋯+𝑎_𝑘1𝐼_𝑛+𝑘1−𝑎_𝑚𝐼^{𝑛−𝑚}−𝑎_𝑚−1𝐼^{𝑛−𝑚+1}−⋯−𝑎_𝑘2+1𝐼^{𝑛−𝑘}₂₋₁.(62) It remains to prove that (62) is orthogonal to the data. (1)It is clear from (6) and (15) that 𝐼_𝑛 to 𝐼_𝑛+𝑘1 and 𝐼^{𝑛−𝑚} to 𝐼^{𝑛−𝑘}₂₋₁ are orthogonal to the data 𝑠_𝑛−1 to 𝑠_𝑛−𝑘2. Therefore the error in (62) is orthogonal to 𝑠_𝑛−𝑘 for 1≤𝑘≤𝑘₂. (2)Further from (43) and regarding that 𝐼^{𝑛−𝑚} to 𝐼^{𝑛−𝑘}₂₋₁ are orthogonal to the data 𝑠_𝑛+1 to 𝑠_𝑛+𝑘1 according to (15), we see that the error in (62) is orthogonal to 𝑠_𝑛+𝑘 for 1≤𝑘≤𝑘₁.

Therefore the error is orthogonal to the data and the proof is completed.

From (32), (36), and (62), the MSE is𝐸𝑒^2𝑛=𝐸𝑒_𝑛𝑠_𝑛=1∑^𝑘1_𝑘=0𝑎^2𝑘−∑^{𝑚𝑘=𝑘}₂₊₁𝑎^2𝑘⋅−𝑅𝑎_𝑠𝐼𝑚[0]+𝑎₁𝑅_𝑠𝐼[−1]+⋯+𝑅_𝑠𝐼−𝑘₁−𝑎𝑅_𝐼^′_𝑠𝑘2+1[−𝑚]−𝑎_𝑚−1𝑅_𝐼^′_𝑠[−𝑚+1]⋯=𝑅𝜎_𝐼^′_𝑠−𝑘₂−1₂∑^𝑘1_𝑘=0𝑎^2𝑘−∑^{𝑚𝑘=𝑘}₂₊₁𝑎^2𝑘.(63) For the case 𝑘₁=𝑘₂, 2𝑘₁≥𝑚−1, 𝑘₁≤𝑚−1, we can replace 𝑘₂ by 𝑘₁ in (61) to achieve the following.𝑠_𝑛̂==−𝐸𝑠_𝑛∣𝑠_𝑛+𝑘1,−𝑘₁≤𝑘≤𝑘₁,𝑘≠0∑^𝑘1_𝑘=0𝑎^2𝑘−∑^{𝑚𝑘=𝑘}₁₊₁𝑎^2𝑘⋅⎛⎜⎜⎝^𝑘1_𝑘=_𝑚−𝑘1⎛⎜⎜⎝^𝑘1−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛−𝑘+_{𝑚−𝑘1−1}+𝑠_𝑛+𝑘_𝑘=1⎛⎜⎜⎝^𝑘1−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘−^𝑚−𝑘_𝑝=_𝑘1+1𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛−𝑘+𝑠_𝑛+𝑘⎞⎟⎟⎠.(64) As expected, we see that the data with the same distance from 𝑠_𝑛 participate with the same weight.

Now, consider the case that the distance between 𝑠_𝑛 and the farthest data on the right side is more than the distance between 𝑠_𝑛 and the farthest data on the left side. It can be handled by the symmetry of the problem. More clearly, if we replace 𝑠_𝑛−𝑘 by 𝑠_𝑛+𝑘 and vice versa in (61), we get the following.𝑠_𝑛̂==−𝐸𝑠_𝑛∣𝑠_𝑛−𝑘1,−𝑘₂≤𝑘≤𝑘₁,𝑘≠0∑^𝑘1_𝑘=0𝑎^2𝑘−∑^{𝑚𝑘=𝑘}₂₊₁𝑎^2𝑘×⎛⎜⎜⎝^𝑘1_𝑘=_𝑚−𝑘2⎛⎜⎜⎝^𝑘1−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛−𝑘+^{𝑚−𝑘2−1}_𝑘=1⎛⎜⎜⎝^𝑘1−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘−^𝑚−𝑘_𝑝=_𝑘2+1𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛−𝑘+^{𝑚−𝑘1−1}_𝑘=1⎛⎜⎜⎝^𝑘1_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘−^𝑚−𝑘_𝑝=𝑘_2−𝑘+1𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛+𝑘+^𝑘2_𝑘=_𝑚−𝑘1⎛⎜⎜⎝^𝑘2−𝑘_𝑝=0𝑎_𝑝𝑎_𝑝+𝑘⎞⎟⎟⎠𝑠_𝑛+𝑘⎞⎟⎟⎠.(65) Again in (65), 𝑘₁≤𝑘₂≤𝑚−1 and 𝑘₁+𝑘₂≥𝑚−1. The estimation error in this case is𝑒_𝑛=1∑^𝑘1_𝑘=0𝑎^2𝑘−∑^{𝑚𝑘=𝑘}₂₊₁𝑎^2𝑘⋅𝐼^𝑛+𝑎₁𝐼^𝑛−1+⋯+𝑎_𝑘1𝐼^{𝑛−𝑘}₁−𝑎_𝑚𝐼_𝑛+𝑚−𝑎_𝑚−1𝐼_{𝑛+𝑚−1}−⋯−𝑎_𝑘2+1𝐼_𝑛+𝑘2+1.(66) The MSE is the same as (63). We could not find a simple general form for the case 𝑘₁+𝑘₂<𝑚−1.

9. A Detailed Example

In this section we deal with a detailed example. The optimal linear estimation of the following process is desired.𝑠_𝑛+0.8𝑠_𝑛−1+0.3𝑠_𝑛−2−0.1𝑠_𝑛−3=𝐼_𝑛.(67)𝐼_𝑛 is the innovation noise with the unit variance 𝜎=1. We have 𝑎₁=0.8, 𝑎₂=0.3 and 𝑎₃=−0.1. The process is the response of the following 3rd order (𝑚=3) all-pole filter to the innovation noise.𝐻(𝑧)=11+0.8𝑧⁻¹+0.3𝑧⁻²−0.1𝑧⁻³.(68) The poles of this system are 𝑝₁=0.2 and 𝑝_2,3=−0.5±𝑗0.5. Taking inverse 𝑍-transform from 𝑆_𝑠(𝑧)=𝐻(𝑧)𝐻(𝑧⁻¹), we get the following autocorrelation function.𝑅_𝑠[𝑘]==𝑟_𝑘62=5𝐸𝑠_𝑛𝑠_𝑛−𝑘13542×5^{−∣𝑛∣}+40×2_{−∣𝑛∣/2}2257103cos34𝜋𝑛−26sin34𝜋|𝑛|.(69) From (69), we have 𝑟₀=1.8716, 𝑟₁=−1.1339, 𝑟₂=0.2322, 𝑟₃=0.3415, 𝑟₄=−0.4563, 𝑟₅=0.2858, and 𝑟₆=−0.0576. Now, we consider different cases.

9.1. Prediction with Finite Data

We want to derive the following optimal linear prediction.𝑠_𝑛̂=𝐸𝑠_𝑛∣𝑠_𝑛−1,𝑠_𝑛−2=𝐴₁𝑠_𝑛−1+𝐴₂𝑠_𝑛−2.(70) Using (53), we have𝑠_𝑛̂=−11−0.01(0.8+0.1×0.3)𝑠^𝑛−1=−0.8384𝑠_𝑛−1+(0.3+0.1×0.8)𝑠_𝑛−2−0.3838𝑠_𝑛−2.(71) If we want to verify the solution using the orthogonality equations, we have𝐸𝑠_𝑛−𝐴₁𝑠_𝑛−1−𝐴₂𝑠_𝑛−2𝑠_𝑛−𝑘=0,𝑘=1,2.(72) Expanding (72), we get𝑟𝑟₀₁𝐴₁+𝑟₁𝐴₂=𝑟₁,𝐴₁+𝑟₀𝐴₂=𝑟₂,(73) where 𝑟_𝑘’s come from (69). Replacing 𝑟_𝑘’s from (69) in (73), we get−1.18.711363𝐴9𝐴₁−₁1.1339𝐴₂=−1.1339,+1.8716𝐴₂=0.2322,(74) Solving (74), we get the same result as (71).

9.2. Interpolation with Finite Data

Consider the following problem.𝑠_𝑛̂=𝐸𝑠_𝑛∣𝑠_𝑛−1,𝑠_𝑛+1=𝐴₁𝑠_𝑛−1+𝐴^1𝑠_𝑛+1(75) It is the symmetric case of 𝑘₁=𝑘₂=1 and we have 2𝑘₁=2=𝑚−1. Using (64), we have𝑠_𝑛̂=−[1×0.8−0.3×(−0.1)]1+0.64−0.09−0.01𝑠^𝑛−1=−0.5390𝑠_𝑛−1+𝑠_𝑛+1+𝑠_𝑛+1.(76) Let us rederive the solution of (75) using the orthogonality conditions. We have𝐸𝑠_𝑛−𝐴₁𝑠_𝑛−1−𝐴^1𝑠_𝑛+1𝑠_𝑛−𝑘=0,𝑘=1,−1.(77) Expanding (77), we get the following.𝑟𝑟₀₂𝐴₁+𝑟₂𝐴^1=𝑟₁,𝐴₁+𝑟₀𝐴^1=𝑟₁.(78) Solving (78), we get the same answer as (76).

Now, consider the nonsymmetric following problem.𝑠_𝑛̂=𝐸𝑠_𝑛∣𝑠_𝑛−2,𝑠_𝑛−1,𝑠_𝑛+1=𝐴₁𝑠_𝑛+1+𝐴^1𝑠_𝑛−1+𝐴^2𝑠_𝑛−2(79) which is the case of 𝑘₁=1<𝑘₂=2≤𝑚−1, and 𝑘₁+𝑘₂≥𝑚−1. From (61), we get the following results.𝑠_𝑛̂=−1⋅11×+00..86𝑠4−_𝑛+10.01×𝑠_𝑛−1+(1×0.8+0.8×0.3−0.3×(−0.1))=−0.4908+𝑠1_𝑛+1×0.3𝑠_𝑛−2−0.6564𝑠_𝑛−1−0.1840𝑠_𝑛−2.(80) Now, we want to obtain the solution of (79) using the matrix equations and we expect the same answer as (80). The orthogonality condition is𝐸𝑠_𝑛−𝐴₁𝑠_𝑛+1−𝐴^1𝑠_𝑛−1−𝐴^2𝑠_𝑛−2𝑘𝑠=−_𝑛−𝑘1,=1,02,.(81) It follows that𝑟𝑟₀₂𝐴₁+𝑟₂𝐴^1+𝑟₃𝐴^2=𝑟₁,𝑟₃𝐴₁+𝑟₀𝐴^1+𝑟₁𝐴^2=𝑟₁,𝐴₁+𝑟₁𝐴^1+𝑟₀𝐴^2=𝑟₂.(82) The result of (82) is the same as (80).

9.3. Interpolation with Infinite Data on the Left Side

We want to obtain the following estimation.𝑠_𝑛̂==𝐸𝐴₁𝑠_𝑛∣𝑠_𝑛+𝑖,𝑖≤1,𝑖≠0𝑠_𝑛+1+𝐴^1𝑠_𝑛−1+𝐴^2𝑠_𝑛−2+𝐴^3𝑠_𝑛−3.(83) We can do it if we let 𝑘₁=1 in (40). It follows that𝑠_𝑛̂=−1⋅11×+00..86𝑠4_𝑛+1+(1×0.8+0.8×0.3)𝑠_𝑛−1=−+0.4(187×80𝑠.3_𝑛+1+0.8×(−0.1))𝑠_𝑛−2+1×(−0.1)𝑠_𝑛−3−0.6341𝑠_𝑛−1−0.1341𝑠_𝑛−2+0.0610𝑠_𝑛−3.(84)

Now we verity (84) using the orthogonality conditions.𝐸𝑠_𝑛−𝐴₁𝑠_𝑛+1−𝐴^1𝑠_𝑛−1−𝐴^2𝑠_𝑛−2−𝐴^3𝑠𝑘_𝑛−3=𝑠−1,_𝑛−𝑘1,=2,03,.(85) The following set of equations is obtained𝑟𝑟₀₂𝐴₁+𝑟₂𝐴^1+𝑟₃𝐴^2+𝑟₄𝐴^3=𝑟₁,𝑟₃𝐴₁+𝑟₀𝐴^1+𝑟₁𝐴^2+𝑟₂𝐴^3=𝑟₁,𝑟₄𝐴₁+𝑟₁𝐴^1+𝑟₀𝐴^2+𝑟₁𝐴^3=𝑟₂,𝐴₁+𝑟₂𝐴^1+𝑟₁𝐴^2+𝑟₀𝐴^3=𝑟₃.(86)

Note that the coefficient matrix of (86) is not Toeplitz. The result of (86) is the same as (84).

10. Conclusion

We introduced anticausal LTI model besides the known causal LTI model for AR processes. Using these models and the related innovation noises, we achieved the optimal linear interpolations for different cases. Specifically, we extracted the formulae when there are infinite data on the right, or the left sides of the variable to be estimated. We also obtained the linear prediction or interpolation with finite data. The number of data must be at least the order of the process minus one. We could not find a general simple form when fewer data are available. For the proofs of our solutions, the innovation noises and the orthogonality principle are essential.