Research Article  Open Access
Baogui Xin, Wei Peng, "Prediction for Chaotic Time SeriesBased AECNN and Transfer Learning", Complexity, vol. 2020, Article ID 2680480, 9 pages, 2020. https://doi.org/10.1155/2020/2680480
Prediction for Chaotic Time SeriesBased AECNN and Transfer Learning
Abstract
It has been a hot and challenging topic to predict the chaotic time series in the mediumtolong term. We combine autoencoders and convolutional neural networks (AECNN) to capture the intrinsic certainty of chaotic time series. We utilize the transfer learning (TL) theory to improve the prediction performance in mediumtolong term. Thus, we develop a prediction scheme for chaotic time seriesbased AECNN and TL named AECNNTL. Our experimental results show that the proposed AECNNTL has much better prediction performance than any one of the following: AECNN, ARMA, and LSTM.
1. Introduction
Chaos is the unity of extrinsic randomness and intrinsic certainty, which widely exists in social fields and natural systems. However, in many cases, we cannot represent chaotic systems by establishing analytical mathematical models but by observing the time series [1]. With the development of big data acquisition and research technology, we can obtain largescale chaotic time series from a wide range of social fields and natural systems, which highlight the necessity of research on the prediction of chaotic time series. Of course, the prediction of chaotic systems has always been a hot and challenging topic in many fields. However, the acquisition of chaotic time series data usually accompanies errors, which make the prediction of chaotic time series more difficult.
The most significant characteristic of a chaotic system is its initial sensitivity; that is, a slight change in input will produce random and unpredictable difference after long enough evolution, which determines the difficulty in mediumtolongterm prediction for a chaotic system. Similarly, the error in chaotic time series may be amplified to an unacceptable degree due to its initial sensitivity, which will lead to the failure of prediction. However, the orbit of the chaotic system also has intrinsic certainty such as selfsimilarity, which indicates that a chaotic system can be predicted to a certain extent, even in the mediumtolong term by using fractal geometry and other related theories. But the selfsimilarity and other intrinsic certainty of a chaotic system are very difficult to be measured accurately. So far, there is no commonly accepted theory that can be employed to accurately measure the intrinsic certainty such as selfsimilarity and realize the mediumtolongterm prediction of a chaotic system. Inspired by Xia et al. [2], we try to overcome the following difficulties to effectively predict the mediumtolongterm time series: (1) how to make full use of the intrinsic certainty of chaotic time series? It needs us to propose or borrow a universal approximation framework [3], which has a powerful representation of deep learning and can extract highorder features from input information for mediumtolongterm prediction of chaotic time series; (2) how to extract intrinsic highorder features from a chaotic time series and transfer them to its coupled time series to strengthen its prediction performance.
To solve abovementioned difficulties, we will mainly implement the following contributions:(1)Employ a convolutional neural network (CNN) to extract highorder features of a chaotic time series for mediumtolongterm prediction. The CNN has a strong representation learning ability, so we can use it to classify the input information according to its hierarchical structure and extract highorder features from the input information.(2)Utilize the transfer learning (TL) theory to improve the mediumtolongterm prediction performance. We extract the intrinsic highorder features from the chaotic time series and use the transfer learning theory to overlay them with their highorder features to enhance its prediction performance.
The remainder of this article is organized as follows: in Section 2, we review the related work; in Section 3, we describe the proposed the AECNNTL in detail; in Section 4, we implement some experiments and discuss the results; and in Section 5, we draw a conclusion and propose some future works.
2. Related Work
2.1. Classification of Prediction Methods for Chaotic Time Series
According to the step number of prediction, we can divide the prediction methods into the singlestep prediction [4] and multistep prediction [5, 6]. In the singlestep prediction, the model can be only used to predict the sequence values of the next step. In the multistep prediction, the model can be employed to predict the sequence values of the multistep, which sets higher requirements. There are two strategies of the multistep prediction: direct approach and iterative approach. Based on the direct approach, the prediction of chaotic time series can complete longterm prediction once by constructing a function mapping between input and multistep output. The idea of the direct approach is simple and easy to implement, but the direct approach needs more complex function mapping structure, which is a significant challenge. The iterative approach is to predict the time series value after only one step at a time and then take the predicted value as known and finally iterate to predict the next step. The iterative approach has low requirements for the model, but in the iterative process, the prediction error is also input into the model, which results in the error accumulation and the prediction accuracy decrease in mediumtolong term. The proposed AECNNTL constructed in this article belongs to the direct approach.
2.2. Prediction of Chaotic Time Series Based on Deep Learning of Direct Strategies
Taking accurate multistep predictions with a novel recurrent neural network (RNN), Min et al. [5] constructed a new scheme to implement longterm prediction. Yi et al. [7] introduced a deep neural network (DNN) based method to predict the air quality in medium term. Kuo and Huang [8] presented a DNN model (EPNet), which combines CNN and LSTM to predict the electricity price in medium term. Mujeeb et al. [9] built a deep LSTM model for short and mediumterm prediction. Shen et al. [10] set up a prediction model for time series named SeriesNet, which includes an LSTM and a dilated causal convolution network. Mudassir et al. [11] established the highperformance machine learning (ANN, SANN, SVM, and LSTM) based classification and regression models to predict Bitcoin price in shorttomedium term. Hussain et al. [12] put forward a onedimensional convolutional neural network (1DCNN) and extreme learning machine (ELM) to predict the onestepahead streamflow in short and medium terms. Chimmula and Zhang [13] constructed an LSTM network to predict COVID19 cases. Livieris et al. [14] designed a CNNLSTM model to predict gold prices. Zhang and Dong [15] provided a CRNN model consisting of CNN and RNN to predict temperature changes from historical data. Li [16] proposed the KLS method fusing Kalman filter, LSTM, and SVM to forecast the carbon emission. Combining multikernel convolutional layers and convolutional LSTM layers, Asadi and Regan [17] propose a deep learning framework for spatiotemporal forecasting problems.
3. Proposal
As we know, we can employ an autoencoder (AE) to learn a representation (encoding) for a time series by training the network to ignore signal “noise,” i.e., denoising. Considering CNN’s advantages of shift invariant or space invariant, the CNN is most commonly applied to classify images, cluster images by similarity, and perform object recognition within scenes. In this article, we will employ CNN to extract the intrinsic highorder features including similarity from the chaotic time series to implement predictions. By combining the advantages of AE and CNN, we can use AECNN to predict a chaotic time series, as shown in Figure 1.
If there are a chaotic time series Y associated with X, we can extract the highorder intrinsic features from Y and utilize the transfer learning theory to overlay with the highorder intrinsic features from X to enhance the prediction performance. Thus, we can develop a prediction scheme for chaotic time series of mediumtolongtermbased AECNN and transfer learning, named AECNNTL, as shown in Figure 2.
We assume that both time series and with length , where and .
From Figure 2, AECNNTL consists of four processes: denoising, feature extracting, transferring learning, and forecasting. At the denoising stage, a typical AE is firstly used for data filtering so as to remove the noises from the chaotic time series. At the feature extracting stage, the denoised chaotic time series and original time series would be taken as the inputs of CNNs to extract deep representations. On the transferring learning stage, the deep representations from can be transferred to them from and overlapped together. At the forecasting stage, we would obtain the prediction of through a fully connected layer.
The optimization of AECNNTL is to minimize the reconstruction error of AE and the training error of the whole model. At the denoising stage, the output of AE is an approximate copy of input. Therefore, we have to minimize the reconstruction error between input and output, which could maintain the direct significance of temperature. To obtain a sound model, it is necessary to minimize the training error of the whole model. We formulate the objective functions aswhere and are activation functions, and are weights, and and are biases.
It is necessary to obtain a sound model to minimize the training error of the whole model. We formulate the objective function of the whole model aswhere denotes the predicted value.
4. Experiments and Results
4.1. Data
In this article, we use the Beijing PM2.5 dataset (https://archive.ics.uci.edu/ml/datasets/Beijing+PM2.5+Data). The hourly data span the period between January 1, 2010, and December 31, 2014. Each series in this dataset contains 43824 data points. After removing missing values, 41757 data points were finally used in the experiment. The time series we used is PM2.5 concentration () and temperature (°C). The descriptive statistics for PM2.5 concentration and temperature is shown in Table 1.

To obtain a desirable model, we divide the experimental data into three parts: 70% training dataset, 10% validation dataset, and 20% test dataset. The training dataset is to reach a sound model, the validation dataset is to further determine the parameters of the whole network, and the test dataset is to test the generalization ability of the model.
4.2. Evaluation Index
The root mean square error (RMSE) is a widely used evaluation index for continuous type forecasting. Generally speaking, the smaller the RMSE value is, the better its prediction performance is. The RMSE is defined as follows:where denotes the length of test dataset.
4.3. 01 Test for Chaos in the Beijing PM2.5 Dataset
According to the 01 test algorithm for chaos, we sample data points from the Beijing PM2.5 dataset after taking every 10^{th} data point and set a random number drawn from a uniform distribution on satisfying the following translation equations:where denotes the time series of PM2.5 or temperature.
Then, we calculate the following test value of chaos:where , , and the covariance is written with vectors and of length as follows:
Figure 3 shows plots of the test value vs. random number for the time series of temperature. implies that there is a chaotic attractor in the time series of temperature.
Figure 4 shows Brownianlike trajectories and denotes there is also a chaotic attractor in the time series of temperature. That is to say, the time series of temperature is chaotic.
Similar to Figure 3, Figure 5 also shows plots of the test value vs. random number for the PM2.5 concentration time series. also implies there is a chaotic attractor in the time series of PM2.5 concentration.
Similar to Figure 4, Figure 6 also shows Brownianlike trajectories and denotes there is also a chaotic attractor in the time series of PM2.5 concentration. That is to say, the time series of PM2.5 concentration is chaotic.
Figure 7 demonstrates the phase portrait of temperature vs. PM2.5 concentration, in which Brownianlike (unbounded) trajectories imply there exists a chaotic attractor.
In short, Figures 3–7 prove that there exist chaotic attractors in the Beijing PM2.5 dataset.
4.4. Experimental Results
In this part, we test the prediction performance of the proposed AECNNTL for chaotic time series.
To compare the prediction performance of AECNN and AECNNTL, we firstly employ AECNN to predict PM2.5 concentration without TL from temperature, as shown in Figure 8. Then, we utilize AECNNTL to predict PM2.5 concentration, as shown in Figure 9. In the same way, we firstly employ AECNN to predict temperature without TL from PM2.5 concentration, as shown in Figure 10. Then, we utilize AECNNTL to predict temperature, as shown in Figure 11. Owing to the continuous value prediction, we employ RMSE to measure the prediction performance. The smaller the RMSE value is, the better the prediction performance is. And these corresponding RMSEs of prediction are shown in Table 2.

From Table 2, we can catch that the prediction accuracy of PM2.5 concentration is enhanced to a large extent with taking TL from temperature. And the prediction accuracy of temperature is also improved significantly with taking TL from PM2.5 concentration. Therefore, we can conclude that AECNNTL is much better in the prediction performance than any one of the following: AECNN, ARMA, and LSTM.
5. Conclusions
In this article, we develop a prediction scheme for chaotic time series of mediumtolongtermbased AECNN and TL, named AECNNTL, which has much better prediction performance than any one of the following: AECNN, ARMA, and LSTM. This prediction scheme can be used in chaotic time series of mediumtolong term of many natural fields and social systems in the real world.
In fact, AECNNTL can also be used to reveal the dynamic link between PM2.5 concentration and temperature. What is more, AECNNTL can be used to explore the Granger causality [18] between PM2.5 concentration and predict PM2.5 concentration and temperature with a good performance. By means of experiments, we find a bidirectional Granger causality between PM2.5 concentration and temperature, which is consist with previous researches.
We know that the Beijing PM2.5 dataset originates from the nature field. In fact, we can try to employ the proposed AECNNTL to predict a chaotic time series of mediumtolong term which comes from some artificial systems, such as the game system [19–21] and the financial system [22].
Data Availability
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation, to any qualified researcher.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this article.
Acknowledgments
This work was supported by the Natural Science Foundation of Shandong Province (Grant no. ZR2016FM26) and the National Social Science Foundation of China (Grant no. 16FJY008).
References
 J. Ma, Y. Chen, and B. Xin, “Study on prediction methods for dynamic systems of nonlinear chaotic time series,” Applied Mathematics and Mechanics, vol. 25, pp. 605–611, 2004. View at: Publisher Site  Google Scholar
 M. Xia, W. A. Liu, K. Wang, X. Zhang, and Y. Xu, “Nonintrusive load disaggregation based on deep dilated residual network,” Electric Power Systems Research, vol. 170, pp. 277–285, 2019. View at: Publisher Site  Google Scholar
 K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, no. 5, pp. 359–366, 1989. View at: Publisher Site  Google Scholar
 W. Ji and K. C. Chee, “Prediction of hourly solar radiation using a novel hybrid model of ARMA and TDNN,” Solar Energy, vol. 85, no. 5, pp. 808–817, 2011. View at: Publisher Site  Google Scholar
 H. Min, X. Jianhui, X. Shiguo, and Y. FuLiang, “Prediction of chaotic time series based on the recurrent predictor neural network,” IEEE Transactions on Signal Processing, vol. 52, pp. 3409–3416, 2004. View at: Publisher Site  Google Scholar
 X. Wu, Y. Wang, J. Mao, Z. Du, and C. Li, “Multistep prediction of time series with random missing data,” Applied Mathematical Modelling, vol. 38, no. 14, pp. 3512–3522, 2014. View at: Publisher Site  Google Scholar
 X. Yi, J. Zhang, Z. Wang, T. Li, and Y. Zheng, “Deep distributed fusion network for air quality prediction,” in Proceedings of Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 965–973, London, UK, 2018. View at: Publisher Site  Google Scholar
 P.H. Kuo and C.J. Huang, “An electricity price forecasting model by hybrid structured deep neural networks,” Sustainability, vol. 10, no. 4, p. 1280, 2018. View at: Publisher Site  Google Scholar
 S. Mujeeb, N. Javaid, M. Ilahi, Z. Wadud, F. Ishmanov, and M. Afzal, “Deep long shortterm memory: a new price and load forecasting scheme for big data in smart cities,” Sustainability, vol. 11, no. 4, p. 987, 2019. View at: Publisher Site  Google Scholar
 Z. Shen, Y. Zhang, J. Lu, J. Xu, and G. Xiao, “A novel time series forecasting model with deep learning,” Neurocomputing, vol. 396, pp. 302–313, 2020. View at: Publisher Site  Google Scholar
 M. Mudassir, S. Bennbaia, D. Unal, and M. Hammoudeh, “Timeseries forecasting of bitcoin prices using highdimensional features: a machine learning approach,” Neural Computing and Applications, 2020. View at: Publisher Site  Google Scholar
 D. Hussain, T. Hussain, A. A. Khan, S. A. A. Naqvi, and A. Jamil, “A deep learning approach for hydrological timeseries prediction: a case study of gilgit river basin,” Earth Science Informatics, vol. 13, no. 3, pp. 915–927, 2020. View at: Publisher Site  Google Scholar
 V. K. R. Chimmula and L. Zhang, “Time series forecasting of covid19 transmission in Canada using LSTM networks,” Chaos, Solitons & Fractals, vol. 135, Article ID 109864, 2020. View at: Publisher Site  Google Scholar
 I. E. Livieris, E. Pintelas, and P. Pintelas, “A CNNLSTM model for gold price timeseries forecasting,” Neural Computing and Applications, 2020. View at: Publisher Site  Google Scholar
 Z. Zhang and Y. Dong, “Temperature forecasting via convolutional recurrent neural networks based on timeseries data,” Complexity, vol. 2020, Article ID 3536572, 8 pages, 2020. View at: Publisher Site  Google Scholar
 Y. Li, “Forecasting Chinese carbon emissions based on a novel time series prediction method,” Energy Science & Engineering, vol. 8, no. 7, pp. 2274–2285, 2020. View at: Publisher Site  Google Scholar
 R. Asadi and A. C. Regan, “A spatiotemporal decomposition based deep neural network for time series forecasting,” Applied Soft Computing, vol. 87, Article ID 105963, 2020. View at: Publisher Site  Google Scholar
 W. Peng, “DLI: a deep learningbased granger causality inference,” Complexity, vol. 2020, Article ID 5960171, 8 pages, 2020. View at: Publisher Site  Google Scholar
 B. Xin, F. Cao, W. Peng, and A. A. Elsadany, “A bertrand duopoly game with longmemory effects,” Complexity, vol. 2020, Article ID 2924169, 7 pages, 2020. View at: Publisher Site  Google Scholar
 B. Xin, W. Peng, and Y. Kwon, “A discrete fractionalorder cournot duopoly game,” Physica A: Statistical Mechanics and its Applications, vol. 558, Article ID 124993, 2020. View at: Publisher Site  Google Scholar
 B. Xin and M. Sun, “A differential oligopoly game for optimal production planning and water savings,” European Journal of Operational Research, vol. 269, no. 1, pp. 206–217, 2018. View at: Publisher Site  Google Scholar
 B. Xin, W. Peng, Y. Kwon, and Y. Liu, “Modeling, discretization, and hyperchaos detection of conformable derivative approach to a financial system with market confidence and ethics risk,” Advances in Difference Equations, vol. 2019, no. 1, p. 138, 2019. View at: Publisher Site  Google Scholar
Copyright
Copyright © 2020 Baogui Xin and Wei Peng. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.