Abstract
Echo state networks (ESNs), as efficient and powerful computational models for approximating nonlinear dynamical systems, have been successfully applied in financial time series forecasting. Reservoir constructions in standard ESNs rely on trials and errors in real applications due to a series of randomized model building stages. A novel form of ESN with deterministically constructed reservoir is competitive with standard ESN by minimal complexity and possibility of optimizations for ESN specifications. In this paper, forecasting performances of deterministic ESNs are investigated in stock price prediction applications. The experiment results on two benchmark datasets (Shanghai Composite Index and S&P500) demonstrate that deterministic ESNs outperform standard ESN in both accuracy and efficiency, which indicate the prospect of deterministic ESNs for financial prediction.
1. Introduction
Prediction in financial or stock markets is a challenging research topic since the stock market is mostly complex (nonlinear) and volatile. Various technical, fundamental, and statistical indicators have been proposed; however, none of them or combination of techniques has been successful enough. The application of artificial neural network (ANN) in prediction problems is very promising for its general function approximation property and effective learning algorithms. Among them, recurrent neural networks (RNNs) offer arbitrary approximation for dynamical systems with arbitrary precision theoretically compared to other traditional ANNs. However, it is difficult for traditional RNNs based on gradient descent approach to develop in application because of slow convergence and high computational cost. A new form of RNN training methods, echo state network (ESN), has been proposed by Jaeger and Haas [1], which is simple and applicable for time series prediction with high accuracy and computational efficiency. During these ten years, ESNs have been applied in many areas, including time series forecasting [2–7], wireless communication [1], robot control [8], and speech recognition [9]. Lin et al. investigated the effectiveness of ESN to predict the future stock prices in a short term. The experimental results on nearly all stocks in S&P 500 indicated that ESNs outperformed some conventional neural networks in most cases [10, 11]. Although it is verified that standard ESNs can predict future stock prices, the parameters have been determined by trials and errors due to their randomly constructed reservoir. Recently, a deterministically constructed form of ESN has been proposed, which has competitive performance with minimum complexity compared to standard model [12]. Since deterministic ESN model is relatively new, few application researches have been studied until now [13, 14]. This study investigates the performance of deterministic ESN for time series forecasting and its application of stock price prediction. The experiments are based on two benchmark datasets (Shanghai Composite Index and S&P 500) and three types of deterministic ESNs (DLR, DLRB, and SCR) are compared with standard ESNs in forecasting accuracy, stability, and efficiency. The experimental results demonstrate that deterministic ESNs outperform standard ESN in both accuracy and efficiency, which indicate the prospect of deterministic ESNs for financial prediction.
Section 2 briefly introduces standard ESNs and deterministic ESNs; Section 3 presents the experiments and the comparison results of applying deterministic ESNs for stock price forecasting on Shanghai Composite Index and S&P 500 stock datasets. Finally, it is concluded in Section 4.
2. Background
2.1. Echo State Networks
Echo state network is a recurrent discrete-time neural network with input, internal units (dynamic reservoir), and output. In the dynamic reservoir, there are large numbers of sparsely connected neurons where the weights are random and fixed. The states of the dynamic reservoir, called “echo states,” reflect history information of inputs. The memoryless readout is the only part that needs to be trained linearly with echo states; thus the training becomes a simple linear regression, which makes the application of RNNs fast and easy.
2.1.1. Architecture
The basic architecture of ESN with K input units, N neurons in the dynamic reservoir, and L output units is shown in Figure 1. The solid arrows in Figure 1 represent the synaptic connections whose weights are randomly generated and fixed, while the dashed arrows are the trainable output weights.
2.1.2. Training Algorithm
Input. A training set of (input/output) , , and are -dimensional and -dimensional vectors.
Output. A trained ESN is such that its output of the input is close to desired output , where , , , and are input, internal, feedback, and output weight matrices, respectively, with suitable sizes.(1)Initialize a random RNN which has the echo state property (ESP); spectral radius of should be smaller than unity.(2)Activate the reservoir to the dynamic -dimensional states of the RNN by where is a state activation function which is usually a sigmoid function and is the internal recurrent connection weight matrix, of which the spectral radius should be smaller than unity in order to ensure the ESN to work; in other words, should satisfy the ESP.(3)Compute the output weights as the linear regression weights of -dimensional outputs on the states by where is an output activation function.
2.2. Deterministic Echo State Networks
As explained previously, the reservoirs in standard ESNs are constructed randomly so that it is hard to construct ESN and unlikely to optimize the setup. To simplify the reservoir construction, deterministically constructed ESNs (Figure 2) have been proposed [12].
(a)
(b)
(c)
Three reservoir templates with fixed topologies concerned in this study are described as follows.(1)Delay line reservoir (DLR): the units in reservoir are arranged in a line, as shown in Figure 2(a). Obviously, the reservoir matrix has the property that nonzero values are located on the lower subdiagonal of ; that is, for , where is the weight of all the feedforward connections.(2)DLR with backward connections (DLRB): each neuron in reservoir is connected to both the preceding neuron and the successive one, as shown in Figure 2(b). Thus, the lower subdiagonal and upper subdiagonal of both have nonzero elements, and , where is the weight of the feedback connections.(3)Simple cycle reservoir (SCR): neurons in reservoir like in Figure 2(c) are organized in a cycle, where and .
Except the reservoir construction, the input layer in deterministic ESN is fully connected to the reservoir and all the input connections have the same absolute weight value in contrast to randomly generated input weights in standard ESN. The sign of is determined randomly by a random draw from Bernoulli distribution of mean 1/2.
Generally, the deterministic form of ESNs has the following properties, which simplifies the ESN construction by only setting two free parameters and enables a more thorough theoretical analysis of the ESN performance [12]:(1)a simple fixed nonrandom reservoir topology with full connectivity from inputs to the reservoir,(2)a single fixed absolute weight value for all reservoir connections, and(3)a single weight value for input connections, with aperiodic pattern of input signs.
3. Stock Price Forecasting Based on Deterministic Echo State Networks
In this study, to investigate performances of deterministic ESNs for stock price prediction, three types of deterministic ESNs described previously, DLR, DLRB, and SCR, are compared to standard ESN. Since stock price fluctuates violently, in order to make the experiments fair and sound, two benchmark datasets from Shanghai Composite Index and S&P 500 are adopted in this study.
3.1. Experimental Data
1. Shanghai Composite Index Stock Price Dataset. The trading price data from 14 December, 2007 to 31 December, 2012 (totally 1230 days) of fifty stocks in the benchmark Shanghai composite index is adopted.
2. S&P 500 Stock Price Dataset. S&P 500 is usually considered as the benchmark for the United States’ equity performance; hence it is popular for stock price prediction research. The trading price data from 1 June, 2010 to 20 February, 2014 (totally 930 days) of 374 stocks in S&P 500 is also adopted in the experiments. All data come from Yahoo Finance.
In the experiments, 80% of data points in each stock price data are adopted as training set, and the remaining 20% are used for testing.
3.2. Data Preprocessing
In most time series prediction researches, the original data are usually normalized in order to make the preprocessed data in a desired range. In this study, all the original data are normalized as follows: where is the original data, is the minimum data, and is the maximum value in the data, while and are two coefficients to make the preprocessed data in a desired range.
3.3. Prediction Accuracy Evaluation Measure
In this study, a commonly used measure, mean squared error (MSE) shown as follows, is utilized to evaluate forecasting accuracy: where denotes a given time series, , , and denotes the forecast of value at time .
3.4. Experiments and Results
3.4.1. Experiments on Shanghai Composite Index Stock Price Dataset
In this experiment, each ESN model (standard ESN, DLR, DLRB, and SCR) is built with 96 internal units and spectral radius is set to be 0.8. For each stock in this dataset, the next trading day closing price is predicted by its previous 30 closing prices. These forecasting models are investigated according to different values of input weight matrix .
Random Input Weight Matrix . The input weight matrix is randomly set to be in the uniform distributed interval of . The parameter in internal weight matrix of DLR, DLRB, and SCR models is set to be , respectively. Accordingly, the parameter in of DLRB is set to be .
Because of the randomness of , each forecasting model is tested for 1000 times. Each deterministic ESN model is compared with standard ESN model by average forecasting accuracy. The experimental results are shown in Tables 1, 2, and 3. In these tables, the stocks of which the predicting performances are improved as the parameters increase are italic, while reduced ones are with asterisks. From the comparative results, the forecasting performances of DLR, DLRB, and SCR are increased by 19.90%, 19.48%, and 19.71%, respectively, which indicate that the deterministic ESNs outperform the standard ESN in forecasting accuracy. In addition, it is found that the forecasting performances of deterministic ESNs tend to change in one direction (increase or decrease) as the parameter values increase.
Deterministic Input Weight Matrix . The forecasting performances of these ESN models are compared under the condition of the same value. In this experiment, all the values in are , and the sign of is the probability of 50% randomly determined. In DLR, DLRB, and SCR, all the values in internal weight matrix are and the feedback connection values in of DLRB are . The average forecasting accuracy in MSE for 1000 times tests for each ESN model is shown in Table 4, which indicates that SCR, DLR, and DLRB outperform standard ESN in average accuracy by 15.94%, 15.93%, and 15.51%, respectively. During the experiment, for the first 500 tests, it is found that the values of forecasting accuracy (MSE) for the ESN models differ significantly, which indicate that the forecasting performances of the ESN models are unstable. To further investigate the stability, another 500 tests have been done, of which the results also show the instability. Therefore, the stability of each ESN model is evaluated by MSE for the 1000 tests. As shown in Table 4, compared to the standard ESN model, the stabilities of SCR, DLRB, and DLR models are improved by 53.17%, 52.73%, and 52.70%, respectively.
3.4.2. Experiments on S&P 500 Stock Price Dataset
The forecasting models in this experiment are constructed the same as the experiment described in Section 3.4.1. This experiment aims to further investigate the forecasting performance of each ESN model for various stocks since there are more representative stocks in S&P 500 dataset than in Shanghai Composite Index dataset. Meanwhile, time complexities are compared by computing the average running time. The experimental results are shown in Table 5 partly. Although the ESN models perform competitively in forecasting accuracy, the deterministic ESN models have obvious advantages in efficiency which is indicated by the average time saving of 23.46%, 23.45%, and 23.41% for SCR, DLRB, and DLR, respectively.
4. Conclusions
A new method for stock price forecasting by applying deterministic ESN models is presented in this paper. In standard ESNs, the reservoirs are randomly constructed, which makes researchers and practitioners rely on trials and errors. Differently, the deterministic ESNs have their reservoirs constructed deterministically so that the structures of ESN models are simple and easy to be applied. Therefore, deterministic ESNs (three typical types) have been applied to predict stock price. The forecasting performances have been investigated on two benchmark datasets (Shanghai Composite Index and S& P500). The experimental results on Shanghai Composite Index dataset show that deterministic ESNs outperform standard ESN in accuracy by about 20% and stability by 52% averagely. Further investigation on S&P 500 dataset shows that the deterministic ESNs have improvement in efficiency by about 23% while having insignificant improvement in forecasting accuracy. Since deterministic ESNs have advantages of higher forecasting accuracy and efficiency as indicated by the experimental results, they have great prospects in applications of stock price prediction and even other time series forecasting applications. In this work, the deterministic ESN models are constructed by hand. The problem of parameter settings will be considered in future work by applying optimization algorithms.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
The research work is partially supported by the Fundamental Research Funds for the Central Universities in China (CDJZR12180004) and the Major State Basic Research Development Program of China (973 Program) (no. 2013CB329100).