Abstract
Accurate solar irradiance prediction plays an important role in ensuring the security and stability of renewable energy systems. Solar irradiance modeling is usually a timedependent dynamic model. As a new kind of recurrent neural network, echo state network (ESN) shows excellent performance in the field of time series prediction. However, the memory length of classical ESN is fixed and finite, which makes it hard to map sufficient features of solar irradiance with longrange dependency. Therefore, a novel deep echo state network with variable memory pattern (VMPDESN) is proposed in this brief. VMPDESN consists of multiple connected reservoirs in series, and there exist different types of memory modes in VMPDESN. To remember more input history information in the states, the time delay links are added in each reservoir and between every two reservoirs. The VMPDESN is more flexible to deal with different input signals due to its variable memory modes in the reservoir states. Additionally, the effect of different memory patterns on the VMPDESN performance is discussed in detail, including the antidisturbance ability, memory capacity, and prediction accuracy. Finally, the effectiveness of VMPDESN is evaluated by predicting the real solar irradiance task.
1. Introduction
In recent years, accurate solar irradiance prediction has played an increasingly important role in the management, dispatch, and security of renewable energy systems [1]. To predict the solar irradiance series, various techniques have been proposed, such as statistical models [2, 3], artificial neural network (ANN) [4, 5], hybrid methods [6, 7], and so forth. Among those approaches, ANN has become the most popular method for solar irradiance prediction due to its strong nonlinear approximation ability [8].
Among ANN, one powerful algorithm is the recurrent neural network (RNN) [9]. Compared with feedforward neural network (FNN), the RNN has dynamic characteristics and memory performance, by introducing connection loops in the hidden layer. Thus, it is more appropriate for RNN to model the timedependent solar irradiance series than FNN. Echo state network, a simple yet improved variant of RNN [10], adopts a dynamic reservoir as the hidden layer. The dynamic reservoir contains a large number of randomly sparsely connected neurons, which can encode the input signal from the lowdimensional input space to the highdimensional state space. In addition, only the output connection weights in ESN need to be trained by linear regression algorithms [11], while the inputreservoir and reservoirreservoir weights are usually initialized randomly and remain unchanged in the training process. Therefore, ESN avoids the drawbacks of high computational complexity and gradient disappearance in traditional RNN. Due to those advantages, ESN has been widely applied to time series prediction [12–14], pattern recognition [15, 16], and control fields [17–19].
For those time series modeling with longterm dependency, the traditional ESN still shows limitations. On the one hand, the reservoir state update equation of ESN is fixed, which can only express the relation between partial input history and the current state. However, the input features far away from the current time cannot be preserved in the current states of ESN. If the states cannot contain enough input history, then the expected value will not be accurately fitted in output layer. On the other hand, the traditional ESN only contains one reservoir layer, which makes it difficult to fully extract input features. Thus, different variants of ESN have been developed for the performance improvement, for instance, expanding singlereservoir to multiple reservoirs [20–23], changing the state update rules of the reservoir [24–26], and so on. The authors in [20, 21, 23] propose deep ESN (DESN) with multiple reservoirs in series array, which can process the input signal layer by layer. The experimental results show that the DESN has better prediction accuracy than traditional ESN due to the strong feature extraction ability of its deep topology. In [24, 25], some improved ESNs with variable state update equation are proposed, by adding the leaky integrator units in the reservoir. For these ESNs, more input and state history are preserved in the current state. Therefore, they have higher memory capacity than ESN, which also has been illustrated in some experiments. In addition, some other variants of ESN have been established and have been successfully applied to solar irradiance prediction, such as the chainstructure ESN (CESN) with multiple independent ESN modules [27, 28], multitimescale ESN (MTSESN) with multiple reservoirs in parallel array [29], and so forth. These ESNs mainly focus on the topology changes for performance improvement, whose memory patterns are single and fixed.
It can be noticed that the aforementioned ESNs modify the topology or neuron model to improve the performance. Although some effective results have been achieved, they still face some challenges in practical complex application. Firstly, the memory capacity of DESN is finite and continuous in spite of the strong deep learning ability. The single memory pattern will affect the flexibility of DESN to process different kinds of input signals. Furthermore, the leaky ESNs mainly focus on singlereservoir topology although they have larger and variable memory capacity. The singlereservoir ESN has limited feature extraction capability, which will influence the modeling accuracy.
Therefore, a novel deep echo state network with variable memory pattern (VMPDESN) is proposed in this paper in order to handle aforementioned challenges. To encode the input feature into a richer state space, VMPDESN adopts multiple subreservoirs in series as the hidden layer. To remember more input history information, the time delay links are added in each reservoir and every two reservoirs. The VMPDESN can be regarded as an extension of the DESN. Unlike the DESN, the memory capacity of VMPDESN is variable due to the selective memory mode. Compared with singlereservoir leaky ESNs, the hidden layer of VMPDESN consists of multiple subreservoirs, which can hierarchically process the input information. Therefore, it is more flexible and advantageous for VMPDESN to deal with different input signals.
In summary, the main contributions of this paper are summarized as follows:(i)A novel VMPDESN model is proposed for solar irradiance prediction, which has multiple reservoirs topology and selective memory patterns. The VMPDESN is studied in terms of mathematical model, pattern classifications, and stability analysis, respectively.(ii)The time delay links are added in each reservoir and between every two reservoirs in order to preserve more input history in the states. It is more flexible for VMPDESN to process different input signals.(iii)The effect of different memory patterns on VMPDESN is quantitatively and qualitatively analyzed, including the prediction accuracy, antidisturbance ability, and memory capacity.
The remainder of this paper is organized as follows. Section 2 introduces the proposed VMPDESN methodology in detail. In Section 3, the performance of VMPDESN is comprehensively evaluated in terms of prediction accuracy, antidisturbance ability, and memory capacity. Finally, some conclusions are summarized in Section 4.
2. Methodology of a Novel VMPDESN
The memory length of traditional ESN is usually finite and fixed, which cannot extract sufficient features of input signals with longterm dependency. Therefore, a novel deep echo state network with variable memory pattern (VMPDESN) is proposed in this study to preserve more input history features. The developed VMPDESN is more flexible to deal with different kinds of input signals, as a result of variable memory capacity. This section will describe the principle of the proposed VMPDESN methodology.
2.1. VMPDESN Model
As shown in Figure 1, VMPDESN is composed of an input layer, a hidden layer with multiple subreservoirs in series and time delay links, and an output layer. On the one hand, the input signals can be encoded into a richer state space as a result of the hierarchical topology in VMPDESN. On the other hand, it is more flexible to deal with various input signals as different types of memory patterns are included in VMPDESN. To remember more input history features from the states, the time delay links are added in both each reservoir and between every two reservoirs. Therefore, VMPDESN has selectively variable memory modes, which will be helpful for the reservoirs to map more characteristics of input signals with longterm dependency.
Assume VMPDESN has subreservoirs with the same neurons for simplification, and are the number of input and output neurons, respectively. Denote the input and output signals at time step as and . The global reservoir states are given as , where represents the states of reservoir layer , . The input signal and reservoir states satisfy the compact sets through this paper. is the input connection weights matrix, denotes the internal connection weights matrix for reservoir , and is the external connection weights matrix between every two adjacent subreservoirs. denotes the delayed time between every two adjacent reservoirs, while is the delayed time in each reservoir.
The reservoir states of VMPDESN are updated according to
It can be noticed that the input signal of layer is the delayed state value of reservoir layer . The above equations can be generalized as follows:
The network outputs of VMPDESN are expressed as follows:where is the output connection weights matrix.
The method of training VMPDESN is similar to classical ESN. That is to say, only the output connection matrix needs to be learned in VMPDESN, while other connection matrices remain fixed after proper initialization. The ridge regression training mechanism [30] is adopted to compute by minimizing the cost function as follows:
The solution of equation (4) is given bywhere is a by reservoir state matrix, is a by teacher signal vector matrix, and is the training length. denotes a by identity matrix, and is the regulation parameter. During the training stage, and are collected as follows:where the tth column of includes the state signal of all reservoir layers at time and the tth column of denotes the corresponding expected output signal.
The training procedure of VMPDESN is given in Algorithm 1. According to Algorithm 1, the model parameters are firstly initialized by trial and error. Then, the states are updated according to equation (1), when the system is driven by input signal. At the same time, the reservoir states matrix and corresponding expected signal are collected according to equations (6)(7), respectively. The output connection weights are finally computed as equation (5).

Remark 1. When the delayed time are set to and , the DESN can be obtained. On this basis, if the number of reservoir is further set to 1 (i.e., M = 1), the DESN model will degrade to the standard ESN. In addition, the variable memory pattern ESN (VMPESN) can be obtained when , , and . DESN, ESN, and VMPESN are special cases of VMPDESN, which will be taken as benchmarks to verify the effectiveness of VMPDESN in later comparative simulation.
Remark 2. The proposed VMPDESN includes multiple reservoirs in series array and different types of time delay links. On the one hand, the input signals can be encoded into a richer state space representation due to the hierarchical structure of VMPDESN. On the other hand, the reservoir state update equation of VMPDESN is variable and designable, which can deal with different input signals more flexibly.
2.2. Pattern Classification of VMPDESN
From the dynamics of VMPDESN in equation (2), it can be noticed that the VMPDESN mainly includes two types of delayed links, i.e., the delay of internal state in each reservoir and the delay of state information transmission between every two subreservoirs. In order to explore the impact of different memory patterns on the network performance, three various VMPsDESN are discussed in this current study, as shown in Table 1.
2.2.1. Pattern I: VMP1DESN
To test the delay of internal state in each reservoir on network performance, the VMP1DESN is developed with and . The update rules of VMP1DESN are rewritten as follows:
It can be seen that only one kind of memory mode is retained in VMP1DESN. VMP1DESN is a special case of VMPDESN (equation (2)) which can also be regarded as an extension of DESN. The ridge regression training mechanism in equation (4) can also be used to regulate of VMP1DESN.
2.2.2. Pattern II: VMP2DESN
In order to verify the delay of state information transmission between every two subreservoirs on network performance, the VMP2DESN is established with and . The update equations of VMP2DESN are expressed as follows:
Obviously, the VMP2DESN is also an extension of DESN. Compared with traditional ESN and DESN, the VMP2DESN can obtain richer asynchronous state signals, which can express more input history features.
2.2.3. Pattern III: VMP3DESN
VMP3DESN includes two kinds of delayed links, i.e., the delay of internal state in each reservoir and the delay of state information transmission between every two reservoirs. The dynamics of VMP3DESN are formulated as follows:
In VMP3DESN, it should be satisfied that and . It can be observed that the VMP3DESN model is an extension of DESN, VMP1DESN, and VMP2DESN.
Remark 3. In order to study the influence of different memory patterns on network performance, VMP1DESN, VMP2DESN, and VMP3DESN are developed, respectively, which correspond to different reservoir state update equation. VMP1DESN, VMP2DESN, and VMP3DESN are three special forms of VMPDESN (equation (2)) under different constraints. Therefore, the method to train the above three VMPsDESN can refer to Algorithm 1. Furthermore, the effect of three various VMPsDESN on network performance will be discussed and compared in later simulation.
2.3. Echo State Property Analysis of VMPDESN
A valid ESN should satisfy the echo state property (ESP), which plays an important role in ensuring the asymptotical stability [31]. ESP denotes that the reservoir states only depend on the input signal with time and independent of its initial values. Similarly, the ESP of the VMPDESN system is mainly discussed in this section, to guarantee the asymptotically stable operation. To facilitate the theoretical analysis of ESP, the dynamics of VMPDESN in equation (2) are rewritten as follows:where the input signal is defined asand the input connection matrices is denoted as
Then, the Euclidean distance between and randomly initialized from and is computed by
According to Lagrange’s mean value theorem and the fact , one can obtain that and are randomly initialized state values, i.e., is bounded. According to equation (15), it can be obviously obtained that , when and . Therefore, can ensure the ESP of reservoir , as a result of ( denotes the spectral radius of ). Furthermore, the necessary condition to guarantee the ESP of VMPDESN model is deduced as follows:
3. Experimental Design and Results
In this section, the effectiveness of the proposed VMPDESN is substantiated by solar irradiance forecasting. The solar irradiance datasets in the whole year 2017 with hourly time interval are used for simulation, which are obtained from California Irrigation Management Information System (CIMIS) [32]. CIMIS provides public access to data on the solar irradiance and other details about the stations. For each dataset, the training period and testing period are shown in Table 2.
3.1. Performance Evaluation Metrics
In this current study, the performance of the proposed methodology is comprehensively evaluated through quantitative and qualitative analysis. Four different external metrics in [1] are used to quantitatively evaluate the accuracy of VMPsDESN, i.e., root mean square error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), and normalized root mean square error (nRMSE), as shown in equations (17)–(20).where is the predicted value, is the target value, is the mean of actual value, and denotes the length of testing step. In addition, the antidisturbance ability and memory capacity of VMPDESN will be further qualitatively analyzed in later simulation. The influence of disturbance on one network with strong antidisturbance ability will disappear quickly, which plays an important role in practical applications. Memory capacity refers to the information length that one network can remember in a short time.
3.2. Modeling Accuracy of VMPDESN
The prediction performance of VMPDESN is verified by onehourahead solar irradiance prediction, and only the historical solar irradiance data are adopted to model in this study. In order to provide appropriate input and output signals for VMPDESN, the time series analysis [23] is used to characterize the historical solar irradiance data. The network parameters are determined by trial and error, and they are set as follows: the total reservoir size is set as 200, the number of reservoir layer is 4, the input number is 2, and the output layer is 1; connection weights in matrices and are sampled from a uniform distribution over [−0.1,0.1], while the spectral radius of matrices is set as 0.85 to meet the ESP condition. The regularization parameter is .
The influence of different delayed time on the fourlayer VMPDESN is firstly studied. Evaluation period II in Seeley station is adopted to test the prediction accuracy versus delayed time. The iteration process is stopped when the testing error is greater than the initial error. The statistical results are shown in Table 3, in terms of RMSE, nRMSE, MAE, and MAPE, respectively. In addition, the effect of different memory patterns on VMPDESN performance is also compared in Table 3. Note that VMPESN with corresponds to standard ESN and VMP1ESN with corresponds to traditional DESN. From the results in Table 3, it can be obtained that (1) the prediction accuracy of DESN is higher than that of ESN with the same reservoir units; (2) the proposed VMPDESN and VMPESN perform better than DESN and ESN; (3) VMP1DESN, VMP2DESN, and VMP3DESN have equivalent prediction accuracy, indicating that the performance of DESN can be effectively improved by adding these three different memory patterns. Additionally, the prediction error of VMPESN, VMP1DESN, and VMP3DESN varies rapidly with the delayed time, and it starts to increase when the delayed time is greater than 3. However, the prediction error of VMP2ESN varies slowly with the delayed time, and it starts to increase when . The reason behind this phenomenon may be that the delayed link in reservoir can speed up the network response, while the delayed link between every two subreservoirs can stabilize the network performance. The performance of classical ESN, DESN, VMPESN, and VMPsDESN with the best prediction accuracy is compared in Figure 2(a), and the corresponding relative errors are shown in Figure 2(b). Obviously, experimental results clearly show that the proposed VMPDESN could match the actual value better than ESN, DESN, and VMPESN, especially for some inflection points.
Taking Seeley and Blythe NE stations as two examples, the performance of VMPsDESN is further verified. The statistical results of different prediction models are recalculated and reported in Table 4, in terms of RMSE, nRMSE, MAE, and MAPE. According to the results obtained, it can be seen that VMPsDESN can achieve higher accuracy than other models for the most testing evaluation periods. Comparative results between ESN and DESN indicate the advantage of hierarchical topology, while the comparative results between ESN and VMPESN verify the effectiveness of variable memory. In addition, the equivalent prediction accuracy of VMP1DESN, VMP2DESN, and VMP3DESN further demonstrates the validity of three different types of memory patterns in improving the accuracy of DESN.
3.3. Antidisturbance Ability of VMPDESN
The antidisturbance ability of VMPDESN will be qualitatively evaluated in this subsection. The effect of two kinds of disturbances on the networks is mainly analyzed, i.e., the disturbances caused by differences in state initialization and noise in the input signal. For a certain model, the first kind of disturbance can be constructed by initializing two different reservoir states, while other model parameters remain the same. Similarly, the second type of disturbance can be built by adding small disturbance to the input signal, while the model parameters remain consistent. The effect caused by disturbance is expressed by the difference between reservoir states. In this current study, Euclidean distance is adopted to characterize the state differences, as shown in the following equation:where and are the reservoir states generated under different conditions. If the distance is close to zero as the iteration progresses, it can indicate that the difference between the two states has disappeared. That is to say, the corresponding model has the antidisturbance capability. The faster the difference disappears, the stronger the antidisturbance ability of a model, and vice versa.
The impact of state initialization differences on above prediction models is firstly studied. The only variable is the initialized reservoir states, while other model parameters remain the same. The antidisturbance ability of ESN, DESN, VMPESN, and VMPDESN is compared in Figure 3, with the initialized state values as and . From Figure 3, it can be observed that (1) the state Euclidean distance of six prediction models gradually decreases to zero as the training progresses; (2) the disturbance curves of VMPESN decay are significantly slower than the other five models, and the decay speed of the disturbance curves corresponding to ESN, DESN, and VMP2DESN is faster than the other three models; and (3) the state difference of ESN, DESN, and VMP2DESN has decayed to zero, when the training reaches the time step 30. Therefore, it can be concluded that each of the six models has a certain antidisturbance ability, and the antidisturbance ability of ESN, DESN, and VMP2DESN is stronger than the other three models. The reason is that ESN, DESN, and VMP2DESN contain their own delayed states, which affect the ability to resist such kind of disturbance caused by initial states.
Then, the ability of six models to resist the second type of disturbance is further explored. The single variable is input signal, while the other model parameters remain unchanged. Assume that the input signal at time step receives a small disturbance and changes from the original value to . For better presentation, the semilogarithmic Euclidean distance curves of six models under input disturbance are plotted and compared in Figure 4. As can be seen from Figure 4, there is an immediate difference between states at time step 200, when the disturbance is added to the input signal. Simultaneously, the difference between states reaches the maximum at time step 200. As the iteration goes on, the impact of input disturbance on six models gradually decreases. In addition, the disturbance curves corresponding to DESN, VMPDESN2, and ESN decay faster than the other three models, further indicating the stronger ability to resist input disturbance.
The quantitative results in Subsection 3.2 have illustrated that VMP1DESN, VMP2DESN, and VMP3DESN corresponding to the three different types of memory patterns can achieve equivalent performance in improving the prediction accuracy. However, the qualitative analysis results in this subsection show that VMP2DESN performs better than VMP1DESN and VMP3DESN in resisting the input and initial states disturbances.
3.4. Memory Capacity of VMPDESN
The memory capacity (MC) of several models is investigated in this section. MC refers to the information length that one network can remember in a short time, i.e., the ability to recall the input signal. According to [33], MC is denoted and computed as follows:where is the actual network output and is the input signal with a delay . denotes the correlation coefficient between and . The goal is to train the readout of a network to recall the input signal with a delay . That is to say, is the target output with the input signal . The closer is to 1, the higher the accuracy of a network is to recall the delayed input signal.
In this experiment, MC of traditional ESN, DESN, VMPESN, and the proposed VMPsDESN is calculated and compared. The input signal is sampled from a uniform distribution , and the upper limit of is set as 200 for practical consideration. The number of reservoir layer is 4, and the total reservoir size is set as 200, while other model parameters remain unchanged.
Taking a fourlayer VMP2DESN as an example, the influence of delayed time on memory capacity is discussed. Figure 5 shows the changes of with under different in VMP2DESN. As illustrated in Figure 5, is almost monotonously decreasing over delay , when is relatively small, such as and . However, decreases to zero through a series of discrete time spans, when is large, such as and . The reason behind this phenomenon may be the special memory mode in VMP2DESN. When is relatively small, there exist duplicate memories in different reservoir layers, and only a small amount of past input information can be recalled accurately by VMP2DESN. Duplicate memories denote that some input features preserved in former reservoir layer are also contained in the latter reservoir layer. When is large, the duplicate memories among reservoirs in VMP2DESN can be effectively decreased to a certain extent. The input features far away from the current time can also be recalled accurately under this circumstance.
Table 5 and Figure 6 further compare the MC of several models. From the results in Table 5, it can be seen that (1) the MC of DESN is larger than that of ESN due to multiple reservoir layers; (2) VMPDESN has higher MC than ESN because of the adjustable memory intervals. In addition, VMP2DESN has the highest MC value among three different types of VMPDESN, which further illustrates the advantages of adding delay links between every two subreservoirs. Figure 6 shows versus delay in ESN, DESN, and VMP2DESN. Obviously, VMP2DESN has a higher MC than ESN and DESN. For ESN and DESN, the memory capacity is limited under constant reservoir size. However, the memory of this proposed VMPDESN can be designable in advance by adjusting the delayed time. Therefore, VMPDESN can obtain higher memory capacity under proper memory pattern and delayed time.
4. Conclusion
A novel deep echo state network is proposed in this paper, i.e., VMPDESN. This model introduces different types of memory patterns, which is helpful to extract more input features of solar irradiance with longrange dependency. Compared with traditional ESN and DESN, the memory length of VMPDESN is variable and designable. Therefore, it is more flexible for VMPDESN to deal with various input signals. The effect of three memory patterns on VMPDESN is explored by quantitative and qualitative analysis in detail. The quantitative simulation results illustrate that the VMPsDESN under different memory modes can achieve equivalent performance in improving the prediction accuracy. From the qualitative analysis results, VMPDESN under the second memory pattern has a stronger antidisturbance ability and a higher memory capacity than the other two modes.
In this current study, the delayed time between every two subreservoirs are set to be the same with the delayed time in each reservoir for simplification. For more complex modeling in practice, the parameters can be different. However, it should be noted that different delayed time will increase the difficulty of parameter optimization.
Data Availability
Data can be found at http://cimis.water.ca.gov.
Conflicts of Interest
The authors declare that there are no conflicts of interest.
Authors’ Contributions
Qian Li conceptualized the study; Qian Li and Zhijun Guo proposed the methodology; Qian Li, Tao Li, and Zhijun Guo provided software; Qian Li and Tao Li validated the study; Dayong Yang was responsible for formal analysis; Jiangang Ouyang investigated the study; Qian Li was responsible for resources; Qian Li performed data curation; Qian Li wrote original draft; Zhijun Guo reviewed and edited the manuscript; Qian Li visualized the study; Zhijun Guo and Dayong Yang supervised the study; Zhijun Guo was responsible for project administration; Qian Li was responsible for funding acquisition. All authors have read and agreed to the published version of the manuscript.
Acknowledgments
This research was supported by the National Natural Science Foundation of China (62163026).