Building Recurrent Neural Networks to Implement Multiple Attractor Dynamics Using the Gradient Descent Method

Abstract

The present paper proposes a recurrent neural network model and learning algorithm that can acquire the ability to generate desired multiple sequences. The network model is a dynamical system in which the transition function is a contraction mapping, and the learning algorithm is based on the gradient descent method. We show a numerical simulation in which a recurrent neural network obtains a multiple periodic attractor consisting of five Lissajous curves, or a Van der Pol oscillator with twelve different parameters. The present analysis clarifies that the model contains many stable regions as attractors, and multiple time series can be embedded into these regions by using the present learning method.

1. Introduction

Recurrent neural networks (RNNs) have been successfully applied to the modeling of various types of dynamical systems. Since the universal approximation ability of multilayer neural networks has been proved, RNNs can model arbitrary dynamical systems and turing machines [1–3]. However, applying RNNs to a desired model may be very difficult even if such RNNs exist [4]. For example, building RNNs to implement required multiple attractor dynamics is a difficult problem for standard training, such as the gradient descent method. Doya and Yoshizawa [5] demonstrated that RNNs can acquire two limit cycles in the gradient descent method using initialization with small connection weights, whereas learning for more than three limit cycles is difficult [6]. This is due to the fact that the learning of several time series causes a conflict with respect to the changing of the connection weights. How to form RNN models that can learn several temporal sequence patterns has proved to be a challenging problem.

There have been some approaches to this problem. In order to avoid conflicts in the change of parameters, the mixture-of-experts-type architecture has been investigated [7, 8]. The mixture-of-experts model consists of RNNs as experts and a hierarchical gating mechanism. At the end of successful learning, each expert implements attractor dynamics as locally represented knowledge, and a gating mechanism chooses only one expert at any time. The system can acquire many attractor patterns although there is a disadvantage in that the system does not have the generalization ability on the attractor patterns. As the other approach to implement multiple patterns, the parametric bias (PB) method has been developed to improve the learning capability of RNNs [9, 10]. In an RNN that employs the PB method (RNNPB), PB values provide the information needed in order to individualize each sequence. It has been reported that the number of time series that RNNPBs can learn is greater than that which RNNs without PB can learn. However, the PB method cannot avoid the conflict caused by each attractor learning. Therefore, learning multiple time series by an RNNPB tends to fail when the number of time series increases.

In the present study, we will focus on the training method for RNNs to learn multiple attractor dynamics. Furthermore, we will show that the present research is related to research into RNNs with contraction transition functions. In recent years, RNNs with contraction transition mapping have been investigated with respect to the performance of time series learning [11–13], generalization ability [14], and memory capacity [15]. Jaeger [11, 12] demonstrated that an “echo state network,” which is an RNN with contraction mapping, successfully learns the Mackey-Glass chaotic time series, a well-known benchmark system for time series prediction. In order to formally express the generalization ability, Hammer and Tiňo proved that RNNs with contraction are distribution-independent learnable in the probably approximately correct (PAC) sense [14]. From the above results, RNNs with contraction might be regarded as powerful tools for modeling dynamical systems. However, RNNs with contraction have difficulty in representing multiple attractor dynamics because dynamic states governed by the contraction transition function are globally attracted to one point. In this paper, the representation capability of RNNs with contraction mapping will be improved such that the RNNs can obtain multiple attractor dynamics.

We start by defining the concepts of the RNN and the training method for multiple attractor dynamics. The RNN has the Elman net-type architecture, and the training method for RNNs is basically based on the backpropagation through-time (BPTT) algorithm [16]. We then show in numerical simulation that the RNNs can acquire multiple periodic attractors constituted by five Lissajous curves, or a Van der Pol oscillator with twelve different parameters. Moreover, we consider why the RNNs successfully learn multiple attractors and how the performance of learnability depends on parameters of the RNNs. Finally, we link the results obtained herein to other learning strategies, and consider other advanced research topics.

2. Model

2.1. Recurrent Neural Network

We first consider a neural network model with recurrent connection, such as the Elman net [17] (see Figure 1). The RNN contains I/O units, orthogonal units, and internal units. We denote the dynamic states of I/O units, orthogonal units, and internal units at time step 𝑛 by 𝐱_𝑛∈ℝ^𝑁1,𝐫_𝑛∈ℝ^𝑁2, and 𝐮_𝑛∈ℝ^𝑁3, respectively. The RNN is defined by functions 𝑓_𝜽 and 𝑔_𝜽 with a parameter 𝜽≡(𝐖₁,𝐖₂,𝐖₃,𝐕,𝐛,𝐝), where 𝑓_𝜽∶ℝ^𝑁1×ℝ^𝑁2×ℝ^𝑁3→ℝ^𝑁3 and 𝑔_𝜽∶ℝ^𝑁3→ℝ^𝑁1×ℝ^𝑁2 are of the forms𝑓_𝜽(𝐱,𝐫,𝐮)=(1−𝜖)𝐮+𝜖(𝐖₁⋅𝐱+𝐖₂⋅𝐫+𝐖₃𝑔⋅𝐹(𝐮)+𝐛),_𝜽(𝐮)=𝐹(𝐕⋅𝐹(𝐮)+𝐝), (1)(2) where 𝐖₁∈ℝ^𝑁1×𝑁3,𝐖₂∈ℝ^𝑁2×𝑁3,𝐖₃∈ℝ^𝑁3×𝑁3, and 𝐕∈ℝ^{𝑁3×𝑁1𝑁2} are matrices, 𝑏∈ℝ^𝑁3 and 𝑑∈ℝ^𝑁1𝑁2 are vectors, 𝜖∈ℝ is a time constant that satisfies 0≤𝜖≤1, and 𝐹 denotes a componentwise application such as 𝐹_𝑖=tanh.

Figure 1 

Architecture of the recurrent neural network. Solid arrows, dotted arrows, and boxes represent fixed connections, adjustable connections, and network states, respectively.

Dynamic states of the RNN at time step 𝑛 are updated according to(𝐱_𝑛,𝐫_𝑛)=𝑔_𝜽(𝐮_𝑛𝐮),_𝑛+1=𝑓_𝜽(𝐱_𝑛,𝐫_𝑛,𝐮_𝑛)=𝑓_𝜽(𝑔_𝜽(𝐮_𝑛),𝐮_𝑛). (3) From these equations, the RNN can be represented by an 𝑁₃-dimensional dynamical system.

We now define bistability for the RNN.Definition 1. Assume 𝑓_𝜽∶ℝ^𝑁1×ℝ^𝑁2×ℝ^𝑁3→ℝ^𝑁3 is as above. The function 𝑓_𝜽 is bistable with respect to the third variable 𝐮 if a real value 𝜔>1 and an integer 𝑁_𝑠 exist such that𝑤_𝑖𝑗=𝜔if𝑖≤𝑁_𝑠𝑤,𝑖=𝑗,_𝑖𝑗=0if𝑖≤𝑁_𝑠,𝑖≠𝑗,|𝑤_𝑖𝑗1|<𝑁₃otherwise, (4)for every element 𝑤_𝑖𝑗 of the matrix 𝐖₃.

The bistability of a function 𝑓_𝜽 is a key concept of our learning method. We will show in Section 4.1 that the bistable function 𝑓_𝜽 plays an important role in the learning of multiple attractor dynamics.

2.2. Learning Method

We present a formulation of the training procedure for the RNN with a multiple teacher I/O time series. For every 1≤𝑘≤𝑚 and 𝐿_𝑘∈ℕ, we assume that (𝐱₁^(𝑘),…,𝐱_𝐿^(𝑘)_𝑘) is a sequence of teacher I/O of length 𝐿_𝑘.

Initialization of Parameters
We initialize every element of matrices 𝐖₁(0), 𝐖₂(0), and 𝐕(0) and vectors 𝐛(0) and 𝐝(0) randomly from the uniform distribution in the interval (−1/𝑁₃,1/𝑁₃). A matrix 𝐖₃(0) is randomly assigned such that 𝑓_𝜽 is bistable. For all 1≤𝑘≤𝑚,𝐮₁^(𝑘)(0) is randomly initialized in the interval [−1,1].
Assume that 𝐫_𝑛^(𝑘)(0) is an 𝑚-tuple of vectors (𝐬_𝑛^(𝑘,1)(0),…,𝐬_𝑛^(𝑘,𝑚)(0)) for 1≤𝑘≤𝑚 and 1≤𝑛≤𝐿_𝑘, and that the dimension of 𝐬_𝑛^(𝑘,𝑙)(0) is equivalent to that of 𝐬^{(𝑘′,𝑙′)}_𝑛′(0) if 𝑙=𝑙^′. We initialize 𝐬_𝑛^(𝑘,𝑙)(0) such that𝐬_𝑛^(𝑘,𝑙)(0)=𝟎if𝑘=𝑙,−𝟏otherwise. (5)

Run Network with Teacher I/O and Compute Error Function
For every 1≤𝑘≤𝑚, the sequence (𝐱₁^(𝑘)(𝑡),…,𝐱_𝐿^(𝑘)_𝑘(𝑡)) of I/O units of the RNN at learning step 𝑡 is defined by(𝐱_𝑛^(𝑘)(𝑡),𝐫̂_𝑛^(𝑘)(𝑡))=𝑔_𝜽(𝐭)(𝐮_𝑛^(𝑘)𝐮(𝑡)),_(𝑘)𝑛+1(𝑡)=𝑓_𝜽(𝑡)(𝐱_𝑛^(𝑘),𝐫_𝑛^(𝑘)(𝑡),𝐮_𝑛^(𝑘)(𝑡)). (6) The error function 𝐸^(𝑘)(𝑡) of the RNN at learning step 𝑡 with the 𝑘th teacher I/O time series is defined by𝐸^(𝑘)(𝑡)=^𝐿𝑘_𝑛=1𝑒(𝐱_𝑛^(𝑘)(𝑡),𝐱_𝑛^(𝑘))+𝑒(𝐫̂_𝑛^(𝑘)(𝑡),𝐫_𝑛^(𝑘)(𝑡)), (7)where 𝑒 denotes the mean square error function 𝑒(𝐱,𝐱^′)=(1/2)(𝐱−𝐱^′)^T⋅(𝐱−𝐱^′). Finally, the error function 𝐸(𝑡) at learning step 𝑡 is defined by𝐸(𝑡)=^𝑚_𝑘=1𝐸^(𝑘)(𝑡). (8)

Update Parameters
Let 𝝆(𝑡)∈{𝐖₁(𝑡),𝐖₂(𝑡),𝐕(𝑡),𝐛(𝑡),𝐝(𝑡)} be a parameter of the RNN at learning step 𝑡. We determine the parameter 𝝆(𝑡+1) by𝝆(𝑡+1)=𝝆(𝑡)+𝛼Δ𝝆(𝑡),Δ𝝆(𝑡)=𝛽Δ𝝆(𝑡−1)−𝜕𝐸(𝑡),𝜕𝝆(𝑡) (9) where Δ𝝆(0)=0;𝛼 and 𝛽 are the constants of the learning rate and momentum, respectively. On the other hand, a connection matrix 𝐖₃(𝑡) is not changed as 𝐖₃(𝑡+1)=𝐖₃(𝑡) in order to hold the bistability condition. We compute the initial state 𝐮₁^(𝑘)(𝑡+1) of the internal units at learning step 𝑡+1 such that𝐮₁^(𝑘)(𝑡+1)=𝐮₁^(𝑘)(𝑡)+𝛼^′Δ𝐮₁^(𝑘)(𝑡),Δ𝐮₁^(𝑘)(𝑡)=𝛽Δ𝐮₁^(𝑘)(𝑡−1)−𝜕𝐸(𝑡)𝜕𝐮₁^(𝑘),(𝑡) (10) where Δ𝐮₁^(𝑘)(0)=0, and 𝛼^′ is the constant of the learning rate of the initial state. Assume that 𝐬_𝑛^(𝑘,𝑙)(𝑡) is a vector as a component of the orthogonal units 𝐫_𝑛^(𝑘)(𝑡), such as 𝐫_𝑛^(𝑘)(𝑡)=(𝐬_𝑛^(𝑘,1)(𝑡),…,𝐬_𝑛^(𝑘,𝑚)(𝑡)). The vector 𝐬_𝑛^(𝑘,𝑙)(𝑡+1) is defined by𝐬_𝑛^(𝑘,𝑙)𝐬(𝑡+1)=^{𝑛(𝑘,𝑙)}(𝑡)+𝛼^′′Δ𝐬_𝑛^(𝑘,𝑙)(𝑡)if𝑘=𝑙,−𝟏otherwise,Δ𝐬_𝑛^(𝑘,𝑙)(𝑡)=𝛽Δ𝐬_𝑛^(𝑘,𝑙)(𝑡−1)−𝜕𝐸(𝑡)𝜕𝐬_𝑛^(𝑘,𝑙),(𝑡) (11)(12) where Δ𝐬_𝑛^(𝑘,𝑙)(0)=0, and 𝛼^′′ is the constant of the learning rate of the orthogonal units.
Note that the maximum value of the error function 𝐸(𝑡) depends on the number of units and the length of the teacher I/O time series. Thus, we should scale the learning rates 𝛼,𝛼^′, and 𝛼^′′ with the number of units and length of sequences. In the present paper, we consider parameters 𝛾,𝛾^′, and 𝛾^′′ such that 𝛼=𝛾/(𝑁₁+𝑁₂)∑_𝑘𝐿_𝑘,𝛼^′=𝛾^′/(𝑁₁+𝑁₂), and 𝛼^′′=𝛾^′′/(𝑁₁+𝑁₂).

3. Numerical Experiments

In this section, we conduct two types of experiments as examples of using the training method for RNNs proposed in Section 2. The first experiment shows the learning of five Lissajous curves. The second experiment shows the training of multiple attractors of a Van der Pol oscillator with 12 different parameters.

3.1. Experiment 1: Lissajous Curves

3.1.1. Teacher I/O Time Series

Our first task is to learn the five Lissajous curves defined by𝑥⁽¹⁾_𝑛,1=45sin(2𝜋𝑛𝑀),𝑥⁽¹⁾_𝑛,2=45cos(2𝜋𝑛𝑀𝑥),_(2)𝑛,1=45sin(4𝜋𝑛𝑀),𝑥⁽²⁾_𝑛,2=45cos(2𝜋𝑛𝑀𝑥),_(3)𝑛,1=𝑥⁽²⁾_𝑛,2,𝑥⁽³⁾_𝑛,2=𝑥⁽²⁾_𝑛,1,𝑥⁽⁴⁾_𝑛,1=45sin(2𝜋𝑛𝑀),𝑥⁽⁴⁾_𝑛,2=45cos(4𝜋𝑛𝑀𝑥),_(5)𝑛,1=𝑥⁽⁴⁾_𝑛,2,𝑥⁽⁵⁾_𝑛,2=𝑥⁽⁴⁾_𝑛,1, (13) and we consider constants 𝑀=32 and 𝐿_𝑘=200 for all 1≤𝑘≤5 (see Figure 2).

(a)

(b)

(c)

(d)

(e)

Figure 2 

Trajectories of the teacher I/O time series in experiment 1.

3.1.2. Learning and Testing

We now describe the specific conditions applied to RNN training. The time constant 𝜖 is set to 0.1. The number 𝑁₂ of orthogonal units is 10, and the dimension of a vector 𝐬_𝑛^(𝑘,𝑙) is 2 for all 1≤𝑙≤5. Suppose that 𝑓_𝜽 is bistable with 𝑁₃=30,𝑁_𝑠=15, and 𝜔=2.5. The learning rates and momentum are given by 𝛾=0.1,𝛾^′=𝛾^′′=0.01, and 𝛽=0.9, respectively.

Figure 3 shows the error function 𝐸^(𝑘)(𝑡) for 20 000 learning steps. We also show the Kullback-Leibler divergence between the teacher I/O time series and a sequence of I/O units in the RNN computed by (3) which do not use external perturbation by the teaching sequences. We use the Kullback-Leibler divergence as a measure of the discrepancy between two sequences. Formally, the Kullback-Leibler divergence between two probability distributions 𝑝 and 𝑞 is defined as𝑑_KL(𝑝,𝑞)=(log𝑝(𝑥)−log𝑞(𝑥))𝑝(𝑥)𝑑𝑥. (14)By definition, in order to compute the Kullback-Leibler divergence, it is necessary to obtain probability distributions of the teacher I/O time series and a sequence of I/O units. However, obtaining the probability distribution of a sequence of I/O units is very difficult. Therefore, we quantize a time series of real-valued vectors into a symbolic sequence such that if the real value is less than 0, then the symbol 0 is appropriated, and otherwise the symbol 1 is appropriated. In addition, we use the probability distribution whereby sub-blocks with a block length of 𝑙 appear in the symbolic sequence given by the above quantization.

(a)

(b)

Figure 3 

Error and Kullback-Leibler divergence between the teaching sequences and output generated by the RNN for 20 000 learning steps in experiment 1.

Figure 4 describes attractors of the trained RNN computed by (3) of which the initial state of internal units is 𝐮₁^(𝑘)(𝑡) for each 1≤𝑘≤5. By comparing the attractors with the teacher I/O time series displayed in Figure 2, we can see that the RNN can generate sequences similar to training data.

(a)

(b)

(c)

(d)

(e)

Figure 4 

Time series 𝐱𝑛 generated by the trained RNN in experiment 1. For each time series, only the initial state 𝐮0 is different.

In Figure 5, examples of attractors for the RNN with random initial states are displayed. This shows that, in addition to the attractors corresponding to teacher I/O time series, there exist many attractors of the RNN.

Figure 5 

Time series 𝐱𝑛 generated by the trained RNN with random initial state 𝐮0 in experiment 1.

3.2. Experiment 2: Van der Pol Attractors

3.2.1. Teacher I/O Time Series

Our second task is to learn multiple attractors given by the Van der Pol oscillator with different parameters. The Van der Pol oscillator defined by𝑑²𝑦𝑑𝑡²−𝜇(1−𝑦²)𝑑𝑦𝑑𝑡+𝑦=0 (15)is a model of an electronic circuit that appeared in very early radios. It is well known that there exists a limit cycle for the Van der Pol oscillator. In this experiment, we consider twelve teacher I/O time series, where the 𝑘th teacher I/O time series 𝐱_𝑛^(𝑘) is given by𝑥^(𝑘)_𝑛,1=𝑎𝑦(𝑡)+𝑏_𝑘,𝑥^(𝑘)_𝑛,2=𝑎𝑑𝑦(𝑡)𝑑𝑡+𝑐_𝑘𝑛,𝑡=𝜏_𝑘, (16)for 𝜇=0.25 and 𝑎=0.15, where 𝑏_𝑘 and 𝑐_𝑘 are constant parameters representing the center position of the limit cycle, and 𝜏_𝑘 is a time constant of the oscillator. We assume that the parameters 𝑏_𝑘,𝑐_𝑘, and 𝜏_𝑘 are given by combining the values of 𝑏_𝑘=±0.4, 𝑐_𝑘=±0.4, and 𝜏_𝑘=2,4,6. Figure 6 shows the teacher I/O time series given by (16). The length of training data is 𝐿_𝑘=200 for 1≤𝑘≤12.

(a)

(b)

Figure 6 

Teaching sequences of experiment 2. (a) Trajectories on ℝ2. (b) Temporal trajectories of teaching sequences.

3.2.2. Learning and Testing

The parameters for learning are set as follows. Let 𝑓_𝜽 be bistable with 𝑁₃=40 and 𝑁_𝑠=20. The dimension of the vector 𝐬_𝑛^(𝑘,𝑙) is 1 for every 1≤𝑙≤12 so that 𝑁₂=12. Other parameters are the same as in experiment 1.

The error function and the Kullback-Leibler divergence for 200 000 learning steps are displayed in Figure 7. Figure 8 shows attractors of the trained RNN, and the initial state of the internal units of which is set to 𝐮₁^(𝑘)(𝑡) for every 1≤𝑘≤12.

(a)

(b)

Figure 7 

Error and Kullback-Leibler divergence between the teaching sequences and output generated by the RNN for 200 000 learning steps in experiment 1.

(a)

(b)

Figure 8 

Time series 𝐱𝑛 generated by the trained RNN in experiment 2. For each time series, the initial state 𝐮0 is the same as the training phase. (a) Trajectories on ℝ2. (b) Temporal trajectories.

This result allows us to consider that the RNN acquires multiple periodic attractors constituted by the teacher I/O time series.

4. Numerical Analysis

4.1. Contraction and Bistability

Assume that 𝑋 and 𝑈 are sets and that 𝑈 is equipped with a metric structure. A function 𝑓∶𝑋×𝑈→𝑈 is a contraction with respect to 𝑈 if a real value 𝐶∈[0,1) exists such that the inequality∀𝑢₁,𝑢₂∈𝑈∀𝑥∈𝑋|𝑓(𝑥,𝑢₁)−𝑓(𝑥,𝑢₂)|≤𝐶|𝑢₁−𝑢₂| (17)holds for all 𝑥∈𝑋 and 𝑢₁,𝑢₂∈𝑈. Lemma 2. Let one consider a dynamical system on ℝ^𝑁3 defined by the transition function 𝐮_𝑛+1=𝑓_𝜽(𝑔_𝜽(𝐮_𝑛),𝐮_𝑛), where 𝑓_𝜽 and 𝑔_𝜽 are defined in (1) and (2), respectively. Assume that each element 𝑤_𝑖𝑗 of the matrix 𝐖₃ satisfies (4), and 𝜈∈ℝ is the maximum absolute value of elements in 𝐖₁, 𝐖₂, and 𝐛. If there exist three solutions of𝑥−𝜔tanh(𝑥)+𝑣(𝑁₁+𝑁₂+1)=0,𝑥∈ℝ, (18)then
(1) there are 2^𝑁𝑠 invariant sets of a dynamical system 𝐮_𝑛+1=𝑓_𝜽(𝑔_𝜽(𝐮_𝑛),𝐮_𝑛);(2) suppose that 𝑈⊂ℝ^𝑁3 is an invariant set of the dynamical system; then, the restriction of 𝑓_𝜽 to ℝ^𝑁1×ℝ^𝑁2×𝑈 is a contraction with respect to 𝑈 and the maximum norm |⋅|_∞, where |𝐮−𝐮^′|_∞≡max_{1≤𝑖≤𝑁3}|𝑢_𝑖−𝑢^′_𝑖|.Proof. We suppose that (18) has three solutions, such as 𝑥₁>𝑥₂>𝑥₃ (see Figure 9). In general, 𝑥₁,𝑥₂>0 and 𝑥₃<0.
(1) Assume 1≤𝑖≤𝑁_𝑠 and 𝑥₁≥𝑢_𝑛^(𝑖)≥𝑥₂. Then, the expression𝑢_𝑛^(𝑖)−𝜔tanh(𝑢_𝑛^(𝑖))+𝑣(𝑁₁+𝑁₂+1)≤0⇒𝑢_𝑛^(𝑖)−𝜔tanh(𝑢_𝑛^(𝑖))−𝑂_𝑛^(𝑖)≤0⟺−𝜖(𝑢_𝑛^(𝑖)−𝜔tanh(𝑢_𝑛^(𝑖))−𝑂_𝑛^(𝑖))≥0⟺(1−𝜖)𝑢_𝑛^(𝑖)+𝜖(𝜔tanh(𝑢_𝑛^(𝑖))+𝑂_𝑛^(𝑖))≥𝑢_𝑛^(𝑖)⟺𝑢^(𝑖)_𝑛+1≥𝑢_𝑛^(𝑖) (19)is satisfied for all 𝐱_𝑛∈ℝ^𝑁1,𝐫_𝑛∈ℝ^𝑁2, and 𝐮_𝑛,𝐮^′_𝑛∈ℝ^𝑁3, where 𝑂_𝑛^(𝑖) is the 𝑖th element of the vector 𝐎_𝑛=𝐖₁⋅𝐱_𝑛+𝐖₂⋅𝐫_𝑛+𝐛. Hence, 𝑢^(𝑖)_𝑛+1≥𝑢_𝑛^(𝑖) if 𝑥₁≥𝑢_𝑛^(𝑖)≥𝑥₂. Furthermore, if 𝑢_𝑛^(𝑖)≥𝑥₁, then 𝑢^(𝑖)_𝑛+1≥𝑥₁ because𝑢_𝑛^(𝑖)≥𝑥₁⇒tanh(𝑢_𝑛^(𝑖))≥tanh(𝑥₁)(because𝑥₁>0)⇒(1−𝜖)𝑢_𝑛^(𝑖)+𝜖(𝜔tanh(𝑢_𝑛^(𝑖))+𝑂_𝑛^(𝑖))≥(1−𝜖)𝑥₁+𝜖(𝜔tanh(𝑥₁)+𝑂_𝑛^(𝑖))⇒𝑢^(𝑖)_𝑛+1≥𝑥₁(because𝑥₁≥𝑢_𝑛^(𝑖)≥𝑥₂⇒𝑢^(𝑖)_𝑛+1≥𝑢_𝑛^(𝑖)). (20)Therefore, the region [𝑥₁,∞) is a stable set of the 𝑖th element of vector 𝐮_𝑛 satisfying the fact that if 𝑢_𝑛^(𝑖)∈[𝑥₁,∞) then 𝑢^(𝑖)_𝑛+𝑘∈[𝑥₁,∞) for any 𝑘∈ℕ.
Similarly, we can easily show that if 𝑢_𝑛^(𝑖)≤−𝑥₁, then 𝑢^(𝑖)_𝑛+1≤−𝑥₁. Thus, there are two stable regions of the 𝑖th element of vector 𝐮_𝑛 for each 1≤𝑖≤𝑁_𝑠. Then, there are 2^𝑁𝑠 invariant sets.
(2) Let 𝑈⊂ℝ^𝑁3 be the invariant set presented above. Assume that 𝐮_𝑛,𝐮^′_𝑛∈𝑈.
For any 1≤𝑖≤𝑁_𝑠, the inequality|𝑢^(𝑖)_𝑛+1−𝑢^′(𝑖)_𝑛+1|=|(1−𝜖)(𝑢_𝑛^(𝑖)−𝑢_𝑛^′(𝑖))+𝜖𝜔(tanh(𝑢_𝑛^(𝑖))−tanh(𝑢_𝑛^′(𝑖)))|<|(1−𝜖)(𝑢_𝑛^(𝑖)−𝑢_𝑛^′(𝑖))+𝜖(𝑢_𝑛^(𝑖)−𝑢_𝑛^′(𝑖))|=|𝑢_𝑛^(𝑖)−𝑢_𝑛^′(𝑖)| (21)holds because if |𝑥|≥𝑥₁, then 𝜔|𝑑tanh(𝑥)/𝑑𝑥|<1.
On the other hand, for every 𝑁_𝑠≤𝑖≤𝑁₃,|𝑢^(𝑖)_𝑛+1−𝑢^′(𝑖)_𝑛+1|=|(1−𝜖)(𝑢_𝑛^(𝑖)−𝑢_𝑛^′(𝑖))+𝜖_𝑗𝑤_𝑖𝑗(tanh(𝑢_𝑛^(𝑗))−tanh(𝑢_𝑛^′(𝑖)))|≤(1−𝜖)|𝑢_𝑛^(𝑖)−𝑢_𝑛^′(𝑖)|+𝜖|_𝑗𝑤_𝑖𝑗(tanh(𝑢_𝑛^(𝑗))−tanh(𝑢_𝑛^′(𝑗)))|≤(1−𝜖)|𝑢_𝑛^(𝑖)−𝑢_𝑛^′(𝑖)|+𝜖|_𝑗𝑤_𝑖𝑗(𝑢_𝑛^(𝑗)−𝑢_𝑛^′(𝑗))|≤(1−𝜖)|𝑢_𝑛^(𝑖)−𝑢_𝑛^′(𝑖)|+𝜖𝑁₃max_𝑗|𝑤_𝑖𝑗(𝑢_𝑛^(𝑗)−𝑢_𝑛^′(𝑗))|≤(1−𝜖)|𝑢_𝑛^(𝑖)−𝑢_𝑛^′(𝑖)|+𝜖𝑁₃max_𝑗|𝑤_𝑖𝑗|max_𝑗|𝑢_𝑛^(𝑗)−𝑢_𝑛^′(𝑗)|≤((1−𝜖)+𝜖𝑁₃max_𝑗|𝑤_𝑖𝑗|)max_𝑗|𝑢_𝑛^(𝑗)−𝑢_𝑛^′(𝑗)|<max_𝑗|𝑢_𝑛^(𝑗)−𝑢_𝑛^′(𝑗)|. (22)
Then, |𝐮_𝑛+1−𝐮^′_𝑛+1|_∞<|𝐮_𝑛−𝐮^′_𝑛|_∞ is obtained for any 𝐮_𝑛,𝐮^′_𝑛∈𝑈. Accordingly, the restriction of 𝑓_𝜽 to ℝ^𝑁1×ℝ^𝑁2×𝑈 is a contraction with respect to 𝑈 and the maximum norm |⋅|_∞.

Figure 9 

Schematic diagram of (18).

For any 𝑁₁,𝑁₂∈ℕ and 𝜈≥0, there is a real number 𝑞 such that if 𝜔≥𝑞, then (18) has three solutions. Thus, if 𝜔 is large enough and matrices 𝐖₁ and 𝐖₂ represent small connection weights, then 𝑓_𝜽 contains 2^𝑁𝑠 invariant sets, and each restriction of 𝑓_𝜽 to an invariant set is a contraction with respect to a third input. Moreover, the integer 𝑁₃−𝑁_𝑠 is the effective degree of freedom for each contraction mapping restricted to an invariant set. If 𝑁₃−𝑁_𝑠 is a large value, then RNN can acquire a more complex time sequence. In Figures 10 and 11, we plot the Kullback-Leibler divergence of the trained RNN for parameters 𝜔 and 𝑁_𝑠, in which the training data are the same as those for experiment 1. These results imply that it is necessary that 𝜔,2^𝑁𝑠, and 𝑁₃−𝑁_𝑠 be large values in order to learn multiple attractor dynamics.

Figure 10 

Kullback-Leibler divergence between the teaching sequences and output generated by the trained RNN with 𝜖=0.1,𝑁2=10 (dim𝐬𝑛(𝑘,𝑙)=2), 𝑁3=30, and 𝑁𝑠=15.

(a)

(b)

(c)

Figure 11 

Kullback-Leibler divergence between the teaching sequences and output generated by the trained RNN with 𝜖=0.1,𝜔=2.5, and 𝑁2=10 (dim𝐬𝑛(𝑘,𝑙)=2). (a) 𝑁3=20, (b) 𝑁3=30, and (c) 𝑁3=40.

4.2. Orthogonality

In the last paragraph of the previous section, we have shown that RNNs have many stable regions, and the existence of the stable regions plays an important role in the learning of multiple sequences. However, the existence of multiple stable regions is not sufficient for success in the multiple attractor learning because if the change of parameters corresponding to each time series influences other changes, each time series cannot necessarily be embedded into each region. Similarly, this problem appears in the method of RNNPB.

In the training algorithm defined in Section 2, each state of orthogonal units 𝐫=(𝐬⁽¹⁾,…,𝐬^(𝑚)) is trained by (5) and (11). Thus, firing of 𝐬^(𝑘) only occurs in the generation of the 𝑘th teaching sequence. This implies that orthogonal units allow the conflict of parameter changes caused by multiple time series learning to be avoided because orbits corresponding to each teaching I/O time series run around the orthogonal state space of the trained RNN.

In order to show the effect of the orthogonal units on the conflict among teaching sequences, we consider the 𝑘th learning ratio 𝑑𝑣^(𝑘)_𝑖𝑗(𝑡) defined by𝑑𝑣^(𝑘)_𝑖𝑗(𝑡)=𝜕𝐸^(𝑘)(𝑡)𝜕𝑣_𝑖𝑗(𝜕𝐸(𝑡)𝜕𝑣_𝑖𝑗)⁻¹, (23)where 𝑣_𝑖𝑗 is an element of the matrix 𝑉. If 𝑑𝑣^(𝑘)_𝑖𝑗(𝑡) is nearly equal to 1, then the change in 𝑣_𝑖𝑗 is approximately independent of teaching sequences rather than the 𝑘th sequence. In Figure 12, we plot the value 𝑑𝑉^(𝑘)(𝑡) determined by𝑑𝑉^(𝑘)1(𝑡)=(𝑁₃−𝑁_𝑠)|𝑅^(𝑘)|_{𝑖∈𝑅(𝑘)}^𝑁3_{𝑗=𝑁𝑠}𝑑𝑣^(𝑘)_𝑖𝑗(𝑡), (24)where 𝑅^(𝑘) is a set of indices corresponding to the elements of the vector 𝐬^(𝑘). The value 𝑑𝑉^(𝑘)(𝑡) represents the average of the 𝑘th learning ratio for connections between internal units and orthogonal units 𝐬^(𝑘). In this numerical experiment, for each learning step, 𝑑𝑉^(𝑘)(𝑡) is clearly larger than 1/𝑚=0.2, where 𝑚 is the number of teaching sequences. Then, the sum of the 𝑘th learning ratios of connection weights between internal units and orthogonal units 𝐬^(𝑘) is dominant. Therefore, in changing matrix 𝑉, there is no conflict generated by multiple teaching sequences. However, we could not find a strong bias of the learning ratio for the matrices 𝑊₁ and 𝑊₂ and every element 𝑣_𝑖𝑗 of 𝑉 with 𝑖<𝑁_𝑠. Thus, we consider that connection weights between internal units and orthogonal units encode information on an individual time series, and other connection weights encode whole information.

Figure 12 

Average 𝑑𝑉(𝑘)(𝑡) of the 𝑘th learning ratio for the connections between internal units and orthogonal units 𝐬(𝑘) for 200 000 learning steps in experiment 1.

5. Discussion

In this report, we have investigated a method of embedding multiple time series into a single RNN. In order to clarify the characteristics of the proposed approach, we compare the proposed approach with other approaches with respect to information representation of multiple sequences in the models. The mixture-of-RNN-experts-type model composes local representation in an RNN for each sequence. The local representation provides robustness against changing the parameters in learning, but it lacks the ability to extract common patterns included in the sequences because of the independency of the local representation. In the proposed model, the local representation is constructed into orthogonal units, while the global representation is also constructed into internal units using the connection weights between I/O units and internal units. Since each sequence generated by the proposed model shares the state space and connection weights, the model can extract common patterns of the sequences as well as conventional neural networks.

Another characteristic, which clarifies the difference between our model and other models, is whether the classification of each time series is self-organized into the state space. For example, in the mixture-of-RNN-experts-type model, the allocation of time series to each RNN is determined automatically. As another example, in the RNNPB model, PB values are self-organized such that the PB can individualize each time series. On the other hand, the proposed model needs the information of orthogonalization for each time series. Since the sparse firing patterns which appear in orthogonal units, corresponding to time series, are given as teaching information externally, the classification of sequences is not self-organized. The characteristic whereby the time series cannot be automatically classified is a disadvantage of the proposed model. However, the time series can be classified using other clustering techniques before applying the proposed method. Thus, by combining the proposed method and other clustering techniques, an algorithm that automatically classifies and generates multiple time series can be constructed.

6. Conclusion

In this paper, we have presented an RNN model and a learning algorithm that can acquire the ability to generate multiple sequences. The RNN model consists of two distinct properties called bistability and orthogonality. Bistability guarantees the existence of multiple attractor structures in RNNs, and provides the RNNs with contraction transition mapping. Orthogonality, which is given as a function of the orthogonal vectors of RNNs, helps prevent conflicts with respect to parameter changes caused by multiple training sequences. In the numerical experiments, RNNs which have bistability and orthogonality can learn multiple periodic attractors constituted by five Lissajous curves or 12 Van der Pol oscillators. Based on these results, the proposed model can be applied to the modeling of various types of dynamical systems that include multiple attractors.