Advances in Artificial Neural Systems
Volume 2009 (2009), Article ID 846040, 11 pages
doi:10.1155/2009/846040
Research Article

Building Recurrent Neural Networks to Implement Multiple Attractor Dynamics Using the Gradient Descent Method

Jun Namikawa and Jun Tani

Brain Science Institute, RIKEN, 2-1 Hirosawa, Wako City, Saitama 351-0198, Japan

Received 31 March 2008; Accepted 22 August 2008

Academic Editor: Akira Imada

Copyright © 2009 Jun Namikawa and Jun Tani. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

The present paper proposes a recurrent neural network model and learning algorithm that can acquire the ability to generate desired multiple sequences. The network model is a dynamical system in which the transition function is a contraction mapping, and the learning algorithm is based on the gradient descent method. We show a numerical simulation in which a recurrent neural network obtains a multiple periodic attractor consisting of five Lissajous curves, or a Van der Pol oscillator with twelve different parameters. The present analysis clarifies that the model contains many stable regions as attractors, and multiple time series can be embedded into these regions by using the present learning method.

1. Introduction

Recurrent neural networks (RNNs) have been successfully applied to the modeling of various types of dynamical systems. Since the universal approximation ability of multilayer neural networks has been proved, RNNs can model arbitrary dynamical systems and turing machines [1–3]. However, applying RNNs to a desired model may be very difficult even if such RNNs exist [4]. For example, building RNNs to implement required multiple attractor dynamics is a difficult problem for standard training, such as the gradient descent method. Doya and Yoshizawa [5] demonstrated that RNNs can acquire two limit cycles in the gradient descent method using initialization with small connection weights, whereas learning for more than three limit cycles is difficult [6]. This is due to the fact that the learning of several time series causes a conflict with respect to the changing of the connection weights. How to form RNN models that can learn several temporal sequence patterns has proved to be a challenging problem.

There have been some approaches to this problem. In order to avoid conflicts in the change of parameters, the mixture-of-experts-type architecture has been investigated [7, 8]. The mixture-of-experts model consists of RNNs as experts and a hierarchical gating mechanism. At the end of successful learning, each expert implements attractor dynamics as locally represented knowledge, and a gating mechanism chooses only one expert at any time. The system can acquire many attractor patterns although there is a disadvantage in that the system does not have the generalization ability on the attractor patterns. As the other approach to implement multiple patterns, the parametric bias (PB) method has been developed to improve the learning capability of RNNs [9, 10]. In an RNN that employs the PB method (RNNPB), PB values provide the information needed in order to individualize each sequence. It has been reported that the number of time series that RNNPBs can learn is greater than that which RNNs without PB can learn. However, the PB method cannot avoid the conflict caused by each attractor learning. Therefore, learning multiple time series by an RNNPB tends to fail when the number of time series increases.

In the present study, we will focus on the training method for RNNs to learn multiple attractor dynamics. Furthermore, we will show that the present research is related to research into RNNs with contraction transition functions. In recent years, RNNs with contraction transition mapping have been investigated with respect to the performance of time series learning [11–13], generalization ability [14], and memory capacity [15]. Jaeger [11, 12] demonstrated that an “echo state network,” which is an RNN with contraction mapping, successfully learns the Mackey-Glass chaotic time series, a well-known benchmark system for time series prediction. In order to formally express the generalization ability, Hammer and Tiňo proved that RNNs with contraction are distribution-independent learnable in the probably approximately correct (PAC) sense [14]. From the above results, RNNs with contraction might be regarded as powerful tools for modeling dynamical systems. However, RNNs with contraction have difficulty in representing multiple attractor dynamics because dynamic states governed by the contraction transition function are globally attracted to one point. In this paper, the representation capability of RNNs with contraction mapping will be improved such that the RNNs can obtain multiple attractor dynamics.

We start by defining the concepts of the RNN and the training method for multiple attractor dynamics. The RNN has the Elman net-type architecture, and the training method for RNNs is basically based on the backpropagation through-time (BPTT) algorithm [16]. We then show in numerical simulation that the RNNs can acquire multiple periodic attractors constituted by five Lissajous curves, or a Van der Pol oscillator with twelve different parameters. Moreover, we consider why the RNNs successfully learn multiple attractors and how the performance of learnability depends on parameters of the RNNs. Finally, we link the results obtained herein to other learning strategies, and consider other advanced research topics.

2. Model

2.1. Recurrent Neural Network

We first consider a neural network model with recurrent connection, such as the Elman net [17] (see Figure 1). The RNN contains I/O units, orthogonal units, and internal units. We denote the dynamic states of I/O units, orthogonal units, and internal units at time step 𝑛 by 𝐱 𝑛 ∈ ℝ 𝑁 1 , 𝐫 𝑛 ∈ ℝ 𝑁 2 , and 𝐮 𝑛 ∈ ℝ 𝑁 3 , respectively. The RNN is defined by functions 𝑓 𝜽 and 𝑔 𝜽 with a parameter 𝜽 ≡ ( 𝐖 1 , 𝐖 2 , 𝐖 3 , 𝐕 , 𝐛 , 𝐝 ) , where 𝑓 𝜽 ∶ ℝ 𝑁 1 × ℝ 𝑁 2 × ℝ 𝑁 3 → ℝ 𝑁 3 and 𝑔 𝜽 ∶ ℝ 𝑁 3 → ℝ 𝑁 1 × ℝ 𝑁 2 are of the forms where 𝐕 ∈ ℝ 𝑁 3 × 𝑁 1 𝑁 2 , and 𝑏 ∈ ℝ 𝑁 3 are matrices, 𝑑 ∈ ℝ 𝑁 1 𝑁 2 and 𝜖 ∈ ℝ are vectors, 0 ≤ 𝜖 ≤ 1 is a time constant that satisfies 𝐹 , and 𝐹 𝑖 = t a n h denotes a componentwise application such as 𝑛 .

846040.fig.001
Figure 1: Architecture of the recurrent neural network. Solid arrows, dotted arrows, and boxes represent fixed connections, adjustable connections, and network states, respectively.

Dynamic states of the RNN at time step 𝐱 𝑛 , 𝐫 𝑛 = 𝑔 𝜽 𝐮 𝑛 , 𝐮 𝑛 + 1 = 𝑓 𝜽 𝐱 𝑛 , 𝐫 𝑛 , 𝐮 𝑛 = 𝑓 𝜽 𝑔 𝜽 𝐮 𝑛 , 𝐮 𝑛 . ( 3 ) are updated according to From these equations, the RNN can be represented by an 𝑓 𝜽 ∶ ℝ 𝑁 1 × ℝ 𝑁 2 × ℝ 𝑁 3 → ℝ 𝑁 3 -dimensional dynamical system.

We now define bistability for the RNN.Definition 1. Assume 𝑓 𝜽 is as above. The function 𝐮 is bistable with respect to the third variable 𝜔 > 1 if a real value 𝑁 𝑠 and an integer 𝑤 𝑖 𝑗 = 𝜔 i f 𝑖 ≤ 𝑁 𝑠 𝑤 , 𝑖 = 𝑗 , 𝑖 𝑗 = 0 i f 𝑖 ≤ 𝑁 𝑠 | | 𝑤 , 𝑖 ≠ 𝑗 , 𝑖 𝑗 | | < 1 𝑁 3 o t h e r w i s e , ( 4 ) exist such thatfor every element 𝐖 3 of the matrix 𝑓 𝜽 .

The bistability of a function 𝑓 𝜽 is a key concept of our learning method. We will show in Section 4.1 that the bistable function 1 ≤ 𝑘 ≤ 𝑚 plays an important role in the learning of multiple attractor dynamics.

2.2. Learning Method

We present a formulation of the training procedure for the RNN with a multiple teacher I/O time series. For every 𝐿 𝑘 ∈ ℕ and ( 𝐱 1 ( 𝑘 ) , … , 𝐱 𝐿 ( 𝑘 ) 𝑘 ) , we assume that 𝐿 𝑘 is a sequence of teacher I/O of length 𝐖 1 ( 0 ) .

Initialization of Parameters
We initialize every element of matrices 𝐖 2 ( 0 ) , 𝐕 ( 0 ) , and 𝐛 ( 0 ) and vectors 𝐝 ( 0 ) and ( − 1 / 𝑁 3 , 1 / 𝑁 3 ) randomly from the uniform distribution in the interval 𝐖 3 ( 0 ) . A matrix 𝑓 𝜽 is randomly assigned such that 1 ≤ 𝑘 ≤ 𝑚 , 𝐮 1 ( 𝑘 ) ( 0 ) is bistable. For all [ − 1 , 1 ] is randomly initialized in the interval 𝐫 𝑛 ( 𝑘 ) ( 0 ) .
Assume that 𝑚 is an ( 𝐬 𝑛 ( 𝑘 , 1 ) ( 0 ) , … , 𝐬 𝑛 ( 𝑘 , 𝑚 ) ( 0 ) ) -tuple of vectors 1 ≤ 𝑘 ≤ 𝑚 for 1 ≤ 𝑛 ≤ 𝐿 𝑘 and 𝐬 𝑛 ( 𝑘 , 𝑙 ) ( 0 ) , and that the dimension of 𝐬 ( 𝑘 ′ , 𝑙 ′ ) 𝑛 ′ ( 0 ) is equivalent to that of 𝑙 = 𝑙 if 𝐬 𝑛 ( 𝑘 , 𝑙 ) ( 0 ) . We initialize 𝐬 𝑛 ( 𝑘 , 𝑙 ) ( 0 ) = 𝟎 i f 𝑘 = 𝑙 , − 𝟏 o t h e r w i s e . ( 5 ) such that

Run Network with Teacher I/O and Compute Error Function
For every ( ̂ 𝐱 1 ( 𝑘 ) ̂ 𝐱 ( 𝑡 ) , … , 𝐿 ( 𝑘 ) 𝑘 ( 𝑡 ) ) , the sequence 𝑡 of I/O units of the RNN at learning step ̂ 𝐱 𝑛 ( 𝑘 ) ̂ 𝐫 ( 𝑡 ) , 𝑛 ( 𝑘 ) ( 𝑡 ) = 𝑔 𝜽 ( 𝐭 ) 𝐮 𝑛 ( 𝑘 ) , 𝐮 ( 𝑡 ) ( 𝑘 ) 𝑛 + 1 ( 𝑡 ) = 𝑓 𝜽 ( 𝑡 ) 𝐱 𝑛 ( 𝑘 ) , 𝐫 𝑛 ( 𝑘 ) ( 𝑡 ) , 𝐮 𝑛 ( 𝑘 ) . ( 𝑡 ) ( 6 ) is defined by The error function 𝑡 of the RNN at learning step 𝑘 with the 𝐸 ( 𝑘 ) ( 𝑡 ) = 𝐿 𝑘 𝑛 = 1 𝑒 ̂ 𝐱 𝑛 ( 𝑘 ) ( 𝑡 ) , 𝐱 𝑛 ( 𝑘 ) ̂ 𝐫 + 𝑒 𝑛 ( 𝑘 ) ( 𝑡 ) , 𝐫 𝑛 ( 𝑘 ) ( 𝑡 ) , ( 7 ) th teacher I/O time series is defined bywhere 𝑒 ( 𝐱 , 𝐱 ) = ( 1 / 2 ) ( 𝐱 − 𝐱 ) T ⋅ ( 𝐱 − 𝐱 ) denotes the mean square error function 𝐸 ( 𝑡 ) . Finally, the error function 𝑡 at learning step 𝐸 ( 𝑡 ) = 𝑚 𝑘 = 1 𝐸 ( 𝑘 ) ( 𝑡 ) . ( 8 ) is defined by

Update Parameters
Let 𝐝 ( 𝑡 ) } 𝑡 be a parameter of the RNN at learning step 𝝆 ( 𝑡 + 1 ) . We determine the parameter 𝝆 ( 𝑡 + 1 ) = 𝝆 ( 𝑡 ) + 𝛼 Δ 𝝆 ( 𝑡 ) , Δ 𝝆 ( 𝑡 ) = 𝛽 Δ 𝝆 ( 𝑡 − 1 ) − 𝜕 𝐸 ( 𝑡 ) , 𝜕 𝝆 ( 𝑡 ) ( 9 ) by where 𝛽 and 𝐖 3 ( 𝑡 ) are the constants of the learning rate and momentum, respectively. On the other hand, a connection matrix 𝐖 3 ( 𝑡 + 1 ) = 𝐖 3 ( 𝑡 ) is not changed as 𝐮 1 ( 𝑘 ) ( 𝑡 + 1 ) in order to hold the bistability condition. We compute the initial state 𝑡 + 1 of the internal units at learning step 𝐮 1 ( 𝑘 ) ( 𝑡 + 1 ) = 𝐮 1 ( 𝑘 ) ( 𝑡 ) + 𝛼 Δ 𝐮 1 ( 𝑘 ) ( 𝑡 ) , Δ 𝐮 1 ( 𝑘 ) ( 𝑡 ) = 𝛽 Δ 𝐮 1 ( 𝑘 ) ( 𝑡 − 1 ) − 𝜕 𝐸 ( 𝑡 ) 𝜕 𝐮 1 ( 𝑘 ) , ( 𝑡 ) ( 1 0 ) such that where 𝛼 , and 𝐬 𝑛 ( 𝑘 , 𝑙 ) ( 𝑡 ) is the constant of the learning rate of the initial state. Assume that 𝐫 𝑛 ( 𝑘 ) ( 𝑡 ) is a vector as a component of the orthogonal units 𝐫 𝑛 ( 𝑘 ) ( 𝑡 ) = ( 𝐬 𝑛 ( 𝑘 , 1 ) ( 𝑡 ) , … , 𝐬 𝑛 ( 𝑘 , 𝑚 ) ( 𝑡 ) ) , such as 𝐬 𝑛 ( 𝑘 , 𝑙 ) ( 𝑡 + 1 ) . The vector 𝐬 𝑛 ( 𝑘 , 𝑙 ) 𝐬 ( 𝑡 + 1 ) = 𝑛 ( 𝑘 , 𝑙 ) ( 𝑡 ) + 𝛼 Δ 𝐬 𝑛 ( 𝑘 , 𝑙 ) ( 𝑡 ) i f 𝑘 = 𝑙 , − 𝟏 o t h e r w i s e , ( 1 1 ) Δ 𝐬 𝑛 ( 𝑘 , 𝑙 ) ( 𝑡 ) = 𝛽 Δ 𝐬 𝑛 ( 𝑘 , 𝑙 ) ( 𝑡 − 1 ) − 𝜕 𝐸 ( 𝑡 ) 𝜕 𝐬 𝑛 ( 𝑘 , 𝑙 ) , ( 𝑡 ) ( 1 2 ) is defined by where 𝐸 ( 𝑡 ) , and 𝛼 , 𝛼 is the constant of the learning rate of the orthogonal units.
Note that the maximum value of the error function 𝛼 depends on the number of units and the length of the teacher I/O time series. Thus, we should scale the learning rates 𝛾 , 𝛾 , and 𝛾 with the number of units and length of sequences. In the present paper, we consider parameters 𝛼 = 𝛾 / ( 𝑁 1 + 𝑁 2 ) ∑ 𝑘 𝐿 𝑘 , 𝛼 = 𝛾 / ( 𝑁 1 + 𝑁 2 ) , and 𝛼 = 𝛾 / ( 𝑁 1 + 𝑁 2 ) such that 𝑥 ( 1 ) 𝑛 , 1 = 4 5 s i n 2 𝜋 𝑛 𝑀 , 𝑥 ( 1 ) 𝑛 , 2 = 4 5 c o s 2 𝜋 𝑛 𝑀 , 𝑥 ( 2 ) 𝑛 , 1 = 4 5 s i n 4 𝜋 𝑛 𝑀 , 𝑥 ( 2 ) 𝑛 , 2 = 4 5 c o s 2 𝜋 𝑛 𝑀 , 𝑥 ( 3 ) 𝑛 , 1 = 𝑥 ( 2 ) 𝑛 , 2 , 𝑥 ( 3 ) 𝑛 , 2 = 𝑥 ( 2 ) 𝑛 , 1 , 𝑥 ( 4 ) 𝑛 , 1 = 4 5 s i n 2 𝜋 𝑛 𝑀 , 𝑥 ( 4 ) 𝑛 , 2 = 4 5 c o s 4 𝜋 𝑛 𝑀 , 𝑥 ( 5 ) 𝑛 , 1 = 𝑥 ( 4 ) 𝑛 , 2 , 𝑥 ( 5 ) 𝑛 , 2 = 𝑥 ( 4 ) 𝑛 , 1 , ( 1 3 ) , and 𝑀 = 3 2 .

3. Numerical Experiments

In this section, we conduct two types of experiments as examples of using the training method for RNNs proposed in Section 2. The first experiment shows the learning of five Lissajous curves. The second experiment shows the training of multiple attractors of a Van der Pol oscillator with 12 different parameters.

3.1. Experiment 1: Lissajous Curves
3.1.1. Teacher I/O Time Series

Our first task is to learn the five Lissajous curves defined by and we consider constants 1 ≤ 𝑘 ≤ 5 and 𝜖 for all 0 . 1 (see Figure 2).

fig2
Figure 2: Trajectories of the teacher I/O time series in experiment 1.
3.1.2. Learning and Testing

We now describe the specific conditions applied to RNN training. The time constant 𝑁 2 is set to 1 0 . The number 𝐬 𝑛 ( 𝑘 , 𝑙 ) of orthogonal units is 2 , and the dimension of a vector 1 ≤ 𝑙 ≤ 5 is 𝑓 𝜽 for all 𝑁 3 = 3 0 , 𝑁 𝑠 = 1 5 . Suppose that 𝜔 = 2 . 5 is bistable with 𝛾 = 0 . 1 , 𝛾 = 𝛾 = 0 . 0 1 , and 𝛽 = 0 . 9 . The learning rates and momentum are given by 𝐸 ( 𝑘 ) ( 𝑡 ) , and 𝑝 , respectively.

Figure 3 shows the error function 𝑞 for 20 000 learning steps. We also show the Kullback-Leibler divergence between the teacher I/O time series and a sequence of I/O units in the RNN computed by (3) which do not use external perturbation by the teaching sequences. We use the Kullback-Leibler divergence as a measure of the discrepancy between two sequences. Formally, the Kullback-Leibler divergence between two probability distributions 𝑑 K L ( 𝑝 , 𝑞 ) = l o g 𝑝 ( 𝑥 ) − l o g 𝑞 ( 𝑥 ) 𝑝 ( 𝑥 ) 𝑑 𝑥 . ( 1 4 ) and 0 is defined asBy definition, in order to compute the Kullback-Leibler divergence, it is necessary to obtain probability distributions of the teacher I/O time series and a sequence of I/O units. However, obtaining the probability distribution of a sequence of I/O units is very difficult. Therefore, we quantize a time series of real-valued vectors into a symbolic sequence such that if the real value is less than 1 , then the symbol 𝑙 is appropriated, and otherwise the symbol 𝐮 1 ( 𝑘 ) ( 𝑡 ) is appropriated. In addition, we use the probability distribution whereby sub-blocks with a block length of 1 ≤ 𝑘 ≤ 5 appear in the symbolic sequence given by the above quantization.

fig3
Figure 3: Error and Kullback-Leibler divergence between the teaching sequences and output generated by the RNN for 20 000 learning steps in experiment 1.

Figure 4 describes attractors of the trained RNN computed by (3) of which the initial state of internal units is 𝐱 𝑛 for each 𝐮 0 . By comparing the attractors with the teacher I/O time series displayed in Figure 2, we can see that the RNN can generate sequences similar to training data.

fig4
Figure 4: Time series 𝐱 𝑛 generated by the trained RNN in experiment 1. For each time series, only the initial state 𝐮 0 is different.

In Figure 5, examples of attractors for the RNN with random initial states are displayed. This shows that, in addition to the attractors corresponding to teacher I/O time series, there exist many attractors of the RNN.

846040.fig.005
Figure 5: Time series 𝑑 2 𝑦 𝑑 𝑡 2 − 𝜇 1 − 𝑦 2 𝑑 𝑦 𝑑 𝑡 + 𝑦 = 0 ( 1 5 ) generated by the trained RNN with random initial state 𝑘 in experiment 1.
3.2. Experiment 2: Van der Pol Attractors
3.2.1. Teacher I/O Time Series

Our second task is to learn multiple attractors given by the Van der Pol oscillator with different parameters. The Van der Pol oscillator defined byis a model of an electronic circuit that appeared in very early radios. It is well known that there exists a limit cycle for the Van der Pol oscillator. In this experiment, we consider twelve teacher I/O time series, where the 𝑥 ( 𝑘 ) 𝑛 , 1 = 𝑎 𝑦 ( 𝑡 ) + 𝑏 𝑘 , 𝑥 ( 𝑘 ) 𝑛 , 2 = 𝑎 𝑑 𝑦 ( 𝑡 ) 𝑑 𝑡 + 𝑐 𝑘 𝑛 , 𝑡 = 𝜏 𝑘 , ( 1 6 ) th teacher I/O time series 𝜇 = 0 . 2 5 is given byfor 𝑏 𝑘 and 𝑐 𝑘 , where 𝜏 𝑘 and 𝑏 𝑘 , 𝑐 𝑘 are constant parameters representing the center position of the limit cycle, and 𝜏 𝑘 is a time constant of the oscillator. We assume that the parameters 𝑏 𝑘 = ± 0 . 4 , and 𝑐 𝑘 = ± 0 . 4 are given by combining the values of 𝜏 𝑘 = 2 , 4 , 6 , 𝐿 𝑘 = 2 0 0 , and 1 ≤ 𝑘 ≤ 1 2 . Figure 6 shows the teacher I/O time series given by (16). The length of training data is ℝ 2 for 𝑓 𝜽 .

fig6
Figure 6: Teaching sequences of experiment 2. (a) Trajectories on 𝑁 3 = 4 0 . (b) Temporal trajectories of teaching sequences.
3.2.2. Learning and Testing

The parameters for learning are set as follows. Let 𝑁 𝑠 = 2 0 be bistable with 𝐬 𝑛 ( 𝑘 , 𝑙 ) and 1 . The dimension of the vector 1 ≤ 𝑙 ≤ 1 2 is 𝑁 2 = 1 2 for every 𝐮 1 ( 𝑘 ) ( 𝑡 ) so that 1 ≤ 𝑘 ≤ 1 2 . Other parameters are the same as in experiment 1.

The error function and the Kullback-Leibler divergence for 200 000 learning steps are displayed in Figure 7. Figure 8 shows attractors of the trained RNN, and the initial state of the internal units of which is set to 𝐱 𝑛 for every 𝐮 0 .

fig7
Figure 7: Error and Kullback-Leibler divergence between the teaching sequences and output generated by the RNN for 200 000 learning steps in experiment 1.
fig8
Figure 8: Time series ℝ 2 generated by the trained RNN in experiment 2. For each time series, the initial state 𝑋 is the same as the training phase. (a) Trajectories on 𝑈 . (b) Temporal trajectories.

This result allows us to consider that the RNN acquires multiple periodic attractors constituted by the teacher I/O time series.

4. Numerical Analysis

4.1. Contraction and Bistability

Assume that 𝑈 and 𝑓 ∶ 𝑋 × 𝑈 → 𝑈 are sets and that 𝑈 is equipped with a metric structure. A function 𝐶 ∈ [ 0 , 1 ) is a contraction with respect to ∀ 𝑢 1 , 𝑢 2 | | ∈ 𝑈 ∀ 𝑥 ∈ 𝑋 𝑓 ( 𝑥 , 𝑢 1 ) − 𝑓 ( 𝑥 , 𝑢 2 ) | | | | 𝑢 ≤ 𝐶 1 − 𝑢 2 | | ( 1 7 ) if a real value 𝑥 ∈ 𝑋 exists such that the inequalityholds for all ℝ 𝑁 3 and 𝐮 𝑛 + 1 = 𝑓 𝜽 ( 𝑔 𝜽 ( 𝐮 𝑛 ) , 𝐮 𝑛 ) . Lemma 2. Let one consider a dynamical system on 𝑓 𝜽 defined by the transition function 𝑔 𝜽 , where 𝑤 𝑖 𝑗 and 𝐖 3 are defined in (1) and (2), respectively. Assume that each element 𝜈 ∈ ℝ of the matrix 𝐖 1 satisfies (4), and 𝐖 2 is the maximum absolute value of elements in 𝐛 , 𝑁 𝑥 − 𝜔 t a n h ( 𝑥 ) + 𝑣 1 + 𝑁 2 + 1 = 0 , 𝑥 ∈ ℝ , ( 1 8 ) , and 2 𝑁 𝑠 . If there exist three solutions of then
(1) there are 𝑈 ⊂ ℝ 𝑁 3 invariant sets of a dynamical system 𝑓 𝜽 ;(2) suppose that ℝ 𝑁 1 × ℝ 𝑁 2 × 𝑈 is an invariant set of the dynamical system; then, the restriction of 𝑈 to | ⋅ | ∞ is a contraction with respect to | 𝐮 − 𝐮 | ∞ ≡ m a x 1 ≤ 𝑖 ≤ 𝑁 3 | 𝑢 𝑖 − 𝑢 𝑖 | and the maximum norm 𝑥 1 > 𝑥 2 > 𝑥 3 , where 𝑥 1 , 𝑥 2 > 0 .Proof. We suppose that (18) has three solutions, such as 𝑥 3 < 0 (see Figure 9). In general, 1 ≤ 𝑖 ≤ 𝑁 𝑠 and 𝑥 1 ≥ 𝑢 𝑛 ( 𝑖 ) ≥ 𝑥 2 .
(1) Assume 𝑢 𝑛 ( 𝑖 ) 𝑢 − 𝜔 t a n h 𝑛 ( 𝑖 ) 𝑁 + 𝑣 1 + 𝑁 2 + 1 ≤ 0 ⟹ 𝑢 𝑛 ( 𝑖 ) 𝑢 − 𝜔 t a n h 𝑛 ( 𝑖 ) − 𝑂 𝑛 ( 𝑖 ) 𝑢 ≤ 0 ⟺ − 𝜖 𝑛 ( 𝑖 ) 𝑢 − 𝜔 t a n h 𝑛 ( 𝑖 ) − 𝑂 𝑛 ( 𝑖 ) ≥ 0 ⟺ ( 1 − 𝜖 ) 𝑢 𝑛 ( 𝑖 ) 𝑢 + 𝜖 𝜔 t a n h 𝑛 ( 𝑖 ) + 𝑂 𝑛 ( 𝑖 ) ≥ 𝑢 𝑛 ( 𝑖 ) ⟺ 𝑢 ( 𝑖 ) 𝑛 + 1 ≥ 𝑢 𝑛 ( 𝑖 ) ( 1 9 ) and 𝐱 𝑛 ∈ ℝ 𝑁 1 , 𝐫 𝑛 ∈ ℝ 𝑁 2 . Then, the expressionis satisfied for all 𝑂 𝑛 ( 𝑖 ) , and 𝑖 , where 𝐎 𝑛 = 𝐖 1 ⋅ 𝐱 𝑛 + 𝐖 2 ⋅ 𝐫 𝑛 + 𝐛 is the 𝑢 ( 𝑖 ) 𝑛 + 1 ≥ 𝑢 𝑛 ( 𝑖 ) th element of the vector 𝑥 1 ≥ 𝑢 𝑛 ( 𝑖 ) ≥ 𝑥 2 . Hence, 𝑢 𝑛 ( 𝑖 ) ≥ 𝑥 1 if 𝑢 ( 𝑖 ) 𝑛 + 1 ≥ 𝑥 1 . Furthermore, if 𝑢 𝑛 ( 𝑖 ) ≥ 𝑥 1 𝑢 ⟹ t a n h 𝑛 ( 𝑖 ) 𝑥 ≥ t a n h 1 b e c a u s e 𝑥 1 > 0 ⟹ ( 1 − 𝜖 ) 𝑢 𝑛 ( 𝑖 ) 𝑢 + 𝜖 𝜔 t a n h 𝑛 ( 𝑖 ) + 𝑂 𝑛 ( 𝑖 ) ≥ ( 1 − 𝜖 ) 𝑥 1 𝑥 + 𝜖 𝜔 t a n h 1 + 𝑂 𝑛 ( 𝑖 ) ⟹ 𝑢 ( 𝑖 ) 𝑛 + 1 ≥ 𝑥 1 ( b e c a u s e 𝑥 1 ≥ 𝑢 𝑛 ( 𝑖 ) ≥ 𝑥 2 ⟹ 𝑢 ( 𝑖 ) 𝑛 + 1 ≥ 𝑢 𝑛 ( 𝑖 ) . ( 2 0 ) , then [ 𝑥 1 , ∞ ) becauseTherefore, the region 𝐮 𝑛 is a stable set of the 𝑢 𝑛 ( 𝑖 ) ∈ [ 𝑥 1 , ∞ ) th element of vector 𝑢 ( 𝑖 ) 𝑛 + 𝑘 ∈ [ 𝑥 1 , ∞ ) satisfying the fact that if 𝑘 ∈ ℕ then 𝑢 𝑛 ( 𝑖 ) ≤ − 𝑥 1 for any 𝑢 ( 𝑖 ) 𝑛 + 1 ≤ − 𝑥 1 .
Similarly, we can easily show that if 𝑖 , then 𝐮 𝑛 . Thus, there are two stable regions of the 1 ≤ 𝑖 ≤ 𝑁 𝑠 th element of vector 2 𝑁 𝑠 for each 𝑈 ⊂ ℝ 𝑁 3 . Then, there are 𝐮 𝑛 , 𝐮 𝑛 ∈ 𝑈 invariant sets.
(2) Let 1 ≤ 𝑖 ≤ 𝑁 𝑠 be the invariant set presented above. Assume that | | 𝑢 ( 𝑖 ) 𝑛 + 1 − 𝑢 ( 𝑖 ) 𝑛 + 1 | | = | | 𝑢 ( 1 − 𝜖 ) 𝑛 ( 𝑖 ) − 𝑢 𝑛 ( 𝑖 ) + 𝜖 𝜔 t a n h ( 𝑢 𝑛 ( 𝑖 ) 𝑢 − t a n h 𝑛 ( 𝑖 ) | | < | | 𝑢 ( 1 − 𝜖 ) 𝑛 ( 𝑖 ) − 𝑢 𝑛 ( 𝑖 ) 𝑢 + 𝜖 𝑛 ( 𝑖 ) − 𝑢 𝑛 ( 𝑖 ) | | = | | 𝑢 𝑛 ( 𝑖 ) − 𝑢 𝑛 ( 𝑖 ) | | ( 2 1 ) .
For any | 𝑥 | ≥ 𝑥 1 , the inequalityholds because if 𝑁 𝑠 ≤ 𝑖 ≤ 𝑁 3 , then | | 𝑢 ( 𝑖 ) 𝑛 + 1 − 𝑢 ( 𝑖 ) 𝑛 + 1 | | = | | | | 𝑢 ( 1 − 𝜖 ) 𝑛 ( 𝑖 ) − 𝑢 𝑛 ( 𝑖 ) + 𝜖 𝑗 𝑤 𝑖 𝑗 𝑢 t a n h 𝑛 ( 𝑗 ) 𝑢 − t a n h 𝑛 ( 𝑖 ) | | | | | | 𝑢 ≤ ( 1 − 𝜖 ) 𝑛 ( 𝑖 ) − 𝑢 𝑛 ( 𝑖 ) | | | | | | + 𝜖 𝑗 𝑤 𝑖 𝑗 𝑢 t a n h 𝑛 ( 𝑗 ) 𝑢 − t a n h 𝑛 ( 𝑗 ) | | | | | | 𝑢 ≤ ( 1 − 𝜖 ) 𝑛 ( 𝑖 ) − 𝑢 𝑛 ( 𝑖 ) | | | | | | + 𝜖 𝑗 𝑤 𝑖 𝑗 𝑢 𝑛 ( 𝑗 ) − 𝑢 𝑛 ( 𝑗 ) | | | | | | 𝑢 ≤ ( 1 − 𝜖 ) 𝑛 ( 𝑖 ) − 𝑢 𝑛 ( 𝑖 ) | | + 𝜖 𝑁 3 m a x 𝑗 | | 𝑤 𝑖 𝑗 𝑢 𝑛 ( 𝑗 ) − 𝑢 𝑛 ( 𝑗 ) | | | | 𝑢 ≤ ( 1 − 𝜖 ) 𝑛 ( 𝑖 ) − 𝑢 𝑛 ( 𝑖 ) | | + 𝜖 𝑁 3 m a x 𝑗 | | 𝑤 𝑖 𝑗 | | m a x 𝑗 | | 𝑢 𝑛 ( 𝑗 ) − 𝑢 𝑛 ( 𝑗 ) | | ≤ ( 1 − 𝜖 ) + 𝜖 𝑁 3 m a x 𝑗 | | 𝑤 𝑖 𝑗 | | m a x 𝑗 | | 𝑢 𝑛 ( 𝑗 ) − 𝑢 𝑛 ( 𝑗 ) | | < m a x 𝑗 | | 𝑢 𝑛 ( 𝑗 ) − 𝑢 𝑛 ( 𝑗 ) | | . ( 2 2 ) .
On the other hand, for every | 𝐮 𝑛 + 1 − 𝐮 𝑛 + 1 | ∞ < | 𝐮 𝑛 − 𝐮 𝑛 | ∞ ,
Then, 𝑓 𝜽 is obtained for any ℝ 𝑁 1 × ℝ 𝑁 2 × 𝑈 . Accordingly, the restriction of 𝑈 to | ⋅ | ∞ is a contraction with respect to 𝑁 1 , 𝑁 2 ∈ ℕ and the maximum norm 𝜈 ≥ 0 .

846040.fig.009
Figure 9: Schematic diagram of (18).

For any 𝑞 and 𝜔 ≥ 𝑞 , there is a real number 𝜔 such that if 𝐖 1 , then (18) has three solutions. Thus, if 𝐖 2 is large enough and matrices 𝑓 𝜽 and 2 𝑁 𝑠 represent small connection weights, then 𝑓 𝜽 contains 𝑁 3 − 𝑁 𝑠 invariant sets, and each restriction of 𝑁 3 − 𝑁 𝑠 to an invariant set is a contraction with respect to a third input. Moreover, the integer 𝜔 is the effective degree of freedom for each contraction mapping restricted to an invariant set. If 𝑁 𝑠 is a large value, then RNN can acquire a more complex time sequence. In Figures 10 and 11, we plot the Kullback-Leibler divergence of the trained RNN for parameters 𝜔 , 2 𝑁 𝑠 and 𝑁 3 − 𝑁 𝑠 , in which the training data are the same as those for experiment 1. These results imply that it is necessary that 𝜖 = 0 . 1 , 𝑁 2 = 1 0 , and d i m 𝐬 𝑛 ( 𝑘 , 𝑙 ) = 2 be large values in order to learn multiple attractor dynamics.

846040.fig.010
Figure 10: Kullback-Leibler divergence between the teaching sequences and output generated by the trained RNN with 𝑁 3 = 3 0 ( 𝑁 𝑠 = 1 5 ), 𝜖 = 0 . 1 , 𝜔 = 2 . 5 , and 𝑁 2 = 1 0 .
fig11
Figure 11: Kullback-Leibler divergence between the teaching sequences and output generated by the trained RNN with d i m 𝐬 𝑛 ( 𝑘 , 𝑙 ) = 2 , and 𝑁 3 = 2 0 ( 𝑁 3 = 3 0 ). (a) 𝑁 3 = 4 0 , (b) 𝐫 = ( 𝐬 ( 1 ) , … , 𝐬 ( 𝑚 ) ) , and (c) 𝐬 ( 𝑘 ) .
4.2. Orthogonality

In the last paragraph of the previous section, we have shown that RNNs have many stable regions, and the existence of the stable regions plays an important role in the learning of multiple sequences. However, the existence of multiple stable regions is not sufficient for success in the multiple attractor learning because if the change of parameters corresponding to each time series influences other changes, each time series cannot necessarily be embedded into each region. Similarly, this problem appears in the method of RNNPB.

In the training algorithm defined in Section 2, each state of orthogonal units 𝑘 is trained by (5) and (11). Thus, firing of 𝑘 only occurs in the generation of the 𝑑 𝑣 ( 𝑘 ) 𝑖 𝑗 ( 𝑡 ) th teaching sequence. This implies that orthogonal units allow the conflict of parameter changes caused by multiple time series learning to be avoided because orbits corresponding to each teaching I/O time series run around the orthogonal state space of the trained RNN.

In order to show the effect of the orthogonal units on the conflict among teaching sequences, we consider the 𝑑 𝑣 ( 𝑘 ) 𝑖 𝑗 ( 𝑡 ) = 𝜕 𝐸 ( 𝑘 ) ( 𝑡 ) 𝜕 𝑣 𝑖 𝑗 𝜕 𝐸 ( 𝑡 ) 𝜕 𝑣 𝑖 𝑗 − 1 , ( 2 3 ) th learning ratio 𝑣 𝑖 𝑗 defined bywhere 𝑑 𝑣 ( 𝑘 ) 𝑖 𝑗 ( 𝑡 ) is an element of the matrix 1 . If 𝑣 𝑖 𝑗 is nearly equal to 𝑘 , then the change in 𝑑 𝑉 ( 𝑘 ) ( 𝑡 ) is approximately independent of teaching sequences rather than the 𝑑 𝑉 ( 𝑘 ) 1 ( 𝑡 ) = ( 𝑁 3 − 𝑁 𝑠 ) | 𝑅 ( 𝑘 ) | 𝑖 ∈ 𝑅 ( 𝑘 ) 𝑁 3 𝑗 = 𝑁 𝑠 𝑑 𝑣 ( 𝑘 ) 𝑖 𝑗 ( 𝑡 ) , ( 2 4 ) th sequence. In Figure 12, we plot the value 𝑅 ( 𝑘 ) determined bywhere 𝑑 𝑉 ( 𝑘 ) ( 𝑡 ) is a set of indices corresponding to the elements of the vector 𝑘 . The value 𝐬 ( 𝑘 ) represents the average of the 𝑑 𝑉 ( 𝑘 ) ( 𝑡 ) th learning ratio for connections between internal units and orthogonal units 1 / 𝑚 = 0 . 2 . In this numerical experiment, for each learning step, 𝑚 is clearly larger than 𝑘 , where 𝐬 ( 𝑘 ) is the number of teaching sequences. Then, the sum of the 𝑉 th learning ratios of connection weights between internal units and orthogonal units 𝑊 1 is dominant. Therefore, in changing matrix 𝑊 2 , there is no conflict generated by multiple teaching sequences. However, we could not find a strong bias of the learning ratio for the matrices 𝑣 𝑖 𝑗 and 𝑉 and every element 𝑖 < 𝑁 𝑠 of 𝑑 𝑉 ( 𝑘 ) ( 𝑡 ) with 𝑘 . Thus, we consider that connection weights between internal units and orthogonal units encode information on an individual time series, and other connection weights encode whole information.

846040.fig.012
Figure 12: Average 𝐬 ( 𝑘 ) of the th learning ratio for the connections between internal units and orthogonal units for 200 000 learning steps in experiment 1.

5. Discussion

In this report, we have investigated a method of embedding multiple time series into a single RNN. In order to clarify the characteristics of the proposed approach, we compare the proposed approach with other approaches with respect to information representation of multiple sequences in the models. The mixture-of-RNN-experts-type model composes local representation in an RNN for each sequence. The local representation provides robustness against changing the parameters in learning, but it lacks the ability to extract common patterns included in the sequences because of the independency of the local representation. In the proposed model, the local representation is constructed into orthogonal units, while the global representation is also constructed into internal units using the connection weights between I/O units and internal units. Since each sequence generated by the proposed model shares the state space and connection weights, the model can extract common patterns of the sequences as well as conventional neural networks.

Another characteristic, which clarifies the difference between our model and other models, is whether the classification of each time series is self-organized into the state space. For example, in the mixture-of-RNN-experts-type model, the allocation of time series to each RNN is determined automatically. As another example, in the RNNPB model, PB values are self-organized such that the PB can individualize each time series. On the other hand, the proposed model needs the information of orthogonalization for each time series. Since the sparse firing patterns which appear in orthogonal units, corresponding to time series, are given as teaching information externally, the classification of sequences is not self-organized. The characteristic whereby the time series cannot be automatically classified is a disadvantage of the proposed model. However, the time series can be classified using other clustering techniques before applying the proposed method. Thus, by combining the proposed method and other clustering techniques, an algorithm that automatically classifies and generates multiple time series can be constructed.

6. Conclusion

In this paper, we have presented an RNN model and a learning algorithm that can acquire the ability to generate multiple sequences. The RNN model consists of two distinct properties called bistability and orthogonality. Bistability guarantees the existence of multiple attractor structures in RNNs, and provides the RNNs with contraction transition mapping. Orthogonality, which is given as a function of the orthogonal vectors of RNNs, helps prevent conflicts with respect to parameter changes caused by multiple training sequences. In the numerical experiments, RNNs which have bistability and orthogonality can learn multiple periodic attractors constituted by five Lissajous curves or 12 Van der Pol oscillators. Based on these results, the proposed model can be applied to the modeling of various types of dynamical systems that include multiple attractors.

References

  1. K. Funahashi and Y. Nakamura, “Approximation of dynamical systems by continuous time recurrent neural networks,” Neural Networks, vol. 6, no. 6, pp. 801–806, 1993. View at Publisher · View at Google Scholar
  2. H. T. Siegelmann and E. D. Sontag, “Analog computation via neural networks,” Theoretical Computer Science, vol. 131, no. 2, pp. 331–360, 1994. View at Publisher · View at Google Scholar · View at MathSciNet
  3. H. T. Siegelmann and E. D. Sontag, “On the computational power of neural nets,” Journal of Computer and System Sciences, vol. 50, no. 1, pp. 132–150, 1995. View at Publisher · View at Google Scholar · View at MathSciNet
  4. Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 157–166, 1994. View at Publisher · View at Google Scholar · View at PubMed
  5. K. Doya and S. Yoshizawa, “Memorizing oscillatory patterns in the analog neuron network,” in Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN '89), vol. 1, pp. 27–32, Washington, DC, USA, June 1989. View at Publisher · View at Google Scholar
  6. F.-S. Tsung, Modeling dynamical systems with recurrent neural networks, Ph.D. thesis, Department of Computer Science, University of California, San Diego, Calif, USA, 1994.
  7. D. M. Wolpert and M. Kawato, “Multiple paired forward and inverse models for motor control,” Neural Networks, vol. 11, no. 7-8, pp. 1317–1329, 1998. View at Publisher · View at Google Scholar
  8. J. Tani and S. Nolfi, “Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems,” Neural Networks, vol. 12, no. 7-8, pp. 1131–1141, 1999. View at Publisher · View at Google Scholar
  9. J. Tani, “Learning to generate articulated behavior through the bottom-up and the top-down interaction processes,” Neural Networks, vol. 16, no. 1, pp. 11–23, 2003. View at Publisher · View at Google Scholar
  10. J. Tani and M. Ito, “Self-organization of behavioral primitives as multiple attractor dynamics: a robot experiment,” IEEE Transactions on Systems, Man and Cybernetics Part A, vol. 33, no. 4, pp. 481–488, 2003. View at Publisher · View at Google Scholar
  11. H. Jaeger, “Short term memory in echo state networks,” National Research Center for Information Technology, Bremen, German, 2001.
  12. H. Jaeger and H. Haas, “Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication,” Science, vol. 304, no. 5667, pp. 78–80, 2004. View at Publisher · View at Google Scholar · View at PubMed
  13. W. Maass, T. Natschläger, and H. Markram, “A fresh look at real-time computation in generic recurrent neural circuits,” Institute for Theoretical Computer Science, TU Graz, Graz, Austria, 2002.
  14. B. Hammer and P. Tiňo, “Recurrent neural networks with small weights implement definite memory machines,” Neural Computation, vol. 15, no. 8, pp. 1897–1929, 2003. View at Publisher · View at Google Scholar
  15. O. L. White, D. D. Lee, and H. Sompolinsky, “Short-term memory in orthogonal neural networks,” Physical Review Letters, vol. 92, no. 14, Article ID 148102, 4 pages, 2004. View at Publisher · View at Google Scholar
  16. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1: Foundations, D. E. Rumelhart and J. L. McLelland, Eds., pp. 318–362, MIT Press, Cambridge, Mass, USA, 1986.
  17. J. L. Elman, “Finding structure in time,” Cognitive Science, vol. 14, no. 2, pp. 179–211, 1990. View at Publisher · View at Google Scholar