Abstract
This paper studies a small neural network with three neurons. First, the activation function takes the sign function. Although the network is a simple hybrid system with all subsystems being exponentially stable, we find that it can exhibit very complex dynamics such as limit cycles and chaos. Since the sign function is a limit case of sigmoidal functions, we find that chaos robustly exists with some different activation functions, which implies that such chaos in this network is more related to its weight matrix than the type of activation functions. For chaos, we present a rigorous computer-assisted study by virtue of topological horseshoe theory.
1. Introduction
Since substantial evidence of chaos is found in biological studies of natural neuronal systems, researchers have realized that chaos is much helpful for neural networks escaping the local minima and may play an essential role in the storage and retrieval of information [1–3]. Thus, a thorough investigation on chaotic dynamics of neural networks is significant for neural networks studies, which has become a popular research field in recent decades. A lot of artificial neural networks have been proposed in order to realize chaotic and hyperchaotic attractors [4–12].
Generally, neural networks in real world have very high dimension, which is too hard to study. Fortunately, research in anatomy and physiology shows that neurons in biological brains are grouped together into functional circuits [13, 14]. This implies that a first step before studying chaos in high-dimensional neural networks should be to have detailed investigations on chaos in low-dimensional networks with only a few nerons [15–19].
The nonlinearity of neural systems usually comes from the activation functions, which is the reason causing chaos. There are many types of sigmoid functions used in literature, such as the hyperbolic tangent function, piecewise linear functions, the Logistic (sigmoid) function, and the sign functions. So, we are interested in whether chaos in a neural network can take place with all these functions or whether chaos can take place for any type sigmoidal activation function.
In order to answer the two questions, this paper will take a limit of the sigmoidal functions by zooming out the input scale and study a small Hopfield neural network (HNN) with hard switches. Such sign function is not only extremely easy to implement, but also of dynamical and biological significance in gene regulatory networks [4, 5]. An interesting phenomenon we find in this paper is that the HNN can demonstrate chaos, although it is a switching system that only consisted of stable subsystems; such chaos still exists even when we replace the sign function with many other activation functions.
The following paper is organized as follows. Section 2 presents the HNN and demonstrates its chaotic behavior with different activation functions; Section 3 first recalls some theoretical results of topological horseshoe and then presents computer-assisted proof of the existence of chaos; Section 4 draws conclusions.
2. Chaos in the Small Network with Different Activation Functions
The Hopfield neural network is described by [20] where is a sigmoidal activation function and is an matrix, called weight matrix or connection matrix describing the strength of connections between neurons.
In this paper, we only consider three neurons, that is, and take , then the small network can be written with the following equation in vector form: where , .
Now let , , , and where is a parameter. Then we can easily solve the isolated equilibrium points of (2) while the parameter from to , as illustrated in Table 1. Since (2) is symmetric with respect to the origin, its equilibria appears in pairs, and the origin is always an equilibrium. Numerical computation suggests that the origin is always unstable. Since only takes value of , , and , it is easy to see from the equation that the other equilibria are always exponentially stable.
To explore complex dynamics in (2), we simulate the system with then numerically find that there are limit circles and strange attractors, as illustrated with the bifurcation plot in Figure 1. The dynamics relies greatly on its initial condition; that is, different initial value may result in different kind of dynamics. During our simulation, we chose . The bifurcation diagram indicates that the system (2) is very likely to be chaotic when is near to zero.
After careful numerical computation, we obtain an invariant set called attractor in the sense that almost every trajectory with initial point near this set tends to this set, while this set contains a trajectory, that is, dense in it, as illustrated in Figure 2 for . It seems a chaotic attractor. In the next section, we will prove that this attractor is indeed chaotic with the topological horseshoe theory.
Since the sign function is a limit of most sigmoidal functions with large-scale input, we in this section will study existence of chaos for the HNN (1) with other sigmoidal activation function. For this purpose, we considerate the following equations: where is a positive scale factor. Replacing with , we have the equivalent system
So for any type sigmoidal activation function with its output range from to , if is great enough, will be sufficiently close to the discontinuous sign function. The robustness of (2) suggests that (4) could exhibit chaos with the same weight matrix (3). Taking the consideration of the bias, (1) may exhibit chaos for any type sigmoidal activation function when .
To illustrate this fact, we will give examples with a hyperbolic tangent function, a Logistic function, a piecewise linear function, and a very complicated function.
Case 1. takes the hyperbolic tangent function. Let . In order to make the neural system (5) chaos, we take . Then we get an attractor shown in Figure 3. The Lyapunov exponents are , , and . The first one is positive showing that the attractor is most likely chaotic.
Case 2. takes the Logistic (sigmoid) function. Take , then . From the previous subsection, we have a Hopfield neural network with the Logistic (sigmoid) activation function:
Here, is the bias.
Case 3. is a piecewise linear function. Let . In order to make the neural system (5) chaos, we also take . Then we get an attractor shown in Figure 4. The three Lyapunov exponents are , , and . The first one is positive suggesting the attractor is chaotic.
Case 4. is a complicated piecewise linear function. For the system (4) and the weight matrix (3) at , we take a more complicated randomly as shown in Figure 5. Let , a chaotic attractor appears as shown in Figure 6. Since the activation function is too complicate, it is not easy to compute the Lyapunov exponent of the attractor in Figure 6 with enough accuracy; so, in next section we will prove that this attractor is indeed chaotic with the topological horseshoe theory.
It is clear to see from Figures 3 to 6 that although their activation functions are much different, the chaotic attractor exists in all four cases and they all look similar. All the above four cases suggest that the existence of chaos in this neural network is robust on the type of activation functions. In another word, the complex dynamics of a small neural network should be more related to its weight matrix.
3. Computer-Assisted Proof of Chaos
The existence of a topological horseshoe is recognized as one of the most important signatures of chaos. With the horseshoe theory, one can not only prove chaos rigorously but also reveal the mechanism of chaos with its invariant set. So we will present computer-assisted proof of chaos in the HNN in this section. First, let us recall a theorem on topological horseshoe and the m-shift and then present our main results.
Let be a metric space, is a compact subset of , and is map satisfying the assumption that there exist mutually disjoint compact subsets of , the restriction of to each , that is, is continuous.
Definition 1. Let be a compact subset of , such that for each , is nonempty and compact; then is called a connection with respect to . Let be a family of connections s with respect to satisfying property: . Then is said to be -connected family with respect to .
Theorem 2. Suppose that there exists a -connected family with respect to . Then there exists a compact invariant set , such that is semiconjugate to -shift.
In this theorem, the -shift is also called Bernoulli shift sometimes, denoted by , where is the collection of all bi-infinite sequences and the shift map is defined as
It is well known that is a Cantor set, which is compact, totally disconnected, and perfect. As a dynamical system defined on , has a countable infinity of periodic orbits consisting of orbits of all periods, an uncountable infinity of aperiodic orbits, and a dense orbit. A direct consequence of these three properties is that the dynamics generated by the shift map is sensitive to initial conditions. Since is topologically semiconjugate to , which means that there exists a continuous surjection such that , must be also sensitive to initial conditions. Mathematically, the complexity of the system can be measured by its topological entropy, which roughly means the exponential growth rate of the number of distinguishable orbits as time advances. As another result of the semiconjugate, the topological entropy of , denoted by , is not less then . When , , therefore the system is chaotic. For more details of the above symbolic dynamics and horseshoe theory, we refer the reader to [21–24].
In what follows we will study existence of horseshoes embedded in the attractor in Figure 2. For this purpose, we will utilize the technique of cross section and the corresponding Poincaré map. Consider the section plane , as shown in Figure 2. The Poincaré map is chosen as follows. For each , is taken to be the second return point in under the flow with the initial condition .
To find the horseshoe, we use the efficient method proposed in [21, 25] which has been implemented with a MATLAB toolbox called “a toolbox for finding horseshoes in 2D map” (download: http://www.mathworks.com/matlabcentral/fileexchange/14075). The method is so powerful that it has been successfully applied in a number of chaotic systems [26–29], a fractional-order system [30],even a hyperchaotic system [31].
After many attempts, we carefully pick two quadrangles and in this section plane with their vertices as follows:
By means of interval analysis, we compute the Poincaré map on and and then have the following statement.
Theorem 3. For the Poincaré map corresponding to the cross sections and , there exists a closed invariant set for which is semiconjugate to the 2-shift map.
Proof. To prove this statement, we will find two disjoint compact subsets and of , such that the existence of a -connected family can be easily derived.
The first subset takes as shown in Figure 7, and the Poincaré map sends this subset to its image as follows:
showing that is mapped left of the side and is mapped right of the side . In this case, we say that the image lies wholly across the quadrangles and with respect to and .
The second subset takes as shown in Figure 8, and the Poincaré map sends this subset to its image as follows:
showing that is mapped left of the side and is mapped right of the side . In this case, we say that the image lies wholly across the quadrangles and with respect to and .
Generally, (2) can be regarded as a switching system consisting of eight very simple continuous subsystems and 12 quarter planes called switching planes; for detail, see [5]. In this system, every trajectory in the neighborhood of the attractor transversely intersects with the switching plane, and the Poincaré map can be regarded as a composition of a series of continuous submaps by the subsystems. Since each subsystem is linear, it is easy to prove that for the quadrangle region or all submaps are continuous; consequently, is continuous.
Note that the subsets and are mutually disjointed. It is easy to see from the whole acrossness of and with respect to and that there exists a -connected family with respect to and . In view of Theorem 2, this means that the Poincaré map is semiconjugate to 2-shift map.
It is easy to see from Theorem 3 that the entropy of is not less than , so the attractor in Figure 2 is a chaotic attractor indeed.
So we can prove the existence of chaos in the attractor shown in Figure 6 by the same way in Section 3. The new vertices of the two subsets and are
The images of and are shown in Figure 9, from which we can see easily that the images of and across simultaneously and , that is, analogous to Figures 7 and 8. From Theorem 2, we infer that there exists a -connected family with respect to and , where is the corresponding Poincaré map. From the topological horseshoe theory we have after similar arguments. The positive entropy suggests that the attractor in Figure 6 is chaotic indeed.
4. Conclusions
In this paper, we have studied a 3D Hopfield neural network with the sign activation function. Computer simulation shows that this HNN can exhibit chaotic attractors and limit cycles with respect to . In order to verify the chaotic behavior, we present a computer-assisted verification for the existence of horseshoes imbedded in this system. We also show evidence that chaos could possibly be exhibited by the HNN with any type of sigmoidal activation function. In another word, such chaos should be more related to its weight matrix than the type of activation functions. In addition, since the HNN is a switching system only consisting of stable subsystems, this fact suggests that the dynamics of a hybrid system could be much more complex than we used to think. This may be of interest to researchers of neural networks, nonlinear dynamics, and so on.
Acknowledgments
This work is supported in part by the National Natural Science Foundation of China (61104150), Natural Science Foundation Project of Chongqing (cstcjjA40044), and Doctoral Fund of CQUPT (A2009-12).