Abstract
As can be represented by neurons and their synaptic connections, attractor networks are widely believed to underlie biological memory systems and have been used extensively in recent years to model the storage and retrieval process of memory. In this paper, we propose a new energy function, which is nonnegative and attains zero values only at the desired memory patterns. An attractor network is designed based on the proposed energy function. It is shown that the desired memory patterns are stored as the stable equilibrium points of the attractor network. To retrieve a memory pattern, an initial stimulus input is presented to the network, and its states converge to one of stable equilibrium points. Consequently, the existence of the spurious points, that is, local maxima, saddle points, or other local minima which are undesired memory patterns, can be avoided. The simulation results show the effectiveness of the proposed method.
1. Introduction
Memory is a fundamental component of our human brain; how to simulate the human memory process has attracted many scientists attention in the research of cognitive systems and architectures [1, 2]. Attractor networks [3–6] have been one of the most popular models for memory storage and retrieval in recent decades since the hypothesis of attractor dynamics is supported and observed in the neocortex and hippocanpus in various memory experiments [6–9]. In general, an attractor network is a network of recurrently connected nodes in a biological network, whose states may settle to some stable patterns. One distinguish advantage is that the network can be represented by neurons and their synaptic connections. The particular pattern of such a recurrent network, which is called its “attractor” [10–13], can be stationary, time varying, or even stochastic. In theoretical neuroscience, different kinds of attractor neural networks have been associated with different functions, such as memory, motor behavior, and classification. In this paper, we consider the patterns or the so-called attractors as the stationary memory patterns stored in the dynamic system, which allows us to employ methods in dynamical systems to quantitatively analyze the characteristics such as the stability and robustness of the network.
Usually researchers design an attractor network and then propose its energy function, often called Lyapnov function, to analyze the network [14–16], since the energy function plays a very important role in analyzing the network stability and robustness. In this paper, we design the memory dynamics in an opposite direction. We first propose an energy function and then design the attractor network. The energy function actually contains the desired memory patterns we wish to store. Different from Hopfield networks [10, 14], we introduce multiplicative algebra into the attractor network. This is biologically possible since both addition algebra and multiplication algebra are the simplest and the most widespread of all operations in the nervous system [17]. In addition, the value of our proposed energy function is nonnegative and attains zero values only at the patterns stored in the network. This makes it easy to distinguish the desired memory patterns from some other possible undesired patterns which are called spurious points [18]. It is shown that the memory patterns are stored as the stable equilibrium points of the dynamical attractor networks, which are also the local minimum points of the energy function. Compared with existing results in attractor networks [7, 10, 14], the patterns are not necessarily binary and uncorrelated in this paper. Binary patterns simplify the network design significantly as seen in [14]. Also, the uncorrelated patterns give a minimum interactions of the interactions of the network, which makes its behavior analysis much easier as seen in [7, 19].
On the other hand, when a stimulus pattern is presented as an initial input of the dynamic system, the states of the system will converge to a particular stored attractor iteratively. This process is called associative memory retrieval. The “associative” means that the memory we retrieved was by its informational content rather than by names, addresses, or relative positions. One very touchy problem in the retrieval process is how to overcome the problem that the system states may converge to spurious points; see, for example, [18, 20]. Usually, there are two kinds of equilibrium points for a dynamic system, the stable ones and unstable ones. It is shown that only the stable equilibrium points, that is, the local minimum points, of the energy function exist in the proposed designed system. We also prove that those local minima are only the memory patterns stored in the network. Thus, the spurious points, that is, local maxima, saddle points, or other local minima which are the undesired memory patterns, can be avoided.
The contributions of this paper are summarized as follows. Firstly, we have proposed a new energy function different from the energy function in Hopfield networks. This makes it easy to differentiate memory patterns from possible spurious points. Secondly, we have presented an attractor network design based on the proposed energy function. The patterns stored in the attractor network can be nonbinary and either correlated or uncorrelated. Finally, we have proven that, when an arbitrary input stimulus is presented to the designed attractor network, the states converge to one of the stored patterns. This implies that there are no spurious points in the designed dynamical systems.
The rest of the paper is organized as follows. Some background knowledge is reviewed in Section 2. Section 3 introduces the main design method of the proposed attractor network. The convergence properties are analyzed in Section 4. The simulation examples are shown in Section 5. Finally, the paper is concluded in Section 6.
2. Background Knowledge
2.1. Multiplication Algebra in Nervous Systems
Addition algebra is both the simplest and one of the most widespread of all operations in nervous systems. However, as pointed in [17], a number of biological mechanisms could, in theory, implement a multiplication algebra. Actually multiplication can be implemented based on addition. For example, when we multiply two signals and , we can logarithmically transform the two, add the result,and then apply an exponential: Thus, later it can be seen that the memory dynamics in our proposed attractor network could be implemented in a network with neurons and synapses.
2.2. Notations and Definitions
Denote a nonlinear function and a pattern where is the dimension. For a nonlinear dynamical system , we have the following definitions.
Definition 1. A pattern is called an equilibrium point of , if is a zero vector at , which is denoted as .
Definition 2. A pattern is called a stable equilibrium point of , if at and the Jacobian matrix at is a positive definite matrix.
Definition 3. A pattern is called an unstable equilibrium point (saddle point) of , if at while the Jacobian matrix at is a negative semidefinite or an indefinite matrix.
3. A New Energy Function and the Attractor Network
A classical energy based attractor network is the Hopfield network invented by Hopfield [10, 14], which serves as content addressable memory systems with binary threshold nodes. Although Hopfield networks are guaranteed to converge to a local minimum, they may converge to a spurious point (undesired memory pattern) rather than a stored pattern (desired memory pattern). To solve this problem, we propose a new energy function, and an attractor network is then designed based on the proposed energy function, in which the patterns are not necessarily binary and uncorrelated. Finally it is concluded that the memory patterns are the stable equilibrium points of the dynamic system and spurious points can be avoided.
Assume that with are different stationary patterns that we wish to store. Our objective is to design an attractor network to store these patterns such that for can be retrieved when an input stimulus is located around the neighborhood of . Before presenting our proposed method, we would like to point out that an ideal attractor network should preserve the following two properties.
Property 1. are stable equilibrium points of the attractor network.
Property 2. are the only stable equilibrium points of the attractor network.
The energy function is designed as the following form:where is an Euclidian squared distance between and . The energy function is the product of the distance for . For the energy function , it can be checked that , , and if and only if for .
Remark 4. As mentioned in the Introduction, compared with the energy function in the Hopfield network [10, 14], the value of the proposed energy function is nonnegative and attains zero value only on the memory patterns stored in the network. This makes it easy to distinguish the memory patterns from spurious points [18] which may exist in an attractor network.
Note that the energy function attains its minimum only on the memory patterns . This inspires us to design a dynamical system which can make the energy function decrease iteratively. The gradient and the Hessian matrix are given bywhere is a dimensional identity matrix. Let be a random noise vector such thatwhere is a chosen small positive constant. Now an attractor network that decreases the energy function iteratively is represented aswhere the scalar constant is chosen asWe rewrote the differential equation for each neuron as for with Equations (7) and (8) imply that th neuron updates its state according to the synaptic inputs collected from other neurons via synaptic connection strength , in the presence of a random noise .
Remark 5. The Hopfield model has the advantage that it can be represented by neurons and their synaptic connections. Through carefully observing (5)–(8), we note that our proposed attractor network can also be represented by neurons and their synaptic connections by introducing the multiplication algebra in Section 2.1. In addition, the value of our proposed energy function is nonnegative and attains zero values only at the patterns stored in the network. This makes it easy to distinguish the memory patterns from some other possible undesired patterns which are called spurious points [18]. It is shown that the memory patterns are stored as the stable equilibrium points of the dynamical system, which are also the local minimum points of the energy function. Compared with existing results in attractor networks [7, 10, 14], the patterns are not necessarily binary and uncorrelated in this paper. Binary patterns simplify the network design significantly as seen in [14]. Also, the uncorrelated patterns give a minimum interaction of the interactions of the network, which makes its behavior analysis much easier as seen in [7, 19].
Remark 6. As mentioned earlier, there are two kinds of equilibrium points for a dynamic system: stable ones and unstable ones. In the next section, it will be shown that only the stable equilibrium points exist in the above dynamical system. For an arbitrary initial stimulus, the states of the dynamical system converge to one of its stable equilibrium points, that is, the local minimum points of the energy function, which cannot be its local maximum points or saddle points of the energy function. Thus, Property 1 can be achieved in the design. In the next section, it will also be proven that Property 2 can also be achieved in the design; that is, are the only local minimum points of .
4. Convergence Analysis
Lemma 7. For square matrix , assume that the eigenvalues of are not all the same. If , cannot be a positive semidefinite or positive definite matrix.
Proof. Denote the eigenvalues of as . Note that . If , then such that . If , there also exists such that since cannot be identically equal. Then this lemma holds.
Lemma 8. For conformable matrices , , and , . Also, if matrices and are addable, [21].
Proof. To prove , actually we only need to prove that . As and , this lemma holds.
Theorem 9. The states of the attractor network in (5)–(8) converge to an stable equilibrium point such that , , and is positive definite which is denoted as .
Proof. By Definition 1, the equilibrium points of system (7) are such thatAs is a random vector and is chosen by (6), a point is an equilibrium point of system (7) if and only if gives us that all the possible equilibrium points of (7) are those points at which the values of energy function are zero; that is, . In addition, the Jacobian matrix at a point is the Hessian matrix . Then, it is easy to obtain thatSo is a stable equilibrium point of the attractor network in (5) to (8) based on Definition 2.
Theorem 10. For the case that the dimension , are the only local minimum points of .
Proof. Obviously, are local minimum points (and also global minimum points) of . We now prove that has no other local minimum points. LetIt can be obtained that . It is also known that the dimension of diagonal matrix is ; thenAlso, Lemma 8 gives thatNow we assume that there is a point which is different from but satisfies that . This implies that . This is to say that where Thus, we have if is different from . Let and be and at . Combining (13)-(14) and (16), it can be obtained thatSince are all different from each other, the eigenvalues of cannot be all the same. When , we have , which implies since is a positive definite matrix. Similarly, we have when . From Lemma 7, matrix cannot be a positive definite matrix. It is a seminegative definite or an indefinite matrix. This implies that cannot be a local minimum point though . It can be a local maximum point or a saddle point of .
Remark 11. If the dimension , does not directly imply that is a negative semidefinite or an indefinite matrix. Let be a full-row rank projection matrix, which projects a vector into a two-dimensional plane. The following theorem shows that are still the only local minimum points of .
Theorem 12. For the case that , are also the only local minimum points of .
Proof. Let and be the corresponding projection of the local minimum points in the two-dimensional plane. Then, and become and , respectively, on this two-dimensional plane. It can be obtained thatIf and , then and , which means that the local minimum points of in a higher dimensional space must be also the local minimum points of in the two-dimensional plane while the converse is not true. Obviously, are the local minima of .
Assume that there is a point which is different from but a local minimum of . There exists a full-row rank projection matrix such that . The full-row rank of implies that is different from , where for are the local minimum points of . However, this is impossible by Theorem 10. Thus, there is no other local minimum point of . If a point satisfies that but it is different from , it can be only a local maximum point or a saddle point. So this theorem holds.
5. Simulation Results
Example 1. Design a nonlinear dynamical system whose attractors are with , .
This corresponds to the one-dimensional case. The energy function is constructed as from (2) with and being given byrespectively. The dynamic system can be then designed in (5)–(8). By solving , we have , , or as analyzed in Section 3. From is a local maximum point of . So and are the only two local minimum points of . The energy function is shown in Figure 1 where “” denotes the local minimum point and “” denotes the local maximum or the saddle point. From Figure 1, when the initial stimulus , the states converge to ; when , the states converge to . If the initial stimulus , the states can converge to either or , which depends on the random noise in (7).
Example 2. Example 1, analyze the dynamic system when , and , , respectively.
Case 1. and . Similar to Example 1, the energy function is constructed as with and being given bySolving gives , , or . When , we haveThe two eigenvalues of are and . This is consistent with Theorem 10, which gives . Figure 2 shows the contour map of the energy function in a two-dimensional space. The contour lines around a saddle point look like a horse saddle. It can be concluded that and are two attractors while is not an equilibrium point of system (5)–(8) by Theorem 9. The term guarantees that the attractor network (5)–(8) cannot stay at the point . But from Theorem 10, we know that is a saddle point of .
Case 2. and . In this case, , . When , we haveThen is still not an equilibrium point of system (5)–(8) but a saddle point of as the eigenvalue of is now , , and . But in this case, we have . So we cannot determine whether is a saddle point or a local minimum point of only based on Theorem 10. However, we can determine that it is also a saddle point by Theorem 12. This is consistent with what we observed in simulation, as the eigenvalues of above is , and thus is a saddle point and cannot be a local minimum point of .
Example 3. Design a nonlinear dynamic system whose attractors are , , and .
We have and in this example. The energy function is constructed as withwith its dynamical property described by the attractor network in (5)–(8). Theorem 9 tells us that , , and are the stable equilibrium points of and the attractor network in (5)–(8). However, has more saddle points but these saddle points are not equilibrium points (saddle points) of (5)–(8). To illustrate this, firstly, we find all the points such that . If , then , which meanswhere
We know that gives , , or . But implies that Thus, will be a saddle point of but not the attractor network in (5)–(8) if while is different from . As the attractor network in (5)–(8) does not have any saddle points from Theorem 9, usually, a saddle point of is located in between two local minimum points. As seen from Figure 3, two saddle points of are in between the three local minimum points on the plane. One is located about and the other is about .
6. Conclusion
The contributions of this paper are summarized as follows.(1)We have proposed a new energy function which includes the information of the stored patterns, and it is different from the energy function in Hopfield network. The proposed energy function makes it easy to differentiate memory patterns from possible spurious points.(2)We have presented an attractor network design based on the proposed energy function. The patterns stored in the attractor network can be nonbinary and either correlated or uncorrelated. The memory patterns are the attractors of the network and the equilibrium points of the dynamic system.(3)When an arbitrary input stimulus is presented to the designed attractor network, it has been proved that the states converge to one of the stored patterns. There are no spurious states in the designed dynamic systems.Our future work is to construct a biological plausible dynamical system in hardware (neuromorphic chip) which can stimulate the behavior of the designed network. This sheds new lights on the research towards the realization of artificial cognitive memory.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgment
This work is funded by Brain Inspired Computing Research, Tsinghua University (20141080934).