Variational Analysis, Optimization, and Fixed Point TheoryView this Special Issue
Neural Network Dynamics without Minimizing Energy
Content-addressable memory (CAM) has been described by collective dynamics of neural networks and computing with attractors (equilibrium states). Studies of such neural network systems are typically based on the aspect of energy minimization. However, when the complexity and the dimension of neural network systems go up, the use of energy functions might have its own limitations to study CAM. Recently, we have proposed the decirculation process in neural network dynamics, suggesting a step toward the reshaping of network structure and the control of neural dynamics without minimizing energy. Armed with the decirculation process, a sort of decirculating maps and its structural properties are built here, dedicated to showing that circulation breaking taking place in the connections among many assemblies of neurons can collaborate harmoniously toward the completion of network structure that generates CAM.
Hopfield in 1982 proposed a neural network model using a global energy function to provide absolute stability of global pattern formation . Since then, the concept of content-addressable memory (CAM) has been widely developed, showing that neural networks are capable of yielding an entire memory item on the basis of sufficient partial information [2–6]. However, related lines of research in switched linear networked systems have shown that networked systems can be asymptotically stable, but no common quadratic Lyapunov function exists through the use of a theoretical result of optimal joint spectral radius range for the simultaneous contractibility of coupled matrices  (see also [8, 9]). This implies a limitation of the use of global energy functions to explain the formation of CAM when the complexity and the dimension of networked systems go up.
The above limitation motivates us to search for another logical strategy to study neural network dynamics. More recently, we have proposed the decirculation process in neural network dynamics , in which a criterion that describes and quantifies perturbations of network structure and neural updating is given. The decirculation process is stated as “the occurrence of a loop of neuronal active states leads to a change in neural connections, which feeds back to reinforce neurons to tend to break the circulation of neuronal active states in this loop.” Furthermore, in the study of operator control underlying the decirculation process , we have introduced the decirculating maps of loops of neuronal active states, with each measuring the effects of connection changes and displaying the threshold of circulation breaking. The study of the decirculation process suggests an initial but critical step toward the reshaping of network structure and the control of neural dynamics without minimizing energy.
Here we wish to use the decirculating maps to show that circulation breaking taking place in the connections among many assemblies of neurons can collaborate harmoniously toward the completion of network structure that generates CAM. Thus, in contrast with the explicit construction of global energy functions, the theoretical framework of local decirculating maps reflects, in a neural ensemble sense, that CAM can be derived from the cooperation of connection changes in neural assemblies.
In Section 2 we introduce the decirculating maps and show the structural properties of the symmetric part of the decirculating maps. In Section 3 we describe the neural network dynamics and determine the network structure for circulation breaking. In Section 4 we prove that circulation breaking taking place in the connections among many assemblies of neurons can collaborate harmoniously toward the completion of network structure that generates CAM.
2. Decirculating Maps
We introduce the decirculating maps defined in  and show that the symmetric part of the decirculating maps has nonzero entries relative to some symmetric difference sets and is positive semidefinite.
For this, let denote the binary code consisting of all -strings of fixed-length . Denote by a loop of states in , meaning that , , , and for some . For every , we assign an integer, denoted , according to the rule
We refer to the resulting matrix as the decirculating map of . For example, let Ω = [1111100000, 0011111000, 0000111110, 0111110000, 0001111100, 1111100000]. Then
Denote by and the symmetric part and the skew-symmetric part of , respectively.
Consider the Hilbert space of all real matrices endowed with the Hilbert-Schmidt inner product , that is, if and , then . Let us recall that the symmetric difference of two sets and is the set , each of whose elements belongs to but not to , or belongs to but not to . For any 01-string we define
Theorem 1. Let be a loop of states in . Then (i) if ;(ii) is positive semidefinite.
Proof. According to [11, Lemma 1] with , the assertion of part (i) follows, so we need to prove part (ii). Let . Then Suppose is positive semidefinite. Then we have Combining (4) and (5) shows that if is positive semidefinite, then Let . Then is positive semidefinite and, by (6), we have showing that is positive semidefinite.
3. Network Structure for Circulation Breaking
For network description, name the neurons . The dynamical system of the coupled neurons is modeled by the nonlinear equation [10, 12] where is a vector of neuronal active states denoting the population response of neurons at time , is the coupling matrix of the network, denotes the neurons that adjust their activity at time , and is a function whose th component is defined by otherwise , where is the threshold of neuron and the function is the Heaviside function: for , otherwise 0, which describes an instantaneous unit pulse. On each subsequent time , the network generates a vector of neuronal active states according to (8), resulting in the dynamic flow , .
Theorem 2. Let be a loop of states in . If satisfies then for any threshold , any initial neural active state , and any updating , , the resulting dynamic flow of (8) cannot behave in for each .
Proof. Suppose, by contradiction, that there exist , , , , and such that , . Let Then and . Indeed, if or , then contradicting the loop assumption for some . Since and for each , we conclude from (4) and (8) that contradicting (10), and that completes the proof.
4. Harmonious Collaboration for CAM
We now proceed to the proof that circulation breaking taking place in the connections among many assemblies of neurons can collaborate harmoniously toward the completion of network structure that generates CAM.
We shall first introduce the Schur product theorem.
If and , then the Schur product of and is the matrix . We have the following well-known theorem.
Theorem 3 (Schur product theorem). If are positive semidefinite, then is also positive semidefinite.
Let and . Denote by the principal submatrix of relative to .
Theorem 4. Let be mutually disjoint subsets of . If is symmetric and is positive semidefinite for each , then(i) for each loop satisfying for , (ii)for any threshold , any initial neural active state , and any updating , , satisfying the resulting dynamic flow of the network modeled by (8) will converge to an equilibrium state.
Proof. To prove (i), let be a loop satisfying (15). Then, by Theorem 1(i), we have if . Furthermore, since is symmetric, it follows that
Since is positive semidefinite by Theorem 1(ii) and is positive semidefinite for each , we conclude from the Schur product theorem that is positive semidefinite. Let be a vector with all components equal to . Then for each ,
implying that .
To prove (ii), let , , satisfying (16), and be the resulting dynamic flow of the network modeled by (8). Suppose, by contradiction, that there exist and such that forms a loop of states in . Since satisfies (16), it follows that for each ,
Thus, by Theorem 4(i), we have . By Theorem 2, we see that the dynamic flow cannot form the loop of states in the period of time , which is a contradiction, and that completes the proof.
Let for . If is symmetric with nonnegative diagonal entries, then is positive semidefinite for each . Thus, by Theorem 4(ii), we obtain the following basic theorem for CAM, showing that a network structure can be harmoniously collaborated by taking circulation breaking in all the loops satisfying for each .
Theorem 5 (Hopfield ). If is symmetric with nonnegative diagonal entries, then each dynamic flow of the network modeled by (8), with each neuron adjusting randomly and asynchronously (i.e., for each ), will converge to an equilibrium state.
This work was supported by the National Science Council of Taiwan.
J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities,” Proceedings of the National Academy of Sciences of the United States of America, vol. 79, no. 8, pp. 2554–2558, 1982.View at: Publisher Site | Google Scholar | MathSciNet
M. A. Cohen and S. Grossberg, “Absolute stability of global pattern formation and parallel memory storage by competitive neural networks,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 13, no. 5, pp. 815–826, 1983.View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
S. Grossberg, “Nonlinear neural networks: principles, mechanisms, and architectures,” Neural Networks, vol. 1, no. 1, pp. 17–61, 1988.View at: Google Scholar
J. J. Hopfield, “Pattern recognition computation using action potential timing for stimulus representation,” Nature, vol. 376, no. 6535, pp. 33–36, 1995.View at: Google Scholar
R. Perfetti and E. Ricci, “Recurrent correlation associative memories: a feature space perspective,” IEEE Transactions on Neural Networks, vol. 19, no. 2, pp. 333–345, 2008.View at: Publisher Site | Google Scholar
M.-H. Shih and F.-S. Tsai, “Hamming star-convexity packing in information storage,” Fixed Point Theory and Applications, vol. 2011, Article ID 615274, 17 pages, 2011.View at: Google Scholar | Zentralblatt MATH | MathSciNet
T. Ando and M.-H. Shih, “Simultaneous contractibility,” SIAM Journal on Matrix Analysis and Applications, vol. 19, no. 2, pp. 487–498, 1998.View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
M.-H. Shih and C.-T. Pang, “Simultaneous Schur stability of interval matrices,” Automatica, vol. 44, no. 10, pp. 2621–2627, 2008.View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
A. Jadbabaie, J. Lin, and A. S. Morse, “Coordination of groups of mobile autonomous agents using nearest neighbor rules,” IEEE Transactions on Automatic Control, vol. 48, no. 6, pp. 988–1001, 2003.View at: Publisher Site | Google Scholar | MathSciNet
M.-H. Shih and F.-S. Tsai, “Decirculation process in neural network dynamics,” IEEE Transactions on Neural Networks and Learning Systems, vol. 23, no. 11, pp. 1677–1689, 2012.View at: Google Scholar
M.-H. Shih and F.-S. Tsai, “Operator control of inter-neural computing machines,” In Press.View at: Google Scholar
M.-H. Shih and F.-S. Tsai, “Growth dynamics of cell assemblies,” SIAM Journal on Applied Mathematics, vol. 69, no. 4, pp. 1110–1161, 2009.View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet