Neural Network Dynamics without Minimizing Energy

Abstract

Content-addressable memory (CAM) has been described by collective dynamics of neural networks and computing with attractors (equilibrium states). Studies of such neural network systems are typically based on the aspect of energy minimization. However, when the complexity and the dimension of neural network systems go up, the use of energy functions might have its own limitations to study CAM. Recently, we have proposed the decirculation process in neural network dynamics, suggesting a step toward the reshaping of network structure and the control of neural dynamics without minimizing energy. Armed with the decirculation process, a sort of decirculating maps and its structural properties are built here, dedicated to showing that circulation breaking taking place in the connections among many assemblies of neurons can collaborate harmoniously toward the completion of network structure that generates CAM.

1. Introduction

Hopfield in 1982 proposed a neural network model using a global energy function to provide absolute stability of global pattern formation [1]. Since then, the concept of content-addressable memory (CAM) has been widely developed, showing that neural networks are capable of yielding an entire memory item on the basis of sufficient partial information [2–6]. However, related lines of research in switched linear networked systems have shown that networked systems can be asymptotically stable, but no common quadratic Lyapunov function exists through the use of a theoretical result of optimal joint spectral radius range for the simultaneous contractibility of coupled matrices [7] (see also [8, 9]). This implies a limitation of the use of global energy functions to explain the formation of CAM when the complexity and the dimension of networked systems go up.

The above limitation motivates us to search for another logical strategy to study neural network dynamics. More recently, we have proposed the decirculation process in neural network dynamics [10], in which a criterion that describes and quantifies perturbations of network structure and neural updating is given. The decirculation process is stated as “the occurrence of a loop of neuronal active states leads to a change in neural connections, which feeds back to reinforce neurons to tend to break the circulation of neuronal active states in this loop.” Furthermore, in the study of operator control underlying the decirculation process [11], we have introduced the decirculating maps of loops of neuronal active states, with each measuring the effects of connection changes and displaying the threshold of circulation breaking. The study of the decirculation process suggests an initial but critical step toward the reshaping of network structure and the control of neural dynamics without minimizing energy.

Here we wish to use the decirculating maps to show that circulation breaking taking place in the connections among many assemblies of neurons can collaborate harmoniously toward the completion of network structure that generates CAM. Thus, in contrast with the explicit construction of global energy functions, the theoretical framework of local decirculating maps reflects, in a neural ensemble sense, that CAM can be derived from the cooperation of connection changes in neural assemblies.

In Section 2 we introduce the decirculating maps and show the structural properties of the symmetric part of the decirculating maps. In Section 3 we describe the neural network dynamics and determine the network structure for circulation breaking. In Section 4 we prove that circulation breaking taking place in the connections among many assemblies of neurons can collaborate harmoniously toward the completion of network structure that generates CAM.

2. Decirculating Maps

We introduce the decirculating maps defined in [11] and show that the symmetric part of the decirculating maps has nonzero entries relative to some symmetric difference sets and is positive semidefinite.

For this, let {0,1}^𝑛 denote the binary code consisting of all 01-strings of fixed-length 𝑛. Denote by Ω=[𝑥⁰,𝑥¹,…,𝑥^𝑝] a loop of states in {0,1}^𝑛, meaning that 𝑝>1, 𝑥⁰,𝑥¹,…,𝑥^𝑝∈{0,1}^𝑛, 𝑥⁰=𝑥^𝑝, and 𝑥^𝑖≠𝑥^𝑗 for some 𝑖,𝑗∈{1,2,…,𝑝}. For every 𝑖,𝑗=1,2,…,𝑛, we assign an integer, denoted 𝑐_𝑖𝑗(Ω), according to the rule 𝑐_𝑖𝑗(Ω)=𝑥⁰_𝑗𝑥⁰_𝑖−𝑥¹_𝑖+𝑥¹_𝑗𝑥¹_𝑖−𝑥²_𝑖+⋯+𝑥_𝑗^𝑝−1𝑥_𝑖^𝑝−1−𝑥^𝑝_𝑖. (1)

We refer to the resulting matrix 𝐶(Ω)=(𝑐_𝑖𝑗(Ω)) as the decirculating map of Ω. For example, let Ω = [1111100000, 0011111000, 0000111110, 0111110000, 0001111100, 1111100000]. Then 𝐶(Ω)=11100−1−1−10012210−1−2−2−10012100−1−2−100011000−1−100000000000−1−1−10011100−1−2−2−10122100−1−2−100121000−1−10001100000000000. (2)

Denote by 𝐶_SY(Ω)=(1/2)(𝐶(Ω)+𝐶(Ω)^𝑇) and 𝐶_SK(Ω)=(1/2)(𝐶(Ω)−𝐶(Ω)^𝑇) the symmetric part and the skew-symmetric part of 𝐶(Ω), respectively.

Consider the Hilbert space 𝑀_𝑛(ℝ) of all real 𝑛×𝑛 matrices endowed with the Hilbert-Schmidt inner product ⟨⋅,⋅⟩, that is, if 𝐴=(𝑎_𝑖𝑗) and 𝐵=(𝑏_𝑖𝑗)∈𝑀_𝑛(ℝ), then ⟨𝐴,𝐵⟩=tr(𝐴𝐵^𝑇)=∑_𝑖,𝑗𝑎_𝑖𝑗𝑏_𝑖𝑗. Let us recall that the symmetric difference of two sets 𝑈 and 𝑉 is the set 𝑈▵𝑉, each of whose elements belongs to 𝑈 but not to 𝑉, or belongs to 𝑉 but not to 𝑈. For any 01-string 𝑥=𝑥₁𝑥₂⋯𝑥_𝑛 we define 𝟏(𝑥)=𝑖;𝑥_𝑖,𝟎(𝑥)==1,1≤𝑖≤𝑛𝑖;𝑥_𝑖=0,1≤𝑖≤𝑛. (3)

Theorem 1. Let Ω=[𝑥⁰,𝑥¹,…,𝑥^𝑝] be a loop of states in {0,1}^𝑛. Then (i)𝐶_𝑆𝑌(Ω)_𝑖𝑗=0 if (𝑖,𝑗)∉⋃_{0≤𝑚<𝑝}((𝟏(𝑥^𝑚)▵𝟏(𝑥^𝑚+1))×(𝟏(𝑥^𝑚)▵𝟏(𝑥^𝑚+1)));(ii)𝐶_𝑆𝑌(Ω) is positive semidefinite.

Proof. According to [11, Lemma 1] with 𝐴=0, the assertion of part (i) follows, so we need to prove part (ii). Let 𝐴=(𝑎_𝑖𝑗)∈𝑀_𝑛(ℝ). Then ⟨𝐴,𝐶(Ω)⟩=_𝑖,𝑗𝑎_{𝑖𝑗0≤𝑚<𝑝}𝑥^𝑚_𝑗𝑥^𝑚_𝑖−_{0≤𝑚<𝑝}𝑥^𝑚_𝑗𝑥_𝑖^𝑚+1=_{0≤𝑚<𝑝𝑖,𝑗}𝑎_𝑖𝑗𝑥^𝑚_𝑗𝑥^𝑚_𝑖−_𝑖,𝑗𝑎_𝑖𝑗𝑥^𝑚_𝑗𝑥_𝑖^𝑚+1=_{0≤𝑚<𝑝}𝐴𝑥^𝑚,𝑥^𝑚−𝐴𝑥^𝑚,𝑥^𝑚+1. (4) Suppose 𝐴 is positive semidefinite. Then we have _{0≤𝑚<𝑝}𝐴𝑥^𝑚,𝑥^𝑚−𝐴𝑥^𝑚,𝑥^𝑚+1=12_{0≤𝑚<𝑝}𝐴𝑥^𝑚,𝑥^𝑚+𝐴𝑥^𝑚+1,𝑥^𝑚+1−_{0≤𝑚<𝑝}𝐴𝑥^𝑚,𝑥^𝑚+1−𝐴𝑥^𝑚+1,𝑥^𝑚=12_{0≤𝑚<𝑝}𝐴𝑥^𝑚−𝑥^𝑚+1,𝑥^𝑚−𝑥^𝑚+1≥0. (5) Combining (4) and (5) shows that if 𝐴 is positive semidefinite, then 𝐴,𝐶_SK(Ω)=𝐴,𝐶_SK(Ω)+𝐶_SY(Ω)=⟨𝐴,𝐶(Ω)⟩≥0. (6) Let 𝑦∈ℝ^𝑛. Then (𝑦_𝑖𝑦_𝑗)∈𝑀_𝑛(ℝ) is positive semidefinite and, by (6), we have 𝐶_SK(Ω)𝑦,𝑦=_𝑖𝑗𝑦_𝑖𝑦_𝑗𝐶_SK(Ω)_𝑖𝑗=𝑦_𝑖𝑦_𝑗,𝐶_SK(Ω)≥0, (7) showing that 𝐶_SY(Ω) is positive semidefinite.

3. Network Structure for Circulation Breaking

For network description, name the neurons 1,2,…,𝑛. The dynamical system of the 𝑛 coupled neurons is modeled by the nonlinear equation [10, 12] 𝑥(𝑡+1)=𝐻_𝐴(𝑥(𝑡),𝑠(𝑡)),𝑡=0,1,…, (8) where 𝑥(𝑡)=(𝑥₁(𝑡),𝑥₂(𝑡),…,𝑥_𝑛(𝑡))∈{0,1}^𝑛 is a vector of neuronal active states denoting the population response of neurons at time 𝑡, 𝐴=(𝑎_𝑖𝑗)∈𝑀_𝑛(ℝ) is the coupling matrix of the network, 𝑠(𝑡)⊂{1,2,…,𝑛} denotes the neurons that adjust their activity at time 𝑡, and 𝐻_𝐴(⋅,𝑠(𝑡)) is a function whose 𝑖th component is defined by 𝐻_𝐴(𝑥,𝑠(𝑡))_𝑖=𝟙^𝑛_𝑗=1𝑎_𝑖𝑗𝑥_𝑗−𝑏_𝑖if𝑖∈𝑠(𝑡), (9) otherwise [𝐻_𝐴(𝑥,𝑠(𝑡))]_𝑖=𝑥_𝑖, where 𝑏_𝑖∈ℝ is the threshold of neuron 𝑖 and the function 𝟙 is the Heaviside function: 𝟙(𝑢)=1 for 𝑢≥0, otherwise 0, which describes an instantaneous unit pulse. On each subsequent time 𝑡=0,1,…, the network generates a vector of neuronal active states according to (8), resulting in the dynamic flow 𝑥(𝑡), 𝑡=0,1,….

Theorem 2. Let Ω=[𝑥⁰,𝑥¹,…,𝑥^𝑝] be a loop of states in {0,1}^𝑛. If 𝐴∈𝑀_𝑛(ℝ) satisfies ⟨𝐴,𝐶(Ω)⟩≥0, (10) then for any threshold 𝑏∈ℝ^𝑛, any initial neural active state 𝑥(0)∈{0,1}^𝑛, and any updating 𝑠(𝑡)⊂{1,2,…,𝑛}, 𝑡=0,1,…, the resulting dynamic flow 𝑥(𝑡) of (8) cannot behave in 𝑥(𝑇)=𝑥⁰,𝑥(𝑇+1)=𝑥¹,…,𝑥𝑇+𝑝=𝑥^𝑝 (11) for each 𝑇=0,1,….

Proof. Suppose, by contradiction, that there exist 𝑏∈ℝ^𝑛, 𝑥(0)∈{0,1}^𝑛, 𝑠(𝑡)⊂{1,2,…,𝑛}, 𝑡=0,1,…, and 𝑇≥0 such that 𝑥(𝑇)=𝑥⁰, 𝑥(𝑇+1)=𝑥¹,…,𝑥(𝑇+𝑝)=𝑥^𝑝. Let Λ⁺=(𝑥(𝑡(𝑥(,Λ𝑡;𝟎))∩𝟏𝑡+1))≠∅,𝑇≤𝑡<𝑇+𝑝₋=𝑡;𝟏(𝑥(𝑡))∩𝟎(𝑥(𝑡+1))≠∅,𝑇≤𝑡<𝑇+𝑝. (12) Then Λ⁺≠∅ and Λ⁻≠∅. Indeed, if Λ⁺=∅ or Λ⁻=∅, then 𝑥(𝑇)=𝑥(𝑇+1)=⋯=𝑥𝑇+𝑝, (13) contradicting the loop assumption 𝑥(𝑇+𝑖)≠𝑥(𝑇+𝑗) for some 𝑖,𝑗∈{1,2,…,𝑝}. Since 𝟎(𝑥(𝑡))∩𝟏(𝑥(𝑡+1))⊂𝑠(𝑡) and 𝟏(𝑥(𝑡))∩𝟎(𝑥(𝑡+1))⊂𝑠(𝑡) for each 𝑡=0,1,…, we conclude from (4) and (8) that ⟨𝐴,𝐶(Ω)⟩=_{0≤𝑚<𝑝}()(−⟨()((⟨𝐴𝑥𝑇+𝑚,𝑥𝑇+𝑚)⟩𝐴𝑥𝑇+𝑚,𝑥𝑇+𝑚+1)⟩)=_{0≤𝑚<𝑝}⟨𝐴𝑥(𝑇+𝑚),𝑥(𝑇+𝑚)−𝑥(𝑇+𝑚+1)⟩<−_{𝑡∈Λ+𝑗∈𝟎(𝑥(𝑡))∩𝟏(𝑥(𝑡+1))}𝑏_𝑗+_{𝑡∈Λ−𝑗∈𝟏(𝑥(𝑡))∩𝟎(𝑥(𝑡+1))}𝑏_𝑗=_{0≤𝑚<𝑝}⟨𝑏,𝑥(𝑇+𝑚)−𝑥(𝑇+𝑚+1)⟩=0, (14) contradicting (10), and that completes the proof.

4. Harmonious Collaboration for CAM

We now proceed to the proof that circulation breaking taking place in the connections among many assemblies of neurons can collaborate harmoniously toward the completion of network structure that generates CAM.

We shall first introduce the Schur product theorem.

If 𝐴=(𝑎_𝑖𝑗) and 𝐵=(𝑏_𝑖𝑗)∈𝑀_𝑛(ℝ), then the Schur product of 𝐴 and 𝐵 is the matrix 𝐴∘𝐵=(𝑎_𝑖𝑗𝑏_𝑖𝑗)∈𝑀_𝑛(ℝ). We have the following well-known theorem.

Theorem 3 (Schur product theorem). If 𝐴,𝐵∈𝑀_𝑛(ℝ) are positive semidefinite, then 𝐴∘𝐵 is also positive semidefinite.

Let 𝐴∈𝑀_𝑛(ℝ) and 𝐼⊂{1,2,…,𝑛}. Denote by 𝐴(𝐼) the principal submatrix of 𝐴 relative to 𝐼.

Theorem 4. Let 𝐼₁,𝐼₂,…,𝐼_𝑞 be mutually disjoint subsets of {1,2,…,𝑛}. If 𝐴∈𝑀_𝑛(ℝ) is symmetric and 𝐴(𝐼_𝑘) is positive semidefinite for each 𝑘=1,2,…,𝑞, then(i)⟨𝐴,𝐶(Ω)⟩≥0 for each loop Ω=[𝑥⁰,𝑥¹,…,𝑥^𝑝] satisfying for 𝑚=0,1,…,𝑝−1, 𝟏𝑥^𝑚▵𝟏𝑥^𝑚+1⊂𝐼_𝑘𝑓𝑜𝑟𝑠𝑜𝑚𝑒𝑘∈1,2,…,𝑞; (15)(ii)for any threshold 𝑏∈ℝ^𝑛, any initial neural active state 𝑥(0)∈{0,1}^𝑛, and any updating 𝑠(𝑡)⊂{1,2,…,𝑛}, 𝑡=0,1,…, satisfying 𝑠(𝑡)⊂𝐼_𝑘𝑓𝑜𝑟𝑠𝑜𝑚𝑒𝑘∈1,2,…,𝑞, (16) the resulting dynamic flow 𝑥(𝑡) of the network modeled by (8) will converge to an equilibrium state.

Proof. To prove (i), let Ω=[𝑥⁰,𝑥¹,…,𝑥^𝑝] be a loop satisfying (15). Then, by Theorem 1(i), we have 𝐶_SY(Ω)_𝑖𝑗=0 if (𝑖,𝑗)∈^╱⋃_{1≤𝑘≤𝑞}(𝐼_𝑘×𝐼_𝑘). Furthermore, since 𝐴 is symmetric, it follows that ⟨𝐴,𝐶(Ω)⟩=𝐴,𝐶_SY(Ω)+𝐶_SK(Ω)=𝐴,𝐶_SY(Ω)=_{1≤𝑘≤𝑞}𝐴𝐼_𝑘,𝐶_SY(Ω). (17) Since 𝐶_SY(Ω) is positive semidefinite by Theorem 1(ii) and 𝐴(𝐼_𝑘) is positive semidefinite for each 𝑘=1,2,…,𝑞, we conclude from the Schur product theorem that 𝐴(𝐼_𝑘)∘𝐶_SY(Ω) is positive semidefinite. Let 𝑦∈ℝ^𝑛 be a vector with all components equal to 1. Then for each 𝑘=1,2,…,𝑞, 𝐴𝐼_𝑘,𝐶_SY(Ω)=𝐴𝐼_𝑘∘𝐶_SY(Ω)𝑦,𝑦≥0, (18) implying that ⟨𝐴,𝐶(Ω)⟩≥0.
To prove (ii), let 𝑏∈ℝ^𝑛, 𝑥(0)∈{0,1}^𝑛, 𝑠(𝑡) satisfying (16), and 𝑥(𝑡) be the resulting dynamic flow of the network modeled by (8). Suppose, by contradiction, that there exist 𝑝>1 and 𝑇≥0 such that Ω(𝑇,𝑝)=[𝑥(𝑇),𝑥(𝑇+1),…,𝑥(𝑇+𝑝)] forms a loop of states in {0,1}^𝑛. Since 𝑠(𝑡) satisfies (16), it follows that for each 𝑚=0,1,…,𝑝−1, 𝟏(𝑥(𝑇+𝑚))▵𝟏(𝑥(𝑇+𝑚+1))⊂𝐼_𝑘forsome𝑘∈1,2,…,𝑞. (19)
Thus, by Theorem 4(i), we have ⟨𝐴,𝐶(Ω(𝑇,𝑝))⟩≥0. By Theorem 2, we see that the dynamic flow 𝑥(𝑡) cannot form the loop Ω(𝑇,𝑝) of states in the period of time 𝑡=𝑇,𝑇+1,…,𝑇+𝑝, which is a contradiction, and that completes the proof.

Let 𝐼_𝑘={𝑘} for 𝑘=1,2,…,𝑛. If 𝐴∈𝑀_𝑛(ℝ) is symmetric with nonnegative diagonal entries, then 𝐴(𝐼_𝑘) is positive semidefinite for each 𝑘=1,2,…,𝑛. Thus, by Theorem 4(ii), we obtain the following basic theorem for CAM, showing that a network structure can be harmoniously collaborated by taking circulation breaking in all the loops Ω=[𝑥⁰,𝑥¹,…,𝑥^𝑝] satisfying ♯(𝟏(𝑥^𝑚)▵𝟏(𝑥^𝑚+1))≤1 for each 𝑚=0,1,…,𝑝−1.

Theorem 5 (Hopfield [1]). If 𝐴∈𝑀_𝑛(ℝ) is symmetric with nonnegative diagonal entries, then each dynamic flow 𝑥(𝑡) of the network modeled by (8), with each neuron adjusting randomly and asynchronously (i.e., ♯𝑠(𝑡)=1 for each 𝑡=0,1,…), will converge to an equilibrium state.

Acknowledgment

This work was supported by the National Science Council of Taiwan.