A Novel Chaotic Neural Network Using Memristive Synapse with Applications in Associative Memory

Abstract

Chaotic Neural Network, also denoted by the acronym CNN, has rich dynamical behaviors that can be harnessed in promising engineering applications. However, due to its complex synapse learning rules and network structure, it is difficult to update its synaptic weights quickly and implement its large scale physical circuit. This paper addresses an implementation scheme of a novel CNN with memristive neural synapses that may provide a feasible solution for further development of CNN. Memristor, widely known as the fourth fundamental circuit element, was theoretically predicted by Chua in 1971 and has been developed in 2008 by the researchers in Hewlett-Packard Laboratory. Memristor based hybrid nanoscale CMOS technology is expected to revolutionize the digital and neuromorphic computation. The proposed memristive CNN has four significant features: (1) nanoscale memristors can simplify the synaptic circuit greatly and enable the synaptic weights update easily; (2) it can separate stored patterns from superimposed input; (3) it can deal with one-to-many associative memory; (4) it can deal with many-to-many associative memory. Simulation results are provided to illustrate the effectiveness of the proposed scheme.

1. Introduction

Over the last three decades, chaos has been extensively studied by many researchers to understand complex dynamical phenomenon of nonlinear circuits. It has been found that chaotic dynamics may exist in biological neurons in human brain and play important role in associative memory [1, 2]. To mimic biological neuron, a chaotic neuron model has been proposed by Aihara et al. and subsequently many associative chaotic neural network models have been developed by other researchers [3–13].

In the traditional study of associative chaotic neural networks, researchers generally considered that each historic event works on the network with equal degree, which is, however, hardly the situation in reality. In fact, it has been observed that long-term memory that stores historic events in human brain always decays with time at an exponential rate. Moreover, for traditional chaotic neural network, its structure is very complex, its size is difficult to expand, and its capacity of information processing is limited. Those characteristics pose significant obstacles in CNN’s further development. Based on our previous work [10–13], in this paper, we propose a novel associative chaotic neural network with a new exponential decay spatiotemporal effect and a part of spatiotemporal summation whose behavior is more similar to the physiological senses. A memristive neuronal synapse has been designed taking advantage of the variable memristance (resistance of memristor) and memory ability (even if power off) of the memristor [14, 15]. This is expected to simplify the connections between neurons and reduce the complexity of chaotic neural network greatly.

A memristor or memristive device is essentially a basic two-terminal electronic element with nonvolatile continuously variable resistance which can be modulated by the flux or the charge through it. In 1971, Chua theoretically formulated and defined memristor and described that the memristance (resistance of memristor) is defined by the 𝜑-𝑞 constitutive relation, that is, 𝑀=𝑑𝜑/𝑑𝑞 [14]. For about four decades, it did not draw much attention until the recent TiO₂-based nanoscale practical memristor was developed by the HP Labs [16, 17]. Since the startling behavior of metal oxide film as a simple memristor structure was reported, it immediately garnered extensive interests among numerous researchers from academia and industry. Due to the perfect characteristics, it has been proposed in a broad range of applications including resistive random access memory (RRAM) [18, 19], artificial neural networks [20–22], chaotic circuits [23, 24], Boolean logic implementation [25, 26], signal processing, and pattern recognition [27–29]. Here, we show that analog memristive effects can be achieved in a nanoscale TiO₂-based memristor and this type of device exhibits reliable synaptic behavior. It is expected to change the way computer works completely. Moreover, it has potential applications in modern intelligent information processing.

In this paper, we present an implementation scheme for a new chaotic neural network with memristive neural synapse (MCNN). The classical CNN proposed by Aihara et al. is reviewed and the dynamical behaviors for this chaotic neuron are analyzed experimentally in Section 2. Following that the MCNN is presented in Section 3. Firstly, the memristor model is described through mathematical theory, its behaviors are simulated by computer, and its principle characteristics are analyzed in detail. Next, we explain the structure of the MCNN and its dynamical equations also with the simplified memristive synaptic circuit. Associative memory abilities of the MCNN are introduced finally. In order to illustrate the effectiveness of the MCNN, we design a series of computer simulations in Section 4. Reported results indicate that the MCNN can deal with the separation of superimposed patterns, one-to-many and many-to-many association successfully. By integrating the advantages of memristor, chaos, and neural networks, this design will provide theoretical and practical foundations for developing intelligent information processing system.

2. Chaotic Neural Networks with Dynamical Behaviors Analysis

In this section, we briefly review chaotic neural networks. A series of numerical analysis of the firing rate, the bifurcation structure, and the Lyapunov exponent demonstrate the rich dynamical behaviors of the CNN.

2.1. CNN Based on Aihara’s Model

The dynamical equation of a chaotic neuron model with continuous functions described by [1], 𝑥(𝑡+1)=𝑓𝐴(𝑡)−𝛼^𝑡_𝑑=0𝑘^𝑑𝑔(𝑥(𝑡−𝑑))−𝜃, (2.1) where 𝑥(𝑡+1) is the output of the chaotic neuron at the discrete time 𝑡+1, which is an analogue value between 0 and 1; 𝑓 is a continuous output function, such as logistic function 𝑓(𝑦)=1/(1+𝑒^−𝑦/𝜀) with the steepness parameter 𝜀;𝑔 is a function describing the relationship between the analog output and the refractoriness which can be defined as 𝑔(𝑥)=𝑥;𝐴(𝑡) is the external stimulation; 𝛼,𝑘, and 𝜃 are positive refractory scaling parameter, the damping factor, and the threshold, respectively. If defining the internal state 𝑦(𝑡+1) as 𝑦(𝑡+1)=𝐴(𝑡)−𝛼^𝑡_𝑑=0𝑘^𝑑𝑔(𝑥(𝑡−𝑑))−𝜃 (2.2) we can reduce (2.1) to the following 𝑦()(𝑡)𝑓𝑦(𝑡)(𝑡),𝑥()𝑦(),𝑡+1=𝑘𝑦−𝛼𝑔+𝑎𝑡+1=𝑓𝑡+1 (2.3) where 𝑎(𝑡) is called bifurcation parameter and defined by 𝑎(𝑡)=𝐴(𝑡)−𝑘𝐴(𝑡−1)−𝜃(1−𝑘). (2.4)

The neuron model with chaotic dynamics described above can be generalized as an element of neural network called chaotic neural network (CNN). The dynamics of the 𝑖th chaotic neuron with spatiotemporal summation of feedback inputs and externally applied inputs in a CNN composed of 𝑀 chaotic neurons and 𝑁 externally applied inputs can be modeled as 𝑥_𝑖(𝑡+1)=𝑓𝜉_𝑖(𝑡+1)+𝜂_𝑖(𝑡+1)+𝜁_𝑖(𝑡+1), (2.5) where the external inputs 𝜉_𝑖(𝑡+1), the feedback inputs 𝜂_𝑖(𝑡+1), and the refractoriness 𝜁_𝑖(𝑡+1) are defined as (2.6)–(2.8), respectively. 𝜉_𝑖(𝑡+1)=^𝑁_𝑗=1𝑉_𝑖𝑗𝐴_𝑗(𝑡)+𝑘_𝑒𝜉_𝑖(𝑡)=^𝑁_𝑗=1𝑉^𝑡_{𝑖𝑗𝑑=0}𝑘^𝑑_𝑒𝐴_𝑗(),𝜂𝑡−𝑑_𝑖(𝑡+1)=^𝑀_𝑗=1𝑊_𝑖𝑗𝑥_𝑗(𝑡)+𝑘_𝑓𝜂_𝑖(𝑡)=^𝑀_𝑗=1𝑊^𝑡_{𝑖𝑗𝑑=0}𝑘^𝑑_𝑓𝑥_𝑗(),𝜁𝑡−𝑑_𝑖(𝑡+1)=−𝛼𝑔𝑥_𝑖(𝑡)+𝑘_𝑟𝜁_𝑖(𝑡)−𝜃_𝑖=−𝛼^𝑡_𝑑=0𝑘^𝑑_𝑟𝑔𝑥_𝑖(𝑡−𝑑), (2.6)(2.7)(2.8) where 𝑉_𝑖𝑗 serves as the synaptic weight between the external inputs and the neurons. Similarly, 𝑊_𝑖𝑗 indicates the synaptic weights between two neurons and is trained by Hebbian learning algorithm, such as, 𝑊_𝑖𝑗=^𝑀_𝑘=1𝑥^𝑘_𝑖𝑥^𝑘_𝑗or𝑊_𝑖𝑗=1𝑀^𝑀_𝑘=12𝑥^𝑘_𝑖−12𝑥^𝑘_𝑗−1. (2.9)

2.2. Dynamical Behaviors Analysis

The firing rate of a neuron is a fundamental characteristic of the message that conveys to other neurons. It is variable and denotes the intensity of its activation state. As such, it ranges from near zero to some certain maximum depending on its need to convey a particular level of activation. Traditionally, it has been thought that most of the relevant information was contained in the average firing rate of the neuron. The firing rate is usually defined by a temporal average meaning the spikes that occur in a given time window. Division by the length of the time window gives the average firing rate [30]. Here, the average firing rate or excitation number of a single chaotic neuron depicted in (2.3) is defined as 𝜌=lim_𝑛→+∞1𝑛^𝑛−1_𝑡=0ℎ(𝑥(𝑡)), (2.10) where ℎ is a transfer function which represents waveform-shaping dynamics of the axon with a strict threshold for propagation of action potentials initiated at the trigger zone and assumed to be ℎ(𝑥)=1 for 𝑥≥0.5 and ℎ(𝑥)=0 for 𝑥<0.5. By adjusting the bifurcation parameter 𝑎(𝑡) from 0 to 1, when the other parameters are set as 𝑘=0.7,  𝛼=1.0, 𝜀=0.02, 𝑦(0)=0.5, and the average firing rate is shown in Figure 1.

Figure 1 

The average firing rate of the chaotic neuron.

It is a characteristic of chaotic systems that initially nearby trajectories separate exponentially with time. The Lyapunov exponent or Lyapunov characteristic exponent of a dynamical system is a quantity that characterizes the rate of separation of infinitesimally close trajectories. The rate of separation can be different for different orientations of initial separation vector. Thus, there is a spectrum of Lyapunov exponents—equal in number to the dimensionality of the phase space. It is common to refer to the largest one as the Maximal Lyapunov exponent (MLE), because it determines a notion of predictability for a dynamical system. Specifically, a positive MLE is usually taken as an indication that the system is chaotic [31].

For quantitative analysis of the system (2.3), the Lyapunov exponent 𝜆 is defined as follows, 𝜆=lim_𝑛→+∞1𝑛^𝑛−1_𝑡=0()ln𝑑𝑦𝑡+1(𝑡).𝑑𝑦 (2.11)

Response characteristics of this system are shown in Figure 2. From Figure 2(a), one can see the bifurcation structure clearly. Figure 2(b) shows the Lyapunov exponent spectrum of this system. The values of the parameters employed are set as the same as those in Figure 1.

(a)

(b)

Figure 2 

Response characteristics diagram of (2.3): (a) the bifurcation structure; (b) the Lyapunov exponent.

3. The Chaotic Neural Network with Memristive Neuronal Synapses

In this section, the memristor is introduced at first to describe its typical characteristics. Then a new chaotic neural network (MCNN) with memristive synapses is presented in detail. Finally, the associative memory abilities of the MCNN model are analyzed.

3.1. Memristor Model Description

All two-terminal nonvolatile memory devices based on resistance switching are memristors, regardless of the device material and physical operating mechanisms [15]. Similar to the resistor (which relates voltage 𝑣 and current 𝑖), capacitor (which relates charge 𝑞 and voltage 𝑣) and inductor (which relates flux 𝜑 and current 𝑖), the memristor relates the flux 𝜑, and the charge 𝑞 of a device.

The behaviors of a general memristive device can be described by a pair of coupled equations as following:()̇(),𝑣=𝑅𝑤,𝑖𝑖,𝑤=𝑓𝑤,𝑖 (3.1a)(3.1b)

where (3.1a) is the 𝑖-𝑣 equation for the memristive device and 𝑤 is an (or a group of) internal state variable which is determined by the excitation history of the device. A mathematical model of the typical HP TiO₂-based memristor can be described by [16] 𝑣(𝑡)=𝑀_on𝑤(𝑡)𝐷+𝑀_off𝑤(𝑡)1−𝐷𝑖(𝑡),𝑑𝑤=𝜇𝑑𝑡^𝑣𝑀_on𝐷𝑖(𝑡),𝑤(𝑡)=𝜇_𝑣𝑀_on𝐷^𝑡₀𝑖(𝑡)(0)=𝜇𝑑𝑡+𝑤^𝑣𝑀_on𝐷𝑞(𝑡)+𝑤(0), (3.2) where, 𝑀(𝑡)=𝑀_on(𝑤(𝑡)/𝐷)+𝑀_off(1−𝑤(𝑡)/𝐷) is the total memristance, 𝐷 and 𝑤 indicate the thickness of the whole TiO₂ film and the high conductive region, respectively. If 𝑤=0 or 𝑤=1, memristor reaches the memristance limit value, 𝑀_off or 𝑀_on. 𝑤(0) denotes the initial value of 𝑤. 𝜇_𝑣 is average ion mobility parameter in units of m² s⁻¹ V⁻¹.

Characteristics of the memristor have been analyzed through simulations. By applying a sinusoidal voltage, we can observe the symbolic pinched hysteresis loop (in Figure 3(a)) confined to the first and the third quadrants of the 𝑖-𝑣 plane. The switching characteristic can be seen from this figure [28], that is, memristor can switch between the high-resistance state (“OFF”) and the low-resistance state (“ON”), which has been proposed in binary memory, Boolean calculation, crossbar array, and signal processing abundantly. It should be noted that the pinched hysteresis loop for 𝑖-𝑣 characteristic implies the device is a memristor, but this curve itself is useless as a model since its contour shape changes with both the amplitude and the frequency of any periodic sine wave like applied voltage or current source and thus cannot be used to predict the current response to arbitrarily applied voltage signals, and vice versa [15]. Moreover, it also can be found that under low-frequency AC excitation, there is an obvious 𝑖-𝑣 pinched hysteresis loop, but under high-frequency case, the hysteresis loop collapses and memristor degenerates into a fixed resistor.

(a)

(b)

(c)

(d)

Figure 3 

Characteristics of memristor. (a) Typical memristor 𝑖-𝑣 pinched-hysteresis loop; (b) memristor 𝑞-𝜙 curve; (c) memristance versus state map; (d) memristance changes with a series of voltages pulses.

One of the two ways to predict the response of the device is the 𝑞-𝜑 constitutive relation (in Figure 3(b)). As seen in this figure, the memductance 𝐺(𝜑(𝑡_𝑘)) at the time 𝑡=𝑡_𝑘 is equal to the slope of the 𝑞-𝜑 curve at 𝜑=𝜑(𝑡_𝑘), which is called small-signal memductance. Similarly, if the stimulation is a current source, the memristance 𝑀(𝑞(𝑡_𝑘)) at the time 𝑡=𝑡_𝑘 is equal to the slope of the 𝜑-𝑞 curve at 𝑞=𝑞(𝑡_𝑘), and called small-signal memristance. Assuming the constitutive relation curve is continuous at the time 𝑡=𝑡_𝑘 and the stimulation has sufficiently small amplitude with a fixed frequency, which is called as small-signal condition [15], the small-signal memductance or memristance approximately keeps constant being equal to the curve slope at the time 𝑡_𝑘 over time and would be indistinguishable from a linear resistance.

Memristance versus state map is an equivalent representation as 𝑞-𝜑 curve and also obeys Ohm’s Law except that the memristance is not a constant, as illustrated by the example in Figure 3(c). The memristance decreases continuously with the flux increasing, and vice versa. When the flux reaches the limits, minimum or maximum, for a given memristive device, the memristance remain the ultimate value, 𝑀_off or 𝑀_on.

Figure 3(d) clearly presents the relationship between the memristance and the applied voltage. By applying a series of small voltage pulses with the same magnitude but opposite polarity, we have observed the stair-like decrement (or increment) of memristance correspondingly. If the bias is a big stimulation, memristance changes continuously. Thus, one can tune and choose different memristances flexibly according to the requirements in different applications by controlling the amplitude, polarity, and duration of the stimulation.

3.2. Memristive Chaotic Neural Network

Based on previous works [10–13, 28], we propose a new chaotic neural network with memristive neuronal synapses (MCNN). This MCNN model has four distinguishing features from the traditional CNN: (1) continuously external input replaces initial input, (2) spatiotemporal effect is replaced with a new exponential decreasing effect, and (3) a part of summation replaces the whole spatiotemporal summation. In traditional associative CNN, researchers considered that each historic effect worked on the network with the same equal level, so they usually assumed decay parameters as constant values which are far from the truth in facts. (4) The memristor is exploited for the synaptic weights. Memristors exhibit dynamical resistance modeled by the stimulation history, which is very similar to the biological neuron synapse and has been proposed in artificial neural network (ANN) as synaptic connection. The MCNN with the features above is more suitable to mimic the biological neuron. Moreover, since the memristor is a nanoscale device and it can simplify the synaptic circuit obviously, the MCNN is expected to be realized in physical implementation as a part of a VSLI chip.

Figure 4 shows the structure of the proposed MCNN model that consists of three layers and all of the neurons are fully connected, which is similar to the Hopfield neural networks.

Figure 4 

Structure of the memristive chaotic neural networks.

In the MCNN, the dynamics of the 𝑖th chaotic neuron in the 𝛼th layer is given as follows:(1)while 𝑡>𝑡₀, 𝑥_𝑖^𝛼(𝑡+1)=𝜙^𝐿𝑁_{𝛽=1𝛽≠𝛼𝑗=1}𝑊^𝛼𝛽_𝑖𝑗(𝑡)^𝑡_{𝑑=𝑡−𝑡0}𝑘(𝑟1)ℎ_𝑗𝑥_𝑗(𝑡−𝑑)+^𝑀_𝑗=1𝑊^𝛼𝛼_𝑖𝑗(𝑡)^𝑡_{𝑑=𝑡−𝑡0}𝑘(𝑟2)ℎ_𝑗𝑥_𝑗(𝑡−𝑑)+𝑉^𝛼_𝑖(𝑡)^𝑡_{𝑑=𝑡−𝑡0}𝑘(𝑟3)𝐼_𝑖(𝑡−𝑑)−𝑎^𝑡_{𝑑=𝑡−𝑡0}𝑘(𝑟4)𝑔_𝑖𝑥_𝑖(𝑡−𝑑)−𝜃_𝑖^(𝛼), (3.3)(2)while 𝑡≤𝑡₀, replace 𝑑=𝑡−𝑡₀ with 𝑑=0 in (3.3).

Here, 𝑥_𝑖^𝛼(𝑡+1) is the output of the chaotic neuron. 𝐿,𝑁, and 𝑀 are the number of layers, the chaotic neurons in the 𝛽th layer and in the 𝛼th layer; 𝐼_𝑖(𝑡−𝑑) is the input of the 𝑖th neuron in the 𝛼th layer; 𝑉_𝑖^𝛼(𝑡) is the connection weight between the input and the neurons in the 𝛼th layer; 𝑊^𝛼𝛽_𝑖𝑗(𝑡) indicates the weight between the 𝑖th chaotic neuron in the 𝛼th layer and the 𝑗th neuron in the 𝛽th layer. The decay parameter and transfer function are defined as 𝑘(𝑢)=𝑒^−𝑢 and 𝜙(𝑦)=2/(1+𝑒^−𝑦/𝜀), respectively. For simplicity, 𝑔(𝑥)=𝑥 and ℎ(𝑥)=𝑥. Moreover, we utilize Hebbian learning regulation as learning algorithm of the synaptic weights.

3.3. Memristive Neuronal Synapse in MCNN

We present a simplified memristive synapse circuit using memristors as part of the neuronal synapse taking advantage of the continuously variable memristance and memory ability of memristors. Figure 5 shows the schematic diagram of the memristive synaptic circuit, in which the input voltage source 𝑉_in is assumed as the output of a neuron, and the output of the synaptic circuit 𝑉_out is connected to next neuron as the input signal after weighting. The memristor placed in the feedback path can be programmed by the constant programming current using the circuit’s voltage to current conversion for a big programming voltage 𝑉_pro. Assume that the Op-amp is ideal, and then the memristance after programming by a low-frequency voltage 𝑉_pro for a time width 𝑡_𝑝 is given by 𝑤𝑡_𝑝=𝑢_𝑣𝑀_on𝐷𝑅^𝑡𝑝₀𝑉_pro(0),𝑀𝑡𝑑𝑡+𝑤_𝑝=𝑀_off+𝑀_on−𝑀_off𝑤𝑡_𝑝𝐷. (3.4) The synaptic weight (𝑊_𝑖𝑗 in (3.3)) is realized by the gain of the amplifier: 𝑊_𝑖𝑗𝑣=Gain=^out(𝑡)𝑣_in(𝑡)𝑀𝑡=−^𝑝𝑅. (3.5)

Figure 5 

Simplified schematic diagram of the memristive synaptic circuit.

Different weight values can be obtained easily by changing memristance and the memristance can be set by applying different programming voltages. The synaptic weight keeps constant after the programming operation (the programming voltage cut off) based on the memory ability of memristor. If the synaptic weight should be a positive value, connecting a reversing amplifier to the memristive synaptic circuit may be an effective method.

Next, when the MCNN has been trained by setting its connection weights, it will work normally in some applications such as associative memory. As described in the front, memristors can be considered as a content resistor under a high frequency (or very short operation time) or a small enough external excitation. In this paper, each step of the programs in the simulations uses very short time, about 0.01 s in MATLAB, thus in order to simplify the experiments, we assume that the memristance approximately levels off at the value set in programming stage. At the practical level, for a real memristor, it has some certain threshold voltages presented in literatures [19]. Therefore, in physical implementation of the MCNN, this behavior can be realized by using some voltage-controlling solutions that enable memristance changeable depending on the operation requirement.

3.4. Associative Memory for MCNN

Here, we will primarily analyze the ability of associative memory of the memristive chaotic neural network by undertaking three important applications of the MCNN for examples.

3.4.1. Separation of Superimposed Patterns

Let us assume that a training pattern set {𝑋,𝑌} is memorized in the neural network. When 𝑋 is given to the network as an externally applied input continuously, the network searches around the input pattern, so pattern 𝑋 is recalled. When the superimposed pattern (𝑋+𝑌) is given, the network searches around the 𝑋 and 𝑌. Due to the property that the chaotic neurons change their states by chaos, the 𝑋 and the 𝑌 can be recalled, respectively. We can also simplify the separation procedure as follows: 𝑋+𝑌→⋯→𝑋→⋯→𝑌→⋯→𝑋→⋯. (3.6)

3.4.2. One-to-Many Associations

Based on the property of separation of superimposed patterns, assume the following training set (𝑋₁,𝑌₁,𝑍₁) has been memorized in a three-layered MCNN, When 𝑋₁ is given to the first layer of the network continuously as input, the first layer searches and recalls the 𝑋₁ without doubt, the second and the third layer can search around 𝑌₁ and 𝑍₁ by chaotic itinerancy, respectively. So, 𝑋₁,𝑌₁, and 𝑍₁ can be recalled in three layers, respectively, and one-to-many associations is realized.

3.4.3. Many-to-Many Associations

Let us consider the following training set {(𝑋₁,𝑌₁,𝑍₁),(𝑋₁,𝑌₂,𝑍₂),(𝑋₃,𝑌₃,𝑍₃)}, two patterns of which contain a common term 𝑋₁. When 𝑋₁ is given to the first layer of the network continuously, the internal state of the chaotic neuron in the second layer is determined by the following formula: 𝐼^𝛽=𝑋₁𝑊^(𝛼𝛽)=𝑋₁𝑋^𝑇₁𝑌₁+𝑋₁𝑋^𝑇₁𝑌₂+𝑋₁𝑋^𝑇₃𝑌₃=𝑋₁𝑋^𝑇₁𝑌₁+𝑌₂+𝑋₁𝑋^𝑇₃𝑌₃, (3.7) where 𝑊^(𝛼𝛽) is the connection weight matrix between the first layer and the second layer, 𝑋₁𝑋^𝑇₃𝑌₃ is the noise term, and 𝑌₁+𝑌₂ is the superimposed term resulting from the common term 𝑋₁. Similarly, the superimposed term 𝑍₁+𝑍₂ appears in the third layer. The network searches around 𝑌₁ and 𝑌₂ by chaotic itinerancy in the second layer and searches around 𝑍₁ and 𝑍₂ in the third layer. Then, pattern (𝑋₁,𝑌₁,𝑍₁) and (𝑋₁,𝑌₂,𝑍₂) can be recalled at different time, realizing the many-to-many associations.

4. Simulation Results

In order to demonstrate the effectiveness and associative memory ability of the proposed MCNN, we perform a series of computer simulations. Since bipolar data (−1 or 1) are used in patterns, the weight values can be calculated by simple derivation based on the Hebbian algorithm formula (the first formula in (2.9)) according to the number of the patterns to be remembered, that is, 𝑊∈{−𝑁,−𝑁+2,…,0,…,𝑁−2,𝑁}, where 𝑊 is the weight value and 𝑁(𝑁>1) is the number of the patterns. For example, if there are three patterns used in the weight computation at one time, the weight values belong to {−3,−1,0,1,3}.

In the following experiments, the number of the patterns in the training set used for computing the weights is one, two, and three, respectively. So all of the weight values belong to {−3,−2,−1,0,1,2,3}, where, the value “0” means there is no connection between the two neurons, so we do not need to consider it. Moreover, it is known that the memristance must be positive, so the output of the synaptic circuit is negative (seen from (3.5)). Since the weight values 𝑋 and −𝑋 imply the same memristance state but with an opposite weight sign, so we use a reversing amplifier connected to the synaptic circuit to realize the positive weight. Therefore, for a given resistor 𝑅=10𝑘Ω, the memristor should be programmed to three states 30𝑘Ω,20𝑘Ω,10𝑘Ω, to realize the weights −3,−2,−1 (and 3,2,1). Figure 6 shows such three memristance states which we have got and the corresponding voltage pulses (1 V, 0.36 s), (1 V, 1.02 s), (1 V, 1.42 s). Next, by using the synaptic weights obtained from the memristive synaptic circuit, we will study the associative memory ability of the MCNN.

(a)

(b)

(c)

Figure 6 

Three memristance states and the corresponding programming voltage pulses.

4.1. Separation of Superimposed Patterns of a MCNN

We have designed a MCNN containing 49 neurons to realize the separation of superimposed pattern and the training set is shown in Figure 7. The superimposed pattern 𝐴+𝐵 is input to the network continuously and the number of output is 20. Network Parameters are set as 𝑎=9.8,𝑘𝑟1=0.9,𝑘𝑟2=0,𝑘𝑟3=0.1,𝑘𝑟4=0.9,𝑉=200,𝜃=0, and logistic function with 𝜀=0.02. This MCNN has a layer and the total valid weight values belong to {−3,−1,1,3}.

(a)

(b)

(c)

(d)

Figure 7 

(a)–(c) The training set, (d) the input superimposed pattern.

Figure 8 shows the output of the MCNN for separation of superimposed pattern, in which pattern 𝐴 is separated from the superimposed pattern at step 6, 9, 13, and 16, and pattern 𝐵 is separated at step 5, 8, 11, 14, and 17 successfully. This processing progress takes about 0.01 s in software MATLAB.

Figure 8 

Output of a MCNN for separation of superimposed pattern.

4.2. One-to-Many Associations of a MCNN

We have designed a 3-layered MCNN with each layer containing 81 neurons. The parameters are: 𝑎=10,𝑘𝑟1=0.95,𝑘𝑟2=0.1,𝑘𝑟3=0.1,𝑘𝑟4=0.95,𝑉=100,𝜃=0, and logistic function with 𝜀=0.015. The valid weight values in each layer belong to {−1,1} and those between two layers also belong to {−1,1}.

Figure 9 shows the training set (𝐴,𝑎,1). After the whole pattern set is stored in the MCNN by Hebbian learning algorithm, when a pattern 𝐴 is given to the network as an external input continuously, the whole pattern set can be recalled perfectly (Figure 10) and realize one-to-many associations. This processing progress takes about 0.3 s in software MATLAB.

Figure 9 

Training set.

Figure 10 

Output of a MCNN for one-to-many associations.

4.3. Many-to-Many Associations of a MCNN

A 3-layered 81-81-81 MCNN has been designed for many-to-many associations with a part of spatiotemporal summation (𝑡₀=3 the same as that in the experiments above). The parameters are 𝑎=10,𝑘𝑟1=0.95,𝑘𝑟2=0.1,𝑘𝑟3=0.1,𝑘𝑟4=0.95,𝑉=100,𝜃=0, and logistic function with 𝜀=0.015. The valid weight values in each layer belong to {−2,2} and those between two layers belong to {−3,−1,1,3}.

The training set {(𝐴,𝑎,1),(𝐴,𝑏,2),(𝐶,𝑐,3)} shown in Figure 11 is stored in the MCNN, in which the first two patterns include the common term 𝐴. The terms 𝐴 and 𝐶 are remembered in the first layer, the terms 𝑎,𝑏, and 𝑐 are remembered in the second layer, and 1, 2, and 3 in the third layer. In order to show the outputs clearly, we put the outputs from three layers at the same time together. Figure 12 shows the responses when 𝐴 is given to the first layer as an external input, in which (𝐴,𝑎,1) appears 9 times and (𝐴,𝑏,2) appears 10 times in 30 steps of output. That implies the realization of the many-to-many associative memory of MCNN. This simulation takes about 0.4 s in MATLAB software.

Figure 11 

Training set.

Figure 12 

Output of the proposed MCNN for many-to-many associative memory.

It can be seen from the above experiments that the MCNN can realize associative memory successfully, which indicates it can perform the typical applications as the traditional CNN. Moreover, this MCNN has special characteristics over the traditional ones. First of all, it needs set its connection weights to realize memory ability for a CNN, which is usually called training operation. In order to perform different tasks, these weights should be updated correspondingly. Traditionally, changing these weights is difficult because the synaptic circuit is fixed and complex. In the meantime, this is also a key factor why CNN has not been practically implemented widely. Fortunately, the memristor possesses fast switching (<10 ns), low energy (~1 pJ/operation), high write endurance (10¹⁰), multiple-state operation, scalability, synapse-like behaviors, and CMOS compatibility. The size of a memristor is 1/10 of that of a transistor and requires much less energy, so the memristive synaptic circuit would be more simple and smaller than the traditional circuits. What is more important is it can be very easy to update the connection weights just by reprogramming the memristors in the synaptic circuits with proper programming voltages, which saves the big replacement of the synaptic circuit as in traditional CNN and makes the MCNN be expected to be realized in large scale physical circuit.

5. Conclusions

An implementation scheme for a new Chaotic Neural Network with memristive neuronal synapses (MCNN) is addressed in this paper. Exponential decay spatiotemporal effect and a part of spatiotemporal summation are considered in the MCNN, which is closer to biological neuron behavior and has higher efficiency in information processing such as associative memory. The characteristics of memristors are analyzed including the 𝑖-𝑣 characteristics, 𝑞-𝜑 constitutive relation, and memristance versus state map with the relationship between the applied voltage and memristance. A memristive synaptic circuit is designed based on a variable gain amplifier. The weight value can be updated easily taking advantage of the variable memristance by programming memristor. When an input signal passes through the memristive synaptic circuit, the weighting operation is realized. Reported simulation results show the effectiveness of the memristive chaotic neural network, that is, it can separate stored patterns from superimposed input pattern and can deal with one-to-many associative memory and many-to-many associative memory perfectly. Furthermore, the memristive synaptic circuit makes the network structure of MCNN more simple and more powerful. At the same time, due to the simple structure (two-terminal), small device size (nano-scale) of the memristor, the proposed MCNN model has a brighter future in physical implementation and holds potential to be implemented in VLSI chips.

Acknowledgments

The work was supported by the National Natural Science Foundation of China under Grant 60972155 and 61101233, the Fundamental Research Funds for the Central Universities under Grant XDJK2012A007, the University Excellent Talents supporting Foundations in of Chongqing under Grant 2011-65, the University Key teacher supporting Foundations of Chongqing under Grant 2011-65, the “Spring Sunshine Plan” Research Project of Ministry of Education of China under Grant no. z2011148, the National Science Foundation for Post-doctoral Scientists of China under Grant CPSF20100470116.