Chaotic Hopfield Neural Network Swarm Optimization and Its Application

Abstract

A new neural network based optimization algorithm is proposed. The presented model is a discrete-time, continuous-state Hopfield neural network and the states of the model are updated synchronously. The proposed algorithm combines the advantages of traditional PSO, chaos and Hopfield neural networks: particles learn from their own experience and the experiences of surrounding particles, their search behavior is ergodic, and convergence of the swarm is guaranteed. The effectiveness of the proposed approach is demonstrated using simulations and typical optimization problems.

1. Introduction

The discovery of chaos in astronomical, solar, fluid, and other systems sparked significant research in nonlinear dynamics exhibiting chaos. Chaos was found to be useful and have great potential in many disciplines such as mixing liquids with low power consumption, presenting outages in power systems, biomedical engineering applications involving signals from the brain and heart, to name just a few [1]. Chaotic systems exhibit three important properties. Firstly, a deterministic system is said to be chaotic whenever its evolution sensitively depends on the initial conditions. Secondly, there is an infinite number of unstable periodic orbits embedded in the underlying chaotic set. Thirdly, the dynamics of the chaotic attractor is ergodic, which implies that during its temporal evolution the system ergodically visits small neighborhoods around every point in each one of the unstable periodic orbits embedded within the chaotic attractor. Although it appears to be stochastic, it is generated by a deterministic nonlinear system. Lyapunov exponents characterize quantitatively stochastic properties of the dynamical systems. When the dynamical system is chaotic, there exists at least one lyapunov exponent 𝜆>0. It is reported that chaotic behavior also exists in biological neurons and neural networks [2, 3]. Using chaos to develop novel optimization techniques gained much attention during the last decade. For a given energy or cost function, the chaotic ergodic orbits of a chaotic dynamic system used for optimization may eventually reach the global optimum or a point close to it with high probability [4, 5].

Since Hopfield and Tank [6] applied their neural network to the travelling salesman problem, neural networks have provided a powerful approach to a wide variety of optimization problems [7, 8]. However the Hopfield neural network (HNN) often gets trapped in a local minima. A number of modifications were made to Hopfield neural networks to escape from local minima. Some modifications, based on chaotic neural networks [9] and simulated annealing [10], were proposed to solve global optimization problems [11]. In [12–14] the guaranteed convergence of Hopfield neural networks is discussed.

Particle swarm optimization (PSO), developed by Clerc and Kennedy in 2002 [15], is a stochastic global optimization method which is based on simulation of social behavior. In a particle swarm optimizer, individuals “evolve" by cooperating with other individuals over several generations. Each particle adjusts its flying according to its own flying experiences and the flying experience of its companions. Each individual is named as a particle which, in fact, represents a potential solution to a problem. Each particle is treated as a point in a 𝐷-dimensional space. However, the PSO algorithm is likely to temporarily get stuck and may need a long period of time to escape from a local extremum [16]. It is difficult to guarantee the convergence of the swarm, especially when random parameters are used. In order to improve the dynamical behavior of PSO, one can combine chaos with PSO algorithms to enhance the performance of PSO. In [17–19] chaos were applied to the PSO to avoid the PSO getting trapped in local minima.

PSO is motivated by the behavior of organisms such as fish schooling and bird flocking [20]. During the process, future particle positions (determined by velocity) can be regarded as particle intelligence [21]. Using a chaotic intelligent swarm system to replace the original PSO might be convenient for analysis while maintaining stochastic search properties. Most importantly, the convergence of a particle swarm initialized with random weights is not guaranteed.

In this paper we propose a chaotic Hopfield neural network swarm optimization (CHNNSO) algorithm. The rest of the paper is organized as follows. In Section 2, the preliminaries of Hopfield neural networks and PSO are described. The chaotic Hopfield neural network model is developed in Section 3. In Section 4, the dynamics of the chaotic Hopfield neural network is analyzed. Section 5 provides simulation results and comparisons. The conclusion is given in Section 6.

2. Preliminaries

2.1. Basic Hopfield Neural Network Theory [22]

A Hopfield net is a recurrent neural network having a synaptic connection pattern such that there is an underlying Lyapunov energy function for the activity dynamics. Started in any initial state, the state of the system evolves to a final state that is a (local) minimum of the Lyapunov energy function. The Lyapunov energy function decreases in a monotone fashion under the dynamics and is bounded below. Because of the existence of an elementary Lyapunov energy function for the dynamics, the only possible asymptotic result is a state on an attractor.

There are two popular forms of the model: binary neurons with discrete time which is updated one at a time and continuous time graded neurons. In this paper, the second kind of model is used. The dynamics of a 𝑛-neuron continuous Hopfield neural network is described by 𝑑𝑢_𝑖=𝑑𝑡−𝑢_𝑖𝜏+_𝑗𝑇_𝑖𝑗𝑥_𝑖(𝑡)+𝐼_𝑖. (1) Here, 𝑢_𝑖∈(−∞,∞) is the input of neuron 𝑖, and the output of neuron 𝑖 is 𝑥_𝑖(𝑡)=𝑔_𝑖𝑢_𝑖(𝑡), (2) where 𝑖=1,2,…𝑛, 𝜏 is a positive constant, 𝐼_𝑖 is external inputs (e.g., sensory input or bias current) to neuron 𝑖 and is sometimes called the “firing threshold" when replaced with −𝐼_𝑖. 𝑢_𝑖 is the mean internal potential of the neuron which determines the output of neuron 𝑖. 𝑇_𝑖𝑗 is the strength of synaptic input from neuron 𝑖 to neuron 𝑗. 𝑔 is a monotone function that converts internal potential into firing rate input of the neuron. 𝑇 is the matrix with elements 𝑇_𝑖𝑗. When 𝑇 is symmetric, the Lyapunov energy function is given by 1𝐽=−2_𝑖𝑗𝑇_𝑖𝑗𝑥_𝑖𝑥_𝑗−_𝑖𝐼_𝑖𝑥_𝑖+1𝜏_𝑖^𝑥𝑗₀𝑔⁻¹(𝑍)𝑑𝑍, (3) where 𝑔⁻¹ is the inverse of the gain function 𝑔. There is a significant limiting case of this function when 𝑇 has no diagonal elements and the input-output relation becomes a step, going from 0 to a maximum firing rate (for convenience, scaled to 1). The third term of this Lyapnouv function is then zero or infinite. With no diagonal elements in 𝑇, the minima of 𝐽 are all located at corners of the hypercube 0≤𝑥_𝑗≤1. In this limit, the states of the continuous variable system are stable.

Many optimization problems can be readily represented using Hopfield nets by transforming the problem into variables such that the desired optimization corresponds to the minimization of the respective Lyapunov energy function [6]. The dynamics of the HNN converges to a local Lyapunov energy minimum. If this local minimum is also the global minimum, the solution of the desired optimization task has been carried out by the convergence of the network state.

2.2. Basic PSO Theory

Many real optimization problems can be formulated as the following functional optimization problem: min𝑓𝑋_𝑖,𝑋_𝑖=𝑥¹_𝑖,…,𝑥^𝑛_𝑖,s.t.𝑥_𝑖∈𝑎_𝑖,𝑏_𝑖,𝑖=1,2,…,𝑛. (4) Here 𝑓 is the objective function, and 𝑋_𝑖 is the decision vector consisting of 𝑛 variables.

The original particle swarm algorithm works by iteratively searching in a region and is concerned with the best previous success of each particle, the best previous success of the particle swarm and the current position and velocity of each particle [20]. Every candidate solution of 𝑓(𝑋_𝑖) is called a “particle.” The particle searches the domain of the problem according to 𝑉_𝑖(𝑡+1)=𝜔𝑉_𝑖(𝑡)+𝑐₁𝑅₁𝑃_𝑖−𝑋_𝑖(𝑡)+𝑐₂𝑅₂𝑃_𝑔−𝑋_𝑖(𝑡𝑋),_𝑖(𝑡+1)=𝑋_𝑖(𝑡)+𝑉_𝑖(𝑡+1), (5)(6) where 𝑉_𝑖=[𝑣¹_𝑖,𝑣²_𝑖,…,𝑣^𝑛_𝑖] is the velocity of particle 𝑖; 𝑋_𝑖=[𝑥¹_𝑖,𝑥²_𝑖,…,𝑥^𝑛_𝑖] represents the position of particle 𝑖; 𝑃_𝑖 represents the best previous position of particle 𝑖 (indicating the best discoveries or previous experience of particle 𝑖); 𝑃_𝑔 represents the best previous position among all particles (indicating the best discovery or previous experience of the social swarm); 𝜔 is the inertia weight that controls the impact of the previous velocity of the particle on its current velocity and is sometimes adaptive [17]; 𝑅₁ and 𝑅₂ are two random weights whose components 𝑟^𝑗₁ and 𝑟^𝑗₂ (𝑗=1,2,…,𝑛) are chosen uniformly within the interval [0,1] which might not guarantee the convergence of the particle trajectory; 𝑐₁ and 𝑐₂ are the positive constant parameters. Generally the value of each component in 𝑉_𝑖 should be clamped to the range [−𝑣_max,𝑣_max] to control excessive roaming of particles outside the search space.

3. A Chaotic Hopfield Neural Network Model

From the introduction of basic PSO theory, every particle can be seen as the model of a single fish or a single bird. The position chosen by the particle can be regarded as a state of a neural network with a random synaptic connection. According to (5)-(6), the position components of particle 𝑖 can be thought of as the output of a neural network as shown in Figure 1.

Figure 1 

Particle structure.

In Figure 1, Rand1(⋅) and Rand2(⋅) are two independent and uniformly distributed random variables within the range [0,1], which refer to 𝑟^𝑗₁ and 𝑟^𝑗₂, respectively. 𝑝^𝑗_𝑖 and 𝑝^𝑗_𝑔 are the components of 𝑃_𝑖 and 𝑃_𝑔, respectively. 𝑝^𝑗_𝑔 is the previous best value amongst all particles, and 𝑝^𝑗_𝑖, as an externally applied input, is the 𝑗th element of the best previous position 𝑃_𝑖, and it is coupled with other components of 𝑃_𝑖. The particles migrate toward a new position according to (5)-(6). This process is repeated until a defined stopping criterion is met (e.g., maximum number of iterations or a sufficiently good fitness value).

As pointed out by Clerc and Kennedy [15], the powerful optimization ability of the PSO comes from the interaction amongst the particles. The analysis of complex interaction amongst the particles in the swarm is beyond the scope of this paper which focuses on the construction of a simple particle using a neural network perspective and convergence issues. Artificial neural networks are composed of simple artificial neurons mimicking biological neurons. The HNN has the property that as each neuron in a HNN updates, an energy function is monotonically reduced until the network stabilizes [23]. One can therefore map an optimization problem to a HNN such that the cost function of the problem corresponds to the energy function of the HNN and the result of the HNN thus suggests a low cost solution to the optimization problem. The HNN might therefore be a good choice to model particle behavior.

In order to approach 𝑃_𝑔 and 𝑃_𝑖, the HNN model should include at least two neurons. For simplicity, the HNN model of each particle position component has two neurons whose outputs are 𝑥^𝑗_𝑖(𝑡) and 𝑥^𝑗_𝑖𝑝(𝑡). In order to transform the problem into variables such that the desired optimization corresponds to the minimization of the energy function, the objective function should be determined firstly. As 𝑥^𝑗_𝑖(𝑡) and 𝑥^𝑗_𝑖𝑝(𝑡) should approach 𝑝^𝑗_𝑔 and 𝑝^𝑗_𝑖, respectively, (𝑥^𝑗_𝑖(𝑡)−𝑝^𝑗_𝑔)² and (𝑥^𝑗_𝑖𝑝(𝑡)−𝑝^𝑗_𝑖)² can be chosen as two parts of the energy function. The third part of energy function (𝑥^𝑗_𝑖(𝑡)−𝑥^𝑗_𝑖𝑝(𝑡))² is added to accompany (𝑥^𝑗_𝑖𝑝(𝑡)−𝑝^𝑗_𝑖)² to cause 𝑥^𝑗_𝑖(𝑡) to tend towards 𝑝^𝑗_𝑖. Therefore the HNN Lyapunov energy function for each particle is proposed: 𝐽^𝑗_𝑖(𝑡)=𝐴𝑥^𝑗_𝑖(𝑡)−𝑝^𝑗_𝑔²+𝐵𝑥^𝑗_𝑖𝑝(𝑡)−𝑝^𝑗_𝑖²+𝐶𝑥^𝑗_𝑖(𝑡)−𝑥^𝑗_𝑖𝑝(𝑡)², (7) where 𝐴, 𝐵, and 𝐶 are positive constants.

Here the neuron input-output function is chosen as a sigmoid function, given by (9) and (11). Equations (8) and (10) are the Euler approximation of (1) of the continuous Hopfield neural network [14]. The dynamics of component 𝑗 of particle 𝑖 is described by 𝑢^𝑗_𝑖(𝑡+1)=𝑘𝑢^𝑗_𝑖(𝑡)+𝛼𝜕𝐽^𝑗_𝑖(𝑡)𝜕𝑥^𝑗_𝑖(𝑡)𝑥_𝑗𝑖,(𝑡+1)=𝑔_𝑖𝑢^𝑗_𝑖()=1𝑡+11−𝑒^{𝜉𝑢𝑗𝑖(𝑡+1)},𝑢^𝑗_𝑖𝑝(𝑡+1)=𝑘𝑢^𝑗_𝑖𝑝(𝑡)+𝛼𝜕𝐽^𝑗_𝑖(𝑡)𝜕𝑥^𝑗_𝑖𝑝(𝑡),𝑥^𝑗_𝑖𝑝(𝑡+1)=𝑔_𝑖𝑢^𝑗_𝑖𝑝()=1𝑡+11−𝑒^{𝜉𝑢𝑗𝑖𝑝(𝑡+1)}. (8)(9)(10)(11) According to (5)-(6) and Figure 1, the PSO uses random weights to simulate birds flocking or fish searching for food. When birds flock or fish search for food, they exhibit chaos like behavior, yet (8)–(11) do not generate chaos. Aihara et al. [9] proposed a kind of chaotic neuron, which includes relative refractoriness in the model to simulate chaos in a biological brain. To use this result 𝑧_𝑖(𝑘)(𝑥^𝑗_𝑖(𝑘)−𝐼₀) and 𝑧_𝑖(𝑘)(𝑥^𝑗_𝑖𝑝(𝑘)−𝐼₀) are added to (8) and (10) to cause chaos. Equations (8) and (10) then become 𝑢^𝑗_𝑖(𝑡+1)=𝑘𝑢^𝑗_𝑖(𝑡)+𝛼𝜕𝐽^𝑗_𝑖(𝑡)𝜕𝑥^𝑗_𝑖(𝑡)−𝑧_𝑖(𝑡)𝑥^𝑗_𝑖(𝑡)−𝐼₀𝑢_𝑗𝑖𝑝,(𝑡+1)=𝑘𝑢^𝑗_𝑖𝑝(𝑡)+𝛼𝜕𝐽^𝑗_𝑖(𝑡)𝜕𝑥^𝑗_𝑖𝑝(𝑡)−𝑧_𝑖(𝑡)𝑥^𝑗_𝑖𝑝(𝑡)−𝐼₀. (12)(13) In order to escape from chaos as time evolves, we set 𝑧_𝑖(𝑡+1)=1−𝛽𝑧_𝑖(𝑡). (14) In (8)–(14): 𝜉,  𝛼, and 𝑘 are positive parameters; 𝑧_𝑖(𝑡) is self-feedback connection weight (the refractory strength); 𝛽 is the damping factor of the time-dependent 𝑧_𝑖(𝑡), (0≤𝛽≤1); 𝐼₀ is a positive parameter.

All the parameters are fixed except 𝑧(𝑡) which is varied.

The combination of (9), (11)–(14) is called chaotic Hopfield neural network swarm optimization(CHNNSO) proposed by us. According to (8)–(14), the following procedure can be used for implementing the proposed CHNNSO algorithm.(1) Initialize the swarm, assign a random position in the problem hyperspace to each particle, and calculate the fitness function which is given by the optimization problem whose variables are corresponding to the elements of particle position coordinates.(2) Synchronously update the positions of all the particles using (9), (11)–(14) and change the two states every iteration.(3) Evaluate the fitness function for each particle. (4) For each individual particle, compare the particle’s fitness value with its previous best fitness value. If the current value is better than the previous best value, then set this value as the 𝑝^𝑗_𝑖 and the current particle’s position, 𝑥^𝑗_𝑖, as 𝑝^𝑗_𝑖, else if the 𝑝_𝑖 is updated, then reset 𝑧=𝑧₀. (5) Identify the particle that has the best fitness value. When iterations are less than a certain value, and if the particle with the best fitness value is changed then reset 𝑧=𝑧₀ to keep the particles chaotic to prevent premature convergence.(6) Repeat steps (2)--(5) until a stopping criterion is met (e.g., maximum number of iterations or a sufficiently good fitness value).

As can be seen from (9) and (11), the particle position component 𝑥^𝑗_𝑖 is located in the interval [−1,1]. The optimization problem variable interval must therefore be mapped to [−1,1] and vice versa using 𝑥^𝑗2𝑥=−1+^𝑗−𝑎_𝑗𝑏_𝑗−𝑎_𝑗𝑥,𝑗=1,2,…,𝑛,_𝑗=𝑎_𝑗+12𝑥^𝑗+1𝑏_𝑗−𝑎_𝑗,𝑗=1,2,…,𝑛. (15) Here, 𝑎_𝑗 and 𝑏_𝑗 are the lower boundary and the upper boundary of 𝑥_𝑗(𝑡), respectively, and only one particle is analyzed for simplicity.

4. Dynamics of Chaotic Hopfield Network Swarm Optimization

In this section, the dynamics of the chaotic Hopfield network swarm optimization (CHNNSO) is analyzed. The first subsection discusses the convergence of the chaotic particle swarm. The second subsection discusses the dynamics of the simplest CHNNSO with different parameter values.

4.1. Convergence of the Particle Swarm

Theorem 1 (Wang and Smith [14]). If one has a network of neurons with arbitrarily increasing I/O functions, there exists a sufficient stability condition for a synchronous TCNN (transiently chaotic neural network) equation (12), namely, 𝑘>1,2𝑘𝛽_max>−𝑇^′_min. (16) Here 1/𝛽_max=min(1/𝑔^′_𝑖) (𝑔^′_𝑖 denotes the derivative with respect to time 𝑡 of the neural I/O function for neuron 𝑖, in this paper is the sigmoid function (9). 𝑇^′_min is the minimum eigenvalue of the connected weight matrix of the dynamics of a n-neuron continuous Hopfield neural network).

Theorem 2. A sufficient stability condition for the CHNNSO model is 𝑘>1.

Proof. When 𝑡→∞, 𝑧(𝑡+1)=0. (17) It then follows that the equilibria of (12) and (13) can be evaluated by Δ𝑢^𝑗_𝑖(𝑡+1)=(𝑘−1)𝑢^𝑗_𝑖(𝑡)+𝛼𝜕𝐽^𝑗_𝑖(𝑡)𝜕𝑥^𝑗_𝑖(𝑡),Δ𝑢^𝑗_𝑖𝑝(𝑡+1)=(𝑘−1)𝑢^𝑗_𝑖𝑝(𝑡)+𝛼𝜕𝐽^𝑗_𝑖(𝑡)𝜕𝑥^𝑗_𝑖𝑝(𝑡). (18) According to (7), we get Δ𝑢^𝑗_𝑖(𝑡+1)=(𝑘−1)𝑢^𝑗_𝑖(𝑡)+2𝐴𝛼𝑥^𝑗_𝑖(𝑡)−𝑝^𝑗_𝑔+2𝐶𝛼𝑥^𝑗_𝑖(𝑡)−𝑥^𝑗_𝑖𝑝(𝑡)=0,Δ𝑢^𝑗_𝑖𝑝(𝑡+1)=(𝑘−1)𝑢^𝑗_𝑖𝑝(𝑡)+2𝐵𝛼𝑥^𝑗_𝑖𝑝(𝑡)−𝑝^𝑗_𝑖−2𝐶𝛼𝑥^𝑗_𝑖(𝑡)−𝑥^𝑗_𝑖𝑝(𝑡)𝑇=0,_′min=()=mineigenvalue2𝐴𝛼+2𝐶𝛼−2𝐶𝛼−2𝐶𝛼2𝐵𝛼+2𝐶𝛼𝐴+𝐵+2𝐶𝛼−𝛼(𝐴−𝐵)²+4𝐶²=𝛼𝐴+𝐵+2𝐶−(𝐴−𝐵)²+4𝐶². (19)(20)(21) In this paper, 𝛽_max=1min1/𝑔^′=𝜉4. (22) It is clear that 𝑇^′_min>0 and the stability condition (16) is satisfied when 𝜉>0. The above analysis verifies Theorem 2.

Theorem 3. The particles converge to the sphere with center point 𝑝^𝑗_𝑔 and radius 𝑅=max‖𝑥^𝑗_𝑖𝑒−𝑝^𝑗_𝑖‖ (𝑥^𝑗_𝑖𝑒 is the final convergence equilibria, if the optimization problem is in two-dimensional plane, the particles are finally in a circle).

It is easy to show that the particle model given by (7) and (8)–(14) has only one equilibrium as 𝑡→∞, that is,  𝑧(𝑡)=0. Hence, as 𝑡→∞, 𝑋_𝑖 belongs to the hypersphere whose origin is 𝑃_𝑔 and the radius is 𝑅. Solving (9), (11), (19), and (20) simultaneously, we get (𝑘−1)ln1−1/𝑥^𝑗_𝑖(𝑡)𝜉+2𝐴𝛼𝑥^𝑗_𝑖(𝑡)−𝑃^𝑗_𝑔+2𝐶𝛼𝑥^𝑗_𝑖(𝑡)−𝑥^𝑗_𝑖𝑝(𝑡)()=0,𝑘−1ln1−1/𝑥^𝑗_𝑖𝑝(𝑡)𝜉+2𝐵𝛼𝑥^𝑗_𝑖𝑝(𝑡)−𝑃^𝑗_𝑖−2𝐶𝛼𝑥^𝑗_𝑖(𝑡)−𝑥^𝑗_𝑖𝑝(𝑡)=0. (23)(24) With (23) and (24) satisfied, there must exist the final convergence equilibria 𝑥^𝑗_𝑖𝑒 and 𝑥^𝑗_𝑖𝑝𝑒. So the best place the particle swarm can find is ‖𝑝^𝑗_𝑔−𝑥^𝑗_𝑖𝑒(𝑡+1)‖ and radius is 𝑅=max‖𝑥^𝑗_𝑖𝑒−𝑝^𝑗_𝑖‖.

The above analysis therefore verifies Theorem 3.

4.2. Dynamics of the Simplest Chaotic Hopfield Neural Network Swarm

In this section, the dynamics of the simplest particle swarm model is analyzed. Equations (7) and (8)–(13) are the dynamic model of a single particle with subscript 𝑖 ignored. According to (7) and Theorem 3, the parameters 𝐴, 𝐵, and 𝐶 control the final convergent radius. According to trial and error, the parameters 𝐴, 𝐵, and 𝐶 can be chosen in the range from 0.005 to 0.05. According to (16) and (22), 𝑘>1 and 𝜉>0. In the simulation, the results are better when 𝑘 is in the neighborhood of 1.1 and 𝜉 is in the neighborhood of 150. The parameters 𝛽 and 𝑍(0) control the time of the chaotic period. If 𝛽 is too big and/or 𝑍(0) is too small, the system will quickly escape from chaos and performance will be poor. The parameter 𝐼₀=0.2 is standard in the literature on chaotic neural networks. The simulation showed that the model is not sensitive to the values of parameters 𝜉 and 𝑘, for example, 100<𝜉<200 and 1<𝑘<1.5 are feasible.

Then the values of the parameters in (7)–(14) are set to (0)𝐴=0.02,𝐵=𝐶=0.01,𝜉=150,𝛼=1,𝛽=0.001,𝑍=0.3,𝑘=1.1,𝐼₀=0.2,𝑝_𝑖=0.5,𝑝_𝑔=0.2. (25) Figure 2 shows the time evolution of 𝑥_𝑖(𝑡), 𝑧(𝑡) and the Lyapunov exponent 𝜆 of 𝑥_𝑖(𝑡). The Lyapunov exponent 𝜆 characterizes the rate of separation of infinitesimally close trajectories. A positive Lyapunov exponent is usually taken as an indication that the system is chaotic [1]. Here, 𝜆 is defined as 𝜆=lim_𝑛→+∞1𝑚^𝑚−1_𝑡=0ln𝑑𝑥_𝑖(𝑡+1)𝑑𝑥_𝑖(𝑡). (26) At about 200 steps, 𝑧(𝑡) decays to a small value and 𝑥_𝑖(𝑡) departs from chaos which corresponds with the change of 𝜆 from positive to negative.

(a)

(b)

(c)

Figure 2 

Time evolutions of 𝑥𝑖(𝑡),𝑧(𝑡), and the Lyapunov exponent 𝜆.

According to Figure 2, the convergence process of a simple particle position follows the nonlinear bifurcation making the particle converge to a stable fixed point from a strange attractor. In the following section, it is shown that the fixed point is determined by the best previous position 𝑃_𝑔 among all particles and the best position 𝑃_𝑖 of the individual particle.

Remark 4. The proposed CHNNSO model is a deterministic Chaos-Hopfield neural network swarm which is different from existing PSOs with stochastic parameters. Its search orbits exhibit an evolutionary process of inverse period bifurcation from chaos to periodic orbits then to sink. As chaos is ergodic and the particle is always in a chaotic state at the beginning (e.g., in Figure 2), the particle can escape when trapped in local extrema. This proposed CHNNSO model will therefore in general not suffer from being easily trapped in a the local optimum and will continue to search for a global optimum.

5. Numerical Simulation

To test the performance of the proposed algorithms, two famous benchmark optimization problems and an engineering optimization problem with linear and nonlinear constraints are used. The solutions to the two benchmark problems can be represented in the plane and therefore the convergence of the CHNNSO can be clearly observed. The results of the third optimization problem when compared with other algorithms are displayed in Table 1. We will compare the CHNNSO with the original PSO [20].

Table 1 

The parameters of the Hartmann function (when 𝑛=3 and 𝑞=4).

5.1. The Rastrigin Function

To demonstrate the efficiency of the proposed technique, the famous Rastrigin function is chosen as a test problem. This function with two variables is given by 𝑓(𝑋)=𝑥²₁+𝑥²₂−cos18𝑥₁−cos18𝑥₂,−1<𝑥_𝑖<1,𝑖=1,2. (27) The global minimum is −2 and the minimum point is (0, 0). There are about 50 local minima arranged in a lattice configuration.

The proposed technique is applied with a population size of 20 and the maximum number of iterations is 20000. The chaotic particle swarm parameters are chosen as 𝐴=0.02, 𝐵=𝐶=0.01, 𝛽=0.001, 𝑧(0)=0.3, 𝛼=1, 𝜉=150, 𝑘=1.1,  and  𝐼₀=0.2.

The position of every particle is initialized with a random value. The time evolution of the cost of the Rastrigin function is shown in Figure 3. The global minimum at −2 is obtained by the best particle with (𝑥₁,𝑥₂)=(0,0).

Figure 3 

Time evolutions of Rastrigin function.

From Figure 3, it can be seen that the proposed method gives good optimization results. Since there are two variables in the Rastrigin function, it is easy to show the final convergent particle states in the plane.

In Figure 4, the “∗”s are the best experiences of each particle. The “+”s are the final states of the particles. The global minimum (0,0) is also included in the “∗”s and the “+”s. Most “∗”s and “+”s are overlapped at the global minimum (0,0). According to Theorem 3, the particles will finally converge to a circle finally. For this Rastrigin problem, the particles’ final states converge to the circle as shown in Figure 4, and hence the global convergence of the particles is guaranteed.

Figure 4 

The final states and the best experiences of the particles achieved from the proposed CHNNSO for Rastrigin function.

Figure 5 displays the results when the original PSO [20] was used to optimize the Rastrigin function. In the numerical simulation, the particle swarm population size is also 20 and parameters 𝑐₁ and 𝑐₂ are set to 2 and 𝜔 set to 1. 𝑣_max is set equal to the dynamic range of each dimension. The “∗”s in Figure 5 are the final states of all the particles corresponding to the “+”s in Figure 4. It is easy to see that the final states of the particles are ruleless even though the global minimum of −2 is obtained by the best experience of the particle swarm, that is, (𝑥₁,𝑥₂)=(0,0) as shown in Figure 5.

Figure 5 

The final states of the particles achieved from original PSO for Rastrigin function.

By comparing the results obtained by the proposed CHNNSO in Figure 4 with the results of the original PSO in Figure 5, it can be seen that the final states of the particles of the proposed CHNNSO are attracted to the best experience of all the particles and that convergence is superior. The final states of CHNNSO particles are guaranteed to converge which is not the case for original PSO implications.

When their parameters 𝑐₁ and 𝑐₂ are both set to 1.494 and 𝜔 to a value of 0.729, 𝑣_max is set equal to the dynamic range on each dimension. The constriction factors 𝑐₁, 𝑐₂, 𝜔 are applied to improve the convergence of the particle over time by damping the oscillations once the particle is focused on the best point in an optimal region. The main disadvantage of this method is that the particles may follow wider cycles and may not converge when the individual best performance is far from the neighborhoods best performance (two different regions) [24].

5.2. The Schaffer’s F6 Function

To further investigate the performance of the CHNNSO, 𝑓(𝑋)=0.5+sin𝑥²₁+𝑥²₂²−0.51.0+0.001𝑥²₁+𝑥²₂², (28) the Schaffer’s F6 function [25] is chosen. This function has a single global optimum at (0,0) and 𝑓_min(𝑥,𝑦)=0, and a large number of local optima. The global optimum is difficult to find because the value at the best local optimum differs with only about 10⁻³ from the global minimum. The local optima crowd around the global optimum. The proposed technique is applied with a population size of 30, the iterations are 100000, and the parameters of CHNNSO are chosen as 𝐴=𝐵=𝐶=0.01,𝛽=0.0003,𝑧(0)=0.3,𝜉=155,𝑘=1.1,𝐼₀=0.2. (29)

The position of each particle is initialized with a random value. In Figure 6, the “∗”s are the best experiences of each particle. The global minimum at 2.674×10⁻⁴ is obtained by the best particle with (𝑥₁,𝑥₂)=(−0.00999,0.01294) which is included in the “∗”s. The “+”s are the final states of the particles. According to Theorem 3 the particles’ final states converge in a circle as shown in Figure 6 which proves global convergence. From Figure 6 it is clearly seen that the particles’ final states are attracted to the neighborhoods of the best experiences of all the particles and the convergence is good.

Figure 6 

The final states and the best experiences of the particles achieved from the proposed CHNNSO for Schaffer’ F6 function.

Figure 7 shows the final particle states when the original PSO [20] was used to optimize the Schaffer’s F6 function. In this numerical simulation of the original PSO, the particle swarm population size is also 20 and parameters 𝑐₁ and 𝑐₂ are both set to 2 and set 𝜔 is set to a value of 1. 𝑣_max is set equal to the dynamic range of each dimension. The “∗”s in Figure 7 are the final states of all the particles corresponding to the “+”s in Figure 6. It is easy to see that the final states of the particles are ruleless in Figure 7. The global minimum 6.3299×10⁻⁴ is obtained by the best particle with (𝑥₁,𝑥₂)=(0.01889×10⁻³, −0.4665×10⁻³). The best experience from the original PSO is not as good as the best of the proposed CHNNSO.

Figure 7 

The final states of the particles achieved from original PSO for Schaffer’s F6 function.

Comparing the results obtained from the proposed CHNNSO in Figure 6 and the original PSO in Figure 7, it is clearly seen that the particles’ final states of the proposed CHNNSO are finally attracted to the best experience of all the particles and the convergence is better than that of the original PSO. The CHNNSO can guarantee the convergence of the particle swarm, but the final states of the original PSO are ruleless.

5.3. The Hartmann Function

The Hartmann function when 𝑛=3; 𝑞=4 is given by 𝑓(𝑥)=−^𝑞_𝑖=1𝑐_𝑖exp−^𝑛_𝑗=1𝑎_𝑖𝑗𝑥_𝑗−𝑝_𝑖𝑗², (30) with 𝑥_𝑗 belonging to 𝑆=𝑥∈𝑅^𝑛∣0≤𝑥_𝑗≤1,1≤𝑗≤𝑛, (31) Table 1 shows the parameter values for the Hartmann function when 𝑛=3,𝑞=4.

When 𝑛=3, 𝑋_min=(0.114,0.556,0.882), 𝑓(𝑥_min)=−3.86.

The time evolution of the cost of the Hartmann function is 15000. In Figure 8, only subdimensions are pictured. In Figure 8, the “+”s are the final states of the particles and the “*”s denote the best experiences of all particles. From Figure 8, it can be easily seen that the final states of the particles converge to the circle. The center point is (0.0831,0.5567,0.8522) and the radius is 0.1942. The final particle states confirm Theorem 3, and the final convergency is guaranteed.

Figure 8 

The final states of the particles achieved from original PSO for Hartmann function.

5.4. Design of a Pressure Vessel

The pressure vessel problem described in [26, 27] is an example which has linear and nonlinear constraints and has been solved by a variety of techniques. The objective of the problem is to minimize the total cost of the material needed for forming and welding a cylindrical vessel. There are four design variables: 𝑥₁ (𝑇_𝑠, thickness of the shell), 𝑥₂ (𝑇_ℎ, thickness of the head), 𝑥₃ (𝑅, inner radius), and 𝑥₄ (𝐿, length of the cylindrical section of the vessel). 𝑇_𝑠 and 𝑇_ℎ are integer multiples of 0.0625 inch, which are the available thickness of rolled steel plates, and 𝑅 and 𝐿 are continuous. The problem can be specified as follows: Minimize𝑓(𝑋)=0.6224𝑥₁𝑥₃𝑥₄+1.7781𝑥₂𝑥²₃+3.1661𝑥²₁𝑥₄+19.84𝑥²₁𝑥₃,subjectto𝑔₁(𝑋)=−𝑥₁+0.0193𝑥₃𝑔≤0,₂(𝑋)=−𝑥₂+0.00954𝑥₃𝑔≤0,₃(𝑋)=−𝜋𝑥²₃𝑥₄−43𝜋𝑥₃𝑔+1296000≤0,₄(𝑋)=𝑥₄−240≤0. (32) The following range of the variables were used [27]: 0<𝑥₁≤99,0<𝑥₂≤99,10≤𝑥₃≤200,10≤𝑥₄≤200. (33) de Freitas Vas and de Graça Pinto Fernandes [26] proposed an algorithm to deal with the constrained optimization problems. Here this algorithm [26] is combined with CHNNSO to search for the global optimum. The proposed technique is applied with a population size of 20 and the maximum number of iterations is 20000. Then the values of the parameters in (7)–(14) are set to (0)𝐴=0.02,𝐵=𝐶=0.01,𝜉=150,𝛼=1,𝛽=0.001,𝑍=0.3,𝑘=1.1,𝐼₀𝑝=0.2,_𝑖=0.5,𝑝_𝑔=0.5. (34) From Table 2, the best solution obtained by the CHNNSO is better than the other two solutions previously reported.

Table 2 

Comparison of the results for pressure Vessel design problem.

6. Conclusion

This paper proposed a chaotic neural networks swarm optimization algorithm. It incorporates the particle swarm searching structure having global optimization capability into Hopfield neural networks which guarantee convergence. In addition, by adding chaos generator terms into Hopfield neural networks, the ergodic searching capability is greatly improved in the proposed algorithm. The decay factor introduced in the chaos terms ensures that the searching evolves to convergence to global optimum after globally chaotic optimization. The experiment results of three classic benchmark functions showed that the proposed algorithm can guarantee the convergence of the particle swarm searching and can escape from local extremum. Therefore, the proposed algorithm improves the practicality of particle swarm optimization. As this is a general particle model, some techniques such as the local best version algorithm proposed in [29] can be used together with the new model. This will be explored in future work.

Acknowledgments

This work was supported by China/South Africa Research Cooperation Programme (nos. 78673 and CS06-L02), South African National Research Foundation Incentive Grant (no. 81705), SDUST Research Fund (no. 2010KYTD101), and Key scientific support program of Qingdao City (no. 11-2-3-51-nsh).