Abstract
Being inspired by the Hopfield neural networks (Hopfield (1982) and Hopfield and Tank (1985)) and the nonlinear sigmoid power control algorithm for cellular radio systems in Uykan and Koivo (2004), in this paper, we present a novel discrete recurrent nonlinear system and extend the results in Uykan (2009), which are for autonomous linear systems, to nonlinear case. The proposed system can be viewed as a discrete-time realization of a recently proposed continuous-time network in Uykan (2013). In this paper, we focus on discrete-time analysis and provide various novel key results concerning the discrete-time dynamics of the proposed system, some of which are as follows: (i) the proposed system is shown to be stable in synchronous and asynchronous work mode in discrete time; (ii) a novel concept called Pseudo-SINR (pseudo-signal-to-interference-noise ratio) is introduced for discrete-time nonlinear systems; (iii) it is shown that when the system states approach an equilibrium point, the instantaneous Pseudo-SINRs are balanced; that is, they are equal to a target value. The simulation results confirm the novel results presented and show the effectiveness of the proposed discrete-time network as applied to various associative memory systems and clustering problems.
1. Introduction
Artificial neural networks have been an important research area since 1970s. Since then, various biologically inspired neural network models have been developed. Hopfield Neural Networks [1, 2] have been one of the most widely used neural networks since the early 1980s whose applications vary from combinatorial optimization (e.g., [3, 4]) to image restoration (e.g., [5]) and from various control engineering optimization problems in robotics (e.g., [6]) to associative memory systems (e.g., [7, 8]). For a tutorial and further references about Hopfield neural networks, see, for example, [9, 10].
In [11], we introduce a novel pseudo-signal-to-interference-noise ratio concept for discrete-time autonomous linear systems. Our main motivation in this paper is to investigate a nonlinear extension of [11]. Furthermore, the proposed system can be viewed as a discrete-time realization of a very recently proposed continuous-time network called double-sigmoid continuous-time Hopfield neural network in a brief letter [12]. And our investigations in this paper yield various interesting key novel results in discrete time, some of which are as follows: (i) a novel concept called Pseudo-SINR (pseudo-signal-to-interference-noise ratio) is introduced for discrete-time nonlinear systems; (ii) it is shown that when the network approaches to one of its equilibrium points, the instantaneous Pseudo-SINRs are balanced; that is, they are equal to a target value; (iii) the proposed network outperforms its Hopfield neural network counterpart as applied to various associative memory systems and clustering applications. The disadvantage of the proposed network is that it increases the computational burden.
The paper is organized as follows. The proposed recurrent network and its stability features are analyzed in Section 2. Simulation results are presented in Section 3, followed by conclusions in Section 4.
2. Discrete Pseudo-SINR-Balancing Recurrent Neural Networks
Being inspired by the nonlinear sigmoid power control algorithm for cellular radio systems in [13] and the Hopfield neural networks [2], we propose the following discrete nonlinear recurrent network: where represents the iteration step, , , and are defined as in (2), and , , represents a vectoral mapping from to . For an -dimensional vector , where is chosen as the sigmoid function; that is, for a real number , the output is , where , . We will call the network in (1) as discrete sigmoid-pseudo-SINR-balancing recurrent neural network (D-SP-SNN). The name comes from the fact that the proposed network balances an entity called Pseudo-SINR, as will be seen in the following. In this paper, we choose sigmoid function because it’s used both in Hopfield neural network and the power control algorithm in [13]. Furthermore, the proposed D-SP-SNN can be viewed as a discrete-time implementation of a very recently proposed continuous-time network called double-sigmoid continuous-time Hopfield neural network in the brief [12]. In this paper, we focus on discrete-time analysis and provide various novel key results concerning the discrete-time dynamics of the proposed system:
In (2), shows the self-state-feedback matrix, with zero diagonal shows the interneurons connection weight matrix, and is a threshold vector.
The proposed network includes both the sigmoid power control in [13] and the traditional Hopfield neural network (HNN) as its special cases by choosing the and appropriately. The Euler approximation of the continuous-time HNN is given as
Let us call the network in (3) HNN-Euler, which is a special case of the proposed D-SP-SNN in (1). From (1), where is the step size at time . Let’s define the error signal as
Then, the performance index is defined as -norm of the error vector in (5) as follows:
In what follows, we examine the evolution of the energy function in (6) in synchronous and asynchronous work modes. Asynchronous mode means that at every iteration step, at most only one state is updated, whereas synchronous mode refers to the case that all the states are updated at every iteration step according to (4).
Proposition 1. In asynchronous mode of the proposed network D-SP-SNN in (4) with a symmetric matrix , for a nonzero error vector, the -norm of the error vector in (6) decreases at every step; that is, the error vector goes to zero for any such that
if
where is the global Lipschitz constant of as shown the in Appendix A.
Proof. In asynchronous mode, only one state is updated at an iteration time. Let show the index of the state which is updated at time whose error signal is different than zero; that is, , as defined in (5). Writing (5) in vector form for steps and results in
Using the error signal definition of (5) in (4) gives
So, the error signal for state is obtained using (10) and (11) as follows:
From (12) and (13), if is chosen to satisfy , then
where is a sigmoid function, which is lower and upper bounded. Since the sigmoid function has the same sign as its argument and if and only if , then it is seen that can easily be chosen small enough to satisfy according to the parameter and the slope of sigmoid function .
Above, we examined only the state and its error signal . In what follows, we examine the evolution of the norm of the complete error vector in (10). From the point of view of the norm of the , the worst case is that when decreases, all other elements , increase. So, using (10), (12), and (14), we obtain that if
then
The sigmoid function is a Lipschitz continuous function as shown in Appendix A. So,
where is ’s global Lipschitz constant as shown in Appendix A. From (15) and (17), choosing yields (15), which implies (16). This completes the proof.
Definition 2 (pseudo-SINR). We define the following system variable, which will be called pseudo-SINR, for the D-SP-SNN in (4):
where represents the sigmoid function, that is, , and is a constant, which we call target .
Examining the in (18), we observe that it resembles the traditional signal-to-interference-noise ratio (SINR) definition in cellular radio systems (see, e.g., [14, 15]); therefore we call it Pseudo-SINR.
Definition 3 (prototype vectors). Prototype vectors are defined as those ’s which make , in (18). So, from (18) and (5), the prototype vectors make the error signal zero; that is, given that and .
Proposition 4. In asynchronous mode, choosing the slope of relatively small as compared to and choosing and satisfying (8), the D-SP-SNN in (4) with a symmetric matrix is stable and there exists a finite time constant such that the -norm of the error vector in (6) approaches to an -vicinity of the zero as its steady state, where is a relatively small positive number. If at the converged point, then it corresponds to a prototype vector as defined above.
Proof. Since it is asynchronous mode, (10)–(14) hold where . So, if at time is chosen to satisfy as in (8), then
Note that it is straightforward to choose a sufficiently small to satisfy (8) according to and the slope of sigmoid . Using (10), (12), and (19), it is seen for that if
then
We observe from (12), (20), (21), and (22) the following.
If the , approach to either of the saturation regimes of its sigmoid function , then
since . That satisfies (20) and (21). Therefore, the norm of the error vector in (6) does not go to infinity and is finite for any .
if and only if ; that is,
Examining the (11), (12), and (13) taking the observations and into account, we conclude that any of the , does not go to infinity and is finite for any . So, the D-SP-SNN in (4) with a symmetric matrix is stable for the assumptions in Proposition 4. Because there is a finite number of insaturation states (i.e., the number of all possible insaturation state combinations is finite), which is equal to , there exists a finite time constant such that the -norm of the error vector in (6) approaches to an -vicinity of the zero as its steady state, where is a relatively small positive number.
From (18), if at the converged point, then it corresponds to a prototype vector as defined in the previous section, which completes the proof.
In what follows, we examine the evolution of pseudo-SINR in (18). From (18), let us define the following error signal at time :
Proposition 5. In asynchronous mode, in the D-SP-SNN in (4) with a sufficiently small and with a symmetric matrix , the is getting closer to at those iteration steps where ; that is, , where index shows the state being updated at iteration .
Proof. Let show the state which is updated at time . The pseudo-SINR defined by (18) for nonzero is equal to
Without loss of generality, and for the sake of simplicity, let is take . Then, from (26) and (25)
In asynchronous mode, from (26), . Using this observation and (27),
From (4) and (28),
Provided that , we write, from (5) and (27),
Writing (30) in (29) gives
From (31), since sigmoid function is an odd function, and and ,
As seen from (32), for a nonzero , choosing a sufficiently small satisfying assures that
which completes the proof.
Proposition 6. The results in Propositions 1 and 4 for asynchronous mode hold also for synchronous mode.
In synchronous mode, all the states are updated at every step according to (5). So, from (5)
Using (5) in (34) and writing it elementwise give
From (34) and (35), we obtain which is equal to (15) in Proposition 1 and (20) in Proposition 4.
It is well known that the performance of Hopfield network may highly depend on the parameter setting of the weight matrix (e.g., [8]). There are various ways for determining the weight matrix of the Hopfield networks: gradient-descent supervised learning (e.g., [16]), solving linear inequalities (e.g., [17, 18] among others), Hebb learning rule [19, 20], and so forth. How to design D-SP-SNN is out of the scope of this paper. The methods used for traditional Hopfield NN can also be used for the proposed networks D-SP-SNN. As far as the simulation results in Section 3 are concerned, we determine the matrices , , and by using a Hebb learning-based algorithm [19] presented in Appendix B.
3. Simulation Results
In the simulation part, we examine the performance of the proposed D-SP-SNN in the area of associative memory systems and clustering problem. In Examples 7 and 8, we present some toy examples one with 8 neurons and one with 16 neurons, respectively, where the desired vectors are orthogonal. Lyapunov function of the HNN at time is given as
In Examples 7 and 8, we use discrete-time HNN just for comparison reasons, which is given by where is the weight matrix and is the state at time , and at most one state is updated at a time.
Example 7. In this example of discrete-time networks, there are 8 neurons. The desired prototype vectors are as follows:
The weight matrices and and the threshold vector are obtained as follows by using the outer-product-based design (Hebb-learning [19]) presented in Appendix B and the slopes of sigmoid functions and are set to , , and , , respectively, and , :
Figure 1 shows the percentages of correctly recovered desired patterns for all possible initial conditions , for the proposed networks D-SP-SNN as compared to traditional discrete Hopfield network. In the proposed network D-SP-SNN, is a sigmoid function. Establishing an analogy to the traditional fixed step 1-bit increase/decrease power control algorithm (e.g. [21, 22]), we replace the sigmoid function by the sign function and call corresponding network as fixed-step pseudo-SINR neural network (FSPSNN). For comparison reason its performance is also shown in Figure 1.
As seen from Figure 1 the performance of the proposed network D-SP-SNN is remarkably better than that of the traditional discrete Hopfield network for all Hamming distance cases. The FSP-SNN also considerably outperforms the Hopfield network for 1 and 2 Hamming distance cases while the all the networks perform poorly (less than 20%) at 3-Hamming distance case.
Example 8. The desired prototype vectors are
The weight matrices and and threshold vector are obtained as follows by using the outer-product-based design (Hebb-learning [19]) in Appendix B:
Figure 2 shows the percentage of correctly recovered desired patterns for all possible initial conditions , in the proposed D-SP-SNN and FSP-SNN as compared to discrete Hopfield network.
The total number of different possible combinations for the initial conditions for this example is 64, 480, 2240 and 7280 for 1-, 2-, 3-, and 4-Hamming distance cases, respectively, which could be calculated by , where and 1, 2, 3, and 4.
As seen from Figure 2 the performance of the proposed networks D-SP-SNN and FSP-SNN is the same as that of discrete Hopfield Network for 1-Hamming and 2-Hamming distance cases ( for all networks). However, the D-SP-SNN and FSP-SNN give better performance than the discrete Hopfield network does for 3- and 4- Hamming distance cases.
Typical plots for evolution of states in Example 8 by the D-SP-SNN are shown in Figure 3. The evolution of corresponding pseudo-SINRs is given by Figure 4. The figure shows that the pseudo-SINRs approach to constant value 1 as states converge to the equilibrium point.
Evolutions of the Lyapunov function in (37) for the states of Figure 3 in Example 8 are given in Figure 5. The figure shows that the proposed D-SP-SNN minimizes the Lyapunov function of Hopfield neural network with the same weight matrix.
(a)
(b)
(a)
(b)
Example 9. In Examples 7 and 8, the desired vectors are orthogonal. In this example, the desired vectors represent numbers 1, 2, 3, and 4, which are not orthogonal to each other. The numbers are represented by 25 neurons. The weight matrix is determined by the Hebb learning as in the previous examples. In the rest of the examples in this paper, we set , , , , and , for all .
Figure 6 shows desired pattern 1, a distorted pattern 1 where the Hamming Distance (HD) is 5, the result of HNN-Euler, and the result of the D-SP-SNN using the distorted pattern as initial condition. As seen from the figure, the proposed D-SP-SNN succeeds to recover the number while the HNN-Euler fails for the same parameters and weight matrix.
The evolutions of the Lyapunov function in (37) and the norm of the difference between the state vector and equilibrium point for pattern 1 in Figure 6 are shown in Figure 7. As seen from the figure, (i) the proposed D-SP-SNN minimizes the Lyapunov function of Hopfield neural network, and (ii) the proposed D-SP-SNN converges faster than its HNN-Euler counterpart with the same weight matrix for this example.
Figure 8 shows desired pattern 2, a distorted pattern 2 where the HD is 5, the result of HNN-Euler, and the result of D-SP-SNN using the distorted pattern as initial condition. As seen from the figure, the proposed D-SP-SNN succeeds to recover the number while the HNN-Euler fails for the same parameters and weight matrix.
Evolutions of states in Example 9 for pattern 2 by HNN-Euler and by D-SP-SNN are shown in Figures 9 and 10, respectively. The figures show that the states of proposed D-SP-SNN converge faster than those of its HNN-Euler counterpart for the same parameter settings.
Figure 11 shows the evolutions of pseudo-SINRs of states in Example 9 for pattern 2 by D-SP-SNN. The figure shows that the pseudo-SINRs approach to constant value 1 as states converge to the equilibrium point.
The evolutions of the norm of the difference between the state vector and equilibrium point for pattern 2 in Figure 8 are shown in Figure 12. As seen from the figure, the proposed D-SP-SNN converges much faster than its HNN-Euler counterpart.
Figure 13 shows desired pattern 3, a distorted pattern 3 where the HD is 5, the result of HNN-Euler, and the result of the D-SP-SNN using the distorted pattern as initial condition. As seen from the figure, the proposed D-SP-SNN succeeds to recover the number while its HNN-Euler counterpart fails for the same parameters and weight matrix.
The evolutions of the Lyapunov function and the norm of the difference between the state vector and equilibrium point for pattern 3 in Figure 13 are shown in Figure 14. As seen from the figure, (i) the proposed D-SP-SNN minimizes the Lyapunov function of Hopfield neural network, and (ii) the proposed D-SP-SNN converges faster than its HNN-Euler counterpart with the same weight matrix for this example.
Figure 15 shows desired pattern 4, a distorted pattern 4 where the HD is 5, the result of HNN-Euler, and the result of the D-SP-SNN using the distorted pattern as initial condition. As seen from the figure, the proposed D-SP-SNN succeeds to recover the number while its HNN-Euler counterpart fails for the same parameters settings.
The evolutions of the Lyapunov function and the norm of the difference between the state vector and equilibrium point for pattern 4 in Figure 15 are shown in Figure 16. As seen from the figure, (i) the proposed D-SP-SNN minimizes the Lyapunov function of Hopfield Neural Network, and (ii) the proposed D-SP-SNN converges faster than its HNN-Euler counterpart with the same weight matrix for this example.
(a)
(b)
(c)
(d)
(a)
(b)
(a)
(b)
(c)
(d)
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
(d)
(a)
(b)
(a)
(b)
(c)
(d)
(a)
(b)
Example 10. In this and in the following example, we examine the performance of the proposed D-SP-SNN in clustering problem. Clustering is used in a wide range of applications, such as engineering, biology, marketing, information retrieval, social network analysis, image processing, text mining, finding communities, influencers, and leaders in online or offline social networks. Data clustering is a technique that enables dividing large amounts of data into groups/clusters in an unsupervised manner such that the data points in the same group/cluster are similar and those in different clusters are dissimilar according to some defined similarity criteria. The clustering problem is an NP-complete, and its general solution even for 2-clustering case is not known. It is well known that the clustering problem can be formulated in the form of the Lyapunov function of the HNN. The weight matrix is chosen as the distance matrix of the dataset and is the same for both HNN-Euler and D-SP-SNN.
In what follows, we compare the performance of the proposed D-SP-SNN as compared to its HNN-Euler counterpart as applied to clustering problems for the very same parameter settings. Two-dimensional 16 data points to be bisected are shown in Figure 17. The clustering results are also shown in Figure 17. As seen from the figure, the D-SP-SNN finds the optimum solution for this toy example. HNN-Euler also gives the same solution.
The evolutions of states in the clustering by HNN-Euler and by D-SP-SNN are shown in Figures 18 and 19, respectively. As seen from the figures, the states of the proposed D-SP-SNN converge faster that those of its HNN-Euler counterpart.
The evolutions of psuedo-SINRs of states in the clustering by D-SP-SNN in Example 10 () are given by Figure 20. The figure shows that the pseudo-SINRs approach to constant value 1 as states converge to the equilibrium point.
The evolutions of Lyapunov function and the norm of the difference between the state vector and equilibrium point in Example 10 are given in Figure 21. The figure confirms that (i) the proposed D-SP-SNN minimizes the Lyapunov function of Hopfield neural network and (ii) the proposed D-SP-SNN converges faster than its HNN-Euler counterpart with the same weight matrix.
(a)
(b)
(a)
(b)
(a)
(b)
(a)
(b)
Example 11. In this example, there are 40 data points as shown in Figure 22. The figure also shows the bisecting clustering results by -means algorithm and the proposed D-SP-SNN. As seen from the figure, while -means fail to find the optimum clustering solution for this example (for a randomly given initial values), the proposed D-SP-SNN succeeds in finding the optimum solution (for the same initial values).
Figure 23 shows the evolution of pseudo-SINRs of states by the D-SP-SNN. The figure shows that the pseudo-SINRs approach to constant value 1 as states converge to the equilibrium point, as before.
Evolutions of the Lyapunov function and the norm of the difference between the state vector and equilibrium point are shown in Figure 24. The figure confirms the superior convergence speed of the D-SP-SNN as compared to its HNN-Euler counterpart.
(a)
(b)
(a)
(b)
4. Conclusions
In this paper, we present and analyze a discrete recurrent nonlinear system which includes the Hopfield neural networks [1, 2] and the nonlinear sigmoid power control algorithm for cellular radio systems in [13], as special cases by properly choosing the functions. This paper extends the results in [11], which are for autonomous linear systems, to nonlinear case. The proposed system can be viewed as a discrete-time realization of a recently proposed continuous-time network in [12]. In this paper, we focus on discrete-time analysis and present various novel key results concerning the discrete-time dynamics of the proposed system, some of which are as follows: (i) the proposed network is shown to be stable in synchronous and asynchronous work mode in discrete time; (ii) a novel concept called Pseudo-SINR (pseudo-signal-to-interference-noise ratio) is introduced for discrete-time nonlinear systems; (iii) it is shown that when the network approaches one of its equilibrium points, the instantaneous Pseudo-SINRs become equal to a constant target value.
The simulation results confirm the novel results (e.g., Pseudo-SINR convergence, etc.) presented and show a superior performance of the proposed network as compared to its Hopfield network counterpart in various associative memory systems and clustering examples. Moreover, the results show that the proposed network minimizes the Lyapunov function of the Hopfield neural networks. The disadvantage of the D-SP-SNN is that it increases the computational burden.
Appendices
A. Lipschitz Constant of the Sigmoid Function
In what follows, we will show the sigmoid function () that has the global Lipschitz constant . Since is a differentiable function, we can apply the mean value theorem:
The derivative of is whose maximum is at the point ; that is, . So we obtain the following inequality: where is the global Lipschitz constant of the sigmoid function.
B. Outer Product-Based Network Design
Let us assume that desired prototype vectors are orthogonal and each element of a prototype vector is either −1 or .
Step 1. Calculate the sum of outer products of the prototype vectors (Hebb Rule, [19])
Step 2. Determine the diagonal matrix and as follows:
where is a real number and
where shows the entries of matrix , is the dimension of the vector , and is the number of the prototype vectors (). In (B.2), from (B.1) since is from . It is observed that gives relatively good performance; however, by examining the nonlinear state equations in Section 2, it can be seen that the proposed networks D-SP-SNN and FSP-SNN contain the prototype vectors at their equilibrium points for a relatively large interval of .
Another choice of in (B.2) is which yields . In what follows we show that this choice also assures that are the equilibrium points of the networks.
From (B.1)–(B.3)
where represents the identity matrix. Since , then . Using (B.4) and the orthogonality properties of the set gives the following:
So, the prototype vectors correspond to equilibrium points.