Abstract

For the Hopfield Neural Network problem we consider unbounded monotone nondecreasing activation functions. We prove convergence to zero in an exponential manner provided that we start with sufficiently small initial data.

1. Introduction

Of concern is the following system: where , ,, are continuous functions, and are the activation functions which will be assumed continuous and bounded by some nondecreasing (and possibly unbounded functions).

This system appears in Neural Network theory [1, 2]. As is well-known, Neural Networks are an important tool in business intelligence. Their architecture differs from the one of standard computers in that it consists of a large number of processors (neurons) with high connections between them. In contrast to computers with a single processor, (Artificial) Neural Networks perform their computations in parallel.

Just as the human brain, the neurons receive weighted signals from the neurons in the input layer, sum up these inputs and test against a threshold value. Then they decide to fire or not.

The applications are numerous, we may cite few: modelling soil behavior, design of tunnels, image processing, graph flow, data deconvolution, energy demand forecasting, ecosystem evaluation, scheduling optimization, targeted marketing, medical diagnosis, time series analysis, and stock market.

Neural Networks are able to analyze and evaluate many phenomena in real world business as well as in industry. Some of their advantages over the conventional computers are forecasting, strategy planning, and predicting many phenomena.

Different methods have been used by many authors to study the well-posedness and the asymptotic behavior of solutions [320]. In particular, a lot of efforts are devoted in improving the set of conditions on the different coefficients involved in the system as well as the class of activation functions. Regarding the latter issue, the early assumptions of boundedness, monotonicity, and differentiability have been all relaxed to merely a global Lipschitz condition. Since then, it seems that, this assumption has not been weakened further considerably although there is a great need for that [21]. A slightly weaker condition: , and there exist such that , where and is the equilibrium, has been used in [2224] (see also [2527]).

Here we assume that the activation functions are bounded by continuous monotone nondecreasing functions , that is, The functions are not necessarily Lipschitz continuous and they may be unbounded (like power type functions with powers bigger than one). We can also consider activation functions with discrete delays as is explained below. We prove that, for sufficiently small initial data, solutions decay to zero exponentially.

The local existence and existence of equilibria is standard (see, the Gronwall-type Lemma 1 below) and the global existence follows from the estimation in our theorem below. However, the uniqueness of the equilibrium is not trivial. Here, as we are concerned with the convergence to zero rather than stability of equilibrium, the uniqueness of equilibrium is put aside.

The next section contains the statement and proof of our result as well as a crucial lemma we will be using.

2. Exponential Convergence

In this section it is proved that solutions converge to zero in an exponential manner when the activation functions are (or bounded by) continuous nondecreasing and unbounded functions. To this end we need a lemma due to Bainov and Simeonov [3].

Let and let . We write if is nondecreasing in .

Lemma 1. Let be a positive continuous function in are nonnegative continuous functions for which are nondecreasing in for any fixed are nondecreasing continuous functions in , with for and is a nonnegative continuous functions in . If in , then the inequality implies that where , and is chosen so that the functions are defined for .

For the statement of our theorem we will need the following notation:

Theorem 2. Assume that satisfy for some continuous nondecreasing (and possibly unbounded) functions , in , with for . Assume further that , are continuous functions. If in then, there exists such that where .

Proof. From (1) and our assumption on we see that or where denotes the right Dini derivative. Hence In virtue of (13) we derive and thereafter (see [28]) where Applying Lemma 1 we obtain the existence of such that where , and are as defined in (7)–(9).

Remark 3. To have global existence we need and this is possible when

Remark 4. Assuming that grows up at most polynomially, we see that the rate is exponential.

Remark 5. Note here that our assumptions in the previous remarks involve a smallness condition on the initial data.

3. Applications

Using Kirchhoff's law, Hopfield demonstrated that electrical circuits could behave as a small Neural Network. His original system has the form: where : Capacity, : Resistance, : Bias (external action on the neuron), : Input (voltage) of the th neuron, : Output of the neuron, : The coupling constants of the neuron with the th neuron, and : Activation functions.

are called elements of the weight matrix or connection matrix. This matrix describes the strength of connection between neurons. The expression is sometimes called the feedback factor.

The functions are nonlinear functions characterizing the response of the th neuron to changes in its state. Typical activation functions are the “Step function”, the “Sign function”, the “Gaussian” function, the “Hyperbolic function”, and the “Exponential type function”. However, it has been established that many other activation functions arise in practice which are not of these forms. Therefore there is a need to enlarge these classes of functions to more general ones.

In Neural Network Theory researchers are rather interested in designing models which are globally asymptotically stable. That is, the models must have a unique equilibrium which attracts all the solutions. Of course the rate of convergence is extremely important and it is preferable to have an exponential convergence rate. In the present work (for the case of variable coefficients) we prove that if solutions start close enough to zero then they will be attracted by zero. Our theorem shows that solutions remain bounded by as long as defined as a bound for the interval of existence of the 's (see (8)). In Remark 3 we gave a sufficient condition ensuring the existence of the 's for all time. That is conditions for which . It follows then that, under these conditions, the states actually converge to zero as goes to infinity with an exponential rate in case does not grow too fast and as .

The example below represents a possible practical situation for which our argument applies. Again we establish a sufficient explicit condition leading to exponential convergence to zero provided that the initial data are small enough.

Example 6. Consider the special (but common) functions . The order means . Clearly, in this case , , and for The value will be the largest value of for which for all . As we are interested in the long time behavior of solutions it is necessary that these conditions hold for all . Our theorem then implies that solutions are bounded by the expression which provides us with an exponential decay under some fairly reasonable assumptions.

3.1. Discrete Delays

The case where we have discrete delays in the activation functions, that is, where are different finite delays, can be treated similarly. We use the following functional to get rid of the delayed terms and replace them by terms without delays.

Acknowledgment

The author is grateful for the financial support and the facilities provided by King Fahd University of Petroleum and Minerals through Grant no. IN111052.