International Scholarly Research Notices

International Scholarly Research Notices / 2011 / Article

Research Article | Open Access

Volume 2011 |Article ID 145801 | 12 pages | https://doi.org/10.5402/2011/145801

Lyapunov Stability Analysis of Gradient Descent-Learning Algorithm in Network Training

Academic Editor: L. Simoni
Received17 Mar 2011
Accepted13 May 2011
Published05 Jul 2011

Abstract

The Lyapunov stability theorem is applied to guarantee the convergence and stability of the learning algorithm for several networks. Gradient descent learning algorithm and its developed algorithms are one of the most useful learning algorithms in developing the networks. To guarantee the stability and convergence of the learning process, the upper bound of the learning rates should be investigated. Here, the Lyapunov stability theorem was developed and applied to several networks in order to guaranty the stability of the learning algorithm.

1. Introduction

Science has evolved from an attempt to understand and predict the behavior of the universe and the systems within it. Much of this owes to the development of suitable models, which agree with the observations. These models are either in a symbolic form which the humans use or in mathematical form that are found from physical laws. Most systems are causal, which can be categorized as either static, where the output depends on the current inputs, or dynamic, where the output depends on not only the current inputs but also past inputs and outputs. Many systems also possess unobservable inputs, which cannot be measured, but affect the system’s output, that is, time series systems. These inputs are known as disturbances and aggravate the modeling process.

To cope with the complexity of dynamic systems, there have been significant developments in the field of artificial neural network during last three decades which have been applied for identification and modeling [15]. One major issue that instigates for proposing these different types of networks is to predict the dynamic behavior of many complex systems existing in nature. ANN is a powerful method in approximating a nonlinear system and mapping between input and output data [1]. Recently, wavelet neural networks (WNNs) have been introduced [610]. Such types of networks employ wavelets as the activation function in a hidden layer. Because of the ability of the localized analysis of wavelets collectively in their frequency and time domains and the learning ability of ANN, the WNN prompts a superior system model for complex and seismic applications. The majority of the applications of wavelet function are limited to a small dimension [11] although WNN can handle large-dimension problems as well [6]. Due to the dynamic behavior of recurrent network, they are suitable in dealing with the modeling of dynamic systems as compared with the static behavior of feed-forward network [1219]. It has already been shown that recurrent networks are less sensitive to noise with relatively smaller network size and simpler structure. Their long-term prediction property makes them more powerful in dealing with dynamic systems. Recurrent networks are less sensitive to noise because the recurrent network could recognize and generate periodic waves in spite of the existence of a large amount of noise. This means that the network is able to regenerate the original periodic waves in the process of learning the teachers' signals with noises [2]. For unknown dynamic systems, the recurrent network results in a smaller-sized network as compared with the feed-forward network [12, 20]. For the time-series modeling, it generates a simpler structure [1523] and gives long-term predictions [22, 24]. The recurrent network for system modeling learns and memorizes information in terms of embedded weights [21].

Different methods have been introduced for learning the parameters onnetwork based of the gradient descent. All learning methods like backpropagation-through-time [16, 17] or real-time recurrent learning algorithm [18] can be applied in order to adjust parameters of the feed-forward or recurrent networks. In [19], the quasi-Newton method was applied to improve the rate of convergence. In [9, 23], using the Lyapunov stability theorem, a mathematical way was introduced for calculating the upper bound of the learning rate for recurrent and feed-forward wavelet neural network based on the network parameters. Here, the Lyapunov stability theorem is developed and applied to several networks, and the learning procedure of the proposed networks is considered.

2. Methodology

2.1. Gradient-Descent Algorithm

The Gradient-descent (GD) learning can be achieved by minimizing the performance index 𝐽 as follows:1𝐽=2𝑃𝑦2𝑟𝑃𝑝=1𝑌(𝑝)𝑌(𝑝)2,(2.1) where 𝑦𝑟=(max𝑃𝑝=1𝑌(𝑝)min𝑃𝑝=1𝑌(𝑝)), 𝑌 is the output of the known network, 𝑌 is the actual data, and 𝑃 is the number of dataset. The reason for using a normalized mean square error is that it provides a universal platform for modeling evaluation irrespective of the application and target value specification while selecting an input to the model.

In the batch-learning scheme employing the 𝑃-data set, achange in any parameter is covered by the following equation:Δ𝜐(𝑞)=𝑃𝑝=1Δ𝑝𝜐(𝑞),(2.2) and the parametric update equation is𝜐(𝑞+1)=𝜐(𝑞)+𝜕𝐽𝜕𝜐.(2.3)

2.2. Lyapunov Method in Analysis of Stability

Consider a dynamic system, which satisfies𝑡̇𝑥=𝑓(𝑥,𝑡),𝑥0=𝑥0,𝑥𝑅.(2.4)

The equilibrium point 𝑥=0 is stable (in the sense of Lyapunov) at 𝑡=𝑡0 if for any 𝜀>0 there exists a 𝛿(𝑡0,𝜀)>0 such that 𝑥𝑡0<𝛿𝑥(𝑡)<𝜀,𝑡𝑡0.(2.5)

Lyapunov Stability Theorem
Let 𝑉(𝑥,𝑡) be a nonnegative function with the derivative ̇𝑉 along the trajectories of the system. Then(i)The origin of the system is locally stable (in the sense of Lyapunov) if 𝑉(𝑥,𝑡) is locally positive definite and ̇𝑉(𝑥,𝑡)0 is locally in 𝑥 and for all 𝑡;(ii)The origin of the system is globally uniformly asymptotically stable if 𝑉(𝑥,𝑡) is positive definite and excrescent and ̇𝑉(𝑥,𝑡) is positive definite.
To approve stability analysis of the networks based on GD learning algorithm, we can define discreet function as 1𝑉(𝑘)=𝐸(𝑘)=2[]𝑒(𝑘)2.(2.6) Change of Lyapunov function is 1Δ𝑉(𝑘)=𝑉(𝑘+1)𝑉(𝑘)=2𝑒2(𝑘+1)𝑒2(𝑘).(2.7)from𝑒(𝑘+1)=𝑒(𝑘)+Δ𝑒(𝑘)𝑒2(𝑘+1)=𝑒2(𝑘)+Δ2𝑒(𝑘)+2𝑒(𝑘)Δ𝑒(𝑘).(2.8) Then 1Δ𝑉(𝑘)=Δ𝑒(𝑘)𝑒(𝑘)+2Δ𝑒(𝑘).(2.9) Difference of error is Δ𝑒(𝑘)=𝑒(𝑘+1)𝑒(𝑘)𝜕𝑒(𝑘)𝜕𝜐𝑇Δ𝜐,(2.10) where 𝜐 is the learning parameter and 𝑒(𝑘)=̂𝑦(𝑘)𝑦(𝑘) is error between output of plant and present output of network Δ𝜐=𝜂𝜕𝐽𝜕𝜐.(2.11)
By using (2.10) and (2.1) and putting them in (2.3), Δ𝑉(𝑘)=𝜕𝑒(𝑘)𝜕𝜐𝑇1Δ𝜐𝑒(𝑘)+2𝜕𝑒(𝑘)𝜕𝜐𝑇,Δ𝜐Δ𝑉(𝑘)=𝜕𝑒(𝑘)𝜕𝜐𝑇𝜂𝜕𝐸(𝑘)1𝜕𝜐𝑒(𝑘)+2𝜕𝑒(𝑘)𝜕𝜐𝑇𝜂𝜕𝐸(𝑘),𝜕𝜐Δ𝑉(𝑘)=𝜕𝑒(𝑘)𝜕𝜐𝑇1(𝜂)𝑃𝑦2𝑟𝑒(𝑘)𝜕̂𝑦(𝑘)1𝜕𝜐𝑒(𝑘)+2𝜕𝑒(𝑘)𝜕𝜐𝑇1(𝜂)𝑃𝑦2𝑟𝑒(𝑘)𝜕̂𝑦(𝑘),𝜕𝜐Δ𝑉(𝑘)=𝑒2(𝑘)𝜕̂𝑦(𝑘)𝜕𝜐𝑇1𝜂𝑃𝑦2𝑟𝜕̂𝑦(𝑘)+1𝜕𝜐2𝜕̂𝑦(𝑘)𝜕𝜐𝑇𝜕̂𝑦(𝑘)𝜕𝜐𝑇𝜂21𝑃𝑦2𝑟2𝜕̂𝑦(𝑘)𝜕𝜐2Δ𝑉(𝑘)=𝑒21(𝑘)2𝜂𝑃𝑦2𝑟𝜕̂𝑦(𝑘)𝜕𝜐2𝜂2𝑃𝑦2𝑟𝜕̂𝑦(𝑘)𝜕𝜐2,(2.12) where 𝑦𝑟=(max𝑃𝑝=1𝑦(𝑝)min𝑃𝑝=1𝑦(𝑝)).
Therefore Δ𝑉(𝑘)=𝜆𝑒2(𝑘),(2.13) where 𝜆=(1/2)(𝜂/(𝑃𝑦2𝑟))(𝜕̂𝑦(𝑘)/𝜕𝜐)2{2(𝜂/(𝑃𝑦2𝑟))(𝜕̂𝑦(𝑘)/𝜕𝜐)2}.
From the Lyapunov stability theorem, the stability is guaranteed if 𝑉(𝑘) is positive and 𝑉(𝑘) is negative. From (2.6), 𝑉(𝑘) is already positive. The condition of stability depends on 𝑉(𝑘) being negative. Therefore, 𝜆>0 is considered for all models.
Because (1/2)(𝜂/(𝑃𝑦2𝑟))(𝜕̂𝑦(𝑘)/𝜕𝜐)2>0, then the convergence condition is limited to 𝜂2𝑃𝑦2𝑟𝜕̂𝑦(𝑘)𝜕𝜐2𝜂>0𝑃𝑦2𝑟𝜕̂𝑦(𝑘)𝜕𝜐2<2𝜂<2𝑃𝑦2𝑟(𝜕̂𝑦(𝑘)/𝜕𝜐)2.(2.14) The maximum learning rate 𝜂 changes in a fixed range. Since 2𝑃𝑦2𝑟 does not depend on the model, the value of 𝜂Max guarantees that the convergence can be found by minimizing the term of |𝜕̂𝑦(𝑘)/𝜕𝜐𝐼|. Therefore, 0<𝜂<𝜂Max,(2.15) where 𝜂Max=(2𝑃𝑦2𝑟)/Max(𝜕̂𝑦(𝑘)/𝜕𝜐)2.

3. Experimental Results

In this section, the proposed stability analysis is applied for some networks. The selected networks are neurofuzzy (ANFIA) [25, 26], Wavelet neurofuzzy, and recurrent wavelet network.

3.1. Example 1: Convergence Theorems of the TSK Neurofuzzy Model

TSKmodel has a linear or nonlinear relationship of inputs 𝑤𝑚(𝑋) in the output space. The rules of TSK model are in the following way:𝑅𝑚if𝐱is𝐀mthen𝑦is𝑤𝑚(𝑋).(3.1) A linear form of 𝑤𝑚(𝑋) in (3.1) is as follows:𝑤𝑚(𝑋)=𝑤𝑚0+𝑤𝑚1𝑥1++𝑤𝑚𝑛𝑥𝑛.(3.2) By taking the Gaussian membership function and an equal number of fuzzy sets to the rules with respect to the inputs, the firing strength of rules (3.1) can be written as𝜇𝐴𝑚(𝐱)=𝑛𝑖=1𝑥exp𝑖𝑥𝑚𝑖𝜎𝑚𝑖2,(3.3) where 𝑥𝑚𝑖 and 𝜎𝑚𝑖 are the center and standard deviation of the Gaussian membership functions, respectively. By applying the T-norm (product operator) of the membership functions of the premise parts of the rule and the weighted average gravity method for de-fuzzification, the output of the TSK model can be defined as𝑌=𝑀𝑚=1𝜇𝐴𝑚(𝐱)𝑤𝑚(𝐱)𝑀𝑚=1𝜇𝐴𝑚(𝐱).(3.4)

Theorem 3.1. The asymptotic learning convergence of TSK neurofuzzy is guaranteed if the learning rate for different learning parameters follows the upper bound as will be mentioned below: 0<𝜂𝑤<2𝑃y2r,0<𝜂𝜎<2𝑃𝑦2𝑟max𝑚||||𝑤(X)22/𝜎3min2,0<𝜂𝑥<2𝑃𝑦2𝑟max𝑚||||𝑤(X)22/𝜎2min2.(3.5)

Proof. In equation (2.15) for neurofuzzy models can be written as 0<𝜂𝜐<2𝑃𝑦2𝑟||𝜕𝑌NF||/𝜕𝜐2max.(3.6) Because 𝛽𝑚=𝜇𝐴𝑚(𝐗)/𝑀𝑚=1𝜇𝐴𝑚(𝐗)1 for all 𝑚 and since local models have same variables, that is, 𝐗, therefore, from (3.7), (3.5) easily can be derived 𝜕𝑌NF𝜕𝑤𝑚0=𝛽𝑚,𝜕𝑌NF𝜕𝑤𝑚𝑖=𝑥𝑖𝛽𝑚,𝜕𝑌NF𝜕𝑥𝑚𝑖=𝑤𝑚𝐗𝛽𝑚𝜇𝐴𝑚1𝛽𝑚𝑥2𝑖𝑥𝑚𝑖𝜎2𝑚𝑖,𝜕𝑌NF𝜕𝜎𝑚𝑖=𝑤𝑚𝐗𝛽𝑚𝜇𝐴𝑚1𝛽𝑚𝑥2𝑖𝑥𝑚𝑖2𝜎3𝑚𝑖.(3.7)

3.2. Example 2: Convergence Theorems of Recurrent Wavelet Neuron Models

Each neuron model in the proposed recurrent neuron models is summation or multiplication of Sigmoid Activation Function (SAF) and Wavelet Activation Function (WAF) as shown in Figure 1. Morlet wavelet function is considered in the recurrent models. In the series of developing different recurrent networks and neuron models, the proposed neurons’ model is used in a one-hidden-layer feed-forward neural network as shown in Figure 2.

The output of feed-forward network is given in   the following equation:𝑌WNN=𝐿𝑙=1𝑊𝑙𝑦𝑙,(3.8) where 𝑦𝑙 is the output of S-W neurons, 𝑊𝑙 is the weights between hidden neuron and output neurons, and 𝐿 is the number of hidden neuron,𝑦𝑗(𝑘)=𝑦𝜃𝑗(𝑘)+𝑦𝜓𝑗(𝑘).(3.9)

The functions 𝑦𝜃𝑗 and 𝑦𝜓𝑗 are output of SAF and WAF for 𝑗th S-W neuron, in the hidden layer, respectively. The functions 𝑦𝜃𝑗 and 𝑦𝜓𝑗 are expressed as follow.𝑦𝜃𝑗(𝑘)=𝜃𝑛𝑖=1𝐶𝑗𝑆𝑖𝑥𝑖,𝑦(𝑘)𝜓𝑗(𝑘)=𝜓𝑛𝑖=1𝐶𝑗𝑊𝑖𝑥𝑖.(𝑘)(3.10)𝑥𝑖 is 𝑖th input. 𝐶𝑆 and 𝐶𝑊 are weights to input signal for SAF and WAF, in each hidden neuron, respectively.

To prove convergence of the recurrent networks, these facts are needed:Fact 1: let 𝑔(𝑦)=𝑦𝑒(𝑦2). Then |𝑔(𝑦)|<1,forall𝑦.Fact 2: let 𝑓(𝑦)=𝑦2𝑒(𝑦2). Then |𝑓(𝑦)|<1,forall𝑦.Fact 3: let 𝜃(𝑦)=1/(1+𝑒𝑦)be a sigmoid function. Then |𝜃(𝑦)|<1,forall𝑦Fact 4: let 𝜓𝑎,𝑏(𝑦)=𝑒((𝑦𝑏)/𝑎)2cos(5((𝑦𝑏)/𝑎)) be a Morlet wavelet function. Then |𝜓𝑎,𝑏(𝑦)|<1,forall𝑦,𝑎,𝑏.

(a) Summation Sigmoid-Recurrent Wavelet
Suppose 𝑍=𝑛𝑖=1𝐶𝑗𝑆𝑖𝑥𝑖(𝑘) and 𝑆=𝑛𝑖=1𝐶𝑗𝑊𝑖𝑥𝑖(𝑘)+𝑄𝑗𝑊𝑦𝑗𝜓(𝑘1).
From the facts 3 and 4: For parameter 𝑊 in all models 𝜕̂𝑦𝜕𝑊𝑗=𝑦𝑗<||𝑦𝑗𝜓+𝑦𝑗𝜃||<1+1=2.(3.11) Therefore 0<𝜂𝑊<(2𝑃𝑦2𝑟)/22=(𝑃𝑦2𝑟)/2.
Differential of output of the model for another learning parameter is 𝜕̂𝑦(𝑘)𝜕𝐶𝑗𝑊𝑖=𝑥𝑖(𝑘)𝑊𝑗𝜓𝑛𝑖=1𝐶𝑗𝑊𝑖𝑥𝑖(𝑘)+𝑄𝑗𝑊𝑦𝑗𝜓|||(𝑘1)<112𝑎𝑆𝑏𝑎𝑒((𝑆𝑏)/𝑎)25cos𝑆𝑏𝑎𝑒((𝑆𝑏)/𝑎)25𝑎5sin𝑆𝑏𝑎|||<2𝑎min511+𝑎min1<7.(3.12) Therefore, 0<𝜂𝐶𝑊<(2𝑃𝑦2𝑟)/72=(2𝑃𝑦2𝑟)/49𝜕̂𝑦(𝑘)𝜕𝐶𝑗𝑆𝑖=𝑥𝑖(𝑘)𝑊𝑖𝜃𝑛𝑖=1𝐶𝑗𝑆𝑖𝑥𝑖(𝑘)<11𝜃(𝑧)(1𝜃(𝑧))<11=1.(3.13) Therefore 0<𝜂𝐶𝑆<(2𝑃𝑦2𝑟)/12=2𝑃𝑦2𝑟𝜕̂𝑦(𝑘)𝜕𝑄𝑗𝑊=𝑊𝑗𝑦𝑗𝜓(𝑘1)𝜓𝑛𝑖=1𝐶𝑗𝑊𝑖𝑥𝑖(𝑘)+𝑄𝑗𝑊𝑦𝑗𝜓|||(𝑘1)<112𝑎𝑆𝑏𝑎𝑒((𝑆𝑏)/𝑎)25cos𝑆𝑏𝑎𝑒((𝑆𝑏)/𝑎)25𝑎5sin𝑆𝑏𝑎|||<2𝑎min511+𝑎min1<7.(3.14) Therefore, 0<𝜂𝑄𝑊<(2𝑃𝑦2𝑟)/72=(2𝑃𝑦2𝑟)/49.

(b) Multiplication Sigmoid-Recurrent Wavelet
From facts 3 and 4 suppose 𝑍=𝑛𝑖=1𝐶𝑗𝑆𝑖𝑥𝑖(𝑘) and 𝑆=𝑛𝑖=1𝐶𝑗𝑊𝑖𝑥𝑖(𝑘)+𝑄𝑗𝑊𝑦𝑗𝜓(𝑘1).
For parameter 𝑊 in all networks: 𝜕̂𝑦𝜕𝑊𝑗=𝑦𝑗=𝑦𝑗𝜓𝑦𝑗𝜃<11<1.(3.15) Therefore, 0<𝜂𝑊<(2𝑃𝑦2𝑟)/1<2𝑃𝑦2𝑟𝜕̂𝑦(𝑘)𝜕𝐶𝑗𝑊𝑖=𝑥𝑖(𝑘)𝑊𝑗𝜃𝑛𝑖=1𝐶𝑗𝑆𝑖𝑥𝑖(𝑘)𝜓𝑛𝑖=1𝐶𝑗𝑊𝑖𝑥𝑖(𝑘)+𝑄𝑗𝑊𝑦𝑗𝜓|||(𝑘1)<1112𝑎𝑆𝑏𝑎𝑒((𝑆𝑏)/𝑎)25cos𝑆𝑏𝑎𝑒((𝑆𝑏)/𝑎)25𝑎5sin𝑆𝑏𝑎|||<2𝑎min511+𝑎min1<7.(3.16) Therefore, 0<𝜂𝐶𝑊<(2𝑃𝑦2𝑟)/(7)2=(2𝑃𝑦2𝑟)/49𝜕̂𝑦(𝑘)𝜕𝐶𝑗𝑆𝑖=𝑥𝑖(𝑘)𝑊𝑗𝜃𝑛𝑖=1𝐶𝑗𝑆𝑖𝑥𝑖(𝑘)𝜓𝑛𝑖=1𝐶𝑗𝑊𝑖𝑥𝑖(𝑘)+𝑄𝑗𝑊𝑦𝑗𝜓(𝑘1)<11𝜃(𝑍)(1𝜃(𝑍))1<11<1.(3.17) Therefore, 0<𝜂𝐶𝑆<(2𝑃𝑦2𝑟)/(1)2=2𝑃𝑦2𝑟𝜕̂𝑦(𝑘)𝜕𝑄𝑗𝑊=𝑊𝑗𝑦𝑗𝜓(𝑘1)𝜃𝑛𝑖=1𝐶𝑗𝑆𝑖𝑥𝑖(𝑘)𝜓𝑛𝑖=1𝐶𝑗𝑊𝑖𝑥𝑖(𝑘)+𝑄𝑗𝑊𝑦𝑗𝜓|||(𝑘1)<1112𝑎𝑆𝑏𝑎𝑒((𝑆𝑏)/𝑎)25cos𝑆𝑏𝑎𝑒((𝑆𝑏)/𝑎)25𝑎5sin𝑆𝑏𝑎|||<2𝑎min511+𝑎min1<7.(3.18) Therefore, 0<𝜂𝑄𝑊<(2𝑃𝑦2𝑟)/(7)2=(2𝑃𝑦2𝑟)/49

3.3. Example 3: Convergence Theorems of the Wavelet Nuro-Fuzzy (WNF) Model

The consequent part of each fuzzy rule corresponds to a sub-WNN consisting of wavelet with the specified dilation value, where, in the TSK fuzzy model, a linear function of inputs is used while 𝑤𝑚𝑌(𝑋)=WNN𝑚. Figure 1 shows the proposed WNN model which uses a combination of sigmoid and wavelet activation functions as a hidden neuron (Figure 2 without recurrent part) in the consequent part of each fuzzy rule.

Theorem 3.2. The asymptotic learning convergence is guaranteed if the learning rate for different learning parameters follows the upper bound as will be mentioned below: 0<𝜂𝜎<2𝑃𝑦2𝑟||𝑌WNN||2max2/𝜎3min2,0<𝜂𝑥<2𝑃𝑦2𝑟||𝑌WNN||2max2/𝜎2min2,0<𝜂𝑤<2𝑃𝑦2𝑟||𝜕𝑌WNN||/𝜕𝑤2max,0<𝜂𝐶𝑆<2𝑃𝑦2𝑟||𝜕𝑌WNN/𝜕𝐶𝑆||2max,0<𝜂𝐶𝑊<2𝑃𝑦2𝑟||𝜕𝑌WNN/𝜕𝐶𝑊||2max,(3.19)

where 𝜂𝑤, 𝜂𝐶𝑁, or 𝜂𝐶𝑊 and 𝜂𝜎 or 𝜂𝑥 are the parameters’ learning rates of the consequent and the premise parts of the fuzzy rules. 𝐶𝑆 and 𝐶𝑊 are weights to inputs, signal for sigmoid and wavelet activation functions of local WNNs, in each hidden neuron, respectively. 𝑥𝑚 and 𝜎𝑚 are the center and standard deviation of the Gaussian membership functions of rule number m in WNF model, respectively.

Proof. In equation (2.15) for WNF models can be written as 0<𝜂𝜐<2𝑃𝑦2𝑟||𝜕𝑌WNF||/𝜕𝜐2max,𝜕𝑌WNF𝜕𝑤=𝛽𝑚𝜕𝑌WNN𝑚,𝜕𝑌𝜕𝑤WNF𝜕𝐶𝑁=𝛽𝑚𝜕𝑌WNN𝑚𝜕𝐶𝑁,𝜕𝑌WNF𝜕𝐶𝑊=𝛽𝑚𝜕𝑌WNN𝑚𝜕𝐶𝑊.(3.20) Because 𝛽𝑚=𝜇𝐴𝑚(𝐗)/𝑀𝑚=1𝜇𝐴𝑚(𝐗)1 for all 𝑚, therefore (3.13) to (3.15) are easily derived.
From (2.15) and (3.4) for parameters 𝜎 or 𝑥,there is 𝜕𝑌WNF=𝑌𝜕𝜎WNN𝑚𝛽𝑚𝜇𝐴𝑚1𝛽𝑚𝑥2𝑖𝑥𝑚𝑖2𝜎3𝑚𝑖=𝑌WNN𝑚1𝛽𝑚𝑀𝑚=1𝜇𝐴𝑚𝑥2𝑖𝑥𝑚𝑖2𝜎3𝑚𝑖,𝜕𝑌WNF𝜕𝑥=𝑌WNN𝑚𝛽𝑚𝜇𝐴𝑚1𝛽𝑚𝑥2𝑖𝑥𝑚𝑖𝜎2𝑚𝑖=𝑌WNN𝑚1𝛽𝑚𝑀𝑚=1𝜇𝐴𝑚𝑥2𝑖𝑥𝑚𝑖𝜎2𝑚𝑖(3.21) and therefore (3.19) arederived.

4. Conclusion

In this paper, a developed Lyapunov stability theorem was applied to guarantee the convergence of the gradient-descent learning algorithm in network training. The experimental examples showed that the upper bound of the learning parameter could be easily considered using this theorem. So, an adaptive learning algorithm can guaranty the fast and stable learning procedure.

References

  1. K. S. Narendra and K. Parthasarathy, “Identification and control of dynamical systems using neural networks,” IEEE Transactions on Neural Networks, vol. 1, no. 1, pp. 4–27, 1990. View at: Publisher Site | Google Scholar
  2. S. Z. Qin, H. T. Su, and T. J. McAvoy, “Comparison of four neural net learning methods for dynamic system identification,” IEEE Transactions on Neural Networks, vol. 3, no. 1, pp. 122–130, 1992. View at: Publisher Site | Google Scholar
  3. T. Yabuta and T. Yamada, “Learning control using neural networks,” in Proceedings of the IEEE International Conference on Robotics and Automation, (ICRA '91), pp. 740–745, Sacramento, Calif, USA, April 1991. View at: Google Scholar
  4. P. Frasconi, M. Gori, and G. Soda, “Local feedback multilayered networks,” Neural Computation, vol. 7, no. 1, pp. 120–130, 1992. View at: Google Scholar
  5. J. C. Patra, R. N. Pal, B. N. Chatterji, and G. Panda, “Identification of nonlinear dynamic systems using functional link artificial neural networks,” IEEE Transactions on Systems, Man, and Cybernetics, Part B, vol. 29, no. 2, pp. 254–262, 1999. View at: Publisher Site | Google Scholar
  6. Q. Zhang and A. Benveniste, “Wavelet networks,” IEEE Transactions on Neural Networks, vol. 3, no. 6, pp. 889–898, 1992. View at: Publisher Site | Google Scholar
  7. J. Zhang, G. G. Walter, Y. Miao, and W. Lee, “Wavelet neural networks for function learning,” IEEE Transactions on Signal Processing, vol. 43, no. 6, pp. 1485–1497, 1995. View at: Publisher Site | Google Scholar
  8. T. I. Boubez and R. L. Peskin, “Wavelet neural networks and receptive field partitioning,” in Proceedings of the IEEE International Conference on Neural Networks, pp. 1544–1549, San Francisco, Calif, USA, March 1993. View at: Google Scholar
  9. A. Banakar and M. F. Azeem, “Artificial wavelet neural network and its application in neuro-fuzzy models,” Applied Soft Computing, vol. 8, no. 4, pp. 1463–1485, 2008. View at: Publisher Site | Google Scholar
  10. Q. Zhang, “Using wavelet network in nonparametric estimation,” IEEE Transactions on Neural Networks, vol. 8, no. 2, pp. 227–236, 1997. View at: Publisher Site | Google Scholar
  11. A. Benveniste, B. Juditsky, B. Delyon, Q. Zhang, and P. Y. Glorennec, “Wavelets in identification,” in Proceedings of the 10th IFAC Symposium on System Identification, (SYSID '94), Copenhagen, Denmark, July 1994. View at: Publisher Site | Google Scholar
  12. X. D. Li, J. K. L. Ho, and T. W. S. Chow, “Approximation of dynamical time-variant systems by continuous-time recurrent neural networks,” IEEE Transactions on Circuits and Systems, vol. 52, no. 10, pp. 656–660, 2005. View at: Google Scholar
  13. B. Srinivasan, U. R. Prasad, and N. J. Rao, “Back propagation through adjoints for the identification of nonlinear dynamic systems using recurrent neural models,” IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 213–228, 1994. View at: Publisher Site | Google Scholar
  14. P. Frasconi and M. Gori, “Computational capabilities of local-feedback recurrent networks acting as finite-state machines,” IEEE Transactions on Neural Networks, vol. 7, no. 6, pp. 1520–1525, 1996. View at: Google Scholar
  15. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” Neural Computation, vol. 3, pp. 79–87, 1991. View at: Publisher Site | Google Scholar
  16. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing I, D. E. Rumelhart and J. L. McClelland, Eds., pp. 675–695, MIT Press, Cambridge, UK, 1986. View at: Google Scholar
  17. P. Werbos, “Generalization of backpropagation with application to a recurrent gas Markov model,” Neural Networks, vol. 1, pp. 339–356, 1988. View at: Google Scholar
  18. R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent networks,” Neural Networks, vol. 1, pp. 270–280, 1989. View at: Google Scholar
  19. R. J. Williams and D. Zipser, “Mechanical system modeling using recurrent neural networks via quasi-Newton learning methods,” Applied Mathematical Modeling, vol. 19, no. 7, pp. 421–428, 1995. View at: Publisher Site | Google Scholar
  20. C.-F. Juang, “A TSK-type recurrent fuzzy network for dynamic systems processing by neural network and genetic algorithms,” IEEE Transactions on Fuzzy Systems, vol. 10, no. 2, pp. 155–170, 2002. View at: Publisher Site | Google Scholar
  21. C.-H. Lee and C.-C. Teng, “Identification and control of dynamic systems using recurrent fuzzy neural networks,” IEEE Transactions on Fuzzy Systems, vol. 8, no. 4, pp. 349–366, 2000. View at: Publisher Site | Google Scholar
  22. P. A. Mastorocostas and J. B. Theocharis, “A recurrent fuzzy-neural model for dynamic system identification,” IEEE Transactions on Systems, Man, and Cybernetics, Part B, vol. 32, no. 2, pp. 176–190, 2002. View at: Publisher Site | Google Scholar
  23. S. J. Yoo, Y. H. Choi, and J. B. Park, “Generalized predictive control based on self-recurrent wavelet neural network for stable path tracking of mobile robots: adaptive learning rates approach,” IEEE Transactions on Circuits and Systems, vol. 53, no. 6, pp. 1381–1394, 2006. View at: Publisher Site | Google Scholar | MathSciNet
  24. T. G. Barbounis, J. B. Theocharis, M. C. Alexiadis, and P. S. Dokopoulos, “Long-term wind speed and power forecasting using local recurrent neural network models,” IEEE Transactions on Energy Conversion, vol. 21, no. 1, pp. 273–284, 2006. View at: Publisher Site | Google Scholar
  25. T. Takagi and M. Sugeno, “Fuzzy identification of systems and its applications to modeling and control,” IEEE Transactions on Systems, Man and Cybernetics, vol. 15, no. 1, pp. 116–132, 1985. View at: Google Scholar | Zentralblatt MATH
  26. J. S. R. Jang, “ANFIS: adaptive-network-based fuzzy inference system,” IEEE Transactions on Systems, Man and Cybernetics, vol. 23, no. 3, pp. 665–685, 1993. View at: Publisher Site | Google Scholar

Copyright © 2011 Ahmad Banakar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


More related articles

1587 Views | 485 Downloads | 2 Citations
 PDF  Download Citation  Citation
 Download other formatsMore
 Order printed copiesOrder

Related articles

We are committed to sharing findings related to COVID-19 as quickly and safely as possible. Any author submitting a COVID-19 paper should notify us at help@hindawi.com to ensure their research is fast-tracked and made available on a preprint server as soon as possible. We will be providing unlimited waivers of publication charges for accepted articles related to COVID-19. Sign up here as a reviewer to help fast-track new submissions.