Advances in Artificial Neural Systems

Volume 2015, Article ID 931379, 16 pages

http://dx.doi.org/10.1155/2015/931379

## Stochastic Search Algorithms for Identification, Optimization, and Training of Artificial Neural Networks

Faculty of Management, 21000 Novi Sad, Serbia

Received 6 July 2014; Revised 19 November 2014; Accepted 19 November 2014

Academic Editor: Ozgur Kisi

Copyright © 2015 Kostantin P. Nikolic. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

This paper presents certain stochastic search algorithms (SSA) suitable for effective identification, optimization, and training of artificial neural networks (ANN). The modified algorithm of nonlinear stochastic search (MN-SDS) has been introduced by the author. Its basic objectives are to improve convergence property of the source defined nonlinear stochastic search (N-SDS) method as per Professor Rastrigin. Having in mind vast range of possible algorithms and procedures a so-called method of stochastic direct search (SDS) has been practiced (in the literature is called stochastic local search-SLS). The MN-SDS convergence property is rather advancing over N-SDS; namely it has even better convergence over range of gradient procedures of optimization. The SDS, that is, SLS, has not been practiced enough in the process of identification, optimization, and training of ANN. Their efficiency in some cases of pure nonlinear systems makes them suitable for optimization and training of ANN. The presented examples illustrate only partially operatively end efficiency of SDS, that is, MN-SDS. For comparative method backpropagation error (BPE) method was used.

#### 1. Introduction

The main target of this paper is a presentation of a specific option of direct SS and its application in identification and optimisation of linear and nonlinear objects or processes. The method of stochastic search was introduced by Ashby [1] related to gomeostat. Till 60th of last century the said gomeostat of Ashby’s was adopted mostly as philosophic concept in cybernetics trying to explain the question of stability of rather complex systems having impacts of stochastic nature [2].

The stochastic direct search (SDS) had not been noticed as advanced concurrent option for quite a long time. The researches and developments works of Professor Rastrigin and his associates promoted the SS to be competing method for solving various problems of identification and optimization of complex systems [3].

It has been shown that SDS algorithms besides being competing are even advancing over well-known methods. Parameter for comparing is a property of convergence during solving the set task. For comparing purposes gradient methods were used in reference [4]. The SDS method showed remarkable advance. For systems with noise certain numerical options offer the method of stochastic approximation (MSA) [5]. In some cases procedures of SDS are more efficient than MSA [6].

During the last 20 years, vast interests have been shown for advanced SDS, especially on the case where classical deterministic techniques do not apply. Direct SS algorithms are one part of the SSA family. The important subjects of random search were being made: theorems of global optimization, convergence theorems, and applications on complex control systems [7–9].

The author has been using SDS algorithms (in several of his published papers) regarding identification of complex control systems [10], as well as synthesis and training of artificial neural networks [11–13].

Through experience in application of certain SDS basic definition the author was motivated to introduce the so-called modified nonlinear SDS (MN-SDS) applicable as numerical method for identification and optimization of substantial nonlinear systems. The main reason is rather slow convergence of N-SDS of basic definition and this deficiency has been overcome.

The application of SDS is efficient for both determined and stochastic description of systems.

The SDS algorithm is characterized by introduction of random variables. An applicable option is generator of random numbers [14, 15].

The previously said is enhanced by well-developed modern computer hardware and software providing suitable ambient conditions for creation and implementation of SDS methods and procedures.

#### 2. Method and Materials

##### 2.1. Definition of SDS Method

The solving of theoretical and/or practical problems usually requests firstly an identification task followed by final stage, that is, a system optimization. The analysis and synthesis of systems always consider the previously said [16, 17].

Methods of SDS are ones of competing options for numerical procedures providing solution for identification and optimization of complex control systems [18], but so ANN. Let us start with an internal system description in general form [19]:where and are nonlinear vector functions; and are vector functions of constrains (variables and parameters); is system state vector, is control vector, and is vector of disturbance; are parameters describing the system structure such as constants, matrices, and vectors; is real time; is noise usually added in (2).

A parameters identification of the above system anticipates certain measurements of the system variables observing the criteria function:

The criteria function in (5) is without involvement of constrains and ; in case that constrains are not possible to avoid, is introduced [20]:where and are Langrage multiplicators and , a [20].

When and are rather large and tend to , then both and tend to the same value for variables , that is, corresponding optimal parameters.

Further for the purpose of simplicity of this presentation a function from (5) is to be used and to be the function of one vector variable , so *.*

Methods of optimization start with iterative form where from current state system transfers into by the following rule:

So, is a function stepping into new state where is a step and is vector function of guiding search. For iterative optimization by using gradient method [21]:

In case of SDS the relation (7) gets the next form:where the direction of search is function of and random vector (Figure 1).