About this Journal Submit a Manuscript Table of Contents
ISRN Artificial Intelligence
Volume 2012 (2012), Article ID 486361, 9 pages
http://dx.doi.org/10.5402/2012/486361
Research Article

An Advanced Conjugate Gradient Training Algorithm Based on a Modified Secant Equation

1Department of Mathematics, University of Patras, 26500 Patras, Greece
2Educational Software Development Laboratory, Department of Mathematics, University of Patras, 26500 Patras, Greece

Received 5 August 2011; Accepted 4 September 2011

Academic Editors: T. Kurita and Z. Liu

Copyright © 2012 Ioannis E. Livieris and Panagiotis Pintelas. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Conjugate gradient methods constitute excellent neural network training methods characterized by their simplicity, numerical efficiency, and their very low memory requirements. In this paper, we propose a conjugate gradient neural network training algorithm which guarantees sufficient descent using any line search, avoiding thereby the usually inefficient restarts. Moreover, it achieves a high-order accuracy in approximating the second-order curvature information of the error surface by utilizing the modified secant condition proposed by Li et al. (2007). Under mild conditions, we establish that the proposed method is globally convergent for general functions under the strong Wolfe conditions. Experimental results provide evidence that our proposed method is preferable and in general superior to the classical conjugate gradient methods and has a potential to significantly enhance the computational efficiency and robustness of the training process.

1. Introduction

Learning systems, such as multilayer feedforward neural networks (FNN), are parallel computational models comprised of densely interconnected, adaptive processing units, characterized by an inherent propensity for learning from experience and also discovering new knowledge. Due to their excellent capability of self-learning and self-adapting, they have been successfully applied in many areas of artificial intelligence [15] and are often found to be more efficient and accurate than other classification techniques [6]. The operation of a FNN is usually based on the following equations: 𝑛𝑒𝑡𝑙𝑗=𝑁𝑙1𝑖=1𝑤𝑙1,𝑙𝑖𝑗𝑦𝑖𝑙1+𝑏𝑙𝑗,𝑦𝑙𝑗=𝑓𝑛𝑒𝑡𝑙𝑗,(1) where 𝑛𝑒𝑡𝑙𝑗 is the sum of its weighted inputs for the 𝑗th node in the 𝑙th layer (𝑗=1,,𝑁𝑙), 𝑤𝑙1,𝑙𝑖𝑗 are the weights from the 𝑖th neuron at the (𝑙1) layer to the 𝑗th neuron at the 𝑙th layer, 𝑏𝑙𝑗 is the bias of the 𝑗th neuron at the 𝑙th layer, 𝑦𝑙𝑖 is the output of the 𝑗th neuron that belongs to the 𝑙th layer, and 𝑓(𝑛𝑒𝑡𝑙𝑗) is the 𝑗th neuron activation function.

The problem of training a neural network is to iteratively adjust its weights, in order to globally minimize a measure of difference between the actual output of the network and the desired output for all examples of the training set [7]. More mathematically, the training process can be formulated as the minimization of the error function 𝐸(𝑤), defined by the sum of square differences between the actual output of the FNN, denoted by 𝑦𝐿𝑗,𝑝 and the desired output, denoted by 𝑡𝑗,𝑝, relative to the appeared output, namely,𝐸(𝑤)=𝑃𝑁𝑝=1𝐿𝑗=1𝑦𝐿𝑗,𝑝𝑡𝑗,𝑝2,(2) where 𝑤𝑛 is the vector network weights and 𝑃 represents the number of patterns used in the training set.

Conjugate gradient methods are probably the most famous iterative methods for efficiently training neural networks due to their simplicity, numerical efficiency, and their very low memory requirements. These methods generate a sequence of weights {𝑤𝑘} using the iterative formula𝑤𝑘+1=𝑤𝑘+𝜂𝑘𝑑𝑘,𝑘=0,1,,(3) where 𝑘 is the current iteration usually called epoch, 𝑤0𝑛 is a given initial point, 𝜂𝑘>0 is the learning rate, and 𝑑𝑘 is a descent search direction defined by𝑑𝑘=𝑔0,if𝑘=0,𝑔𝑘+𝛽𝑘𝑑𝑘1,otherwise,(4) where 𝑔𝑘 is the gradient of 𝐸 at 𝑤𝑘 and 𝛽𝑘 is a scalar. In the literature, there have been proposed several choices for 𝛽𝑘 which give rise to distinct conjugate gradient methods. The most well-known conjugate gradient methods include the Fletcher-Reeves (FR) method [8], the Hestenes-Stiefel (HS) method [9], and the Polak-Ribière (PR) method [10]. The update parameters of these methods are, respectively, specified as follows:𝛽HS𝑘=𝑔𝑇𝑘𝑦𝑘1𝑦𝑇𝑘1𝑑𝑘1,𝛽FR𝑘=𝑔𝑘2𝑔𝑘12,𝛽PR𝑘=𝑔𝑇𝑘𝑦𝑘1𝑔𝑘12,(5) where 𝑠𝑘1=𝑥𝑘𝑥𝑘1, 𝑦𝑘1=𝑔𝑘𝑔𝑘1 and denotes the Euclidean norm.

The PR method behaves like the HS method in practical computation and it is generally believed to be one of the most efficient conjugate gradient methods. However, despite the practical advantages of this method, it has the major drawback of not being globally convergent for general functions and as a result it may be trapped and cycle infinitely without presenting any substantial progress [11]. For rectifying the convergence failure of the PR method, Gilbert and Nocedal [12], motivated by Powell’s work [13], proposed to restrict the update parameter 𝛽𝑘 of being nonnegative, namely, 𝛽PR+𝑘=max{𝛽PR𝑘,0}. The authors conducted an elegant analysis of this conjugate gradient method (PR+) and established that it is globally convergent under strong assumptions. Moreover, although that the PR method and the PR+ method usually perform better than the other conjugate gradient methods, they cannot guarantee to generate descent directions, hence restarts are employed in order to guarantee convergence. Nevertheless, there is also a worry with restart algorithms that their restarts may be triggered too often; thus degrading the overall efficiency and robustness of the minimization process [14].

During the last decade, much effort has been devoted to develop new conjugate gradient methods which are not only globally convergent for general functions but also computationally superior to classical methods and are classified in two classes. The first class utilizes second-order information to accelerate conjugate gradient methods by utilizing new secant equations (see [1518]). Sample works include the nonlinear conjugate gradient methods proposed by Zhang et al. [1921] which are based on MBFGS secant equation [15]. Ford et al. [22] proposed a multistep conjugate gradient method that is based on the multistep quasi-Newton methods proposed in [16, 17]. Recently, Yabe and Takano [23] and Li et al. [18] proposed conjugate gradient methods which are based on modified secant equation using both the gradient and function values with higher orders of accuracy in the approximation of the curvature. Under proper conditions, these methods are globally convergent and sometimes their numerical performance is superior to classical conjugate gradient methods. However, these methods do not ensure to generate descent directions; therefore the descent condition is usually assumed in their analysis and implementations.

The second class aims at developing conjugate gradient methods which generate descent directions, in order to avoid the usually inefficient restarts. On the basis of this idea, Zhang et al. [20, 2426] modified the search direction in order to ensure sufficient descent, that is, 𝑑𝑇𝑘𝑔𝑘=𝑔𝑘2, independent of the performed line search. Independently, Hager and Zhang [27] modified the parameter 𝛽𝑘 and proposed a new descent conjugate gradient method, called the CG-DESCENT method. More analytically, they proposed a modification of the Hestenes-Stiefel formula 𝛽HS𝑘 in the following way:𝛽HZ𝑘=𝛽HS𝑘𝑦2𝑘12𝑑𝑇𝑘1𝑦𝑘12𝑔𝑇𝑘𝑑𝑘1.(6) Along this line, Yuan [28] based on [12, 27, 29], proposed a modified PR method, that is,𝛽DPR+𝑘=𝛽PR𝑘𝛽minPR𝑘𝑦,𝐶𝑘12𝑔𝑘14𝑔𝑇𝑘𝑑𝑘1,(7) where 𝐶 is a parameter which essentially controls the relative weight between conjugacy and descent and in case 𝐶>1/4 then the above formula satisfies 𝑔𝑇𝑘𝑑𝑘(11/4𝐶)𝑔𝑘2. An important feature of this method is that it is globally convergent for general functions. Recently, Livieris et al. [3032] motivated by the previous works presented some descent conjugate gradient training algorithms providing some promising results. Based on their numerical experiments, the authors concluded that the sufficient descent property led to a significant improvement of the training process.

In this paper, we proposed a new conjugate gradient training algorithm which has both characteristics of the previous presented classes. Our method ensures sufficient descent independent of the accuracy of the line search, avoiding thereby the usually inefficient restarts. Moreover, it achieves a high-order accuracy in approximating the second-order curvature information of the error surface by utilizing the modified secant condition proposed in [18]. Under mild conditions, we establish the global convergence of our proposed method.

The remainder of this paper is organized as follows. In Section 2, we present our proposed conjugate gradient training algorithm and in Section 3, we present its global convergence analysis. The experimental results are reported in Section 4 using the performance profiles of Dolan and Morè [33]. Finally, Section 5 presents our concluding remarks.

2. Modified Polak-Ribière+ Conjugate Gradient Algorithm

Firstly, we recall that for quasi-Newton methods, an approximation matrix 𝐵𝑘1 to the Hessian 2𝐸(𝑤𝑘1) of a nonlinear function 𝐸 is updated so that a new matrix 𝐵𝑘 satisfies the following secant condition:𝐵𝑘𝑠𝑘1=𝑦𝑘1.(8) Obviously, only two gradients are exploited in the secant equation (8), while the function values available are neglected. Recently, Li et al. [18] proposed a conjugate gradient method based on the modified secant condition𝐵𝑘𝑠𝑘1=̃𝑦𝑘1,̃𝑦𝑘1=𝑦𝑘1+𝜃max𝑘1,0𝑠𝑘12𝑠𝑘1,(9) where 𝜃𝑘1 is defined by𝜃𝑘1𝐸=2𝑘1𝐸𝑘+𝑔𝑘+𝑔𝑘1𝑇𝑠𝑘1,(10) and 𝐸𝑘 denotes 𝐸(𝑤𝑘). The authors proved that this new secant equation (9) is superior to the classical one (8) in the sense that ̃𝑦𝑘1 better approximates 2𝐸(𝑤𝑘)𝑠𝑘1 than 𝑦𝑘1 (see [18]).

Motivated by the theoretical advantages of this modified secant condition (9), we propose a modification of formula (7), in the following way:𝛽MPR+𝑘=𝑔𝑇𝑘̃𝑦𝑘1𝑔𝑘12𝑔min𝑇𝑘̃𝑦𝑘1𝑔𝑘12,𝐶̃𝑦𝑘12𝑔𝑘14𝑔𝑇𝑘𝑑𝑘1,(11) with 𝐶>1/4. It is easy to see from (4) and (11) that our proposed formula 𝛽MPR+𝑘 satisfies the sufficient descent condition𝑔𝑇𝑘𝑑𝑘11𝑔4𝐶𝑘2,(12) independent of the line search used.

At this point, we present a high level description of our proposed algorithm, called modified Polak-Ribière+ conjugate gradient algorithm (MPR+-CG).

Algorithm 1 (modified Polak-Ribière+ conjugate gradient algorithm). Step 1. Initiate 𝑤0, 0<𝜎1<𝜎2<1, 𝐸𝐺 and 𝑘MAX; set 𝑘=0.Step 2. Calculate the error function value 𝐸𝑘 and its gradient 𝑔𝑘.Step 3. If (𝐸𝑘<𝐸𝐺), return 𝑤=𝑤𝑘 and 𝐸=𝐸𝑘.Step 4. If (𝑔𝑘=0), return “Error goal not met”.Step 5. Compute the descent direction 𝑑𝑘 using (4) and (11).Step 6. Compute the learning rate 𝜂𝑘 using the strong Wolfe line search conditions 𝐸𝑤𝑘+𝛼𝑘𝑑𝑘𝑤𝐸𝑘𝜎1𝛼𝑘𝑔𝑇𝑘𝑑𝑘,(13)|||𝑔𝑤𝑘+𝛼𝑘𝑑𝑘𝑇𝑑𝑘|||𝜎2||𝑔𝑇𝑘𝑑𝑘||.(14)Step 7. Update the weights 𝑤𝑘+1=𝑤𝑘+𝜂𝑘𝑑𝑘(15) and set 𝑘=𝑘+1.Step 8. If (𝑘>𝑘MAX) return “error goal not met”, else go to Step 2.

3. Global Convergence Analysis

In order to establish the global convergence result for our proposed method, we will impose the following assumptions on the error function 𝐸.

Assumption 1. The level set ={𝑤𝑛𝐸(𝑤)𝐸(𝑤0)} is bounded.

Assumption 2. In some neighborhood 𝒩, 𝐸 is differentiable and its gradient 𝑔 is Lipschitz continuous, namely, there exists a positive constant 𝐿>0 such that 𝑔𝑤𝑤(𝑤)𝑔𝐿𝑤,and𝑤,𝑤𝒩.(16)

Since {𝐸(𝑤𝑘)} is a decreasing sequence, it is clear that the sequence {𝑤𝑘} is contained in . In addition, it follows directly from Assumptions 1 and 2 that there exist positive constraints 𝐵 and 𝑀, such that 𝑤𝑤𝐵,𝑤,𝑤,(17)𝑔(𝑤)𝑀,𝑤.(18) Furthermore, notice that since the error function 𝐸 is bounded below in 𝑛 by zero, it is differentiable and its gradient is Lipschitz continuous [34]. Assumptions 1 and 2 always hold.

The following lemma is very useful for the global convergence analysis.

Lemma 2 (see [18]). Suppose that Assumptions 1 and 2 hold and the line search satisfies the strong Wolfe line search conditions (13) and (14). For 𝜃𝑘 and ̃𝑦𝑘 defined in (10) and (9), respectively, one has ||𝜃𝑘||𝑠𝐿𝑘2,̃𝑦𝑘𝑠2𝐿𝑘.(19)
Subsequently, we will establish the global convergence of Algorithm MPR+-CG for general functions. Firstly, we present a lemma that Algorithm MPR+-CG prevents the inefficient behavior of the jamming phenomenon [35] from occurring. This property is similar to but slightly different from Property(), which was derived by Gilbert and Nocedal [12].

Lemma 3. Suppose that Assumptions 1 and 2 hold. Let {𝑤𝑘} and {𝑑𝑘} be generated by Algorithm MPR+-CG, if there exists a positive constant 𝜇>0 such that 𝑔𝑘𝜇,𝑘0,(20) then there exist constants 𝑏>1 and 𝜆>0 such that ||𝛽MPR+𝑘||𝑠𝑏,(21)𝑘1||𝛽𝜆MPR+𝑘||1𝑏.(22)

Proof. Utilizing Lemma 2 together with Assumption 2 and relations (12), (14), (17), (18), (20) we have ||𝛽MPR+𝑘||||𝑔𝑇𝑘̃𝑦𝑘1||𝑔𝑘12+𝐶̃𝑦𝑘12𝑔𝑘14||𝑔𝑇𝑘𝑑𝑘1||𝑔𝑘̃𝑦𝑘1𝑔𝑘12+𝐶̃𝑦𝑘12𝜎2||𝑔𝑇𝑘1𝑑𝑘1||𝑔𝑘14𝑠2𝑀𝐿𝑘1𝑔𝑘12+𝐶4𝐿2𝑠𝑘12𝑔𝑘14𝜎211𝑔4𝐶𝑘122𝑀𝐿+4𝐿2𝐵𝐶𝜎2(11/4𝐶)𝜇2𝑠𝑘1𝑠𝐷𝑘1.(23) Therefore, by setting b:= max{2,2𝐷𝐵} and λ:= 1/𝐷𝑏, we have relations (21) and (22) hold. The proof is completed.

Subsequently, we present a lemma which shows that, asymptotically, the search directions 𝑤𝑘 change slowly. This lemma corresponds to Lemma 4.1 of Gilbert and Nocedal [12] and the proof is exactly the same as that of Lemma 4.1 in [12], thus we omit it.

Lemma 4. Suppose that Assumptions 1 and 2 hold. Let {𝑤𝑘} and {𝑑𝑘} be generated by Algorithm MPR+-CG, if there exists a positive constant 𝜇>0 such that (21) holds; then 𝑑𝑘0 and 𝑘1𝑢𝑘𝑢𝑘12<,(24) where 𝑢𝑘=𝑑𝑘/𝑑𝑘.

Next, by making use of Lemmas 3 and 4, we establish the global convergence theorem for Algorithm MPR+-CG under the strong Wolfe line search.

Theorem 5. Suppose that Assumptions 1 and 2 hold. If {𝑤𝑘} is obtained by Algorithm MPR+-CG where the line search satisfies the strong Wolfe line search conditions (13) and (14), then one has lim𝑘𝑔inf𝑘=0.(25)

Proof. We proceed by contraction. Suppose that there exists a positive constant 𝜇>0 such that for all 𝑘0𝑔𝑘𝜇.(26) The proof is divided in the following two steps.Step I
A bound on the step 𝑠𝑘. Let Δ be a positive integer, chosen large enough that Δ4𝐵𝐷,(27) where 𝐵 and 𝐷 are defined in (17) and (23), respectively. For any 𝑙>𝑘𝑘0 with 𝑙𝑘Δ, following the same proof as the case II of Theorem 3.2 in [27], we get 𝑙1𝑗=𝑘𝑠𝑗<2𝐵.(28)
Step II
A bound on the search directions of 𝑑𝑙. It follows from the definition of 𝑑𝑘 in (4) together with (18) and (23), we obtain 𝑑𝑙2𝑔𝑙2+||𝛽MPR+𝑘||2𝑑𝑙12𝑀2+𝐷2𝑠𝑙12𝑑𝑙12.(29)
Now, the remaining argument is standard in the same way as case III in Theorem 3.2 in [27], thus we omit it. This completes the proof.

4. Experimental Results

In this section, we will present experimental results in order to evaluate the performance of our proposed conjugate gradient algorithm MPR+-CG in five famous classification problems acquired by the UCI Repository of Machine Learning Databases [36]: the iris problem, the diabetes problem, the sonar problem, the yeast problem, and the Escherichia coli problem.

The implementation code was written in Matlab 6.5 on a Pentium IV computer (2.4 MHz, 512 Mbyte RAM) running Windows XP operating system based on the SCG code of Birgin and Martínez [37]. All methods are implemented with the line search proposed in CONMIN [38] which employs various polynomial interpolation schemes and safeguards in satisfying the strong Wolfe line search conditions. The heuristic parameters were set as 𝜎1=104 and 𝜎2=0.5 as in [30, 39]. All networks have received the same sequence of input patterns and the initial weights were generated using the Nguyen-Widrow method [40]. For evaluating classification accuracy we, have used the standard procedure called 𝑘-fold cross-validation [41]. The results have been averaged over 500 simulations.

4.1. Training Performance

The cumulative total for a performance metric over all simulations does not seem to be too informative, since a small number of simulations can tend to dominate these results. For this reason, we use the performance profiles proposed by Dolan and Morè [33] to present perhaps the most complete information in terms of robustness, efficiency, and solution quality. The performance profile plots the fraction 𝑃 of simulations for which any given method is within a factor 𝜏 of the best training method. The horizontal axis of each plot shows the percentage of the simulations for which a method is the fastest (efficiency), while the vertical axis gives the percentage of the simulations that the neural networks were successfully trained by each method (robustness). The reported performance profiles have been created using the Libopt environment [42] for measuring the efficiency and the robustness of our method in terms of computational time (CPU time) and function/gradient evaluations (FE/GE). The curves in the following figures have the following meaning.(i)“PR’’ stands for the Polak-Ribière conjugate gradient method.(ii)“PR+’’ stands for the Polak-Ribière+ conjugate gradient method.(iii)“MPR+’’ stands for Algorithm MPR+-CG.

4.1.1. Iris Classification Problem

This benchmark is perhaps the most best known to be found in the pattern-recognition literature [36]. The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. The network architectures constitute of 1 hidden layer with 7 neurons and an output layer of 3 neurons. The training goal was set to 𝐸𝐺0.01 within the limit of 1000 epochs and all networks were tested using 10-fold cross-validation [30].

Figure 1 presents the performance profiles for the iris classification problem, regarding both performance metrics. MPR+ illustrates the best performance in terms of efficiency and robustness, significantly outperforming the classical training methods PR and PR+. Furthermore, the performance profiles show that MPR+ is the only method reporting an excellent (100%) probability of being the optimal training method.

fig1
Figure 1: Log10 scaled performance profiles for the iris classification problem.
4.1.2. Diabetes Classification Problem

The aim of this real-world classification task is to decide whether a Pima Indian female is diabetes positive or not. The data of this benchmark consists of 768 different patterns, each of them having 8 features of real continuous values and a class label (diabetes positive or not). We have used neural networks with 2 hidden layers of 4 neurons each and an output layer of 2 neurons [43]. The training goal was set to 𝐸𝐺<0.14 within the limit of 2000 epochs and all networks were tested using 10-fold cross-validation [44].

Figure 2 illustrates the performance profiles for the diabetes classification problem, investigating the efficiency and robustness of each training method. Clearly, our proposed method MPR+ significantly outperforms the conjugate gradient methods PR and PR+ since the curves of the former lie above the curves of the latter, regarding both performance metrics. More analytically, the performance profiles show that the probability of MPR+ to successfully train a neural network within a factor 3.41 of the best solver is 100%, in contrast with PR and PR+ which have probability 84.3% and 85%, respectively.

fig2
Figure 2: Log10 scaled performance profiles for the diabetes classification problem.
4.1.3. Sonar Classification Problem

This is the dataset used by Gorman and Sejnowski [45] in their study of the classification of sonar signals using a neural network. The dataset contains signals obtained from a variety of different aspect angles, spanning 90 degrees for the cylinder and 180 degrees for the rock. The network architecture for this problem constitutes of 1 hidden layer of 24 neurons and an output layer of 2 neurons [45]. The training goal was set to 𝐸𝐺=0.1 within the limit of 1000 epochs and all networks were tested using 3-fold cross-validation [45].

In Figure 3 are presented the performance profiles for the sonar classification problem, relative to both performance metrics. Our proposed conjugate gradient method MPR+ presents the highest probability of being the optimal training method. Furthermore, MPR+ significantly outperforms PR and is slightly more robust than PR+, regarding both performance metrics.

fig3
Figure 3: Log10 scaled performance profiles for the sonar classification problem.
4.1.4. Yeast Classification Problem

This problem is based on a drastically imbalanced dataset and concerns the determination of the cellular localization of the yeast proteins into ten localization sites. Saccharomyces cerevisiae (yeast) is the simplest Eukaryotic organism. The network architecture for this classification problem consists of 1 hidden layer of 16 neurons and an output layer of 10 neurons [46]. The training goal was set to 𝐸𝐺<0.05 within the limit of 2000 epochs and all networks were tested using 10-fold cross validation [47].

Figure 4 presents the performance profiles for the yeast classification problem, regarding both performance metrics. The interpretation in Figure 4 highlights that our proposed conjugate gradient method MPR+ is the only method exhibiting an excellent (100%) probability of successful training. Moreover, it is worth noticing that PR and PR+ report very poor performance exhibiting 0% and 5% probability of successfully training, respectively, in contrast with our proposed method MPR+ which has successfully trained all neural networks.

fig4
Figure 4: Log10 scaled performance profiles for the yeast classification problem.
4.1.5. Escherichia coli Classification Problem

This problem is based on a drastically imbalanced data set of 336 patterns and concerns the classification of the E. coli protein localization patterns into eight localization sites. E. coli, being a prokaryotic gram-negative bacterium, is an important component of the biosphere. Three major and distinctive types of proteins are characterized in E. coli: enzymes, transporters, and regulators. The largest number of genes encoding enzymes (34%) (this should include all the cytoplasm proteins) is followed by the genes for transport functions and the genes for regulatory process (11.5%) [48]. The network architectures constitute of 1 hidden layer with 16 neurons and an output layer of 8 neurons [46]. The training goal was set to 𝐸𝐺0.02 within the limit of 2000 epochs and all neural networks were tested using 4-fold cross-validation [47].

In Figure 5 are presented the performance profiles for the Escherichia coli classification problem. Similar observations can be made with the previous benchmarks. More specifically, MPR+ significantly outperforms the classical training methods PR and PR+, since the curves of the former lie above the curves of the latter, regarding both performance metrics. Moreover the performance profiles show that the probability of MPR+ is the only method reporting excellent (100%) probability of being the optimal training method.

fig5
Figure 5: Log10 scaled performance profiles for the Escherichia coli classification problem.
4.2. Generalization Performance

In Table 1 are summarized the generalization results of PR, PR+, and MPR+ conjugate gradient methods, measured by the percentage of testing patterns that were classified correctly in the presented classification problems. Each row reports the average performance in percentage for each problem and the best conjugate gradient method for a problem is illustrated in boldface. Moreover, “−” means that the method reported 0% training success.

tab1
Table 1: Generalization results of conjugate gradient methods PR, PR+, and MPR+.

The interpretation on Table 1 illustrates that MPR+ is an excellent generalizer since it manages to have the highest generalization performance, outperforming the classical training methods PR and PR+ in all classification problems.

5. Conclusions

In this paper, we proposed a conjugate gradient method for efficiently training neural networks. An attractive property of our proposed method is that it ensures sufficient descent, avoiding thereby the usually inefficient restarts. Furthermore, it achieves a high-order accuracy in approximating the second-order curvature information of the error surface by utilizing the modified secant equation proposed in [18]. Under mild conditions, we established that our proposed method is globally convergent. Based on our numerical experiments, we concluded that our proposed method outperforms classical conjugate gradient training methods and has a potential to significantly enhance the computational efficiency and robustness of the training process.

References

  1. C. M. Bishop, Neural Networks for Pattern Recognition, Oxford, UK, 1995.
  2. S. Haykin, Neural Networks: A Comprehensive Foundation, Macmillan College, New York, NY, USA, 1994.
  3. A. Hmich, A. Badri, and A. Sahel, “Automatic speaker identification by using the neural network,” in Proceedings of the IEEE International Conference on Multimedia Computing and Systems (ICMCS '11), pp. 1–5, April 2011. View at Publisher · View at Google Scholar
  4. H. Takeuchi, Y. Terabayashi, K. Yamauchi, and N. Ishii, “Improvement of robot sensor by integrating information using neural network,” International Journal on Artificial Intelligence Tools, vol. 12, no. 2, pp. 139–152, 2003.
  5. C. H. Wu, H. L. Chen, and S. C. Chen, “Gene classification artificial neural system,” International Journal on Artificial Intelligence Tools, vol. 4, no. 4, pp. 501–510, 1995.
  6. B. Lerner, H. Guterman, M. Aladjem, and I. Dinstein, “A comparative study of neural network based feature extraction paradigms,” Pattern Recognition Letters, vol. 20, no. 1, pp. 7–14, 1999. View at Publisher · View at Google Scholar
  7. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D. Rumelhart and J. McClelland, Eds., pp. 318–362, Cambridge, Mass, USA, 1986.
  8. R. Fletcher and C. M. Reeves, “Function minimization by conjugate gradients,” Computer Journal, vol. 7, pp. 149–154, 1964.
  9. M. R. Hestenes and E. Stiefel, “Methods for conjugate gradients for solving linear systems,” Journal of Research of the National Bureau of Standards, vol. 49, no. 6, pp. 409–436, 1952.
  10. E. Polak and G. Ribière, “Note sur la convergence de methods de directions conjuguees,” Revue Francais d'Informatique et de Recherche Operationnelle, vol. 16, pp. 35–43, 1969.
  11. M. J. D. Powell, “Nonconvexminimization calculations and the conjugate gradientmethod,” in Numerical Analysis, vol. 1066 of Lecture notes in mathematics, pp. 122–141, Springer, Berlin, Germany, 1984.
  12. J. C. Gilbert and J. Nocedal, “Global convergence properties of conjugate gradient methods for optimization,” SIAM Journal of Optimization, vol. 2, no. 1, pp. 21–42, 1992.
  13. M. J. D. Powell, “Convergence properties of algorithms for nonlinear optimization,” Siam Review, vol. 28, no. 4, pp. 487–500, 1986.
  14. J. Nocedal, “Theory of algorithms for unconstrained optimization,” Acta Numerica, vol. 1, pp. 199–242, 1992.
  15. D. H. Li and M. Fukushima, “A modified BFGS method and its global convergence in nonconvex minimization,” Journal of Computational and Applied Mathematics, vol. 129, no. 1-2, pp. 15–35, 2001. View at Publisher · View at Google Scholar · View at MathSciNet
  16. J. A. Ford and I. A. Moghrabi, “Multi-step quasi-Newton methods for optimization,” Journal of Computational and Applied Mathematics, vol. 50, no. 1-3, pp. 305–323, 1994.
  17. J. A. Ford and I. A. Moghrabi, “Using function-values in multi-step quasi-Newton methods,” Journal of Computational and Applied Mathematics, vol. 66, no. 1-2, pp. 201–211, 1996.
  18. G. Li, C. Tang, and Z. Wei, “New conjugacy condition and related new conjugate gradient methods for unconstrained optimization,” Journal of Computational and Applied Mathematics, vol. 202, no. 2, pp. 523–539, 2007. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  19. L. Zhang, “Two modified Dai-Yuan nonlinear conjugate gradient methods,” Numerical Algorithms, vol. 50, no. 1, pp. 1–16, 2009. View at Publisher · View at Google Scholar
  20. L. Zhang, W. Zhou, and D. Li, “Some descent three-term conjugate gradient methods and their global convergence,” Optimization Methods and Software, vol. 22, no. 4, pp. 697–711, 2007. View at Publisher · View at Google Scholar · View at MathSciNet
  21. W. Zhou and L. Zhang, “A nonlinear conjugate gradient method based on the MBFGS secant condition,” Optimization Methods and Software, vol. 21, no. 5, pp. 707–714, 2006.
  22. J. A. Ford, Y. Narushima, and H. Yabe, “Multi-step nonlinear conjugate gradient methods for unconstrained minimization,” Computational Optimization and Applications, vol. 40, no. 2, pp. 191–216, 2008. View at Publisher · View at Google Scholar
  23. H. Yabe and M. Takano, “Global convergence properties of nonlinear conjugate gradient methods with modified secant condition,” Computational Optimization and Applications, vol. 28, no. 2, pp. 203–225, 2004. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  24. L. Zhang, W. Zhou, and D. Li, “Global convergence of a modified Fletcher-Reeves conjugate gradient method with Armijo-type line search,” Numerische Mathematik, vol. 104, no. 4, pp. 561–572, 2006. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  25. L. Zhang, W. Zhou, and D. Li, “A descent modified Polak-Ribière-Polyak conjugate gradient method and its global convergence,” IMA Journal of Numerical Analysis, vol. 26, no. 4, pp. 629–640, 2006. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  26. L. Zhang and W. Zhou, “Two descent hybrid conjugate gradient methods for optimization,” Journal of Computational and Applied Mathematics, vol. 216, no. 1, pp. 251–264, 2008. View at Publisher · View at Google Scholar
  27. W. W. Hager and H. Zhang, “A new conjugate gradient method with guaranteed descent and an efficient line search,” SIAM Journal on Optimization, vol. 16, no. 1, pp. 170–192, 2005. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  28. G. Yuan, “Modified nonlinear conjugate gradient methods with sufficient descent property for large-scale optimization problems,” Optimization Letters, vol. 3, no. 1, pp. 11–21, 2009. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  29. G. H. Yu., Nonlinear self-scaling conjugate gradient methods for large-scale optimization problems, Ph.D. thesis, Sun Yat-Sen University, 2007.
  30. I. E. Livieris and P. Pintelas, “Performance evaluation of descent CG methods for neural network training,” in Proceedings of the 9th Hellenic European Research on Computer Mathematics & its Applications Conference (HERCMA '09), E. A. Lipitakis, Ed., pp. 40–46, 2009.
  31. I. E. Livieris and P. Pintelas, “An improved spectral conjugate gradient neural network training algorithm,” International Journal on Artificial Intelligence Tools. In press.
  32. I. E. Livieris, D. G. Sotiropoulos, and P. Pintelas, “On descent spectral CG algorithms for training recurrent neural networks,” in Proceedings of the 13th Panellenic Conference of Informatics, pp. 65–69, 2009.
  33. E. Dolan and J. J. Moré, “Benchmarking optimization software with performance profiles,” Mathematical Programming, vol. 91, no. 2, pp. 201–213, 2002. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  34. J. Hertz, A. Krogh, and R. Palmer, Introduction to the Theory of Neural Computation, Addison-Wesley, Reading, Mass, USA, 1991.
  35. M. J. D. Powell, “Restart procedures for the conjugate gradient method,” Mathematical Programming, vol. 12, no. 1, pp. 241–254, 1977. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  36. P. M. Murphy and D. W. Aha, UCI Repository of Machine Learning Databases, University of California, Department of Information and Computer Science, Irvine, Calif, USA, 1994.
  37. E. G. Birgin and J. M. Martínez, “A spectral conjugate gradient method for unconstrained optimization,” Applied Mathematics and Optimization, vol. 43, no. 2, pp. 117–128, 2001. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  38. D. F. Shanno and K. H. Phua, “Minimization of unconstrained multivariate functions,” ACM Transactions on Mathematical Software, vol. 2, pp. 87–94, 1976.
  39. G. Yu, L. Guan, and W. Chen, “Spectral conjugate gradient methods with sufficient descent property for large-scale unconstrained optimization,” Optimization Methods and Software, vol. 23, no. 2, pp. 275–293, 2008. View at Publisher · View at Google Scholar · View at Scopus
  40. D. Nguyen and B. Widrow, “Improving the learning speed of 2-layer neural network by choosing initial values of adaptive weights,” Biological Cybernetics, vol. 59, pp. 71–113, 1990.
  41. R. Kohavi, “A study of cross-validation and bootstrap for accuracy estimation and model selection,” in Proceedings of the IEEE International Joint Conference on Artificial Intelligence, pp. 223–228, AAAI Press and MIT Press, 1995.
  42. J. C. Gilbert and X. Jonsson, LIBOPT-an enviroment for testing solvers on heterogeneous collections of problems—version 1. CoRR, abs/cs/0703025, 2007.
  43. L. Prechelt, “PROBEN1-a set of benchmarks and benchmarking rules for neural network training algorithms,” Tech. Rep. 21/94, Fakultät für Informatik, University of Karlsruhe, 1994.
  44. J. Yu, S. Wang, and L. Xi, “Evolving artificial neural networks using an improved PSO and DPSO,” Neurocomputing, vol. 71, no. 4–6, pp. 1054–1060, 2008. View at Publisher · View at Google Scholar
  45. R. P. Gorman and T. J. Sejnowski, “Analysis of hidden units in a layered network trained to classify sonar targets,” Neural Networks, vol. 1, no. 1, pp. 75–89, 1988.
  46. A. D. Anastasiadis, G. D. Magoulas, and M. N. Vrahatis, “New globally convergent training scheme based on the resilient propagation algorithm,” Neurocomputing, vol. 64, no. 1–4, pp. 253–270, 2005. View at Publisher · View at Google Scholar
  47. P. Horton and K. Nakai, “Better prediction of protein cellular localization sites with the k nearest neighbors classifier,” Intelligent Systems for Molecular Biology, pp. 368–383, 1997.
  48. P. Liang, B. Labedan, and M. Riley, “Physiological genomics of escherichia coli protein families,” Physiological Genomics, vol. 2002, no. 9, pp. 15–26, 2002.