About this Journal Submit a Manuscript Table of Contents
ISRN Artificial Intelligence
Volume 2012 (2012), Article ID 486361, 9 pages
http://dx.doi.org/10.5402/2012/486361
Research Article

An Advanced Conjugate Gradient Training Algorithm Based on a Modified Secant Equation

1Department of Mathematics, University of Patras, 26500 Patras, Greece
2Educational Software Development Laboratory, Department of Mathematics, University of Patras, 26500 Patras, Greece

Received 5 August 2011; Accepted 4 September 2011

Academic Editors: T. Kurita and Z. Liu

Copyright © 2012 Ioannis E. Livieris and Panagiotis Pintelas. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Linked References

  1. C. M. Bishop, Neural Networks for Pattern Recognition, Oxford, UK, 1995.
  2. S. Haykin, Neural Networks: A Comprehensive Foundation, Macmillan College, New York, NY, USA, 1994.
  3. A. Hmich, A. Badri, and A. Sahel, “Automatic speaker identification by using the neural network,” in Proceedings of the IEEE International Conference on Multimedia Computing and Systems (ICMCS '11), pp. 1–5, April 2011. View at Publisher · View at Google Scholar
  4. H. Takeuchi, Y. Terabayashi, K. Yamauchi, and N. Ishii, “Improvement of robot sensor by integrating information using neural network,” International Journal on Artificial Intelligence Tools, vol. 12, no. 2, pp. 139–152, 2003.
  5. C. H. Wu, H. L. Chen, and S. C. Chen, “Gene classification artificial neural system,” International Journal on Artificial Intelligence Tools, vol. 4, no. 4, pp. 501–510, 1995.
  6. B. Lerner, H. Guterman, M. Aladjem, and I. Dinstein, “A comparative study of neural network based feature extraction paradigms,” Pattern Recognition Letters, vol. 20, no. 1, pp. 7–14, 1999. View at Publisher · View at Google Scholar
  7. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D. Rumelhart and J. McClelland, Eds., pp. 318–362, Cambridge, Mass, USA, 1986.
  8. R. Fletcher and C. M. Reeves, “Function minimization by conjugate gradients,” Computer Journal, vol. 7, pp. 149–154, 1964.
  9. M. R. Hestenes and E. Stiefel, “Methods for conjugate gradients for solving linear systems,” Journal of Research of the National Bureau of Standards, vol. 49, no. 6, pp. 409–436, 1952.
  10. E. Polak and G. Ribière, “Note sur la convergence de methods de directions conjuguees,” Revue Francais d'Informatique et de Recherche Operationnelle, vol. 16, pp. 35–43, 1969.
  11. M. J. D. Powell, “Nonconvexminimization calculations and the conjugate gradientmethod,” in Numerical Analysis, vol. 1066 of Lecture notes in mathematics, pp. 122–141, Springer, Berlin, Germany, 1984.
  12. J. C. Gilbert and J. Nocedal, “Global convergence properties of conjugate gradient methods for optimization,” SIAM Journal of Optimization, vol. 2, no. 1, pp. 21–42, 1992.
  13. M. J. D. Powell, “Convergence properties of algorithms for nonlinear optimization,” Siam Review, vol. 28, no. 4, pp. 487–500, 1986.
  14. J. Nocedal, “Theory of algorithms for unconstrained optimization,” Acta Numerica, vol. 1, pp. 199–242, 1992.
  15. D. H. Li and M. Fukushima, “A modified BFGS method and its global convergence in nonconvex minimization,” Journal of Computational and Applied Mathematics, vol. 129, no. 1-2, pp. 15–35, 2001. View at Publisher · View at Google Scholar · View at MathSciNet
  16. J. A. Ford and I. A. Moghrabi, “Multi-step quasi-Newton methods for optimization,” Journal of Computational and Applied Mathematics, vol. 50, no. 1-3, pp. 305–323, 1994.
  17. J. A. Ford and I. A. Moghrabi, “Using function-values in multi-step quasi-Newton methods,” Journal of Computational and Applied Mathematics, vol. 66, no. 1-2, pp. 201–211, 1996.
  18. G. Li, C. Tang, and Z. Wei, “New conjugacy condition and related new conjugate gradient methods for unconstrained optimization,” Journal of Computational and Applied Mathematics, vol. 202, no. 2, pp. 523–539, 2007. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  19. L. Zhang, “Two modified Dai-Yuan nonlinear conjugate gradient methods,” Numerical Algorithms, vol. 50, no. 1, pp. 1–16, 2009. View at Publisher · View at Google Scholar
  20. L. Zhang, W. Zhou, and D. Li, “Some descent three-term conjugate gradient methods and their global convergence,” Optimization Methods and Software, vol. 22, no. 4, pp. 697–711, 2007. View at Publisher · View at Google Scholar · View at MathSciNet
  21. W. Zhou and L. Zhang, “A nonlinear conjugate gradient method based on the MBFGS secant condition,” Optimization Methods and Software, vol. 21, no. 5, pp. 707–714, 2006.
  22. J. A. Ford, Y. Narushima, and H. Yabe, “Multi-step nonlinear conjugate gradient methods for unconstrained minimization,” Computational Optimization and Applications, vol. 40, no. 2, pp. 191–216, 2008. View at Publisher · View at Google Scholar
  23. H. Yabe and M. Takano, “Global convergence properties of nonlinear conjugate gradient methods with modified secant condition,” Computational Optimization and Applications, vol. 28, no. 2, pp. 203–225, 2004. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  24. L. Zhang, W. Zhou, and D. Li, “Global convergence of a modified Fletcher-Reeves conjugate gradient method with Armijo-type line search,” Numerische Mathematik, vol. 104, no. 4, pp. 561–572, 2006. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  25. L. Zhang, W. Zhou, and D. Li, “A descent modified Polak-Ribière-Polyak conjugate gradient method and its global convergence,” IMA Journal of Numerical Analysis, vol. 26, no. 4, pp. 629–640, 2006. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  26. L. Zhang and W. Zhou, “Two descent hybrid conjugate gradient methods for optimization,” Journal of Computational and Applied Mathematics, vol. 216, no. 1, pp. 251–264, 2008. View at Publisher · View at Google Scholar
  27. W. W. Hager and H. Zhang, “A new conjugate gradient method with guaranteed descent and an efficient line search,” SIAM Journal on Optimization, vol. 16, no. 1, pp. 170–192, 2005. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  28. G. Yuan, “Modified nonlinear conjugate gradient methods with sufficient descent property for large-scale optimization problems,” Optimization Letters, vol. 3, no. 1, pp. 11–21, 2009. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  29. G. H. Yu., Nonlinear self-scaling conjugate gradient methods for large-scale optimization problems, Ph.D. thesis, Sun Yat-Sen University, 2007.
  30. I. E. Livieris and P. Pintelas, “Performance evaluation of descent CG methods for neural network training,” in Proceedings of the 9th Hellenic European Research on Computer Mathematics & its Applications Conference (HERCMA '09), E. A. Lipitakis, Ed., pp. 40–46, 2009.
  31. I. E. Livieris and P. Pintelas, “An improved spectral conjugate gradient neural network training algorithm,” International Journal on Artificial Intelligence Tools. In press.
  32. I. E. Livieris, D. G. Sotiropoulos, and P. Pintelas, “On descent spectral CG algorithms for training recurrent neural networks,” in Proceedings of the 13th Panellenic Conference of Informatics, pp. 65–69, 2009.
  33. E. Dolan and J. J. Moré, “Benchmarking optimization software with performance profiles,” Mathematical Programming, vol. 91, no. 2, pp. 201–213, 2002. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  34. J. Hertz, A. Krogh, and R. Palmer, Introduction to the Theory of Neural Computation, Addison-Wesley, Reading, Mass, USA, 1991.
  35. M. J. D. Powell, “Restart procedures for the conjugate gradient method,” Mathematical Programming, vol. 12, no. 1, pp. 241–254, 1977. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  36. P. M. Murphy and D. W. Aha, UCI Repository of Machine Learning Databases, University of California, Department of Information and Computer Science, Irvine, Calif, USA, 1994.
  37. E. G. Birgin and J. M. Martínez, “A spectral conjugate gradient method for unconstrained optimization,” Applied Mathematics and Optimization, vol. 43, no. 2, pp. 117–128, 2001. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  38. D. F. Shanno and K. H. Phua, “Minimization of unconstrained multivariate functions,” ACM Transactions on Mathematical Software, vol. 2, pp. 87–94, 1976.
  39. G. Yu, L. Guan, and W. Chen, “Spectral conjugate gradient methods with sufficient descent property for large-scale unconstrained optimization,” Optimization Methods and Software, vol. 23, no. 2, pp. 275–293, 2008. View at Publisher · View at Google Scholar · View at Scopus
  40. D. Nguyen and B. Widrow, “Improving the learning speed of 2-layer neural network by choosing initial values of adaptive weights,” Biological Cybernetics, vol. 59, pp. 71–113, 1990.
  41. R. Kohavi, “A study of cross-validation and bootstrap for accuracy estimation and model selection,” in Proceedings of the IEEE International Joint Conference on Artificial Intelligence, pp. 223–228, AAAI Press and MIT Press, 1995.
  42. J. C. Gilbert and X. Jonsson, LIBOPT-an enviroment for testing solvers on heterogeneous collections of problems—version 1. CoRR, abs/cs/0703025, 2007.
  43. L. Prechelt, “PROBEN1-a set of benchmarks and benchmarking rules for neural network training algorithms,” Tech. Rep. 21/94, Fakultät für Informatik, University of Karlsruhe, 1994.
  44. J. Yu, S. Wang, and L. Xi, “Evolving artificial neural networks using an improved PSO and DPSO,” Neurocomputing, vol. 71, no. 4–6, pp. 1054–1060, 2008. View at Publisher · View at Google Scholar
  45. R. P. Gorman and T. J. Sejnowski, “Analysis of hidden units in a layered network trained to classify sonar targets,” Neural Networks, vol. 1, no. 1, pp. 75–89, 1988.
  46. A. D. Anastasiadis, G. D. Magoulas, and M. N. Vrahatis, “New globally convergent training scheme based on the resilient propagation algorithm,” Neurocomputing, vol. 64, no. 1–4, pp. 253–270, 2005. View at Publisher · View at Google Scholar
  47. P. Horton and K. Nakai, “Better prediction of protein cellular localization sites with the k nearest neighbors classifier,” Intelligent Systems for Molecular Biology, pp. 368–383, 1997.
  48. P. Liang, B. Labedan, and M. Riley, “Physiological genomics of escherichia coli protein families,” Physiological Genomics, vol. 2002, no. 9, pp. 15–26, 2002.