Table of Contents Author Guidelines Submit a Manuscript
Mathematical Problems in Engineering
Volume 2016, Article ID 3015087, 17 pages
http://dx.doi.org/10.1155/2016/3015087
Research Article

Simplified Information Maximization for Improving Generalization Performance in Multilayered Neural Networks

IT Education Center and School of Science and Technology, Tokai University, 1117 Kitakaname, Hiratsuka, Kanagawa 259-1292, Japan

Received 30 July 2015; Accepted 21 February 2016

Academic Editor: Antonino Laudani

Copyright © 2016 Ryotaro Kamimura. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Linked References

  1. R. Linsker, “Self-organization in a perceptual network,” Computer, vol. 21, no. 3, pp. 105–117, 1988. View at Publisher · View at Google Scholar · View at Scopus
  2. R. Linsker, “How to generate ordered maps by maximizing the mutual information between input and output signals,” Neural Computation, vol. 1, no. 3, pp. 402–411, 1989. View at Publisher · View at Google Scholar
  3. R. Linsker, “Local synaptic learning rules suffice to maximize mutual information in a linear network,” Neural Computation, vol. 4, no. 5, pp. 691–702, 1992. View at Publisher · View at Google Scholar
  4. R. Linsker, “Improved local learning rule for information maximization and related applications,” Neural Networks, vol. 18, no. 3, pp. 261–265, 2005. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at Scopus
  5. H. B. Barlow, “Unsupervised learning,” Neural Computation, vol. 1, no. 3, pp. 295–311, 1989. View at Publisher · View at Google Scholar
  6. H. B. Barlow, T. P. Kaushal, and G. J. Mitchison, “Finding minimum entropy codes,” Neural Computation, vol. 1, no. 3, pp. 412–423, 1989. View at Publisher · View at Google Scholar
  7. Z. Zenadic, “Information discriminant analysis: feature extraction with an information-theoretic objective,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 8, pp. 1394–1407, 2007. View at Publisher · View at Google Scholar · View at Scopus
  8. K. Torkkola, “Feature extraction by non-parametric mutual information maximization,” Journal of Machine Learning Research, vol. 3, no. 7-8, pp. 1415–1438, 2003. View at Publisher · View at Google Scholar · View at MathSciNet
  9. J. M. Leiva-Murillo and A. Artés-Rodríguez, “Maximization of mutual information for supervised linear feature extraction,” IEEE Transactions on Neural Networks, vol. 18, no. 5, pp. 1433–1441, 2007. View at Publisher · View at Google Scholar · View at Scopus
  10. R. Kamimura, T. Kamimura, and T. R. Shultz, “Information theoretic competitive learning and linguistic rule acquisition,” Transactions of the Japanese Society for Artificial Intelligence, vol. 16, no. 2, pp. 287–298, 2001. View at Publisher · View at Google Scholar · View at Scopus
  11. R. Kamimura and F. Yoshida, “Teacher-directed learning: information-theoretic competitive learning in supervised multi-layered networks,” Connection Science, vol. 15, no. 2-3, pp. 117–140, 2003. View at Publisher · View at Google Scholar · View at Scopus
  12. R. Kamimura, “Information theoretic competitive learning in self-adaptive multi-layered networks,” Connection Science, vol. 13, no. 4, pp. 323–347, 2003. View at Google Scholar
  13. R. Kamimura, T. Kamimura, and H. Takeuchi, “Greedy information acquisition algorithm: a new information theoretic approach to dynamic information acquisition in neural networks,” Connection Science, vol. 14, no. 2, pp. 137–162, 2002. View at Publisher · View at Google Scholar
  14. R. Kamimura, “Progressive feature extraction with a greedy network-growing algorithm,” Complex Systems, vol. 14, no. 2, pp. 127–153, 2003. View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet
  15. R. Kamimura, “Information-theoretic competitive learning with inverse Euclidean distance output units,” Neural Processing Letters, vol. 18, no. 3, pp. 163–184, 2003. View at Publisher · View at Google Scholar · View at Scopus
  16. G. E. Hinton, “Learning multiple layers of representation,” Trends in Cognitive Sciences, vol. 11, no. 10, pp. 428–434, 2007. View at Publisher · View at Google Scholar · View at Scopus
  17. Y. Bengio, “Learning deep architectures for AI,” Foundations and Trends® in Machine Learning, vol. 2, no. 1, pp. 1–127, 2009. View at Publisher · View at Google Scholar
  18. G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, vol. 313, no. 5786, pp. 504–507, 2006. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  19. G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, vol. 18, no. 7, pp. 1527–1554, 2006. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet · View at Scopus
  20. G. Hinton, “A practical guide to training restricted boltzmann machines,” Momentum, vol. 9, no. 1, p. 926, 2010. View at Google Scholar
  21. C. Poultney, S. Chopra, Y. L. Cun et al., “Efficient learning of sparse representations with an energy-based model,” in Advances in Neural Information Processing Systems, pp. 1137–1144, MIT Press, 2006. View at Google Scholar
  22. X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier networks,” in Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. JMLR W&CP Volume, vol. 15, pp. 315–323, Fort Lauderdale, Fla, USA, April 2011.
  23. M. A. Ranzato, Y.-L. Boureau, and Y. L. Cun, “Sparse feature learning for deep belief networks,” in Proceedings of the Advances in Neural Information Processing Systems (NIPS '08), pp. 1185–1192, 2008.
  24. H. Lee, C. Ekanadham, and A. Y. Ng, “Sparse deep belief net model for visual area v2,” in Advances in Neural Information Processing Systems, pp. 873–880, MIT Press, 2008. View at Google Scholar
  25. P. Lennie, “The cost of cortical computation,” Current Biology, vol. 13, no. 6, pp. 493–497, 2003. View at Publisher · View at Google Scholar · View at Scopus
  26. E. P. Simoncelli, “4.7 Statistical modeling of photographic images,” in Handbook of Video and Image Processing, Academic Press, 2005. View at Google Scholar
  27. R. Kamimura and S. Nakanishi, “Improving generalization performance by information minimization,” IEICE Transactions on Information and Systems, vol. E78-D, no. 2, pp. 163–173, 1995. View at Publisher · View at Google Scholar
  28. R. Kamimura and S. Nakanishi, “Hidden information maximization for feature detection and rule discovery,” Network: Computation in Neural Systems, vol. 6, no. 4, pp. 577–602, 1995. View at Publisher · View at Google Scholar · View at Scopus
  29. R. Kamimura and T. Kamimura, “Structural information and linguistic rule extraction,” in Proceedings of the International Conference on Neural Information Processing (ICONIP '00), pp. 720–726, Taejon, Republic of Korea, November 2000.
  30. M. Lichman, UCI Machine Learning Repository, University of California, Irvine, Calif, USA, School of Information and Computer Sciences, 2013, http://archive.ics.uci.edu/ml/.
  31. A. Oyefusi, “Oil and the probability of rebel participation among youths in the Niger Delta of Nigeria,” Journal of Peace Research, vol. 45, no. 4, pp. 539–555, 2008. View at Publisher · View at Google Scholar · View at Scopus