Table of Contents Author Guidelines Submit a Manuscript
Computational Intelligence and Neuroscience
Volume 2015, Article ID 721367, 10 pages
http://dx.doi.org/10.1155/2015/721367
Research Article

Deep Neural Networks with Multistate Activation Functions

1School of Technology, Beijing Forestry University, No. 35 Qinghuadong Road, Haidian District, Beijing 100083, China
2School of Information Science and Technology, Beijing Forestry University, No. 35 Qinghuadong Road, Haidian District, Beijing 100083, China
3Institute of Automation, Chinese Academy of Sciences, No. 95 Zhongguancundong Road, Haidian District, Beijing 100190, China
4College of Mathematics Physics and Information Engineering, Zhejiang Normal University, No. 688 Yingbin Road, Jinhua 321004, China

Received 25 March 2015; Accepted 23 August 2015

Academic Editor: Cheng-Jian Lin

Copyright © 2015 Chenghao Cai et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Linked References

  1. G. E. Hinton, “Deep belief networks,” Scholarpedia, vol. 4, no. 5, article 5947, 2009. View at Publisher · View at Google Scholar
  2. A.-R. Mohamed, G. E. Dahl, and G. Hinton, “Acoustic modeling using deep belief networks,” IEEE Transactions on Audio, Speech & Language Processing, vol. 20, no. 1, pp. 14–22, 2012. View at Publisher · View at Google Scholar · View at Scopus
  3. G. E. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition,” IEEE Transactions on Audio, Speech & Language Processing, vol. 20, no. 1, pp. 30–42, 2012. View at Publisher · View at Google Scholar · View at Scopus
  4. D. Yu, G. E. Hinton, N. Morgan, J.-T. Chien, and S. Sagayama, “Introduction to the special section on deep learning for speech and language processing,” IEEE Transactions on Audio, Speech & Language Processing, vol. 20, no. 1, pp. 4–6, 2012. View at Publisher · View at Google Scholar · View at Scopus
  5. M. D. Zeiler, M. Ranzato, R. Monga et al., “On rectified linear units for speech processing,” in Proceedings of the 38th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '13), pp. 3517–3521, Vancouver, BC, Canada, May 2013. View at Publisher · View at Google Scholar · View at Scopus
  6. D. C. Ciregan, U. Meier, and J. Schmidhuber, “Multi-column deep neural networks for image classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '12), pp. 3642–3649, Providence, RI, USA, June 2012. View at Publisher · View at Google Scholar · View at Scopus
  7. R. Collobert and J. Weston, “A unified architecture for natural language processing: deep neural networks with multitask learning,” in Proceedings of the 25th International Conference on Machine Learning (ICML '08), pp. 160–167, Helsinki, Finland, June 2008.
  8. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” in Proceedings of the 20th Annual Conference on Neural Information Processing Systems (NIPS '06), vol. 19, pp. 153–160, Vancouver, Canada, December 2006.
  9. J. Dean, G. Corrado, R. Monga et al., “Large scale distributed deep networks,” in Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a Meeting Held December 3–6, 2012, Lake Tahoe, Nevada, United States, pp. 1232–1240, MIT Press, 2012. View at Google Scholar
  10. S. Zhang, C. Zhang, Z. You, R. Zheng, and B. Xu, “Asynchronous stochastic gradient descent for DNN training,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '13), pp. 6660–6663, IEEE, Vancouver, Canada, May 2013. View at Publisher · View at Google Scholar · View at Scopus
  11. W. Deng, Y. Qian, Y. Fan, T. Fu, and K. Yu, “Stochastic data sweeping for fast DNN training,” in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '14), pp. 240–244, Florence, Italy, May 2014. View at Publisher · View at Google Scholar · View at Scopus
  12. J. Martens, “Deep learning via Hessian-free optimization,” in Proceedings of the 27th International Conference on Machine Learning (ICML '10), pp. 735–742, Haifa, Israel, June 2010. View at Scopus
  13. J. Martens and I. Sutskever, “Training deep and recurrent networks with Hessian-Free optimization,” in Neural Networks: Tricks of the Trade, vol. 7700 of Lecture Notes in Computer Science, pp. 479–535, Springer, Berlin, Germany, 2nd edition, 2012. View at Publisher · View at Google Scholar
  14. S. Wiesler, J. Li, and J. Xue, “Investigations on hessianfree optimization for cross-entropy training of deep neural networks,” in Proceedings of the 14th Annual Conference of the International Speech Communication Association (INTERSPEECH '13), pp. 3317–3321, Lyon, France, August 2013.
  15. Z. You and B. Xu, “Investigation of stochastic Hessian-free optimization in deep neural networks for speech recognition,” in Proceedings of the 9th International Symposium on Chinese Spoken Language Processing (ISCSLP '14), pp. 450–453, IEEE, Singapore, September 2014. View at Publisher · View at Google Scholar · View at Scopus
  16. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, “Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion,” Journal of Machine Learning Research, vol. 11, pp. 3371–3408, 2010. View at Google Scholar · View at MathSciNet
  17. J. Xue, J. Li, and Y. Gong, “Restructuring of deep neural network acoustic models with singular value decomposition,” in Proceedings of the 14th Annual Conference of the International Speech Communication Association (INTERSPEECH '13), pp. 2365–2369, Lyon, France, August 2013. View at Scopus
  18. J. Xue, J. Li, D. Yu, M. Seltzer, and Y. Gong, “Singular value decomposition based low-footprint speaker adaptation and personalization for deep neural network,” in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '14), pp. 6359–6363, Florence, Italy, May 2014. View at Publisher · View at Google Scholar · View at Scopus
  19. T. He, Y. Fan, Y. Qian, T. Tan, and K. Yu, “Reshaping deep neural network for fast decoding by node-pruning,” in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '14), pp. 245–249, Florence, Italy, May 2014. View at Publisher · View at Google Scholar · View at Scopus
  20. H. Sak, A. W. Senior, and F. Beaufays, “Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition,” http://arxiv.org/abs/1402.1128.
  21. S. Wiesler, A. Richard, R. Schlüter, and H. Ney, “Mean-normalized stochastic gradient for large-scale deep learning,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '14), pp. 180–184, IEEE, Florence, Italy, May 2014. View at Publisher · View at Google Scholar · View at Scopus
  22. G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, vol. 18, no. 7, pp. 1527–1554, 2006. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet · View at Scopus
  23. Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 157–166, 1994. View at Publisher · View at Google Scholar · View at Scopus
  24. G. E. Dahl, T. N. Sainath, and G. E. Hinton, “Improving deep neural networks for LVCSR using rectified linear units and dropout,” in Proceedings of the 38th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '13), pp. 8609–8613, IEEE, Vancouver, Canada, May 2013. View at Publisher · View at Google Scholar · View at Scopus
  25. L. K. Hansen and P. Salamon, “Neural network ensembles,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 10, pp. 993–1001, 1990. View at Publisher · View at Google Scholar · View at Scopus
  26. I. H. Witten, E. Frank, and M. A. Hall, Data Mining: Practical Machine Learning Tools and Techniques, Morgan Kaufmann, 3rd edition, 2010.
  27. D. Povey, A. Ghoshal, G. Boulianne et al., “The kaldi speech recognition toolkit,” in Proceedings of the IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU '11), Waikoloa, Hawaii, USA, December 2011.