Table of Contents Author Guidelines Submit a Manuscript
Applied Computational Intelligence and Soft Computing
Volume 2018 (2018), Article ID 4084850, 20 pages
https://doi.org/10.1155/2018/4084850
Research Article

A Comparison Study on Rule Extraction from Neural Network Ensembles, Boosted Shallow Trees, and SVMs

1Department of Computer Science, University of Applied Sciences and Arts Western Switzerland, Rue de la Prairie 4, 1202 Geneva, Switzerland
2Department of Computer Science, University of Geneva, Route de Drize 7, 1227 Carouge, Switzerland
3Department of Computer Science, Meiji University, Tama-ku, Kawasaki, Kanagawa 214-8571, Japan

Correspondence should be addressed to Guido Bologna

Received 27 July 2017; Revised 17 November 2017; Accepted 4 December 2017; Published 9 January 2018

Academic Editor: Erich Peter Klement

Copyright © 2018 Guido Bologna and Yoichi Hayashi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Linked References

  1. M. Golea, On the complexity of rule extraction from neural networks and network querying. In Rule Extraction From Trained Artificial Neural Networks Workshop, Society For the Study of Artificial Intelligence and Simulation of Behavior Workshop Series (AISB), pages 51–59, 1996.
  2. A. A. Freitas, “Comprehensible classification models,” ACM SIGKDD Explorations Newsletter, vol. 15, no. 1, pp. 1–10, 2014. View at Publisher · View at Google Scholar
  3. G. Bologna, “A model for single and multiple knowledge based networks,” Artificial Intelligence in Medicine, vol. 28, no. 2, pp. 141–163, 2003. View at Publisher · View at Google Scholar · View at Scopus
  4. G. Bologna, “FDIMLP: A new neuro-fuzzy model,” in Proceedings of the International Joint Conference on Neural Networks (IJCNN'01), vol. 2, pp. 1328–1333, USA, July 2001. View at Scopus
  5. A. Van Assche and H. Blockeel, “Seeing the Forest Through the Trees: Learning a Comprehensible Model from an Ensemble,” in Machine Learning: ECML 2007, vol. 4701 of Lecture Notes in Computer Science, pp. 418–429, Springer Berlin Heidelberg, Berlin, Heidelberg, 2007. View at Publisher · View at Google Scholar
  6. D. H. Wolpert, “The lack of a priori distinctions between learning algorithms,” Neural Computation, vol. 8, no. 7, pp. 1341–1390, 1996. View at Publisher · View at Google Scholar · View at Scopus
  7. W. W. Cohen, “Fast effective rule induction,” in In Proceedings of the Twelfth International Conference on Machine Learning, pp. 115–123, 1995.
  8. G. Bologna and C. Pellegrini, “Three medical examples in neural network rule extraction,” Physica Medica, vol. 13, no. 1, pp. 183–187, 1997. View at Google Scholar · View at Scopus
  9. W. Duch, R. Adamczak, and K. Gra̧bczewski, “A new methodology of extraction, optimization and application of crisp and fuzzy logical rules,” IEEE Transactions on Neural Networks and Learning Systems, vol. 12, no. 2, pp. 277–306, 2001. View at Publisher · View at Google Scholar · View at Scopus
  10. J. Huysmans, R. Setiono, B. Baesens, and J. Vanthienen, “Minerva: Sequential covering for rule extraction,” IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 38, no. 2, pp. 299–309, 2008. View at Publisher · View at Google Scholar · View at Scopus
  11. K. Odajima, Y. Hayashi, G. Tianxia, and R. Setiono, “Greedy rule generation from discrete data and its use in neural network rule extraction,” Neural Networks, vol. 21, no. 7, pp. 1020–1028, 2008. View at Publisher · View at Google Scholar · View at Scopus
  12. Y. Hayashi and S. Nakano, “Use of a Recursive-Rule eXtraction algorithm with J48graft to achieve highly accurate and concise rule extraction from a large breast cancer dataset,” Informatics in Medicine Unlocked, vol. 1, pp. 9–16, 2015. View at Publisher · View at Google Scholar · View at Scopus
  13. M. Craven and J. W. Shavlik, “Extracting tree-structured representations of trained networks,” In Advances in neural information processing systems, pp. 24–30, 1996. View at Google Scholar
  14. U. Johansson, Obtaining accurate and comprehensible data mining models: An evolutionary approach. Linköping University, Department of Computer and Information Science, 2007.
  15. H. Núñez, C. Angulo, and A. Català, “Rule-based learning systems for support vector machines,” Neural Processing Letters, vol. 24, no. 1, pp. 1–18, 2006. View at Publisher · View at Google Scholar · View at Scopus
  16. G. Bologna and Y. Hayashi, “QSVM: A support vector machine for rule extraction,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): Preface, vol. 9095, pp. 276–289, 2015. View at Publisher · View at Google Scholar · View at Scopus
  17. S. I. Gallant, “Connectionist expert systems,” Communications of the ACM, vol. 31, no. 2, pp. 152–169, 1988. View at Publisher · View at Google Scholar · View at Scopus
  18. R. Andrews, J. Diederich, and A. B. Tickle, “Survey and critique of techniques for extracting rules from trained artificial neural networks,” Knowledge-Based Systems, vol. 8, no. 6, pp. 373–389, 1995. View at Publisher · View at Google Scholar · View at Scopus
  19. J. Diederich, Rule Extraction from Support Vector Machines, vol. 80, Springer Science & Business Media, 2008.
  20. N. Barakat and A. P. Bradley, “Rule extraction from support vector machines: a review,” Neurocomputing, vol. 74, no. 1–3, pp. 178–190, 2010. View at Publisher · View at Google Scholar · View at Scopus
  21. G. Bologna, “A study on rule extraction from several combined neural networks,” International Journal of Neural Systems, vol. 11, no. 3, pp. 247–255, 2001. View at Publisher · View at Google Scholar · View at Scopus
  22. G. Bologna, “Is it worth generating rules from neural network ensembles?” Journal of Applied Logic, vol. 2, no. 3, pp. 325–348, 2004. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  23. Z.-H. Zhou, Y. Jiang, and S.-F. Chen, “Extracting symbolic rules from trained neural network ensembles,” Artificial Intelligence Communications, vol. 16, no. 1, p. 16, 2003. View at Google Scholar
  24. S. I. Ao and V. Palade, “Ensemble of Elman neural networks and support vector machines for reverse engineering of gene regulatory networks,” Applied Soft Computing, vol. 11, no. 2, pp. 1718–1726, 2011. View at Publisher · View at Google Scholar · View at Scopus
  25. R. Setiono, B. Baesens, and C. Mues, “Recursive neural network rule extraction for data with mixed attributes,” IEEE Transactions on Neural Networks and Learning Systems, vol. 19, no. 2, pp. 299–307, 2008. View at Publisher · View at Google Scholar · View at Scopus
  26. A. Hara and Y. Hayashi, “Ensemble neural network rule extraction using Re-RX algorithm,” in Proceedings of the 2012 Annual International Joint Conference on Neural Networks, IJCNN 2012, Part of the 2012 IEEE World Congress on Computational Intelligence, WCCI 2012, Australia, June 2012. View at Publisher · View at Google Scholar · View at Scopus
  27. Y. Hayashi, R. Sato, and S. Mitra, “A new approach to three ensemble neural network rule extraction using recursive-rule extraction algorithm,” in Proceedings of the 2013 International Joint Conference on Neural Networks, IJCNN 2013, USA, August 2013. View at Publisher · View at Google Scholar · View at Scopus
  28. G. Brown, J. Wyatt, R. Harris, and X. Yao, “Diversity creation methods: a survey and categorisation,” Information Fusion, vol. 6, no. 1, pp. 5–20, 2005. View at Publisher · View at Google Scholar · View at Scopus
  29. M. Gashler, C. Giraud-Carrier, and T. Martinez, “Decision Tree Ensemble: Small Heterogeneous Is Better Than Large Homogeneous,” in Proceedings of the 2008 Seventh International Conference on Machine Learning and Applications, pp. 900–905, San Diego, CA, USA, December 2008. View at Publisher · View at Google Scholar
  30. J. H. Friedman and B. E. Popescu, “Predictive learning via rule ensembles,” The Annals of Applied Statistics, vol. 2, no. 3, pp. 916–954, 2008. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  31. L. Breiman, J. Friedman, C. J. Stone, and R. A. Olshen, Classification and Regression Trees, CRC press, 1984.
  32. N. Meinshausen, “Node harvest,” The Annals of Applied Statistics, vol. 4, no. 4, pp. 2049–2072, 2010. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  33. M. Mashayekhi and R. Gras, “Rule extraction from decision trees ensembles: new algorithms based on heuristic search and sparse group lasso methods,” International Journal of Information Technology & Decision Making, pp. 1–21, 2017. View at Publisher · View at Google Scholar · View at Scopus
  34. J. Friedman, T. Hastie, and R. Tibshirani, “A note on the group lasso and a sparse group lasso,” https://arxiv.org/abs/1001.0736.
  35. N. Barakat and J. Diederich, “Learning-based rule-extraction from support vector machines,” in Proceedings of the 14th International Conference on Computer Theory and applications ICCTA'2004, 2004.
  36. D. E. Torres D. and C. M. Rocco S., “Extracting trees from trained SVM models using a TREPAN based approach,” in Proceedings of the HIS 2005: Fifth International Conference on Hybrid Intelligent Systems, pp. 353–358, Brazil, November 2005. View at Publisher · View at Google Scholar · View at Scopus
  37. D. Martens, B. Baesens, T. Van Gestel, and J. Vanthienen, “Comprehensible credit scoring models using rule extraction from support vector machines,” European Journal of Operational Research, vol. 183, no. 3, pp. 1466–1476, 2007. View at Publisher · View at Google Scholar · View at Scopus
  38. D. Martens, B. Baesens, and T. V. Gestel, “Decompositional rule extraction from support vector machines by active learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 2, pp. 178–191, 2009. View at Publisher · View at Google Scholar · View at Scopus
  39. N. Barakat and J. Diederich, “Eclectic rule-extraction from support vector machines,” International Journal of Computational Intelligence, vol. 2, no. 1, pp. 59–62, 2005. View at Google Scholar
  40. N. H. Barakat and A. P. Bradley, “Rule extraction from support vector machines: A sequential covering approach,” IEEE Transactions on Knowledge and Data Engineering, vol. 19, no. 6, pp. 729–741, 2007. View at Publisher · View at Google Scholar · View at Scopus
  41. X. Fu, C. Ong, S. Keerthi, G. G. Hung, and L. Goh, “Extracting the knowledge embedded in support vector machines,” in Proceedings of the International Joint Conference on Neural Networks, pp. 291–296, IEEE, 2004. View at Scopus
  42. H. Núñez, C. Angulo, and A. Català, “Rule extraction from support vector machines,” Esann, pp. 107–112, 2002. View at Google Scholar
  43. Y. Zhang, H. Su, T. Jia, and J. Chu, “Rule Extraction from Trained Support Vector Machines,” in Advances in Knowledge Discovery and Data Mining, vol. 3518 of Lecture Notes in Computer Science, pp. 61–70, Springer, Berlin, Germany, 2005. View at Publisher · View at Google Scholar
  44. L. Breiman, “Bagging predictors,” Machine Learning, vol. 24, no. 2, pp. 123–140, 1996. View at Google Scholar · View at Scopus
  45. L. Breiman, Bias, variance, and arcing classifiers (technical report 460). Statistics Department, University of California, 1996.
  46. V. N. Vapnik, Statistical Learning Theory, Adaptive and Learning Systems for Signal Processing, Communications, and Control, Wiley- Interscience, New York, NY, USA, 1998. View at MathSciNet
  47. C. J. C. Burges, “A tutorial on support vector machines for pattern recognition,” Data Mining and Knowledge Discovery, vol. 2, no. 2, pp. 121–167, 1998. View at Publisher · View at Google Scholar · View at Scopus
  48. R. E. Schapire, “A brief introduction to boosting,” in Proceedings of the 16th International Joint Conference on Artificial Intelligence (IJCAI '99), pp. 1401–1406, Stockholm, Sweden, August 1999. View at Scopus
  49. A. Vezhnevets and V. Vezhnevets, “Modest adaboost-teaching adaboost to generalize better,” in Proceedings of the 15th International Conference on Computer Graphics and Vision, GraphiCon 2005, vol. 12, pp. 987–997, Computer Graphics in Russia, June 2005. View at Scopus
  50. J. Friedman, T. Hastie, and R. Tibshirani, “Additive logistic regression: a statistical view of boosting,” The Annals of Statistics, vol. 28, no. 2, pp. 337–407, 2000. View at Publisher · View at Google Scholar · View at MathSciNet
  51. Y. Freund and R. E. Schapire, “A desicion-theoretic generalization of on-line learning and an application to boosting,” in Proceedings of the European Conference on Computational Learning Theory, pp. 23–37, Springer, 1995.
  52. S. L. Salzberg, “C4.5: Programs for Machine Learning by J. Ross Quinlan. Morgan Kaufmann Publishers, Inc., 1993,” Machine Learning, vol. 16, no. 3, pp. 235–240, 1994. View at Publisher · View at Google Scholar
  53. M. Lichman, UCI machine learning repository, university of california, irvine, school of information and computer sciences, 2013.
  54. J. Alcalá-Fdez, A. Fernández, J. Luengo et al., “Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework,” Journal of Multiple-Valued Logic and Soft Computing, vol. 17, pp. 255–287, 2011. View at Google Scholar · View at Scopus
  55. B. Schölkopf, A. J. Smola, R. C. Williamson, and P. L. Bartlett, “New support vector algorithms,” Neural Computation, vol. 12, no. 5, pp. 1207–1245, 2000. View at Publisher · View at Google Scholar · View at Scopus