Table of Contents Author Guidelines Submit a Manuscript
The Scientific World Journal
Volume 2014, Article ID 484873, 14 pages
http://dx.doi.org/10.1155/2014/484873
Research Article

Towards Emotion Detection in Educational Scenarios from Facial Expressions and Body Movements through Multimodal Approaches

aDeNu Research Group, Artificial Intelligence Department, UNED, Calle Juan del Rosal 16, 28040 Madrid, Spain

Received 31 August 2013; Accepted 11 March 2014; Published 22 April 2014

Academic Editors: J. Shu and F. Yu

Copyright © 2014 Mar Saneiro et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Linked References

  1. E. G. Blanchard, B. Volfson, Y.-J. Hong, and S. P. Lajoie, “Affective artificial intelligence in education: from detection to adaptation,” in Proceedings of the Conference on Artificial Intelligence in Education: Building Learning Systems That Care: from Knowledge Representation to Affective Modelling, pp. 81–88, Amsterdam, The Netherlands, 2009.
  2. O. C. Santos, J. G. Boticario, M. Arevalillo-Herraez et al., “MAMIPEC—affective modeling in inclusive personalized educational scenarios,” Bulletin of the IEEE Technical Committee on Learning Technology, vol. 14, no. 4, pp. 35–38, 2012. View at Google Scholar
  3. O. C. Santos, S. Salmeron-Majadas, and J. G. Boticario, “Emotions detection from math exercises by combining several data sources,” in Artificial Intelligence in Education, vol. 7926 of Lecture Notes in Computer Science, pp. 742–745, Springer, Berlin, Germany, 2013. View at Publisher · View at Google Scholar
  4. O. C. Santos, J. G. Boticario, and A. Manjarrés-Riesco, “An approach for an affective educational recommendation model,” in Recommender Systems for Technology Enhanced Learning: Research Trends & Applications, N. Manouselis, H. Drachsler, K. Verbert, and O. C. Santos, Eds., Springer, Berlin, Germany, 2014. View at Google Scholar
  5. M. Pantic and L. J. M. Rothkrantz, “Expert system for automatic analysis of facial expressions,” Image and Vision Computing, vol. 18, no. 11, pp. 881–905, 2000. View at Publisher · View at Google Scholar · View at Scopus
  6. R. W. Picard, Affective Computing, The MIT Press, Cambridge, UK, 1997.
  7. L. Shen, M. Wang, and R. Shen, “Affective e-Learning: using “emotional” data to improve learning in pervasive learning environment,” Educational Technology & Society, vol. 12, no. 2, pp. 176–189, 2009. View at Google Scholar · View at Scopus
  8. Z. Zeng, Y. Fu, G. Roisman, Z. Wen, Y. Hu, and T. Huang, “Spontaneous emotional facial expression,” Journal of Mutimedia, vol. 1, no. 5, 2006. View at Google Scholar
  9. S. Afzal and P. Robinson, “A study of affect in intelligent turoring,” in Proceedings of the 13th International Conference of Artificial Intelligence in Education (AIED '07), Marina Del Rey, Calif, USA, July 2007.
  10. J. Cohn and P. Ekman, “Measuring facial action by manual coding, facial EMG, and automatic facial image analysis,” in Handbook of Nonverbal Behavior Research Methods in the Affective Sciences, J. A. Harrigan, R. Rosenthal, and K. Scherer, Eds., pp. 9–64, 2005. View at Google Scholar
  11. P. Ekman and W. V. Friesen, Manual for the Facial Action Coding System, Consulting Psychologists Press, Palo Alto, Calif, USA, 1977.
  12. P. Ekman, W. Irwin, and E. L. Rosenberg, The Emotional Facial Action Coding System (EMFACS), 1994.
  13. C. Izard, The Maximally Descriminative Facial Movement Coding System (MAX), Instructional Resource Center, University of Delaware, Newark, Del, USA, 1979.
  14. C. Izard, L. Dougherty, and E. A. Hembree, System for Identifying Affect Expressions by Holistic Judgments (AFFEX), Instructional Resources Center, University of Delaware, Newark, Del, USA, 1983.
  15. J. Ahlberg, “CANDIDE-3—an updated parameterized face,” Tech. Rep. LiTH-ISY-R-2326, Department of Electrical Engineering, Linköping University, Linköping, Sweden, 2001. View at Google Scholar
  16. P. Ekman, E. Rosenberg, and J. Hager, “Facial action coding system affect interpretation database (FACSAID),” 1998, http://nirc.com/Expression/FACSAID/facsaid.html.
  17. S. V. Ioannou, A. T. Raouzaiou, V. A. Tzouvaras, T. P. Mailis, K. C. Karpouzis, and S. D. Kollias, “Emotion recognition through facial expression analysis based on a neurofuzzy network,” Neural Networks, vol. 18, no. 4, pp. 423–435, 2005. View at Publisher · View at Google Scholar · View at Scopus
  18. K. Mase, “Recognition of facial expression from optical flow,” IEICE Transactions on Information and Systems, vol. 74, no. 10, pp. 3474–3483, 1991. View at Google Scholar
  19. A. Lanitis, C. J. Taylor, and T. F. Cootes, “A unified approach to coding and interpreting face images,” in Proceedings of the 5th International Conference on Computer Vision, pp. 368–373, Cambridge, Mass, USA, June 1995. View at Scopus
  20. M. J. Black and Y. Yacoob, “Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion,” in Proceedings of the 5th International Conference on Computer Vision, pp. 374–381, June 1995. View at Scopus
  21. D. Freedman and T. Zhang, “Active contours for tracking distributions,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 518–526, 2004. View at Publisher · View at Google Scholar · View at Scopus
  22. H. T. Nguyen and A. W. M. Smeulders, “Fast occluded object tracking by a robust appearance filter,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 8, pp. 1099–1104, 2004. View at Publisher · View at Google Scholar · View at Scopus
  23. H.-T. Chen, T.-L. Liu, and C.-S. Fuh, “Probabilistic tracking with adaptive feature selection,” in Proceedings of the 17th International Conference on Pattern Recognition (ICPR '04), vol. 2, pp. 736–739, Washington, DC, USA, August 2004. View at Publisher · View at Google Scholar · View at Scopus
  24. A. U. Batur and M. H. Hayes, “Adaptive active appearance models,” IEEE Transactions on Image Processing, vol. 14, no. 11, pp. 1707–1721, 2005. View at Publisher · View at Google Scholar · View at Scopus
  25. P. Corcoran, M. C. Ionita, and I. Bacivarov, “Next generation face tracking technology using AAM techniques,” in Proceedings of the International Symposium on Signals, Circuits and Systems (ISSCS '07), vol. 1, pp. 1–4, Iași, Romania, July 2007. View at Publisher · View at Google Scholar · View at Scopus
  26. B. Fasel and J. Luettin, “Recognition of asymmetric facial action unit activities and intensities,” in Proceedings of the 15th International Conference on Pattern Recognition, pp. 1100–1103, Barcelona, Spain, 2000.
  27. L. Marco-Giménez, M. Arevalillo-Herráez, and C. Cuhna-Pérez, “Eigenexpressions: emotion recognition using multiple eigenspaces,” in Pattern Recognition and Image Analysis, vol. 7887 of Lecture Notes in Computer Science, pp. 758–765, Springer, Berlin, Germany, 2013. View at Publisher · View at Google Scholar
  28. M. Pardàs, A. Bonafonte, and J. L. Landabaso, “Emotion recognition based on MPEG-4 facial animation parameters,” in Proceedings of the IEEE International Conference on Acoustic, Speech, and Signal Processing, pp. 3624–3627, Orlando, Fla, USA, May 2002. View at Scopus
  29. A. Geetha, V. Ramalingam, S. Palanivel, and B. Palaniappan, “Facial expression recognition—a real time approach,” Expert Systems with Applications, vol. 36, no. 1, pp. 303–308, 2009. View at Publisher · View at Google Scholar · View at Scopus
  30. L. Ma and K. Khorasani, “Facial expression recognition using constructive feed forward neural networks,” IEEE Transactions on Systems, Man, and Cybernetics B: Cybernetics, vol. 34, no. 3, pp. 1588–1595, 2004. View at Publisher · View at Google Scholar
  31. I. Cohen, N. Sebe, F. G. Cozman, M. C. Cirelo, and T. S. Huang, “Learning Bayesian network classifiers for facial expression recognition using both labeled and unlabeled data,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. I-595–I-601, June 2003. View at Scopus
  32. S. Dubuisson, F. Davoine, and M. Masson, “A solution for facial expression representation and recognition,” Signal Processing: Image Communication, vol. 17, no. 9, pp. 657–673, 2002. View at Publisher · View at Google Scholar · View at Scopus
  33. X.-W. Chen and T. Huang, “Facial expression recognition: a clustering-based approach,” Pattern Recognition Letters, vol. 24, no. 9-10, pp. 1295–1302, 2003. View at Publisher · View at Google Scholar · View at Scopus
  34. A. Kleinsmith, P. R. de Silva, and N. Bianchi-Berthouze, “Cross-cultural differences in recognizing affect from body posture,” Interacting with Computers, vol. 18, no. 6, pp. 1371–1389, 2006. View at Publisher · View at Google Scholar · View at Scopus
  35. H. K. M. Meeren, C. C. R. J. van Heijnsbergen, and B. de Gelder, “Rapid perceptual integration of facial expression and emotional body language,” Proceedings of the National Academy of Sciences of the United States of America, vol. 102, no. 45, pp. 16518–16523, 2005. View at Publisher · View at Google Scholar · View at Scopus
  36. K. Schindler, L. van Gool, and B. de Gelder, “Recognizing emotions expressed by body pose: a biologically inspired neural model,” Neural Networks, vol. 21, no. 9, pp. 1238–1246, 2008. View at Publisher · View at Google Scholar · View at Scopus
  37. A. Kleinsmith and N. Bianchi-Berthouze, “Recognizing affective dimensions from body posture,” in Affective Computing and Intelligent, Lecture Notes in Computer Science, pp. 48–58, Springer, Berlin, Germany, 2007. View at Publisher · View at Google Scholar
  38. D. Bernhardt, Emotion inference from human body motion [Ph.D. thesis], Selwyn College, University of Cambridge, Cambridge, UK, 2010.
  39. H. Rosenfeld, “Measurement of body motion and orientation,” in Handbook of Methods in Nonverbal Behavior Research, K. Scherer and P. Ekman, Eds., pp. 199–286, Cambridge University Press, New York, NY, USA, 1982. View at Google Scholar
  40. N. Dael, M. Mortillaro, and K. R. Scherer, “The body action and posture coding system (BAP): development and reliability,” Journal of Nonverbal Behavior, vol. 36, no. 2, pp. 97–121, 2012. View at Publisher · View at Google Scholar · View at Scopus
  41. S. D. Kreibig, F. H. Wilhelm, W. T. Roth, and J. J. Gross, “Cardiovascular, electrodermal, and respiratory response patterns to fear- and sadness-inducing films,” Psychophysiology, vol. 44, no. 5, pp. 787–806, 2007. View at Publisher · View at Google Scholar · View at Scopus
  42. D. Glowinski, N. Dael, A. Camurri, G. Volpe, M. Mortillaro, and K. R. Scherer, “Towards a minimal representation of affective gestures,” IEEE Transactions on Affective Computing, vol. 2, no. 2, pp. 106–118, 2011. View at Publisher · View at Google Scholar
  43. J. A. Harrigan, “Proxemics, kinesics, and gaze,” in The New Handbook of Methods in Nonverbal Behavior Research, J. A. Harrigan, R. Rosenthal, and K. Scherer, Eds., pp. 137–198, Oxford University Press, New York, NY, USA, 2005. View at Google Scholar
  44. M. A. Schmuckler, “What is ecological validity? A dimensional analysis,” Infancy, vol. 2, no. 4, pp. 419–436, 2001. View at Google Scholar · View at Scopus
  45. S. Afzal and P. Robinson, “Designing for automatic affect inference in learning environments,” Educational Technology & Society, vol. 14, no. 4, pp. 21–34, 2011. View at Google Scholar · View at Scopus
  46. M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A multimodal database for affect recognition and implicit tagging,” IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 42–55, 2012. View at Publisher · View at Google Scholar · View at Scopus
  47. M. Szwoch, “FEEDB: a multimodal database of facial expressions and emotions,” in Proceedings of the 6th International Conference on Human System Interaction (HSI '13), pp. 524–531, June 2013.
  48. O. C. Santos and J. G. Boticario, “Involving users to improve the collaborative logical framework,” The Scientific World Journal, vol. 2014, Article ID 893525, 15 pages, 2014. View at Publisher · View at Google Scholar
  49. O. C. Santos, J. G. Boticario, R. Cabestrero, and P. Quiros, “Extending collaborative learning modeling with emotional information,” in Proceedings of the Workshops at the 16th International Conference on Artificial Intelligence in Education (AIED '13), vol. 1009 of CEUR Workshop Proceedings, pp. 25–28, Memphis, Tenn, USA, July 2013.
  50. O. C. Santos, A. Rodríguez-Ascaso, J. G. Boticario, S. Salmeron-Majadas, P. Quirós, and R. Cabestrero, “Challenges for inclusive affective detection in educational scenarios,” in Universal Access in Human-Computer Interaction. Design Methods, Tools, and Interaction Techniques for eInclusion, vol. 8009 of Lecture Notes in Computer Science, pp. 566–575, Springer, Berlin, Germany, 2013. View at Publisher · View at Google Scholar
  51. O. C. Santos, J. G. Boticario, E. Raffenne, and R. Pastor, “Why using dotLRN? UNED use cases,” in Proceedings of the FLOSS (Free/Libre/Open Source Systems) International Conference, J. R. Rodríguez and M. Palomo, Eds., pp. 195–212, 2007. View at Google Scholar
  52. O. P. John, E. M. Donahue, and R. L. Kentle, The Big Five Inventory—Versions 4a and 54, University of California, Berkeley Institute of Personality and Social Research, Berkeley, Calif, USA, 1991.
  53. R. Schwarcer, Measurement of Perceived Self-Efficacy. Psychometric Scales for Cross-Cultural Research, Freie Universität, Berlin, Germany, 1993.
  54. D. Watson, L. A. Clark, and A. Tellegen, “Development and validation of brief measures of positive and negative affect: the PANAS scales,” Journal of Personality and Social Psychology, vol. 54, no. 6, pp. 1063–1070, 1988. View at Google Scholar · View at Scopus
  55. J. L. Robison, S. W. McQuiggan, and J. C. Lester, “Developing empirically based student personality profiles for affective feedback models,” in Intelligent Tutoring Systems, vol. 6094 of Lecture Notes in Computer Science, pp. 285–295, Springer, Berlin, Germany, 2010. View at Publisher · View at Google Scholar
  56. M. M. Bradley, “Measuring emotion: the self-assessment manikin and the semantic differential,” Journal of Behavior Therapy and Experimental Psychiatry, vol. 25, no. 1, pp. 49–59, 1994. View at Publisher · View at Google Scholar · View at Scopus
  57. A. Mehrabian, “Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament,” Current Psychology, vol. 14, no. 4, pp. 261–292, 1996. View at Google Scholar · View at Scopus
  58. P. J. Lang, M. M. Bradley, and B. N. Cuthbert, “International affective picture system (IAPS): affective ratings of pictures and instruction manual,” Tech. Rep. A-8, University of Florida, Gainesville, Fla, USA, 2008. View at Google Scholar
  59. S. Salmeron-Majadas, O. C. Santos, and J. G. Boticario, “Exploring indicators from keyboard and mouse interactions to predict the user affective state,” in Proceedings of the 7th International Conference on Educational Data Mining (EDM '14), 2014.
  60. Emotion Markup Language (EmotionML) 1.0, W3C Candidate Recommendation, Cambridge, Mass, USA, M. Schröder, Eds., 2012.
  61. N. Dael, M. Mortillaro, and K. R. Scherer, “The body action and posture coding system (BAP): development and reliability,” Journal of Nonverbal Behavior, vol. 36, no. 2, pp. 97–121, 2012. View at Publisher · View at Google Scholar · View at Scopus
  62. A. Pentland, B. Moghaddam, and T. Starner, “View-based and modular eigenspaces for face recognition,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 84–91, June 1994. View at Scopus
  63. O. C. Santos and J. G. Boticario, “Affective issues in semantic educational recommender systems,” in Proceedings of the 2nd Workshop on Recommender Systems in Technology Enhanced Learning, vol. 896 of CEUR Workshop Proceedings, pp. 71–82, Saarbrücken, Germany, September 2012.
  64. M. Arevalillo-Herráez, S. Moreno-Picot, D. Arnau et al., “Towards enriching an ITS with affective support,” in Proceedings of the Personalization Approaches in Learning Environments (PALE '13), vol. 997 of CEUR Workshop Proceedings, pp. 5–13, 2013.
  65. M. Arevalillo-Herráez, D. Arnau, and L. Marco-Giménez, “Domain-specific knowledge representation and inference engine for an intelligent tutoring system,” Knowledge-Based Systems, vol. 49, pp. 97–105, 2013. View at Publisher · View at Google Scholar
  66. O. C. Santos and J. G. Boticario, “User centred design and educational data mining support during the recommendations elicitation process in social online learning environments,” Expert Systems, 2013. View at Publisher · View at Google Scholar
  67. O. C. Santos and J. G. Boticario, “Affective issues in semantic educational recommender systems,” in Proceedings of the 2nd Workshop on Recommender Systems in Technology Enhanced Learning, vol. 896 of CEUR Workshop Proceedings, pp. 71–82, Saarbrücken, Germany, September 2012.