Table of Contents Author Guidelines Submit a Manuscript
Advances in Human-Computer Interaction
Volume 2013, Article ID 136864, 11 pages
http://dx.doi.org/10.1155/2013/136864
Review Article

Assessment in and of Serious Games: An Overview

1Department of Naval, Electric, Electronic and Telecommunications Engineering, University of Genoa, Via all’Opera Pia 11/a, 16145 Genoa, Italy
2Faculty of Business and Information Technology, University of Ontario Institute of Technology, 2000 Simcoe Street North, Oshawa, Canada L1H 7K4
3Department of Mechanical and Aerospace Engineering, Case Western Reserve University, 10900 Euclid Avenue, Cleveland, OH 44106, USA
4Faculty of Computer Science, Universidad Complutense de Madrid, Ciudad Universitaria, 28040 Madrid, Spain

Received 18 May 2012; Revised 22 January 2013; Accepted 6 February 2013

Academic Editor: Armando Bennet Barreto

Copyright © 2013 Francesco Bellotti et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Linked References

  1. J. P. Gee, What Video Games Have to Teach Us about Learning and Literacy, Palgrave MacMillan, New York, NY, USA, 2007.
  2. F. L. Greitzer, O. A. Kuchar, and K. Huston, “Cognitive science implications for enhancing training effectiveness in a serious gaming context,” ACM Journal on Educational Resources in Computing, vol. 7, no. 3, article 2, 2007. View at Publisher · View at Google Scholar · View at Scopus
  3. F. De Grove, P. Mechant, and J. Van Looy, “Uncharted waters? Exploring experts' opinions on the opportunities and limitations of serious games for foreign language learning,” in Proceedings of the 3rd International Conference on Fun and Games, pp. 107–115, Leuven, Belgium, September 2010. View at Publisher · View at Google Scholar · View at Scopus
  4. R. van Eck, “Digital game-based learning: it's not just the digital natives who are restless,” EDUCAUSE Review, vol. 41, no. 2, pp. 16–30, 2006. View at Google Scholar
  5. J. Cannon-Bowers, “The state of gaming and simulation,” in Proceedings of the Training Conference and Expo, Orlando, Fla, USA, March 2006.
  6. V. Shute, M. Ventura, M. Bauer, and D. Zapata-Rivera, “Melding the power of serious games and embedded assessment to monitor and foster learning: flow and grow,” in Serious Games: Mechanisms and Effects, U. Ritterfeld, M. Cody, and P. Vorderer, Eds., pp. 295–321, Routledge, Taylor and Francis, Mahwah, NJ, USA, 2009. View at Google Scholar
  7. J. Gosen and J. Washbush, “A review of scholarship on assessing experiential learning effectiveness,” Simulation & Gaming, vol. 35, no. 2, pp. 270–293, 2004. View at Publisher · View at Google Scholar · View at Scopus
  8. R. T. Hays, “The effectiveness of instructional games: a literature review and discussion,” Tech. Rep. 2005-004, Naval Air Warfare Center, Training Systems Division, 2005. View at Google Scholar
  9. A. A. Kulik, “School mathematics and science programs benefit from instructional technology,” United States National Science Foundation (NSF), National Center for Science and Engineering Statistics (NCSES), InfroBrief NSF-03-301, November 2002, http://www.nsf.gov/statistics/infbrief/nsf03301/.
  10. R. Blunt, “Do serious games work? Results from three studies,” eLearn Magazine, vol. 2009, no. 12, 2009. View at Publisher · View at Google Scholar
  11. B. Bergeron, Developing Serious Games, Thomson Delmar Learning, Hingham, Mass, USA, 2006.
  12. M. Prensky, Don't Bother Me Mom—I'm Learning!, Paragon House, 2006.
  13. S. Livingston, G. Fennessey, J. Coleman, K. Edwards, and S. Kidder, “The Hopkins games program: final report on seven years of research,” Report No. 155, Johns Hopkins University, Center for Social Organization of Schools, Baltimore, Md, USA, 1973. View at Google Scholar
  14. J. Chin, R. Dukes, and W. Gamson, “Assessment in simulation and gaming: a review of the last 40 years,” Simulation & Gaming, vol. 40, no. 4, pp. 553–568, 2009. View at Publisher · View at Google Scholar · View at Scopus
  15. T. M. Connolly, E. A. Boyle, E. MacArthur, T. Hainey, and J. M. Boyle, “A systematic literature review of the empirical evidence on computer games and serious games,” Computers and Education, vol. 59, no. 2, pp. 661–686, 2012. View at Google Scholar
  16. P. M. Kato, S. W. Cole, A. S. Bradlyn, and B. H. Pollock, “A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial,” Pediatrics, vol. 122, no. 2, pp. e305–e317, 2008. View at Publisher · View at Google Scholar · View at Scopus
  17. S. W. Cole, D. J. Yoo, and B. Knutson, “Interactivity and reward-related neural activation during a serious videogame,” PLoS ONE, vol. 7, no. 3, Article ID e33909, 2012. View at Publisher · View at Google Scholar
  18. S. D. Dandeneau and M. W. Baldwin, “The inhibition of socially rejecting information among people with high versus low self-esteem: the role of attentional bias and the effects of bias reduction training,” Journal of Social and Clinical Psychology, vol. 23, no. 4, pp. 584–602, 2004. View at Publisher · View at Google Scholar · View at Scopus
  19. F. Bellotti, R. Berta, A. De Gloria, A. D'Ursi, and V. Fiore, “A serious game model for cultural heritage,” Journal on Computing and Cultural Heritage, vol. 5, no. 4, pp. 1–27, 2012. View at Google Scholar
  20. F. Bellotti, R. Berta, and A. De Gloria, “Designing effective serious games: opportunities and challenges for research,” International Journal of Emerging Technologies in Learning, vol. 5, pp. 22–35, 2010. View at Google Scholar
  21. G. Bente and J. Breuer, “Making the implicit explicit: embedded measurement in serious games,” in Serious Games: Mechanisms and Effects, U. Ritterfield, M. J. Cody, and P. Vorderer, Eds., pp. 322–343, Routledge, New York, NY, USA, 2009. View at Google Scholar
  22. D. Michael and S. Chen, “Proof of learning: assessment in serious games,” October 2005, http://www.gamasutra.com/view/feature/2433/proof_of_learning_assessment_in_.php.
  23. J. Enfield, R. D. Myers, M. Lara, and T. W. Frick, “Innovation diffusion: assessment of strategies within the diffusion simulation game,” Simulation & Gaming, vol. 43, no. 2, pp. 188–214, 2012. View at Publisher · View at Google Scholar
  24. C. Boston, “The concept of formative assessment,” Practical Assessment, Research & Evaluation, vol. 8, no. 9, 2002. View at Google Scholar
  25. P. Moreno-Ger, D. Burgos, and J. Torrente, “Digital games in eLearning environments: current uses and emerging trends,” Simulation & Gaming, vol. 40, no. 5, pp. 669–687, 2009. View at Google Scholar
  26. C. Sebastian, A. Anantachai, J. H. Byun, and J. Lenox, “Assessing what players learned in serious games: in-situ data collection, information trails, and quantitative analysis,” in Proceedings of the 10th International Conference on Computer Games: AI, Animation, Mobile, Educational and Serious Games, pp. 10–19, 2007.
  27. K. Becker and J. R. Parker, The Guide to Computer Simulations and Games, John Wiley & Sons, Indianapolis, Ind, USA, 2011.
  28. P. Dugard and J. Todman, “Analysis of pre-test-post-test control group designs in educational research,” Educational Psychology, vol. 15, no. 2, pp. 181–198, 1995. View at Google Scholar
  29. National Center for Technology Innovation (NCTI), “Experimental Study Design,” 2012, http://www.nationaltechcenter.org/index.php/products/at-research-matters/experimental-study-design/.
  30. L. Allen, M. Seeney, L. Boyle, and F. Hancock, “The implementation of team based assessment in serious games,” in Proceedings of the 1st Conference in Games and Virtual Worlds for Serious Applications (VS-GAMES '09), pp. 28–35, Coventry, UK, March 2009. View at Publisher · View at Google Scholar · View at Scopus
  31. K. Corti, Game-Based Learning: A Serious Business Application, PIXELearning, Coventry, UK, 2006.
  32. J. H. Brockmyer, C. M. Fox, K. A. Curtiss, E. McBroom, K. M. Burkhart, and J. N. Pidruzny, “The development of the Game Engagement Questionnaire: a measure of engagement in video game-playing,” Journal of Experimental Social Psychology, vol. 45, no. 4, pp. 624–634, 2009. View at Publisher · View at Google Scholar · View at Scopus
  33. W. A. IJsselsteijn, W. van de Hoogen, C. Klimmt et al., “Measuring the experience of digital game enjoyment,” in Proceedings of Measuring Behavior, pp. 88–89, Maastricht, The Netherlands, August 2008.
  34. M. Csikszentmihalyi, Flow: The Psychology of Optimal Experience, Harper & Row, New York, NY, USA, 1990.
  35. B. Cowley, D. Charles, M. Black, and R. Hickey, “Toward an understanding of flow in video games,” Computers in Entertainment, vol. 6, no. 2, pp. 1–28, 2008. View at Publisher · View at Google Scholar · View at Scopus
  36. P. Sweetser and P. Wyeth, “GameFlow: a model for evaluating player enjoyment in games,” ACM Computers in Entertainment, vol. 3, no. 3, pp. 1–24, 2005. View at Google Scholar
  37. F. L. Fu, R. C. Su, and S. C. Yu, “EGameFlow: a scale to measure learners' enjoyment of e-learning games,” Computers and Education, vol. 52, no. 1, pp. 101–112, 2009. View at Publisher · View at Google Scholar · View at Scopus
  38. S. H. Janicke and A. Ellis, “Psychological and physiological differences between the 3D and 2D gaming experience,” in Proceedings of the 3D Entertainment Summit, Hollywood, Calif, USA, September, 2011.
  39. H. F. Jelinek, K. August, H. Imam, A. H. Khandoker, A. Koenig, and R. Riener, “Heart rate asymmetry and emotional response to robot assist task challenges in stroke patients,” in Proceedings of the Computing in Cardiology Conference, Hangzhou, China, September 2011.
  40. J. M. Kivikangas, G. Chanel, B. Cowley et al., “A review of the use of psychophysiological methods in game research,” Journal of Gaming & Virtual Worlds, vol. 3, no. 3, pp. 181–199, 2011. View at Google Scholar
  41. L. E. Nacke, Affective ludology: scientific measurement of user experience in interactive entertainment [Ph.D. thesis], Blekinge Institute of Technology, Karlskrona, Sweden, 2009.
  42. A. Plotnikov, N. Stakheika, A. De Gloria et al., “Exploiting real-time EEG analysis for assessing flow in games,” in Workshop: “Game Based Learning for 21st Century Transferable Skills”, at iCalt 2012, Rome, Italy, June 2012.
  43. L. E. Nacke, “Physiological game interaction and psychophysiological evaluation in research and industry,” Gamasutra Article, June 2011, http://www.gamasutra.com/blogs/LennartNacke/20110628/7867/Physiological_Game_Interaction_and_Psychophysiological_Evaluation_in_Research_and_Industry.php.
  44. M. Salminen and N. Ravaja, “Oscillatory brain responses evoked by video game events: the case of super monkey ball 2,” Cyberpsychology & Behavior, vol. 10, no. 3, pp. 330–338, 2007. View at Publisher · View at Google Scholar · View at Scopus
  45. C. Loh, “Designing online games assessment as information trails,” in Games and Simulations in Online Learning: Research and Development Frameworks, D. Gibson, C. Aldrich, and M. Prensky, Eds., pp. 323–348, Information Science Publishing, Hershey, Pa, USA, 2007. View at Google Scholar
  46. G. N. Yannakakis and J. Hallam, “Evolving opponents for interesting interactive computer games,” in Proceedings of the International Conference on Computer Games: Artificial Intelligence, Design and Education, 2004.
  47. T. W. Malone, “Toward a theory of intrinsically motivating instruction,” Cognitive Science, vol. 5, no. 4, pp. 333–369, 1981. View at Google Scholar · View at Scopus
  48. H. Iida, N. Takeshita, and J. Yoshimura, “A metric for entertainment of boardgames: its implication for evolution of chess variants,” in Proceeding of: Entertainment Computing: Technologies and Applications, IFIP First International Workshop on Entertainment Computing (IWEC '02), R. Nakatsu and J. Hoshino, Eds., pp. 65–72, Kluwer Academic, Boston, Mass, USA, 2003. View at Google Scholar
  49. F. Bellotti, R. Berta, A. De Gloria, and L. Primavera, “Enhancing the educational value of video games,” Computers in Entertainment, vol. 7, no. 2, pp. 23–41, 2009. View at Publisher · View at Google Scholar · View at Scopus
  50. M. Hassenzahl and R. Wessler, “Capturing design space from a user perspective: the repertory grid technique revisited,” International Journal of Human-Computer Interaction, vol. 12, no. 3-4, pp. 441–459, 2000. View at Google Scholar · View at Scopus
  51. N. Zoanetti, “Software for online testing and quizzes,” 2011, http://www.assessmentfocus.com/online-testing.php.
  52. A. Flynn, F. Concannon, and M. Campbell, “An evaluation of undergraduate students' online assessment performances,” Advanced Technology for Learning, vol. 3, no. 1, pp. 15–51, 2006. View at Google Scholar
  53. C. Hewson, “Can online course-based assessment methods be fair and equitable? Relationships between students' preferences and performance within online and offline assessments,” Journal of Computer Assisted Learning, vol. 28, no. 5, pp. 488–498, 2001. View at Publisher · View at Google Scholar
  54. E. Guzmán, R. Conejo, and J. L. Pérez-de-la-Cruz, “Improving student performance using self-assessment tests,” IEEE Intelligent Systems, vol. 22, no. 4, pp. 46–52, 2007. View at Publisher · View at Google Scholar · View at Scopus
  55. J. Hattie and D. Masters, “asTTle—Assessment Tools for Teaching and Learning,” HEFCE JISC, 2006, http://www.jisc.ac.uk/media/documents/projects/asttle_casestudy.pdf.
  56. J. Hattie, G. Brown, P. Keegan et al., “Validation evidence of asTTle reading assessment results: norms and criteria,” Asttle Tech. Rep. 22, University of Auckland/Ministry of Education, November 2003. View at Google Scholar
  57. J. Hattie, “Large-scale assessment of student competencies,” in Symposium: Working in Today's World of Testing and Measurement: Required Knowledge and Skills (Joint ITC/CPTA Symposium); the 26th International Congress of Applied Psychology, Athens, Greece, July 2006.
  58. Questionmark Corporation, “Questionmark Perception Measure Knowledge, Skills and Attitudes Securely for Certification, Regulatory Compliance and successful Learning Outcomes,” 2012. View at Google Scholar
  59. J. Bull and D. Stephens, “The use of question mark software for formative and summative assessment in two universities,” Innovations in Education and Teaching International, vol. 36, no. 2, pp. 128–135, 1999. View at Google Scholar · View at Scopus
  60. G. M. Velan, R. K. Kumar, M. Dziegielewski, and D. Wakefield, “Web-based self-assessments in pathology with Questionmark Perception,” Pathology, vol. 34, no. 3, pp. 282–284, 2002. View at Publisher · View at Google Scholar · View at Scopus
  61. HEFCE JISC, “Case study 5: making the most of a computer-assisted assessment system University of Manchester,” 2010, http://www.jisc.ac.uk/media/documents/programmes/elearning/digiassess_makingthemost.pdf.
  62. M. Wood, Human Computer Collaborative Assessment—Access by Computer (ABC)—University of Manchester, HEFCE JISC, 2009.
  63. HEFCE JISC, “Short answer marking engines,” 2009, http://www.jisc.ac.uk/media/documents/projects/shorttext.pdf.
  64. S. Jordan and T. Mitchell, “e-Assessment for learning? The potential of short-answer free-text questions with tailored feedback,” British Journal of Educational Technology, vol. 40, no. 2, pp. 371–385, 2009. View at Publisher · View at Google Scholar · View at Scopus
  65. F. Noorbehbahani and A. A. Kardan, “The automatic assessment of free text answers using a modified BLEU algorithm,” Computers & Education, vol. 56, no. 2, pp. 337–345, 2011. View at Publisher · View at Google Scholar · View at Scopus
  66. I. D. Beatty and W. J. Gerace, “Technology-enhanced formative assessment: a research-based pedagogy for teaching science with classroom response technology,” Journal of Science Education and Technology, vol. 18, no. 2, pp. 146–162, 2009. View at Publisher · View at Google Scholar · View at Scopus
  67. C. Fies and J. Marshall, “Classroom response systems: a review of the literature,” Journal of Science Education and Technology, vol. 15, no. 1, pp. 101–109, 2006. View at Publisher · View at Google Scholar · View at Scopus
  68. C. Fies and J. Marshall, “The C3 framework: evaluating classroom response system interactions in university classrooms,” Journal of Science Education and Technology, vol. 17, no. 5, pp. 483–499, 2008. View at Publisher · View at Google Scholar · View at Scopus
  69. IMS Global Learning Consortium, “IMS Question & Test Interoperability Specification (QTI),” 2012, http://www.imsglobal.org/question/.
  70. C. Smythe and P. Roberts, An Overview of the IMS Question & Test Interoperability Specification, Computer Aided Assessment, Leicestershire, UK, 2000.
  71. Codility Ltd., “Codility: WE TEST CODERS,” 2009, http://codility.com/.
  72. About IKM, “Overview,” 2011, http://www.ikmnet.com/about/overview.cfm.
  73. M. M. Clarke, G. F. Madaus, C. L. Horn, and M. A. Ramos, “Retrospective on educational testing and assessment in the 20th century,” Journal of Curriculum Studies, vol. 32, no. 2, pp. 159–181, 2000. View at Google Scholar · View at Scopus
  74. L. B. Resnick and D. P. Resnick, Assessing the Thinking Curriculum: New Tools for Educational Reform, Learning Research and Development Center: University of Pittsburgh and Carnegie Mellon University, Pittsburgh, Pa, USA, 1989.
  75. M. Lipman, “Some thoughts on the formation of reflective education,” in Teaching-Thinking Skills: Theory and Practice, J. B. Baron and R. J. Sternberg, Eds., pp. 151–161, W. H. Freeman, New York, NY, USA, 1987. View at Google Scholar
  76. C. Tribune, “Standardized testing will limit students' future,” April 2010, http://articles.chicagotribune.com/2010-04-21/news/chi-100421shafer_briefs_1_standardized-test-scores-teacher-and-principal-evaluations.
  77. Fairtest, “What's Wrong with Standardized Tests?” May 2012, http://www.fairtest.org/facts/whatwron.htm.
  78. E. J. Short, M. Noeder, S. Gorovoy, M. J. Manos, and B. Lewis, “The importance of play in both the assessment and treatment of young children,” in An Evidence-Based Approach to Play in Intervention and Prevention: Integrating Developmental and Clinical Science, S. Russ and L. Niec, Eds., Guilford, London, UK.
  79. A. S. Kaugars and S. W. Russ, “Assessing preschool children's pretend play: preliminary validation of the affect in play scale-preschool version,” Early Education and Development, vol. 20, no. 5, pp. 733–755, 2009. View at Publisher · View at Google Scholar · View at Scopus
  80. F. Bellotti, R. Berta, A. De Gloria, and L. Primavera, “Adaptive experience engine for serious games,” IEEE Transactions on Computational Intelligence and AI in Games, vol. 1, no. 4, pp. 264–280, 2009. View at Publisher · View at Google Scholar · View at Scopus
  81. C. H. Tan, K. C. Tan, and A. Tay, “Dynamic game difficulty scaling using adaptive behavior-based AI,” IEEE Transactions on Computational Intelligence and AI in Games, vol. 3, no. 4, pp. 289–301, 2011. View at Publisher · View at Google Scholar
  82. L. Doucet and V. Srinivasany, “Designing entertaining educational games using procedural rhetoric: a case study,” in Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games, pp. 5–10, Los Angeles, Calif, USA, July 2010. View at Publisher · View at Google Scholar · View at Scopus
  83. J. Froschauer, I. Seidel, M. Gärtner, H. Berger, and D. Merkl, “Design and evaluation of a serious game for immersive cultural training,” in Proceedings of the 16th International Conference on Virtual Systems and Multimedia (VSMM '10), pp. 253–260, IEEE CS Press, Seoul, Republic of Korea, October 2010. View at Publisher · View at Google Scholar · View at Scopus
  84. H. Kelly, K. Howell, E. Glinert et al., “How to build serious games,” Communications of the ACM, vol. 50, no. 7, pp. 44–49, 2007. View at Publisher · View at Google Scholar · View at Scopus
  85. J. Swarz, A. Ousley, A. Magro et al., “CancerSpace: a simulation-based game for improving cancer-screening rates,” IEEE Computer Graphics and Applications, vol. 30, no. 1, pp. 90–94, 2010. View at Publisher · View at Google Scholar · View at Scopus
  86. M. A. Zielke, M. J. Evans, F. Dufour et al., “Serious games for immersive cultural training: creating a living world,” IEEE Computer Graphics and Applications, vol. 29, no. 2, pp. 49–60, 2009. View at Publisher · View at Google Scholar · View at Scopus
  87. M. King and R. Newman, “Evaluating business simulation software: approach, tools and pedagogy,” On the Horizon, vol. 17, no. 4, pp. 368–377, 2009. View at Publisher · View at Google Scholar · View at Scopus
  88. A. J. Stainton, J. E. Johnson, and E. P. Borodzicz, “Educational validity of business gaming simulation: a research methodology framework,” Simulation & Gaming, vol. 41, no. 5, pp. 705–723, 2010. View at Publisher · View at Google Scholar · View at Scopus
  89. U. Ritterfeld, M. Cody, and P. Vorderer, Eds., Serious Games: Mechanisms and Effects, Routledge, New York, NY, USA, 2009.
  90. F. Bellotti, R. Berta, A. De Gloria, and L. Primavera, “A task annotation model for SandBox Serious Games,” in Proceedings of IEEE Symposium on Computational Intelligence and Games (CIG '09), pp. 233–240, Milano, Italy, September 2009. View at Publisher · View at Google Scholar · View at Scopus