Abstract
This study aims at reviewing the published scientific literature on the topics of a game development-based learning (GDBL) method using game development frameworks (GDFs) with the perspective of (a) summarizing a guideline for using GDBL in a curriculum, (b) identifying relevant features of GDFs, and (c) presenting a synthesis of impact factors with empirical evidence on the educational effectiveness of the GDBL method. After systematically going through the available literature on the topic, 34 relevant articles were selected for the final study. We analyzed the articles from three perspectives: (1) pedagogical context and teaching process, (2) selection of GDFs, and (3) evaluation of the GDBL method. The findings from the 34 articles suggest that GDFs have many potential benefits as an aid to teach computer science, software engineering, art design, and other fields and that such GDFs combined with the motivation from games can improve the students’ knowledge, skills, attitudes, and behaviors in contrast to the traditional classroom teaching. Furthermore, based on the results of the literature review, we extract a guideline of how to apply the GDBL method in education. The empirical evidence of current findings gives a positive overall picture and can provide a useful reference to educators, practitioners, and researchers in the area of game-based learning.
1. Introduction
Computer games and video games have become very popular in children and adolescents’ life and play a prominent role in the culture of young people [1]. Games can now be played everywhere in technology-rich environments equipped with laptops, smart phones, game consoles (mobile and stationary), set-top, boxes and other digital devices. From this phenomenon, it is believed that the intrinsic motivation that young people shows towards games can be combined with educational content and objectives into what Prensky calls “digital game based learning” [2].
Besides of an abundant appearance of games in young students life, game development technology has matured and became more advanced than before [3]. Based on various existing game development software, the whole duty of game development process can be divided into several domains and roles such as game programmers, 3D model creators, game designers, musicians, animators, and play writers. Under this situation, some web resources and game engines can simplify the game development process. For instance, Microsoft’s XNA game development kit provides the game loop function to draw and update the game contents, and it also provides convenient game development components to load the different format of graphics, audio, and videos. This makes it possible for students to modify existing games or develop their own new games with or without programming. They can design and implement their own game concepts with these game creation tools, learn the developing skills and relevant knowledge, and accumulate related practical experience.
In this context, not only can a game be used for learning, but also the game development tools be used for studying relevant topics within computer science, software engineering (SE), or game programming through motivating assignments. Generally, games can be integrated in education in three ways [4, 5]. First, games can be used instead of traditional exercises motivating students to put extra effort in doing the exercises and giving the teacher and/or teaching assistants an opportunity to monitor how the students work with the exercises in real time, for example, [6, 7]. Second, games can be played within lectures to improve the participation and motivation of students, for example, [8, 9]. Third, the students are required to modify or develop a game as a part of a course using a game development framework (GDF) to learn skills within computer science and SE, for example, [10]. And we label this third as game development-based learning (GDBL). And the GDFs denote the toolkits that can be used to develop or modify games, for example, game engine, game editors, or game (simulation) platforms, or even any integrated development environment (IDE), like Visual C++, Eclipse, J2ME, and Android SDK since all of them can be used to build games. This paper focuses on using the GDBL method in education, where GDFs are used in student exercises to learn skills, extending the use of GDFs as a teaching aid. The motivation for teaching through game development is to utilize the students’ enthusiasm for games. This GDBL method is not new. The earliest similar application of learning through programming in a game-like environment was in early 1970s. The logo [11], the turtle graphics, is one of the oldest libraries that was used to introduce computing concepts to beginners. The concept was based on a “turtle” that could be moved across a 2D screen with a pen, which could be positioned on or off the screen and, thus, may leave a trace of the turtle’s movements. Programming the turtle to draw different patterns can be used to introduce general computing skill, such as procedural operations, iteration, and recursion. Further, in 1987, Micco presented the usage of writing a tic-tac-toe game for learning [12]. After several years of development, we believe that GDBL methods have been improved through the development of technology. Thus, we investigate how GDFs are being used in education through a literature survey and investigate how traditional lectures can become more dynamic, collaborative, and attractive to the students utilizing the current technology rich environment. However, this assertion needs to be further supported by relevant theory, application experiences, evaluation results, and empirical evidence. Nevertheless, to the best of the authors’ knowledge, there does not exist any comprehensive literature reviews on the application of the GDBL method so far.
The aim of the study is to review the recently published literature on the use of GDFs in education to(a)summarize a guideline for how to use GDBL in a curriculum, (b)identify the features of GDFs related to GDBL,(c)present a synthesis of impact factors with the empirical evidence on the educational effectiveness of the GDBL method.
The study is unique in that it presents an overview of the recently published literature on the use of GDFs in education, while taking into account both game engines and relevant toolkits to create/modify games or game-like systems (e.g., simulators). The study can provide useful guidance to teachers at different educational levels or areas, as well as to educators, practitioners, and researchers in the areas of game-based education.
The paper is organized as follows. Section 2 describes the method used for carrying out the systematic review of articles, Section 3 presents the results from the literature review, Section 4 extracts a guideline for GDBL according to the existing literature, and finally Section 5 concludes the paper.
2. Method
Informed by the established method of systematic review [13, 14], the review was undertaken in distinct stages: the development of review protocol, the identification of inclusion and exclusion criteria, a search for relevant studies, critical appraisal, data extraction, and synthesis.
2.1. Protocol Development
We developed a protocol for the systematic review by following the guidelines, procedures and policies of the Campbell Collaboration (http://www.campbellcollaboration.org/), the Cochrane Handbook for Systematic Reviews of Interventions [13], the University of York’s Centre for Reviews and Dissemination’s guidance for those carrying out or commissioning reviews [14], and also refer to reviews on serious game research [15, 16]. This protocol specified the research aim, search strategy, inclusion, exclusion criteria, data extraction, and methods of synthesis.
2.2. Data Source and Search Strategy
For the purpose of the study, a literature search was undertaken in December 2010 in the following international online bibliographic databases: (a) ACM portal, (b) IEEE Xplore, (c) Springer, (d) Science direct. The search string used was (“Game”) AND (“Learning” OR “Teaching”) AND (“Lecture” OR “Curriculum” OR “Lesson” OR “Course” OR “Exercise”). And “education” was not included in the keyword list since we considered that education was a quite general word and did not help minimize the searching scope. Searches were limited to titles and abstracts of articles published in journals and conference proceedings (some are book chapters), in English, from 2000 and onwards. The latter limitation was posed due to the rapid changes in ICT (Information and Communications Technology) in general, and in computer game development technologies in particular.
2.3. Data Extraction with Inclusion and Exclusion Criteria
Figure 1 shows the complete process of the data extraction. The first step was to identify relevant studies. A number of journal and proceedings articles about GDBL were located during searches in the afore-mentioned databases. The articles were examined and the search resulted in 1155 articles. In step 2, from abstracts of each article, we distinguished learning through game play or game development. And most of the excluded articles were using games directly in the classroom to motivate the students’ interest and attendance rate and using game play instead of traditional exercises to study or review the course content. For instance, these were articles generally addressing using virtual online multiplayer game environments to provide a collaborative learning style, for example, [17, 18], articles which referred to games used in the classroom to motivate attendance and to review the course knowledge, for example, [8]. In addition, the articles related to the economics terms “game theory” and “business game” used as business terms were also excluded from this category. Besides, we excluded articles that depicted novel game concepts that were not computer or video games but physical game activities without any technology support for the lecture. For instance, the article in [19] used a self-made table card game in SE education. Mainly based on these three criteria, a total of 1010 articles were excluded after this step.
In step 3, the whole content of the articles was checked. The inclusion criteria were further limited to the scope: a case study or several case studies in the article to describe GDBL. In particular, it required (a) a relatively detailed description of the lecture design process. The articles without a detailed description of their teaching design or exercise process made it impossible to validate their process of how to integrate GDFs in lectures or exercises. According to this requirement, posters, tutorial presentations, and some short papers without detailed description on teaching process were excluded since they could not provide valuable data for our research aim and made it impossible to validate the effectiveness of the method, for example, [20–25]. This was also a measure to ensure the inclusion of the high quality literature in the review. (b) Articles using development toolkits in the curriculums but did not aim to develop games were also excluded, for example, [26]. (c) Articles emphasizing on other aspects apart from GDBL were excluded as it was difficult to validate how game development was integrated in class, for example, learning in an interactive e-lab [27]. Similarly, articles that presented the development of an educational game framework but did not mention how it was integrated in a specific curriculum were excluded, for example, [28–31]. (d) Articles, which focused on changing the controller of the software or hardware, but without elements of computer game development, were also excluded, for example, [32, 33]. Most of them focus on creating a robot controller to learn algorithms or changing some component of a robot to learn artificial intelligence (AI). In contrast, we included learning from modifying parts of a simulator to create the game elements or a game-like system, for example, [34, 35]. Finally, a total of 105 articles were remaining after this step.
In step 4, we carefully looked through the remaining articles and compared their topics, methods, teaching process, and evaluation quality from the presentation of their concepts. After the comparison, the following studies criteria were included: (1) articles that had collected data from assignments or scores after using the GDBL method; (2) articles that had questionnaires with quantitative data and interviews or feedback with qualitative data; and (3) detailed discussion of the collected data and conclusion. In addition, diverse and innovative articles were not neglected, in order to show the various ways to integrate GDFs in education. However, articles reporting on the use of hardware tools to create game or game-like system, such as real robot hand [34], Wii remote [36], Microsoft surface [37], and a projector-camera system [38] to support teaching or learning environment were not included. Finally, a total of 34 articles were included in the review. And we believe these articles were sufficient to get a complete guideline to explain how to integrate the GDBL method in the curriculums.
2.4. Synthesis of Findings
A typology to categorize the 34 articles has to be devised. The classification scheme proposed by [39] in their review of the general instructional gaming literature was adopted for the needs of the present study. This scheme, which was also used in [40], defines the following five categories [39]: (a) research (systematic approaches in the study of gaming targeted at explaining, predicting or controlling particular phenomena or variables), (b) theory (articles explaining the basic concepts or aspects or derived outcomes of gaming), (c) reviews (syntheses of articles concerning general or specific aspects of gaming), (d) discussion (articles stating or describing experiences or opinions with no empirical or systematically presented evidence), and (e) development (articles discussing the design or development of games or projects involving gaming).
Specifically, for the categorization of the articles, the following criteria were applied in this study. Articles comprising empirical research related to GDBL were assigned to the “Research” category. Articles comprising theoretical analyses of concepts, aspects, or outcomes of GDBL were placed in the “Theory” category. Articles presenting syntheses of articles concerning GDBL conducted according to explicit methodology were placed in the “Review” category. Articles reporting on opinions and experiences regarding GDFs used in teaching, with no empirical or systematically presented evidence, were assigned to the “Discussion” category. Finally, articles mainly reporting on the design or development of GDFs used in the GDBL method were assigned to the “Development” category. The articles were grouped into these five categories according to their primary focus. Of the 34 articles found after step 4, 20 were placed in the “Research” category, 1 in the “Theory” category, 7 in the “Discussion” category, and 6 in the “Development” category, whereas no articles fit the “Review” category, which highlights the usefulness and originality of the present study. Like results from other literature reviews on instructional games [40, 41], in this study there were fewer articles in the “Theory” categories than in the “Research,” “Discussion,” and “Development” categories. This can be explained by the fact that instructional games, including GDBL are a relatively new domain of educational technology.
3. Results
This section presents an overview of the studies of the GDBL method based on the results after step 3 and step 4 in Figure 1.
3.1. Overview of the Study after Step 3
In order to have a complete overview of GDBL, we chose the results from step 3 mainly due to the following: (a) they covered a more complete variation of types of GDFs and contained more information than the 34 articles from step 4. (b) They provided more cases in the diversity of GDFs methods used in teaching, which also presents the potential advantages of using GDF in education. (c) They showed the development tendency of GDF related to other factors (e.g., times and technology). We had a study of 105 articles from step 3 representing the use of GDBL method spanning over 11 years. Figure 2 presents the distribution of these articles related to the publishing year after step 3. The result after step 4 is also presented for reference.
(a)
(b)
The types of GDF are classified as (a) Game engines: it mainly covers the commercial game engines and mature and well-known toolkits mainly to create games. (b) Self-made GDF: it mainly includes the game development frameworks that were made by the authors of the articles for usage in a specific course. (c) Games or game editors: it mainly contains editors or platforms that can be used to modify games. (d) Simulation platform: it mainly includes controllers to create a game-like system for robots or other simulation platforms. (e) Hardware platform: it mainly includes both game hardware and related software to build games (laptops and computers are excluded), like Wii remotes, windows surface with XNA, robotic hand. (f) Others are general IDEs, like Visual C++, J2ME, or unspecified game creation toolkits with no specific requirement for learning. For some articles that covers more than one attribute like self-made GDF and simulation platform, we choose priority adhering the following sequence: game engine, self-made GDF, game editor, simulation platform, hardware, and others. Figure 3 shows distribution of types of GDFs applied in GDBL articles in percentage. Further, the top five in game engine subcategory are XNA (9 articles), First Person Shooter (FPS) game engines (unreal: 2 articles, Torque: 2 articles, half-life: 1 article), Flash (4 articles), Alice (4 articles), and Scratch (3 articles).
From statistics shown in Figures 2 and 3, we discovered the following clues.
(1) Tendency of Popularity
Figures 2(a) and 2(b) present the tendency of increasing the number of publications of GDBL articles from 2000, especially from 2006. Between 2006 and 2009, the number of GDBL publications grew with 3–7 articles per year, up to max of 25 articles in Figure 2(a). Figure 3 shows the distribution of the types of GDFs. From the statistics, game engines are most frequently used in GDBL method. We can infer that the continuous development and improvement of game engines will drive the GDBL’s development further in the near future.
(2) Technology Changes the Ways of Learning
After 2006, there was a rapid increase in the number of GDBL articles published. We have analyzed possible reasons concerning this phenomenon from three perspectives: (a) frequent release of new commercial GDFs free of charge, like XNA (2007), Android SDK (2008), and evolution of software development environments, like Flash (acquired by Adobe in 2007) made game development easier than before. Technology changes or enriches the ways of learning and teaching. (b) Cross-disciplinary curriculum started to be used after 2006, for example, [42, 43]. It provides the possibility to use game development in these topics. (c) The upgrowing generation of students is a part of a game accepting culture where the public has an open mind towards games. This culture does not only focus on negative effects of video games such as violence and sex but embraces the positive aspects of games such as social integration, various improved skills, and usage of games for educational purposes, such as Sim-city and Civilization. Furthermore, students that grew up with games have become teachers in schools and may use games in their teaching. They show how technology changes the learning style. Whether it has positive and negative impact on learning depends on how we adopt the technology (game) and how it is used in teaching and learning.
3.2. Overview of the Study after Step 4
In terms of the classification method used in e-learning literature [44], a subcategory was iteratively developed based on the thematic topics found in the articles. Each subcategory was labeled with the disciplinary area-programming, SE, art, and other topic areas. As already mentioned in the introduction, the intended target audiences of the present study are educators, practitioners, researchers, and game designers who use GDFs in learning. The thematic subcategories should help the readers review teaching design, benefits, empirical findings, and future research topics in their own topic of interest. A similar thematic subcategorization of research articles was also performed in review of the general instructional games literature [41]. The overview of 34 articles after step 4 is shown in Table 1 grouped in four categories and labeled with course topics.
These articles present various GDFs used in GDBL and the covered course topics are summarized in Figure 4. The article T28 in Table 1 presents a study for using mobile game development as a motivational tool and a learning context in the computing curriculums. From their survey, the game development process can be used in the study of AI, database, computer networks, SE, human-computer interaction, computer graphics, algorithms, programming, computer architectures, and operating systems.
Both the data from Figure 4 and article T28 can validate that the GDBL method can be used to teach various topics. Most applications are in the field of computer science, electronic, and basic IT learning. However, there are some innovative examples of other applications as well. Article R18 presents how a web-based game-shell platform is used to create a quiz game to teach pupils literacy with no programming requirement. Article R8 presents how Maya and Photoshop are used to create the digital character and movies that could be used as a video inside of a game.
From Table 1, it also shows that GDBL not only can be used in higher education, but also for basic IT education for kids in middle schools. The article D26 presents how pupils are taught basic ICT (Information and Communications Technology) curriculum by creating games. And the articles R20 and D22 describe how middle school students are taught IT and basic 3D programming by building games. The common GDFs used in the primary and middle schools are some GDFs that do not require much programming experiences for pupils, for example, the game editor. This will be further discussed in Section 4.
4. Findings
The articles collected after step 4 are further discussed in this section to serve the purpose of helping to identify and extract the significant elements to meet our aims, like elements to be used to guide the teaching design process when using GDFs in education. Findings are further presented as three aspects: (1) pedagogical context and teaching process, (2) technical aspects, and (3) evaluation results in relation to the aims of this study.
4.1. Pedagogical Context and Teaching Process
This section focuses on the current design process of integrating GDFs in courses or exercises to make the traditional teaching style become more engaging and diversified. This section also provides the detailed steps of how pedagogical theory can be used to guide the teaching design as well as strategies to aid the teaching.
The articles collected in this section are mainly from “Discussion” part in Table 1, and the rest is from the “Research” and “Development” categories. The “Discussion” articles usually have a more complete description than the articles of other categories and include students background, GDF analysis, course setting and background, and teaching design with strategies. We are also concerned with the diversity and flexibility of using GDBL. The diversity shows not only that standard game engines or game frameworks are used in teaching, for example, XNA, but also that GDFs that are adapted or extended for teaching, for example, in article Dev31 they developed an extended library for the Android platform as a GDF for a specific course. Flexibility shows that (a) the same GDF can be used in different situations, for example, article D22 use XNA to teach software architecture and article R15 use XNA to teach programming, (b) the teaching process can be flexible to include other strategies than just integrating GDFs in the learning. For instance, article R13 adds the competition in game development for the assignments.
4.1.1. Pedagogical Context
Integrating game developments in a course study can provide increased motivation and attractiveness for the students. What is behind this motivation and can any theoretical context explain why GDBL can support learning? We investigated this question in the literature review, mainly focusing on (a) why apply the GDBL method in education and (b) how to apply it in a course in the first place. We found it was common to present the teaching design using a GDF in articles from the perspective of a teacher’s experiences from the course, not thinking this process from a learning theory perspective.
Apart from the fact that games motivate for learning, we do not have strong evidence from the pedagogical theory to explain why it is a good idea to apply game development in education yet. However, there exists literature that explains game development, opposed to game play, as a pedagogical activity in the classroom. El-Nasr and Smith mentioned that Seymour Papert presented a relevant conclusion that programming is one example of the constructionism learning theory [10]. Constructionism involves two activities [45]. The first is the mental construction of knowledge that occurs with world experiences, a view borrowed from Jean Piaget’s constructivist theories of learning and development. The second is a more controversial belief that new knowledge can be constructed with particular effectiveness when people engage in constructing products that are personally meaningful. The important issue is that the design and implementation of products are meaningful to those creating them and that learning becomes active and self-directed through the construction of artifacts. In the GDBL method, creating games with GDFs could be this artifact. This could be the fundamental concept to explain the pedagogical context of the GDBL. We can find support for this view from the articles in Table 1. For instance, article R9 considers the learning activity—modifying/creating a game using GDFs as a design activity that has educational benefits such as learning content, skills, and strategies [46]. Design activities are meaningful and engaging to students for exploring skills (analysis, synthesis, evaluation, revision, planning, and monitoring) and concepts to understand how they can be applied in the real world. Further, GDBL can be considered as a variant of several available construction activities. Similarily, for “learning by design,” the article D23 presents using Flash for students from aspect of “learning by doing”—Dewey’s theory [47, 48]. The article D26 uses “learning by making” to learn basic ICT (Information and Communications Technology) knowledge by making games, and it describes that “game making” has the potential to be a powerful learning environment according to attributes identified by Smeets [49]. These contexts are the evidence to explain the GDBL method as a constructionism activity from a theoretical aspect.
Based on Seymour Papert’s opinion, another question pops up: how to use the pedagogical theory to support the design? A positive response is the article Dev31. It presents a case study on the use of double stimulation [50] to guide the exercise design. It considers that using a GDF in education could be a knowledge construction process and describes how to use double stimulus to guide a teaching activity. In schools, learners face a challenge, a problem, or a task that has been designed for a particular pedagogical purpose or they face situations that are likely to appear in work and public life. In both cases, the purpose of exploiting tools is for the learners to respond to such challenges. Based on constructionism, it constructs the relationship between the educational tasks and the material artifacts. This relationship is at the heart of Vygotskij’s notion of double stimulation [50], a method for studying cognitive processes and not just results. In a school setting, typically the first stimulus would be the problem or challenge to which learners are expected to respond to. The second stimulus would be the available mediating tools, like GDFs. Similarily, other pedagogical strategies are also found to support for the GDBL’s teaching design. Problem-based learning (PBL) presented in the articles R6, R14, D25, and Dev33 are also considered as theoretical reference when using GDBL methods. PBL is a pedagogical model that emphasizes the role of a real-life problem and a collaborative discovery process in learning [51]. Within a typical PBL setting, students are first given a challenging but realistic problem of significant size, relevant to the learning objectives of a given course. They are then encouraged to solve the problem in a group throughout the semester as independently as possible with minimum help from the instructor of the course. Even further, article D25 classified the process into the inception phase of PBL by giving game development requirements; the elaboration phase of PBL by building a rapid game prototype; the construction phase of PBL by implementing a game in a project; and the transition phase of PBL by a results evaluation. Apart from the traditional lecture-oriented teaching approach, PBL puts more emphasis on the instructor’s role as a facilitator to prepare meaningful and interesting problems and to create and organize course materials in a manner that students have a just right dose of information in each class to incrementally develop a final solution based on a GDF to the primary problem of the semester. In addition, the articles R12 and Dev29 proposed to use collaborative learning together with the game creation process, and article D24 proposed using “old model of Aristotle” [52] in the teaching design. All of them are of helpful support for the understanding of the teaching process.
The collections of the above results explain the validity of using a GDF in education from a pedagogical angle. Basically, it explains that applying the course content on GDFs by creating games fits well into a knowledge construction process, and it can be integrated with the pedagogical theory supports, like double stimulus or PBL to achieve an improved learning process and outcome. For instance, when we choose double stimulus as a pedagogical theory support, the learning design can be decomposed into two main elements: one is a problem, task, or goal that is designed by the teacher and the other is a responding learning activity that is implemented by students. From the double stimulus perspective, the first stimulus is tasks or assignments and second stimulus can be chosen as a corresponding tool based on the first stimulus. Its outcome depends on teachers’ capacity to keep the two elements match each other. A good task (first stimulus) with inappropriate GDF (second stimulus) will not optimize the output. With this double stimulus support in mind, teachers should find an appropriate match between tasks and GDFs instead of just focusing on one aspect more than the other, like over focus on the design of task but neglect the effort of selecting the GDF. This is not a correct way for applying double stimulus. Further, if the selected GDF always conflicts with the tasks, we should reconsider changing the tasks or GDFs, or even apply a nongame tool. It implies that double stimulus can support learning activity for both GDBL and non-GDBL methods. The teachers should realize it and analyze which tool is better for the course aim and for the students when they apply double stimulus in teaching.
The number of case studies shows that only 30% of 34 articles include both pedagogical and technological design when applying GDBL. This phenomenon reminds us to improve the teaching process with relevant theoretical support. We believe our analysis points to the necessity for further pedagogical and technological codesign to better facilitate awareness of GDBL, thus better conduct the teaching process.
4.1.2. Teaching Process
How to integrate GDFs in teaching and exercises is a very important process when applying GDBL. This section analyzes the teaching process and exercise designs on various GDFs to achieve learning by implementing/modifying a game using GDFs. From our survey, we found necessary and common steps for the integration of GDFs in a course from the selected articles.
The first step is to identify explicit course aims. Figures 3 and 4 show relationships between GDBL and other fields and provides the case studies of how to integrate GDBL into different courses. After the course aim is clear, a common way to integrate GDBL in the course is that the teacher can design an assignment asking to develop a game. The students should then find a solution to this assignment that is in alignment with the course content. When facing such situation, the teacher should find an entry point in how to integrate GDBL with the course and exercises. If this is not possible, we recommend reading articles about similar courses from the selected examples in Table 1 and getting some inspiration. The second step is the exercise design and selection of GDFs. When applying a GDF in a certain course, the selection usually depends on the course content and exercise types, and so forth. We have recognized three types of exercises: one type is to modify the game or adding component to game platform or simulation platform to achieve a complete game, like in the articles D26 or Dev29. The second type is to create a simple game as an exercise to study or practice one or two concepts from the course content, like in the article R9. The third type is to do a complete game development project applying all concepts from the course. Usually, the first and second types can be used in the beginning of a course as a transition period when students are not familiar with the GDF environment, while the third type exercise can be used as a final exercise. However, there are other special cases, like in article R2 where only one type exercise was selected and applied in the whole process. The main driver of exercise design depends on the course aim and students’ background. Selection of GDFs is separately discussed in Section 4.2. The third step is to do a tutorial lecture where the GDF is introduced to the students. The fourth step is to run an initial exercise, which should be easy to do and let the students get familiar with the development environment. The fifth and final step is to do exercises that include implementation of a game. Usually, it is accompanied with some suggestions that were applied in most of the literature: (a) collaborative learning: the student groups range from 2 to 6 students in our statistics; further article R12 has some discussions about how to locate student members in groups such as regular meetings with instructors and flexible meetings among group members. It is important to keep instant communication with the exercise requirements, which would be positive to the students’ learning towards the GDBL method. For each group member, it would be a tradeoff between cooperative and individual work during the work duty allocation. Further, a workshop is suggested to be held at the end of the course. (b) Support: technical support to help students overcome the technical difficulties they face. It is helpful to give examples in the beginning such as to provide optional examples codes and exercise examples to explain the exercise’s complexity. Also there are other strategies like conducting a pilot study before the formal application of GDBL. This approach only appeared in two articles. After the whole teaching process is completed, usually a survey to evaluate both the teaching process and the used strategies is conducted and a more detailed analysis is performed considering the impact factors described in Section 4.3 based on the evaluation from the literature.
4.2. Technical Solution
The technical aspect of the GDBL method is mainly about GDFs’ features described in the 34 articles. And this section will not go into technical details of development of GDFs due to that it is out of the scope of this paper. On contrary, we mainly analyze the GFDs features in the context of GDBL based on our aim of this study.
4.2.1. GDFs Survey
In order to provide a guide to choose a GDF for GDBL, we classify GDFs into two categories: GDFs for novices and GDFs for developers. The main focus of GDFs for novices, including nonprogrammers, is to provide visual methods for customizing game templates and to allow creating or designing games with little or no programming skills. The main focus of GDFs for developers is to offer toolkits that support development of high quality 2D/3D rendering, special effects, physics, animations, sound playback, and network communication in common programming languages, such as C++, C#, and Java. The 34 articles are classified into Tables 2 and 3 according to the GDFs used in the study. The unspecified GDFs or general SDK have been excluded, for example, the articles R10, R11, R14, D24, D25, and T28.
In addition, one mature GDF selected from step 3 in Figure 1 could be a backup for novices—CeeBot Series (http://www.ceebot.com/ceebot/family-e.php) [35]. The programming language in CeeBot is very similar to Java, C++, and C#. It has been developed especially to make learning programming easier. “CeeBots4 School” is a programming course for middle and high school.
4.2.2. Criteria for Selection of Suitable GDFs
Choosing a GDF is considered to be an important procedure during the preparation work for teaching. This process can be described by the following steps: (a) finding various GDF candidates, (b) analyze each GDF’s features, (c) make criteria to filter GDF candidates, and choose one or more GDFs that fit best with the course content. Although our literature survey shows that different course aims have different requirements for the selection of the GDFs, there are still some common points to share. The article D21 presents a general criteria to choose a suitable GDF for the education in terms of theory—“What makes learning to be fun” by Thomas [53]: for example, easy to learn, allow rapid development, and provide an open development environment to attract students’ curiosity. R1 presents that the GDFs should be chosen based on its cost and license, quality, difficulty, textbooks for guidance, and its main functionality. The article D26 explains that their students were not to become experts in programming, and thus they chose GDFs for novices. The article D27 introduces their self-made GDF and assess their own GDF by comparing it with other GDFs in terms of interactive, amusement, easy to use, using official program language, combine with teaching materials, evolutionary learning mode, census analysis, and storylines. The article Dev31 chose the GDF based on analysis of development environment, tutorial documents, emulator, programming language requirements, test devices, interface of the GDF, and possible ways to share games. Further both articles R3 and Dev31 developed a library for the GDF to make it more suitable for the course context. If we face the condition of only one choice, article D22 presents their effort to improve the only GDF. The article D23 presents how they compared different versions of same GDF and made a choice between the newest version with powerful functions or old version but more stable.
To summarize, there are common and essential guidelines when selecting the GDFs: (a) technical environment and inexpensive (low costs) to use and acquire: the technical environment requirements include required operating system and hardware, what tools are provided, are third-party tools supported, and how difficult it is to install GDF. A typical problem can be, for example, that XNA runs only on Windows and many students now have PCs running Linux or Mac OS X. The technical requirements might also be an economical issue, as the choice of GDF might force hardware upgrades or paying for licenses. (b) Sufficient documentation to guide the usage of the GDF. Students need to explore the GDF as an extra task before they start game development on the GDF. If the resources and materials are sufficient and easy to acquire for beginners, it will help them shorten the time spent on learning the technical environment. Time is an important factor during the whole teaching process, which will be further discussed in Section 4.3. (c) Meets the students programming technique contexts. The GDF must be easy to learn and allow rapid development. This issue is also driven by time constraints. Usually, if learning the GDF is not the major educational goal in the course and only an aid to learn something else, learning a new GDF will steal time from the course schedule. An easy and friendly environment is welcomed in order to save time for the students and to keep the focus on the course content and less on the GDF. (d) Not in conflict with the educational goals of the course, flexibility to combine a GDF with teaching materials, and possible to add/change libraries that can be used within the GDF. All GDFs have constraints related to the course content in how they have been designed or how they are released. One example is in SE education where open source GDFs make it possible to do white-box testing on the GDF, while the source code for other GDFs might not be available. Further, some GDFs might have constraints on how you can design your games, what design and architectural patterns you can use, how event-handling must be managed, the freedom of expanding the GDFs functionality, and more. These constraints must be integrated in the SE teaching to introduce the students to the real world where software rarely is built from scratch. In addition, if GDFs are not easy to use, and not strongly relevant to the course content, we can add/change a library with a user guide to apply course content in the GDF. (e) Using an official programming language. Conditionally, it applies to the types of GDFs for developers using a commercial game engine with widely known programming languages, like C#, Java, and C++, which are familiar to the students. But for the types of GDF for novices, if the course just lets students know the data structure, an official language is not really needed. But special programming languages are not widely accepted and as useful as official programming languages in a long run if the students will do more software programming in the future. (f) Amusement and interactive. The GDF should provide a visual and stable development environment to attract the students’ curiosity and engagement. A game development assignment in a user-friendly game development environment could be a good motivation for the students compared to traditional assignments. For example, most students think it is more interesting to work on a game project than, for example, a system for a bank. (g) Ability to develop games in a cross-platform environment. Conditionally, it applies to the types of GDFs for developers. One good example is XNA where the students can choose developing their games either in PC, mobile (Windows Phone 7), and/or console (Xbox360). Other game engines such as Unity3D also allow developing the game in multiplatform. The advantages are the following. (1) Provide students degrees of freedom in developing their games for the platform of their choice and (2) learn about the strengths and constraints of different platforms (e.g., user interface, viewing screen size, resolutions, resources such as memory and processor power, storage for saving/loading the game, and, etc.) in game development.
We consider the above to be the most important criteria to guide the teachers in selecting one or more GDFs for their courses. And some criteria could be changed according to the specific context of the teaching environments. For instance, the target students are middle school pupils and the course goal is to let students familiarize themselves with information technology, it is not necessary to choose (e). In principle, the course aims and students context are the two fundamental and prioritized attributes to decide the selection of GDFs.
4.3. Evaluation
Besides the pedagogical analysis and the GDFs’ analysis in GDBL, this section summarizes the evaluation data from the articles mainly in the “Research” category. Furthermore, we hope to find empirical evidence to support the effectiveness of GDBL. Specifically, in order to approach the third aim of the study, the following information was drawn from each article (if provided in the article): (a) major empirical findings related to the actual effectiveness of GDFs used as an aid in teaching and (b) factors that impact the teaching outcome in terms of the experiment data from the articles are also posed. It is not a simple process to assess the effectiveness of GDBL and it covers at least two aspects: teachers’ and students’ satisfaction of using this method. The teachers’ concerns are the researcher’s understands of the course (not applied where the researchers and the teachers are the same), the GDFs’ features, matching between the selected GDFs and the course content, and teachers’ expertise on games. The students’ concerns involve having interesting exercises and the difficulty of learning extra content—the GDF. Our literature review focuses on these aspects and Table 4 shows a summary of the evaluation process of GDBL in each article. Comparing the students’ and teachers’ satisfaction, students’ satisfaction could be the most important result since it directly relates to teaching effectiveness. And the following results are extracted from the literature and used to validate the effectiveness of GDBL and the impact factors related to it. The results in Table 4 are mainly shown in three categories: (a) experiment data that describes the collected data and materials for the measurement of the effectiveness of the results, (b) conclusion of effectiveness of GDBL, and (c) impact factor that describes the elements that effect outcomes and is classified into positive, neutral, and negative categories based on the articles’ data and conclusions.
From evaluation data in Table 4, the common expressions of measurements are (a) students’ grade or score on the course exam, (b) project results, including analysis of project size and classes they used in game programming; obtaining certain requirements of exercises by percentage; length of codes; percentage completed of the projects and time spent on the projects or the GDF, and so forth, (c) questionnaire surveys to measure the following aspects: students’ satisfaction about the exercise, course and GDF; students background; students’ interest in game development topic; course, and exercise learned and open questions to get suggestions for the improvement of a course, and so forth, and (d) observation and feedback to perceive the fluency of the teaching process and interaction between students and the teacher.
From Table 4, the effectiveness of each article is collected. Generally, 22 of 23 articles have a positive conclusion about using game development in a course in most of aspects, for example, student motivation, engagement in lectures, and exercises. Only the article R5 presents that learning by game design did not have the expected outcome and that the time constraint was a critical issue. Students indicated that they needed more time than two weeks to write a satisfactory 2D game. And finally, it explained that they did not have an adequate number of participants to have an accurate picture about the effects of game design on students’ motivation and attitudes.
Apart from validating the effectiveness of using GDBL methods, the impact factors that could cause a positive or negative outcome deserve to be analyzed. From Table 4, we have summarized what should be noticed when applying GDBL. The following items are the most common issues that appeared repeatedly in our survey.
(1) Communication between the Researcher and Teacher towards the Understanding of the Course Content
This item is not applied to the condition where the teacher and the researcher is the same person. If the researcher designs the method and the researcher invites the teachers to adopt it in schools, good communication and mutual trust between them are crucial to achieving the desired effect. The article R15 states that the teachers should become comfortable with using GDBL and spend a bit more time on it compared to traditional method in a certain course, otherwise it may cause a misunderstanding or bias against GDBL. Another aspect is that the researcher may worry about the teacher and not totally understands the game effectiveness in education, and how game motivation can be successfully used to improve the course design, which is mentioned in the articles R3, R6, R15, and R18. This indicates that the researchers should help teachers in gaining self-confidence and provide constant support while the decision is made to apply GDBL in the curriculum.
(2) Teamwork
This factor could have both positive and negative effects on the teaching results if students work in groups. First, the team size and working environment must be considered in advance. For instance, it was found in article R10 that a big team size could have positive impact on outsourcing course teaching, and article R1 claims that lab environment with teamwork could help improving the effectiveness of cooperative learning. On contrary, as the team gets larger, it becomes more difficult to set the time for general meetings and joint work hours. Further, it also means complex relations in a large group. A serious issue—bottleneck—could happen in the game development process. If one member of the team does not perform, then the entire game development process slows down. Second, the instant communication in a team has a significant impact. Article R2 mentions that the group work can help weaker students. Article R12 also agrees with this statement, but it describes that unexpected situations can occur during the teamwork to hinder the instant communication which the teacher should take care of. Third, the R14 article concludes that students need more experience in working effectively in teams. Most of the case studies found in the articles provide the evidence that teamwork can be used together with GDBL and the nature of teamwork is suitable for cooperative learning and teacher should take care of the issues that may happen during teamwork. However, most of articles did not mention the strategy of competitive learning in GDBL. Only the articles R1 and R13 apply both cooperative and competitive learning in the exercises with a positive feedback in both cases.
(3) GDF Relevance
The most mentioned aspects related to GDFs that impact the outcome are (a) the articles R2, R3, R12, and R15 present the advantages of using interactive graphical GDFs. It shows that visual graphics can provide instant feedback, making student engaged in programs, (b) the articles R3 and R4 describe how a GDF can improve students’ confidence in programming tasks, and (c) the articles R1, R8, R9 R17, and R19 emphasize the need to analyze the GDF’s features in the light the course content, and detailed GDF tutorials should be conducted before it is used in the later exercises.
(4) Students’ Background
In the article T28 surveys, the students’ background was that most of them had played games as they were growing up. This is a suitable pre-requisite to apply GDBL. But a negative aspect is the addictiveness to games, as mentioned in the articles R16 and R17. Some students may focus too much on the game and game development thus losing focus on what they shall learn in the course. This means that the design of the course and the project must be carried out in such a way that the students are forced to learn and use course content. From the articles R5 and R11, it was also noticed that the diversity of student background causes some difficulty of using GDBL. For instance, the programming experience of the students strongly affects the choice of GDF between the ones for novices and the ones for developers. For instance, to use XNA/XQUEST or Android/Sheep from Table 3 for developers, the students must know object-oriented (OO) programming well and be familiar with OO design patterns and OO principles. And some other GDFs require learning a specialized and simplified programming language for game creation, which is more suitable for students without programming experiences.
(5) Teachers’ Requirements
Teachers’ attitude of applying the GDBL method in the course is an essential aspect in a teaching process. The articles R3, R6, and R15 suggest that the faculty should have relevant technical background about the applied GDFs. The article R14 also mentions that they should prepare and solve the anticipated problems they may face during teaching. It is essential that the course staff have technical experience in the selected GDF to provide help for students and to avoid the focus shifting from the course content to technical matters.
(6) Time Constraints and Workload
This problem has been stressed repeatedly in several articles. Most of articles found that the time was limited. For instance, the article R5 mentions that time constraint caused to cut down the time in beginning phase. The article R13 reports that some students complain about insufficient time to complete the project. So there are some advices correspondingly, like the article R18 proposes some suggestions on the time consumption, and the article R3 suggests reading the background material better before the class in order to save class time for students. To help with the time management, a comprehensive time schedule should be prepared in advance for both the teacher and the students. Specifically, a series countermeasures can be the following: (1) make sure that the students learn, understand, and apply the GDBL-project process; (2) force students to set a mandatory rule for teams to create the schedule (strict milestones and deadlines); (3) get involved with the students early to make sure that they make a realistic goal; (4) teacher continuously monitor their progress and guide them to make adjustments, if needed, in order for them to complete their projects.
Other atypical factors could be found in Table 4. Further, Section 4.3 also provides a reference of how to assess the GDBL method. This indicates that future evaluation data of using GDBL is also beneficial, for example, [54]. As it does not only reveal the efficiency of using the framework along with how much the students actually learn from game projects, but also the social relationships’ investigation of learner-learner, learner-teacher, and teacher-researcher.
5. Conclusions
From the above findings, we summarize a guideline for integrating a GDF in learning with teaching strategies. Figure 5 shows a simplified diagram that gives an overview of the design process of applying GDBL (adapted from article D21 and Section 4.1.2). It contains four elements (course aim, pedagogical theory support, GDF resource pool, and impact factor), two methods (learning by creating and learning by modifying games), and six steps in the teaching process and two subjects (students and teachers).
Basically, the course aim has the fundamental effects on the selection of GDF. And the pedagogical theory (Section 4.1.1) could support the teaching design. The GDF resource pool (Section 4.2.2) could be the reference for the selection of GDFs. Usually, during steps A to B in the teaching process in Figure 5, the pedagogical theory support and GDF resource pool play important roles in these two initial steps. Impact factors concern the whole process, but we suggest considering them at beginning as well. In terms of the course aim, pedagogical theory support and GDFs resource pool, the teaching process (Section 4.1.2) starts with designing the lectures and exercises with the selected GDF. After the lectures and tutorials, the course delivery starts and students begin the design and implementation of their projects. For the evaluation framework (Section 4.3), teachers/researchers are suggested to collect data using surveys. Based on the analysis of collected and teaching experiences, they can improve the teaching process framework. Here, we use a compact case to explain how each element in Figure 5 works in a certain course if the GDBL method is applied. The assumption is that the course aim is to teach basic programming rules for beginners. The choice could be made between “learning by modifying games” using a game editor with scripting, or “learning by creating games” using a GDF for novices. Then, we should consider the relationships between the problems and tools from the perspective of double stimulus or use other pedagogical theories to construct the learning process, for example, PBL. With this in mind and according to the criteria in Section 4.2.2, commonly used tools can be selected from the GDF resource pool—GDFs for novice in Table 2 or use another GDF if no suitable GDF is found in Table 2. After finishing steps of A to B in teaching process we start the lecture and the introduction of both exercises and GDFs. Later, students commence the implementation individually or in groups. During the whole teaching process from A to D, the impact factors are relevant but optional. For instance, we can choose a graphical interactive GDF and estimate time to be spent on lectures and exercises. Applying the impact factors in the teaching process depends on the courses’ situations. That is why we have the evaluation and analysis steps E and F in Figure 5. The feedback data can help to validate the choice in each step—whether we choose a right task or a suitable GDF or focus on the most relevant impact factors in a course. In addition, since many elements interact in GDBL, which makes the real situation more complex to analyze and evaluate. Thus, an effective evaluation helps to validate the whole teaching process, and it is not only judged by teachers’ own experiences, but also get opinions from students’ aspect.
From the experience of accomplishing this paper, we still have the following limitation: (a) the scope of data search and collection from four scientific search engines is relatively limited; (b) due to the game research field is younger than other traditional research fields, the amount of articles with empirical data is still limited in our the survey, it maybe cause the pitfall of the evaluation results, for example, generalization; (c) some topics deserve further discussion. Cross-disciplinary courses, like game development course in article R1 covers programming and art design, and machinima course in article R8 have 3D animation and movie creation. Both of them could be further discussed since GDFs plays different roles—the main tool in article R1 and an innovative auxiliary in article R8.
This study has shown that GDBL do have the potential power to help students to learn different curriculums. We hope that the study will provide useful guidance to educators, practitioners, and researchers in the area of GDBL, as well as to GDF designers, and that it will inform their future professional practices and research.