Table 4: Evaluation data collection and impact factor.

TitleSampleComparison Experiment dataConclusion of effectiveness Outcome of impact factor

R122 studentsNoQuantitative data
(1) Questionnaire of students background
(2) Survey project results of students’ game play preferences
(3) Questionnaire on students satisfactory
(4) Questionnaire on interest level in game development careers
(5) Questionnaire on students assessment of gains
(6) Questionnaire on helpful course elements
(7) Students peer-evaluation
Students generally satisfied the elements in the course and resources (including teamwork)
Positive
(1) Lab environment and teamwork helped to archive the effectiveness of cooperative learning
(2) Teaching game development required a shift from teacher-centered to student-centered learning environment
(3) GDFs provided an environment that students could integrate wide variety of skills and knowledge
(4) Motivation factor: competition
Negative
(5) Poor textbook for GDF provided negative effect

R233 students (28 undergraduates and 5 graduate)NoQuantitative data
(1) General questionnaire
Qualitative data
(2) Students feedback about the course
Generally, students enjoyed the project and it fulfilled all of the criteria of a successful project outlined at the beginning plan Positive
(1) Flexible and interactive simulation platform
(2) Providing examples for the difficulty part in the project that was out of the course aim
(3) Group project and discussion helped weaker students
Neutral
(4) Difficulties at first year, but smoothed out by get more teaching experiences and previous evaluation for the improvement

R321 undergraduates NoQuantitative data
(1) General questionnaire
Qualitative data
(2) Video recording about course process
(3) Faculty feedback
The GDF was an excellent catalyst, enabling faculty to begin exploring teaching with game topics and help students to be more engaged
Positive
(1) Because of the immediate interactive graphical feedback, students were engaged and motivated to experiment with the programs
Neutral
(2) Instructor’s attitude toward the interest in GDF
Negative
(3) Visual feedback, although a powerful learning tool, could also be a source of distraction for students
(4) Time spending should not involve the reading of background material in class (better before class)
(5) Limited classroom time was challenging for students

R435 female students from both preschool and universityNoQuantitative data
(1) Questionnaire: students’ opinions about GDF
(2) Questionnaire: effect of students familiarization with Scratch in using of ICT education
(3) Questionnaire (pre- and post-test) attitudes against Internet in education and application development
Scratch was user friendly and satisfied by the students, and it also has a rather positive effect on students’ opinions and attitudes towards computer programming and ICT educational value in education Positive
Scratch helped to setup confidence of students in exploration of ICT in education.

R520 undergraduatesNoQuantitative data
Questionnaire: Likert’s scale (pre- and post-survey in game design course)
Game design had both positive and negative impact on students’ attitudes about computer science, game design, and further development of programming skillsPositive
(1) Students who had prior programming experience can express interest in game design
Negative
(2) Time constraints: assignment might be better received and increase students’ interest if students were given more time and equal emphasis on other phases
(3) Game design topic course had a negative impact on students’ interest in pursuing a CS degree
(4) Not adequate number of participants to have an accurate picture of true effects of game design on students’ motivation and attitudes

R626 high school students and 8 teachersYesQuantitative data
(1) Questionnaire assignment survey
(2) Questionnaire with pre- and post-survey: self-assessment on art and design
(3) Questionnaire survey on teachers’ attitude
It showed great promise for engaging high school students programming and increasing interest in computer related fields of study. Both teachers and students felt a significant improvement in computer programming and self-confidencePositive
(1) Researchers trained both students and teachers by applying GDBL
Neutral
(2) Teacher attitudes and self-confidence about GDBL’s effect the teaching process

R726 students in experimental group. 29 in control groupYesQuantitative data
(1) Each phase of study
(2) Pre- and posttest score
(3) Learning difference between groups and subgroups
(4) Game statistics
(5) Questionnaire survey of each task
Students in the game-first group felt they spent less time on assignments and all students preferred the learning game to the program Positive
“Wu castle” was more effective than a traditional programming assignment for learning and could help prepare students to create deeper, more robust understanding of computing concepts and improving their perceptions of homework

R8  —NoQuantitative data
(1) Questionnaire to survey students feedback
(2) Compared with whole school average score
Students got higher score in this course than school’s average scorePositive
Assessing the GDF in the starting

R926 studentsNoQuantitative data
(1) Questionnaire to survey assignment difficulty with Likert’s scale
Qualitative data
(2) Observation of students progress
Using game development motivated students to learn and allowing them to apply and visualize the utility and application of the conceptsPositive
(1) GDBL could learn several subjects and concepts
Neutral
(2) Different game engines implicitly stressed the use and development of certain skills

R1040 undergraduate studentsNoQuantitative data
(1) Pre- and posttest questionnaire to survey Changed perception of outsourcing concept
(2) Questionnaire: SE outcomes
Qualitative data
(3) Observation: discoveries in communications
Students improved their understanding of outsourcing, developed better appreciation for the importance of SE techniques, and created ad hoc communication protocols between teamsNeutral
Enlarging the teams’ sizes to other universities to create an inclusive teaching environment, which had limitation that only applied in outsourcing teaching

R1138 students
(19 teams)
NoQuantitative data
(1) Length of codes according to grade
(2) Project size and classes
(3) Methods used in programming
(4) Weekly working hours
(5) Proportion of work discussion, coding, thinking, graphics, and audio
(6) Object-oriented skills applied in code
Positive experience had been gained in teaching the topic by using game frameworkNeutral
(1) To keep the students motivated and teachers tailored the course for each student
(2) Using game development to achieve depth of objects and object interactions training

R12124 studentsNoQuantitative data
(1) Grade
(2) Questionnaire to survey students attitude
Learning by creating game was able to improve the student grades largelyPositive
(1) Object-oriented programming concept became easier to understand after seeing object design visually in the GDF
(2) Students felt happy with using cooperative learning system, games development, and visual design
Negative
(3) The group members’ communication was hindered by the in front of computers
(4) GDBL could help with the passing rate, but still have improving space for graduation aim

R1355 students’No Quantitative data
(1) User survey of game project percentage completed
(2) Login times
(3) Questionnaire with Likert’s scale: student satisfaction
(4) Questionnaire with Likert’s scale: tournament features
Combination of game development and friendly student competition was a significant motivator for the increased student performancePositive
(1) Tournament could increase students’ participation and motivation
Negative
(2) Students’ common complaint of not having adequate time to complete the project

R14  —NoQuantitative data
(1) Individual and group creativity levels perceived by students
(2) Students’ perception of abilities developed at intermediate or high levels
Qualitative data
(3) Future career survey
Game project development with collaborative learning was manageable and effective for increasing students’ teamwork capability and increase the employability confidence
Positive
(1) Project (game project development) based learning motivated their team collaboration
Negative
(2) Teacher attitudes: initial resistance for problems that students’ teams faced could be discouraging to faculty members who did not expect it
(3) Teamwork: students were not born knowing how to work effectively in teams. A flawed team-based instructional model had negative effect

R15CS1: 22 in GTA and 10 in Console
CS2: 18 in GTA and 9 in Console
YesQuantitative data:
(1) Success rate (Passing rate)
(2) Assignment score
(3) Self-reported time spent on assignment
(4) Post Assignment Survey
(5) Pre and Post course survey
Qualitative data:
(6) Feedback from faculty
Interactive graphical assignments could be a good tool for teaching CS1 students. The success of GDBL hinged on the instructor’s expertise and enthusiasmPositive
(1) GDF feature: interactive graphical application supported experimentation and visualization
Negative
(2) Teacher’s background and attitudes towards the games impacted the output of a lecture, faculty “dropped” GDBL in the end at first experiment, but became more comfortable later

R1646 studentsNoQuantitative data
Questionnaire about learning process, tradeoff between technical and architecture problems, integration of game development and course, learning outcome
GDF was easy to use and did not conflict with course aim. A good GDF could save development timeNeutral
GDF selection influenced learning process and extra technical issues, but students could learn a lot through a game project.

R1727 in control group, 43 in experimental groupYesQuantitative data
score of the pre- and posttest by a test sheet
Results showed that the proposed game development activity could have higher learning achievements compared to the traditional lecturingPositive:
(1) GDF issues: choosing modifying game according to course topic with simple scenario. And tutorials for GDF were prepared well. Understanding game topic could make engage learning
Negative
(2) Game was additive for students

R18125 experimental students and 186 control group students YesQuantitative data
(1) GRADE test scores
(pretest, posttest)
Qualitative data
(2) Interviews on teacher’s feedback
Game development helped to improve the student content retention and so forth
Positive
(1) Optimum amount of time to spend at a sitting on game development activities was about 45 min by observation
Negative
(2) Too little time allotted to the development of game and insufficient gaps between each game creation activities

R1933 undergraduateNoQuantitative data
Questionnaire with Likert in general
Using GDBL indicated that the motivation of the students was higher and they understand complex problems easier and exercise could be done more rapidly Positive
GDF was searched and chose based on the requirements

R2022 middle schoolersNoQuantitative data
(1) Questionnaire: pre- and post-surveys of participants information
(2) Programs analysis
Qualitative data
(3) Daily log
(4) Interviews on students
Findings suggested the middle school students could use Alice to make games to build information technology fluencyNeutral
(1) To provide a proper challenge in class
(2) Difficulty in using GDF to finish the assignment

T28NANoQuantitative data
(1) Survey of students background
(2) Relevant application about Mobile GDBL
Mobile game development could be successfully integrated into computer science educationPositive
Students’ background: student lived in game environment and game development exercise could be a good motivation

Dev3019 graduateNoQuantitative data
Questionnaire survey with Likert’s scale and system usability scale survey
XQUEST enhanced XNA in suitability as a teaching aid in SE learningPositive
To design the XQUEST from the previous assessment experiences

Dev3257 in group1, 45 in group2 YesQuantitative data
(1) Questionnaire result of students’ user experience
(2) Score for pre/posttest
SIMPLE improved both learning motivation and programming skills for the studentsPositive
Use GQM approach in developing game metrics for students’ exercise.