Abstract
Most of the scientific literature on computer games aimed at offering or aiding in psychotherapy provides little information on the relationship between the game’s design and the player’s cognitive processes. This article investigates the use of Bloom’s taxonomy in describing a psychotherapeutic game in terms of knowledge level and cognitive processing. It introduces the Revised Bloom Taxonomy and applies this to five psychotherapeutic games (Personal Investigator, Treasure Hunt, Ricky and the Spider, Moodbot, and SuperBetter) in a two-round procedure. In the first round consensus was reached on the Player Actions with Learning Objectives (PALOs) in each game. The second round sought to determine what level of knowledge and cognitive processing can be attributed to the PALOs by placing them in the taxonomy. Our low intercoder reliability in the second round indicates that Bloom’s Revised Taxonomy is not suitable to compare and contrast content between games.
1. Introduction
Over the past decade we have observed the emergence of a modest amount of psychotherapeutic games. With the term psychotherapeutic games, we refer to computer games aimed at offering or aiding in therapy for any psychological disorders or conditions (most often the precursors of depression or anxiety). The use of psychotherapeutic board games as well as existing entertainment computer games during therapy is already widely regarded as good practice in many situations [1]. Innovative game-based therapy has shown a higher chance of engaging younger target audiences than traditional conversational and “paper-based” methods [2]. It is therefore surprising to find only limited information concerning psychotherapeutic videogames in the scientific literature relating to design and content [2, 3]. Horne-Moyer et al. focused their review on high-order design characteristics (games for health-related behaviours or individual therapy, versus games for entertainment used in individual or group therapy) and their general effectiveness [4]. Fleming et al. offered a more comprehensive review but kept their review of the games’ designs still quite basic by only briefly commenting on each game’s “rule, goals and game objectives”; “outcomes and feedback to the user”; “conflict, competition, challenge or opposition”; “interaction”; and “representation or story” [5].
A lack of insight into the design intricacies, gameplay, and their cognitive processes is problematic. It makes it hard to compare content across several games and thus discuss the state of the art of this field amongst designers, researchers, and practitioners beyond any specific game. Finally, having insight into what cognitive processes are addressed within gameplay would make it easier to spot missed opportunities for effective psychotherapeutic game design.
We set out to test the application of an analytical tool for labelling the cognitive elements of a psychotherapeutic game. We first present the tool itself—the Revised Bloom Taxonomy—and describe our approach in using this taxonomy for the analysis of five psychotherapeutic games: Personal Investigator, Treasure Hunt, Ricky and the Spider, Moodbot, and SuperBetter. We evaluated the content of these games independently, which allowed us to perform an intercoder reliability analysis and rigorously evaluate the reliability of applying the taxonomy. We conclude with a critique on Bloom’s Revised Taxonomy, answers to our three research questions, and limitations of our own approach.
This paper builds on an earlier exploration of using Bloom’s Revised Taxonomy as an analytical framework to identify whether psychotherapeutic games include metacognition in their games [6]. From this earlier exploration we hypothesized that Bloom’s Revised Taxonomy might be useful in different ways. This paper seeks to extend this earlier work by testing the robustness of Bloom’s Revised Taxonomy as an analytical framework. We wish to explore whether the framework will be useful to designers, researchers, and psychologists using psychotherapeutic games. Thus our research questions are as follows:(1)For designers: can Bloom’s Revised Taxonomy be used as a checklist during the design of a psychotherapeutic game?(2)For researchers: can Bloom’s Revised Taxonomy allow researchers to make a more objective description of game content and allow for comparisons across psychotherapeutic games?(3)For psychologists: can Bloom’s Revised Taxonomy support psychologists in making a more informed choice concerning psychotherapeutic games that might be included in their therapy?
2. Methodology
2.1. Choosing Bloom’s Revised Taxonomy
Bloom’s Revised Taxonomy has already been considered as the “most popular cognitive approach to Serious Game evaluation” [7]. Bloom’s original taxonomy [8] stems from the field of education and consisted of categories for Knowledge, Comprehension, Application, Analysis, Synthesis, and Evaluation. Bloom’s original taxonomy was a popular tool for objectives-based evaluation as it allowed for a high level of detail when stating learning objectives [9].
However, the original taxonomy was criticized resulting in various revisions by different authors. See de Kock, Sleegers, and Voeten for a classification of learning environments, containing reviews of the revisions [10]. The revision of Anderson et al. [11] as well as Pintrich [12] improves the original taxonomy by including the category of metacognition. They also distinguish between two dimensions: a Knowledge dimension and a Cognitive Process dimension. We feel the inclusion of the metacognition knowledge level reflects the ongoing insight in the field of psychotherapy where Cognitive Behavioural Therapy (CBT) is currently advancing into its “third wave.” The first wave of CBT started in the 1950s and applied classical conditioning and operant learning. The second wave applied information processing and brought CBT to its current worldwide status. Now, a third wave of psychotherapies is developing “a heterogeneous group of treatments, including acceptance and commitment treatment, behavioural activation, cognitive behavioural analysis system of psychotherapy, dialectical behavioural therapy, metacognitive therapy, mindfulness-based cognitive therapy and schema therapy” [13]. These three waves in CBT can be seen to move up along both dimensions of our taxonomy. Different therapy forms in CBT’s third wave are aimed at the metacognitive level and include all the cognitive processing steps up to and including Creation as part of their treatment.
By applying Bloom’s Revised Taxonomy to analyze the content of psychotherapeutic games, we are approaching these games as educational content. We see all therapeutic interaction as part of a learning process; often knowledge is to be acquired, emotions are revised, and behaviour is changed during psychotherapy.
2.2. Bloom’s Revised Taxonomy
Bloom’s revised taxonomy consists of two dimensions with several levels each. The levels within the dimensions have a hierarchical nature, meaning that every higher level presupposes the presence of the lower levels.
On the knowledge dimension, the taxonomy distinguishes between the following levels:(1)Factual Knowledge: the basic elements that students must know to be acquainted with a discipline or solve problems in it.(a)Knowledge of terminology(b)Knowledge of specific details and elements(2)Conceptual Knowledge: the interrelationships between the basic elements within a larger structure that enable them to function together.(a)Knowledge of classifications and categories(b)Knowledge of principles and generalizations(c)Knowledge of theories, models, and structures(3)Procedural Knowledge: how to do something, methods of inquiry, and criteria for using skills, algorithms, techniques, and methods.(a)Knowledge of subject-specific skills and algorithms(b)Knowledge of subject-specific techniques and methods(c)Knowledge of criteria for determining when to use appropriate procedures(4)Metacognitive Knowledge: knowledge of cognition in general as well as awareness and knowledge of one’s own cognition.(a)Strategic knowledge(b)Knowledge about cognitive tasks, including appropriate contextual and conditional knowledge(c)Self-knowledge
On the cognitive process dimension, the taxonomy distinguishes between the following levels:(1)Remember: retrieving (recognizing, recalling) relevant knowledge from long-term memory.(2)Understand: determining (interpreting, exemplifying, classifying, summarizing, inferring, comparing, and explaining) the meaning of instructional messages, including oral, written, and graphic communication.(3)Apply: carrying out (executing) or using (implementing) a procedure in a given situation.(4)Analyze: breaking material into its constituent parts and detecting how the parts relate to one another and to an overall structure or purpose (differentiating, organizing, and attributing).(5)Evaluate: making judgments (checking, critiquing) based on criteria and standards.(6)Create: putting elements together (generating, planning, and producing) to form a novel, coherent whole or make an original product.
Seen together these two dimensions can be visualized in the Taxonomy (Table 1) [14].
2.3. Five Psychotherapeutic Games
We applied the Revised Bloom Taxonomy to five psychotherapeutic games: Personal Investigator, Treasure Hunt, Ricky and the Spider, Moodbot, and SuperBetter. These five games have been specifically selected, as they have been published in scientific journals and are explained in sufficient detail for us to perform an analysis [15–20]. We wanted to perform an analysis that could exist outside of the game—not playing the game ourselves or observing the gameplay of the intended players. This approach allows us to compare games by not starting from the individual perspective—creating more bias—as well as providing a very practical limitation in research effort. Our approach is intended to be performed based on the (design) description of the game content, preferably including the goals of the game designers and/or the therapeutic goals the game content is based on.
2.4. First Round of Analysis
In order to try and answer our three research questions we provide a robust measurement that might support designers and can be used to compare and contrast game content by both researchers and psychologists. In this paper we investigate the application of Bloom’s revised taxonomy by going through the process of applying it and looking for intercoder reliability. Our process started with the following steps:(1)Use coders with a background in psychology and (serious) game design.(2)Select and read literature concerning Bloom’s revised taxonomy.(3)Select and read literature concerning the psychotherapeutic games.(4)Provide instruction on applying the taxonomy.(5)Apply the taxonomy individually.(6)Present and discuss results at a face-to-face intercoder meeting.
All four authors of this paper have backgrounds in psychology and/or social science, with experience in game research, in game design, and in education. We started by independently reading the selected literature describing the taxonomy [6, 11, 14] and the selected literature describing our five psychotherapeutic games [15–20]. The instructions provided to classify the games consisted of three steps per game:
(1) Describe the possible actions by the player needed to proceed in the game.
(2) Place actions in the taxonomy and provide a short argument why you place it there. Please note any reservations, questions, and comments that arise while you do this.
(3) Create one Taxonomy Table per game.
All coders independently went through these three steps.
When we presented the results to each other in a face-to-face meeting, it became evident that our process had yielded wildly different results concerning the identified player actions and their respective placement in the Taxonomy Table. Our subsequent discussion focused on elaborating and clarifying the diverse interpretations of the categories in Bloom’s Revised Taxonomy. Our biggest discrepancy concerned the knowledge dimension, especially the difference in classifying an action as conceptual, procedural, or metacognitive knowledge. For the categories in the cognitive process dimension we found it easier to align our interpretations by using concrete examples. We reaffirmed that any discussion of the cognitive processing or knowledge levels can only discuss the lower bound, i.e., the minimal requirement in knowledge acquisition or application to fulfil what the game is asking of the player. Every assignment in a game could be approached from a higher knowledge level, processed with a deeper understanding and a more overt strategy than where we allocated them in the Taxonomy Table. There is no way of knowing this upper bound without measuring every individual during gameplay. We judged the described game content for what knowledge levels and processes were necessary (lower bound) to fulfil its assignment and can therefore be predicted. In our discussion, this perspective was an important anchoring necessary to make any judgement.
There was also discussion on the interpretation of “self-knowledge” as it might be argued that the subject of therapy is the “self” and therefore all therapeutic interactions deal with self-knowledge. To resolve this, we turned to the description of “metacognitive knowledge is knowledge of [one’s own] cognition and about oneself in relation to various subject matters” [11]. The “knowledge of cognition” here goes beyond the identification of feelings and thoughts that is core to the CBT approach, where the patient is made an observer of his/her own (internal) behaviours. This makes the minimal requirement for most CBT self-observation and not necessarily self-knowledge as meant in the metacognitive definition. We concluded that questions concerning experiences and thoughts might be placed in the Taxonomy Table at the Factual Knowledge level, if the questions do not go beyond self-observation.
2.5. Second Round of Analysis
We did not use the results in the Taxonomy Tables of this first round of assessment for further analysis. To see if Bloom’s Revised Taxonomy can be used as a basis for comparison, intercoder reliability must be established. If we can establish that the same data will be coded in the same way by different observers, we can be confident that objective comparisons can be made. In order to perform any intercoder reliability analysis, we had to accumulate an agreed upon list with the same amount of player actions per game.
We decided to leave actions that are only necessary as part of the game literacy out of the assessment—such as retrieving a key in a game to open a game object. Overall, the activity of playing a game can be seen as belonging to Applying Process Knowledge in the Taxonomy Table “students who successfully play or operate a game are showing understanding of process and task and application of skills” [21].
In search of more homogenous way to describe game content and player actions we returned to the literature, where we found the stipulation that the taxonomy is meant as a structure for learning objectives. This provided a structured way of forming a description. Stating a learning objective requires a verb and an object, where the verb refers to the intended cognitive process and the object refers to the knowledge level that must be acquired or constructed [11]. We decided to use this verb/object structure to formulate Player Actions with a Learning Objective (PALO) and structured our consolidated list accordingly.
In our second round of assessment our process consisted of three steps:(1)Create a consolidated list of PALOs per game.(2)Each coder individually places the PALOs in a Taxonomy Table.(3)Establish intercoder reliability by means of statistical analysis.
Leaving pure game actions out of the scope, merging similar descriptions through open discussion and rephrasing our descriptions of player actions lead to an agreed upon list of PALOs for every game. These PALOs were independently encoded by placing them in the Taxonomy Table and the results were shared in order to calculate intercoder reliability. We now offer a basic description and the PALOs of our five selected psychotherapeutic games.
2.6. Personal Investigator
Personal Investigator is a game based on Solution Focused Therapy (SFT) and aimed at adolescent psychological patients. Coyle et al. [15] present SFT as “a structured rather than a freeform therapeutic model,” similar to CBT. The game is meant to help adolescent patients go through five different conversational steps with their therapists. These five steps are translated into five main areas in the 3D game world, where the player interacts with nonplaying characters. Initial trials proved promising, but further trials would be required to further test the game’s validity [15].
The game is a single-player 3D computer game with role-playing characteristics. In the game the player becomes the personal investigator that “hunts for solutions to personal problems,” keeping a notebook along the way to keep a record of the hunt and the solutions found. It is played over roughly three therapy sessions, taking just over half of the one-hour session each time. During the sessions, the player plays the game on the computer, while the therapist observes and offers explanations if requested.
After discussion a consensus of 13 PALOs was reached:(1)The player is asked to give a detective name to his/her avatar.(2)The player is asked to write down a problem he/she has that they would like to work on in the detective notebook.(3)The player is asked to turn a problem into a goal they would like to achieve—this becomes the goal of the game.(4)The player is encouraged to think about situations in which the problem that is opposite of the goal is absent or less prevalent.(5)The player is encouraged to understand (but we do not know how) what they are doing differently when the problem is absent or less prevalent.(6)The player is asked to set goals for repeating the behaviours that result from action 5 more often.(7)The player is asked to write about how he/she copes with difficult situations.(8)The player is asked to write about positive, active ways of coping that draw on their strengths and interests.(9)The player is asked to identify people that can help achieve the goal (in real life).(10)The player is asked to think about personal strengths and write down in the detective notebook things they are good at and past successes.(11)The player is asked to draw the answer to the Miracle Question in their detective notebook.(12)The player is asked to write down what they and others would think, feel, and do differently after the Miracle.(13)The player is asked to rate on a scale of 1-10 how close they are to achieving this new future.
2.7. Treasure Hunt
Treasure Hunt is a game meant to support CBT for children with both internalizing (e.g., depression, anxiety) and externalizing (e.g. oppositional defiant disorder, conduct disorder) psychological disorders [2]. It specifically supports therapy “by offering electronic homework assignments and rehearsing basic psychoeducational parts of treatment” [16]. Players experience CBT support by going through six levels during gameplay, each corresponding to a certain step of the therapy. Again, initial tests proved promising, but further rigorous trials would be required to test the game’s validity [16].
The game is a single-player, 2.5D adventure computer game on an old ship inhabited by Captain Jones, Felix the ship’s cat, and Polly the ship’s parrot. The captain has found an old treasure map that he needs to decipher. The player helps by completing tasks to obtain sea stars, which will eventually allow him/her and the captain to read the map. Finally, after receiving a certificate signed by the captain and the therapist and summarizing what he/she has learnt, the player will find the treasure. The player plays one level per therapy session, lasting roughly 20 minutes.
The literature provides a very limited description of play. It does provide a clear translation between the cognitive behavioural concepts and the game metaphors chosen. The paper also stipulates the importance of the guidance of the therapist and that the game is not meant as self-help but must be embedded in therapy.
After discussion a consensus of six PALOs was reached:(1)The player receives “psycho-education” within the game. The basic psychological foundations of CBT are laid out: one’s personality is made up of thoughts, feelings, and behaviour; one’s thoughts influence one’s feelings; four basic feelings can be distinguished, i.e., anger, fear, happiness, and sadness.(2)The player is asked to distinguish between helpful and unhelpful thoughts in general.(3)The player is asked to distinguish between helpful and unhelpful thoughts specific to the player.(4)The player is asked to shoot unhelpful thoughts in the form of fish.(5)The player is asked to replace unhelpful thoughts with helpful thoughts in the form of fish.(6)The player is asked to summarize the gameplay on a “sailor certificate” that is then signed by Captain Jones and the therapist.
2.8. Ricky and the Spider
Ricky and the Spider is a game based on CBT for treating obsessive compulsive disorder (OCD) amongst children. Brezinka presents the game as “not a self-help game” but one that “should be played under the guidance of a therapist” [17]. The game’s design foundation is a “child-friendly metaphor” for understanding both OCD and the CBT approach, thereby combining “psycho-education, externalizing techniques and exposure with response prevention.” Players experience the therapy by going through eight levels during gameplay. Data gathered from therapists and patients who purchased the game revealed promising results, but, further, rigorous trials would be required to test the game’s validity more convincingly [17].
The game is a single-player, 3D adventure computer game. In the game the player is confronted by Ricky the Grasshopper and Lisa the Ladybug who (without saying it explicitly) suffer from OCD and need to confront The Spider who has been making demands that they cannot meet. They ask Dr. Owl for advice, who in turn requires the player’s help. There are eight levels of gameplay in which Dr. Owl, Ricky, and/or Lisa explain certain theories and tools and give certain tasks that the player must apply and fulfil. The first four levels are all psychoeducational. The latter four levels are focused on exposure tasks that are called “courage tasks” in the game. With the therapist observing, the player plays one level at the beginning of a therapy session, which takes approximately 15 minutes, and recounts the content of the level after which the therapy session continues from there.
After discussion a consensus of seven PALOs was reached:(1)The player receives “psycho-education” within the game. The well-established metaphor for OCD is the thought-stream, which is discussed. As well as the four-leaf-clover with strategies for behaviour.(2)The player is asked to give the Spider (antagonist) a silly nickname.(3)The player is asked to make his/her own compulsion map with courage tasks to complete.(4)The player is asked to practice the easiest courage task multiple times a day (outside of gameplay).(5)The player is asked to support Lisa in performing additional courage tasks.(6)The player is asked to motivate Ricky to do additional courage tasks with the four-leaf-clover strategies.(7)The player is asked to recount the gameplay in interaction with the therapist.
2.9. Moodbot
Moodbot is a game for adult psychological patients recovering from conditions such as psychosis and attempts to prevent them from relapsing [18]. As such the game is not tied to a single form of psychotherapy but is a more general psychotherapeutic aid. As a relapse prevention aid, the game is based on two assumptions. The first is that “communication between a patient and his/her healthcare worker about the patients’ mental state is important for the patient’s path towards recovery.” The second is that patients exhibit various, unique signs that indicate whether they are likely to relapse that need to be recorded in so-called “alert schemes” so that they may be used to help prevent relapse. Moodbot is therefore primarily a way of identifying and communicating mental states and any indicative signs from a patient to his/her therapist between therapy sessions. The game is apparently being trialled in professional psychotherapeutic practice. As yet, there is no further information available to ascertain the game’s validity.
The game is an online, multiplayer 2D computer game, although the interaction with other players is indirect (similar to well-known online, social games such as FarmVille). In the game the player is on board a highly imaginative ship (a large fish) and has to help keep it moving and steer it towards certain islands [19]. The player can overview all the rooms of the ship and visit individual rooms where he/she can perform specific actions that might earn him/her points (dust bunnies) that can be spent to get the ship moving and to steer it towards an island. The game is played daily for approximately five to ten minutes per day [18]. In doing so the player offers daily updates of his/her mental state as well as signs that could be indicative of a relapse that the therapist can access in a backend interface at any time.
After discussion a consensus of four PALOs was reached.(1)Before gameplay the future player is asked to decide—together with their therapist and based on their alert scheme and goals—on the labels of the dashboard.(2)The player is asked to express how they feel that day by adjusting their moodbot, moodtube, and dashboard.(3)The player has the opportunity to go into other players’ rooms and observe their mood-state. It is possible to leave comments (tips and advice) in these rooms.(4)The game can stipulate real world challenges, provided and monitored by the therapist, which bring in-game fulfilment in the shape of a plant.
2.10. SuperBetter
SuperBetter is a game (or gamified platform) that is available as a web-based tool and an app for mobile devices. It appropriates game mechanics in order to provide a new narrative for accomplishing challenging health and wellbeing related goals [22]. SuperBetter is not specifically designed as a psychotherapeutic game. However, in a random controlled trial SuperBetter proved itself effective in decreasing depressive symptoms in comparison with a waitlist group [20].
In SuperBetter players give themselves a superhero-secret-identity based on their “favourite heroes.” Players then select a goal to work toward and are awarded “resilience points” throughout the game (physical, mental, emotional, and social resilience) and level up. Gameplay ends when the goal is achieved and can be continued by setting new goals. Players can take steps towards achieving their goal by performing Quests: actions that share a common theme. SuperBetter has predetermined Quests that players can select or design their own or select Quests designed by other players. Players can also undertake mood-enhancing activities (power-ups), which are simple and instantly possible actions such as drinking a glass of water or hugging yourself. The platform provides Bad guys to battle. These Bad guys belong to certain Quests or can be copied from other players or designed by the player. Finally, players gather social support (invite allies). Players can invite friends through the SuperBetter platform to help them. SuperBetter offers a mail contact form and a Facebook plug-in to do this. If the friend becomes an Ally they have access to the players’ Quests, Power-Ups, and Bad guys and can suggest new ones.
After discussion a consensus of nine PALOs was reached:(1)The player is asked to create a superhero identity to play the game with.(2)The player is asked to state a goal (epic win).(3)The player has the opportunity to select and perform Quests—a series of actions that help achieve their goal.(4)The player has the opportunity to create and perform a Quest—a series of actions that help achieve their goal.(5)The player has the opportunity to select and perform a Power-Up—simple mood-enhancing activities.(6)The player has the opportunity to create and perform a Power-Up—simple mood-enhancing activities.(7)The player has the opportunity to select and battle Bad Guys—behaviours that are counterproductive to achieving a Quest.(8)The player has the opportunity to create and battle Bad Guys—behaviours that are counterproductive to achieving a Quest.(9)The system asks players to invite Allies as in-game social support through social networks or e-mail.
2.11. PALOs and Intercoder Reliability
In this section, we describe the results of placing the PALOs in Bloom’s Revised Taxonomy Table. The similarity in overall assessment is low. Only one PALO is scored exactly the same by all four coders (PALO 4 of Ricky and the Spider) and on five PALOs three out of four coders agreed (PALOs 1 and 5 from Ricky and the Spider, PALO 1 from Treasure Hunt, PALO 5 from Personal Investigator, and PALO 8 from SuperBetter), twelve PALO showed two coder agreements, and the remaining twenty-one PALOs had no exact agreement between coders.
Concerning the attribution of a Knowledge level, the PALOs of Treasure Hunt, Personal Investigator, and Superbetter were most frequently scored as Conceptual Knowledge. Ricky and the Spider and MoodBot were most frequently scored as Procedural Knowledge.
On the attribution of a Cognitive Processing level, the PALOs of Treasure Hunt and Personal Investigator were most frequently scored as “Understand,” while Ricky and the Spider and MoodBot were scored as “Apply” most and SuperBetter had most PALOs scored as “Create.”
When looking at the selection within dimensions, Knowledge level had three PALOs for which all four coders chose the same knowledge level (PALOs 4-6 from Ricky and the Spider), fourteen had three coder agreements, twenty-one had two coder agreements, and only one had no agreement (PALO 9 from Personal Investigator). Within the cognitive processing dimension three PALOs (PALO 4 from Ricky and the Spider, PALO 5 from Personal Investigator, and PALO 1 from SuperBetter) had complete agreement, twelve had three coder agreements, twenty-one had two coder agreements, and three (PALO 3 from Treasure Hunt, PALO 4 from Personal Investigator, and PALO 3 from Moodbot) had no agreement.
Having arrived at an equal amount of data points, an inference concerning the intercoder reliability could be calculated. The Taxonomy Tables were merged into a single matrix, containing a unique identifying number for each cell in the table per game, making the data suitable for the calculation of Krippendorff’s alpha [23, 24]. This is a statistic that represents the reliability of a variable and its encoding process, which is suitable for cases with any number of coders and variables of all measurement levels (nominal in our case) [25]. Running a Krippendorff analysis for intercoder reliability of all the PALOs resulted in a very low alpha (see Table 2). This indicates that intercoder reliability is virtually nonexistent as the four observers rarely agree on any exact placement in the Taxonomy Table. A Krippendorff alpha of exactly zero would mean that there are no differences between the encoding results and attributing random values to the data. The alpha statistic can also be negative, in which case the encoding results are worse than random and indicate a structural error. In our results we are encoding somewhat above pure chance () but not at the level of the norm for a good reliability test (α = between .60 and .80) [25]. Our results indicate a lot of room for subjectivity and interpretation in the placement of our 39 PALOs in the Taxonomy Table. Agreement on encoding within both dimensions was a little higher than overall agreement but provided no intercoder reliability (see Table 2).
All the analyses were rerun while excluding one coder consecutively. Although the Krippendorff alpha does alter slightly by excluding any one coder, there is no indication for a structural error and the reliability remains far below acceptable. This supports the notion that there is actual disagreement between the encoders and it is distributed equally.
The overall results from the analysis indicate that there is no error in the methodology and that the results are not pure chance but that the encoding by this taxonomy is too subjective to be considered reliable.
3. Discussion
In this article we demonstrated the application of the Revised Bloom Taxonomy in the analysis of five psychotherapeutic games. In a two-round process, we managed to come to a consensus on the Player Actions with Learning Objectives (PALOs) that each game contains in its design. All four researchers coded each PALO of each game independently and compared the results and an intercoder reliability statistic was subsequently calculated.
The process of assessing game content through the lens of the Revised Bloom Taxonomy turned out to be open to interpretation and resulted in such a great variability in statements after the first round that no sensible comparison of assessment could be made. However, a consolidated version of what PALOs should be assessed was achieved during discussion. We found that applying Bloom’s Revised Taxonomy to our five games provided a structured discussion of player actions in relation to cognitive processes, knowledge levels, and design goals. We feel that such discussions would be useful in the design process of psychotherapeutic games. Describing possible player actions in terms of a PALO provided an interesting perspective on the translation of (therapeutic) goals into game content.
The intercoder reliability statistic revealed that no consensus could be achieved on how to interpret the game content within the Revised Bloom Taxonomy. The rating is prone to subjectivity, making it challenging to assign PALOs to appropriate cells in the Taxonomy Table. The work of Karpen and Welch [26] also suffered from low interrater reliability (r = 0.25) and accuracy (46%) when assigning exam questions to the appropriate Bloom cells. Accuracy improved (81.8%) when limiting the encoding to three levels instead of six “a three-tier combination of the Bloom’s levels that would optimally improve accuracy: Knowledge, Comprehension/Application, and Analysis/Synthesis/Evaluation.”
4. Conclusion
4.1. For Designers (RQ1)
Can Bloom’s Revised Taxonomy Be Used as a Checklist during the Design of a Psychotherapeutic Game? Our discussion to formulate PALOs provided a structured way of describing how game design connects player actions to certain objectives and might provide support during the design process of a psychotherapeutic game. Attempting to place these PALOs in the taxonomy establishes a discussion of the game content on the level of cognition. Although categorization of content based on Bloom’s Revised Taxonomy is too subjective to be used as a design checklist, we do feel the process facilitates a discussion of high value.
4.2. For Researchers (RQ2)
Can Bloom’s Revised Taxonomy Allow Researchers to Make a More Objective Description of Game Content and Allow for Comparisons across Psychotherapeutic Games? The low intercoder reliability indicates that applying the Revised Bloom Taxonomy to psychotherapeutic games does not provide a robust structure for objectivity. The classification of game content remains very open to interpretation and the descriptions it provides should not be used to compare content across different psychotherapeutic games. In order to be useful as a means of comparison, the taxonomy needs to be further developed into a protocol in which designers, researchers, and therapists need to be involved until the process yields an acceptable intercoder reliability. We estimate a lot of effort needs to be applied before Bloom’s Revised Taxonomy would be suitable, and it is advisable to first investigate if other models or taxonomies might be of more value.
4.3. For Psychologists (RQ3)
Can Bloom’s Revised Taxonomy Support Psychologists in Making a More Informed Choice Concerning Psychotherapeutic Games That Might Be Included in Their Therapy? By describing content in terms of Player Actions with Learning Objectives, the designers provide a better insight into how their choices relate to the intended processing by the player and the desired overall outcome of gameplay. This gives any therapist a better insight into the level of cognitive engagement envisioned (lower bound) for different game content, which would support a more informed choice. Unfortunately, placing these PALOs in Bloom’s Revised Taxonomy has not provided a reliable classification of game content and cannot be seen as valuable information.
4.4. Overall Conclusion
We have established that Bloom’s Revised Taxonomy cannot be used as an objective classification of game content for psychotherapeutic games due to very low intercoder reliability. We have found the process of describing Player Actions with Learning Objectives of value, as it forces game designers to formalize their intentions.
We remain confident in the usefulness of trying to find a common language between game designers, researchers, and psychologists to describe the content of psychotherapeutic games beyond the level of any individual game.
4.5. Limitations
One of the limitations of this paper is that we only describe five games. Although this analysis is by no means exhaustive, we believe that an overview such as this paper can already be helpful to game designers or practitioners in the field of psychotherapeutic gaming. Moreover, analysis of game content can and should also be extended to include psychotherapeutic VR games.
We are also aware that we do not provide an in-depth analysis where knowledge and actions during gameplay are minutely observed, described, and categorized. As every gameplay is a unique experience to some degree, we expect that when encoding would be based on game-play subjectivity would increase and intercoder reliability would decrease.
Data Availability
The encoding data used in this research is available online through the Open Science Framework platform under the title of this paper.
Conflicts of Interest
No conflicts of interest have been discovered.