Abstract
The use of Arabic mobile games has increased in the last decade. The Arabic mobile game market is rapidly growing as well, where more games are being developed or localized to target Arabic game players and attract newcomers. Little existing work, however, has been done to ensure the delivery of effective, efficient, and satisfying Arabic mobile games. Previous studies have shown that most developers of Arabic mobile games do not evaluate these games because of the lack of usability evaluation methods that are specifically designed for such games. In this paper, we present Arabic Mobile Game Heuristics, which is a list of usability heuristics that were explicitly adapted for evaluating Arabic mobile games. The list was developed by revising and modifying Nielsen’s list of heuristics and the game usability principles. Validating the proposed heuristics was achieved by comparing the outcomes of evaluating two Arabic mobile games using the three lists of usability heuristics. Results indicate that the proposed heuristics allowed evaluators to find additional usability issues and identify more severe problems than using the other two lists.
1. Introduction
Mobile games are getting extra attention worldwide for several reasons including accessibility, evolving technology, reachability, and affordability [1, 2]. In the Arab region market, more practitioners and businesses are rising their investments and studies in mobile games [3]. More countries in this region are included in the top 100 game markets in terms of sales [4]. Many game development companies have been focusing on developing Arabic mobile games to target this growing market. Little, however, has been done in terms of evaluating the usability of such games.
Several studies and methodologies have been presented to evaluate games and the player’s experience [5]. Researchers and practitioners are focusing on the game’s functionality, usability, accessibility, and playability [6–8]. Nevertheless, around 75% of game development companies in the Arab region do not use heuristics to evaluate Arabic mobile games [9]. There is a clear need in the industry for a heuristic list of usability, accessibility, or playability that is adapted to the Arabic mobile games. Although heuristic evaluation is cheap, the available heuristic lists are not comprehensive to be adopted by Arabic mobile game developers [9].
In [10], the authors studied the cultural effect on designing Arabic user interfaces. They used Hofstede’s cultural dimensions to develop design guidelines that are adopted to Arabic user interfaces. Their investigation results show high satisfaction among Arab users about the role of culture in the design of user interface. In addition, the authors in [11] investigated the cultural markers of Arabic interfaces. Their results show that there is a clear influence by the Arabic culture on the design components (color, layout, text, and language) of user interfaces. Several aspects of the Arabic culture and language have an impact on how an interface is perceived and interacted with [10]. For example, the Arabic language is read from right to left, which should clearly reflect on how user interface items are laid out on the screen. Translation of words and phrases can also influence the experience of users with an interface [10]. For example, the phrase “Game Over” is translated to what literally means “Good Luck” or “the End” in several Arabic video games. Visual representations of user interface objects in general, and game objects in particular, should be carefully considered for the Arabic users as well. The dagger, for example, is considered a clothing accessory in the Arabic culture, compared to being a weapon in other culture. Moreover, results of other research studies, such as the one presented in [9], show that game development companies are not conducting usability evaluation of Arabic mobile games because of the lack of a comprehensive list of heuristics that are adapted to the Arab users.
Motivated by the benefits and reputation of heuristic evaluation in video games [12] and the absence of a usability heuristic list that is tailored for Arabic mobile games, we consider revising the available lists to develop usability heuristics that are adapted to overcome the challenges of developing usable Arabic mobile games.
In the research presented in this paper, we carried out three main stages. Firstly, we reviewed the available heuristic lists in terms of popularity and applicability in video games and mobile gaming. Results of the first stage led us to focus on both: (i) Nielsen’s Heuristics [13], which are widely used in different domains, and the (ii) Game Usability Principles [12]. In the second stage, we aimed to revise both lists to compile a new one that is tailored for Arabic mobile games. This stage included remote focused groups and surveys with domain experts, which resulted in creating, eliminating, or rewriting heuristics to develop a new related list. In the last stage, we compared the resulted list with Nielsen’s Heuristics [13] and the Game Usability Principles [12] through the evaluation of two Arabic mobile games: Arabian Heroes by Pixelhunters and Malik Altawseel (King of Delivery) by Tamatem Inc. Both games are widely played in the region and highly rated (Malik Altawseel (King of Delivery), for example, received 4.6 stars out of 5 in Google Play by around 170,000 users).
The paper, in its remaining part, is organized as the following: Section 2 presents the related work in literature, Section 3 discusses details of the research methodology and the proposed list of heuristics, Section 3 presents and discusses the results of evaluating the proposed list, and Section 4 concludes the paper and identifies some future work directions.
2. Related Work
Few studies have been published that investigate the evaluation of Arabic interfaces in general and the mobile Arabic games in particular. In fact, the focus of such articles was mainly on reporting case studies of usability testing with users, such as those presented in [10, 14, 15]. Moreover, several research studies have been conducted to identify the challenges of developing Arabic applications [16–18]. The identification of these challenges and issues has motivated several studies to create guidelines for designers and practitioners in designing Arabic interfaces [19, 20].
Although several studies have been reported in developing usability heuristics [21–23], Nielsen’s list [13] has been the most used one. Nielsen’s list has been followed by the development of several checklists to help evaluators apply the heuristics on interfaces [24]. However, recent studies have shown that Nielsen’s 10 usability heuristics need to be updated to remain consistent with modern usability problems [25]. Several studies have been conducted to propose new lists of heuristics that are specifically tailored to particular fields. In [24], for example, authors recommended a formalized process in creating usability heuristics after investigating 73 related studies. Furthermore, authors in [26] proposed a formal approach to design usability heuristics. Their validation of the proposed approach depended on expert opinions and case studies.
Heuristics of game usability have been the interest of many researchers as well. In [12], authors proposed a list of usability principles to inspect video games. Their methodology was based on three steps: identifying real world usability problems, grouping similar usability problems into categories, and creating heuristics. The language of their heuristics was explicitly chosen to link to the game domain, with some similarities with Nielsen’s lists [12]. Other methods of evaluating video games have been explored by researchers as well. In [27], for example, authors developed a framework to measure user experience satisfaction in playing video games. Other researchers explored the evaluation of emotional valence in video games and its effect on the interactivity of experiences [28]. Playability of games has been also investigated. In [29, 30], for example, authors presented their results of evaluating mobile games using playability heuristics. Other studies [31] proposed other playability heuristics that were based on a lexical analysis of nouns and adjectives in online game reviews.
Moreover, evaluating mobile games using available lists of heuristics has been explored by researchers too. In [32], authors presented their initial study with university students in evaluating mobile games using a set of heuristics. Usability evaluation of mobile games using expert reviews and play testing was also reported in [33], where the expert reviews phase included evaluating the game against a list of available heuristics.
Despite all the related work presented, to date, to evaluate video games, Nielsen’s ten usability heuristics is the most popular list used in evaluating their usability [34]. Therefore, our first stage resulted in selecting Nielsen’s list [13] for further investigation. In addition, the Game Usability Principles [12] were also included in our study as we found out that they are being broadly cited by researchers and are specifically designed for the usability of video games.
3. Methodology
The methodology carried out for this study consisted of three stages. In the first stage, we studied the available sets of related heuristics. Our goal was to find the list(s) that are mostly used by practitioners and researchers in the field of video games. We also aimed to study the applicability of each list to mobile games in particular. The first stage is presented in Section 2. The following subsections discuss the second and third stages of the methodology.
3.1. Proposed List of Arabic Mobile Game Heuristics
In the second stage, several remote focus groups took place with three domain experts in order to revise both lists and compile a new one that is tailored for Arabic mobile games. Some of the heuristics were customized to be applicable for Arabic mobile games. Such customizations were achieved by modifying the title of the reviewed heuristic or its description. Adding new heuristics was considered as well but performed with caution for the purpose of not introducing new heuristics that need to be learned by evaluators of Arabic mobile games. The resulted list was then reviewed by 11 practitioners to assess each heuristic and provide a rate (out of 10) about its significance from their perspective in relation to Arabic mobile games. The review took the form of a questionnaire with closed and open-ended questions. Participants were encouraged to leave their comments regarding each of the new heuristic. Affinity diagraming was used to analyze the qualitative feedback of each participant. Comments provided a good understanding of the motivations behind the participant’s decision and helped in compiling a better description of each heuristic. The 11 practitioners had an average of 8.5 years of experience working on Arabic interfaces in the fields of product management, front-end development and UI designing. Each practitioner evaluated the proposed list independently and was giving the opportunity to elaborate on their ratings. Two remote focus group meetings with all 11 practitioners took place after the end of the second stage to finalize the list and compile all modifications. Table 1 shows the proposed/compiled list of Arabic Mobile Game Heuristics (AGH).
3.2. Experiment/Evaluation Design
In the third stage, we compared the new set of heuristics (Arabic Game Heuristics) to both Nielsen’s list of heuristics [13] and the Game Usability Principles [12]. The focus of the comparison was on two main hypotheses: (i) using the proposed Arabic Mobile Game Heuristics will provide a higher number of usability issues than using Nielsen’s Heuristics or Game Usability Principles for Arabic mobile games and (ii) using the proposed Arabic Mobile Game Heuristics will report more severe usability issues than using Nielsen’s Heuristics or Game Usability Principles for Arabic mobile games. We selected two Arabic mobile games: Arabian Heroes by Pixelhunters (Figure 1) and Malik Altawseel (King of Delivery) by Tamatem Inc. (Figure 2) for the experiment. In our selection, we focused on the number of downloads and star ratings on both Apple’s App Store and Google Play.
Eighteen participants were recruited for the study. None of the participants have reported any sufficient experience with evaluating Arabic mobile games. In fact, it was challenging to find any participant who has such experience. However, all participants were carefully selected to have a minimum of 5 years of experience in evaluating usability (median: 7 years) and at least 3 years of experience in evaluating video games (median: 4 years). The median experience in evaluating Arabic interfaces among those participants is five years. 56% of participants are male () and 44% are female (). Participants were divided into three groups (between-subjects). Each group consisted of six participants. Each of the three groups randomly evaluated both games using either the Arabic Mobile Game Heuristics, Nielsen’s Heuristics, or Game Usability Principles.
4. Results and Discussion
Analysis of the comparison of heuristics has shown promising results of the proposed heuristics in evaluating Arabic mobile games when compared to Nielsen’s Heuristics [13] and the Game Usability Principles [12]. As shown in Figure 3, participants reported more usability issues using the Arabic Mobile Game Heuristics (AGH) in comparison with Nielsen’s Updated Heuristics (NH) and the Game Usability Principles (GUP) on both Arabic mobile games. In particular, participants were able to identify a total of 13 usability issues in Game 1 using the proposed Arabic Mobile Game Heuristics, whereas only 6 usability issues and 10 usability issues were reported for the same game using Nielsen’s Updated Heuristics and the Game Usability Principles, respectively. For Game 2, participants identified 12, 5, and 8 usability issues using the Arabic Mobile Game Heuristics, Nielsen’s Updated Heuristics, and the Game Usability Principles, respectively.
Moreover, Figure 4 shows a comparison between the number of usability issues in both Arabic mobile games and the number of issues identified by participants. Arabic Mobile Game Heuristics (AGH) gave participants the ability to identify 86% of the total number of usability issues. However, this percentage was only 38% for Nielsen’s Updated Heuristics (NH) and 62% for the Game Usability Principles (GUP).
The proposed/compiled set of Arabic Mobile Game Heuristics (AGH) has also shown better performance in detecting severe usability issues when compared to the other two lists. As shown in Figure 5 and Table 2, usability issues of severity level 3 and severity level 4 were better identified by participants using the proposed Arabic Mobile Game Heuristics (AGH) than the other two lists for both Arabic mobile games. In fact, participants were not able to identify any of the usability issues of severity level 4 using Nielsen’s Updated Heuristics, whereas 100% of these issues were identified by participants using the Arabic Mobile Game Heuristics.
Results have shown that using the proposed Arabic Mobile Game Heuristics (AGH) provided a higher number of usability issues than using Nielsen’s Heuristics or Game Usability Principles for Arabic mobile games (hypothesis (i)). Furthermore, the results have revealed that using the proposed Arabic Mobile Game Heuristics, participants reported more severe issues than using Nielsen’s Heuristics or Game Usability Principles for Arabic mobile games (hypothesis (ii)). The results have also shown that the Game Usability Principles performed better than Nielsen’s Updated Heuristics in evaluating Arabic mobile games. We believe this occurred because the Game Usability Principles are more adapted to games than Nielsen’s general list. Our results have also shown that some participants identified some usability issues to be catastrophic (severity level 4) using both the Arabic Game Heuristics and the Game Usability Principles but put the same issues under severity level 3 or level 2 using Nielsen’s Updated Heuristics. We believe that this is also a matter of the generality of Nielsen’s list, which is not explicitly adjusted for games.
As depicted in Figures 3–5 and discussed in this section, we are 90% confident that the proposed Arabic Game Heuristics outperform Nielsen’s Updated Heuristics by 46-50% (CI: 90, MOE: ±2.0%) and outperform the Game Usability Principles by 23.7%-28.3% (CI: 90, MOE: ±2.3%) in the number of identified usability issues when evaluating Arabic mobile games. Furthermore, we are 90% confident that using the proposed Arabic Mobile Game Heuristics, AGH evaluators will be able to identify 62%-71% (CI: 90, MOE: ±4.7%) more major (severity level 3) and catastrophic (severity level 4) usability issues than using Nielsen’s Updated Heuristics and 38-46% (CI: 90, MOE: ±4.1%) when using the Game Usability Principles in evaluating Arabic mobile games. We believe that our new set of usability heuristics will have an important impact in improving the evaluation of Arabic mobile games and thus enhance the quality of these games.
5. Conclusion and Future Work
This paper presented a new set of usability heuristics to evaluate Arabic mobile games. This set, Arabic Mobile Game Heuristics, was developed by revising the usability heuristic lists introduced by Nielsen [13] as well as the Game Usability Principles [12] to explicitly adapt to Arabic mobile games. Unlike other lists, our focus has been on the usability evaluation of Arabic mobile games. To test the proposed set, eighteen participants evaluated two Arabic mobile games using the three sets of usability heuristics. The preliminary evaluation of the heuristics has shown promising results for the new set in evaluating Arabic mobile games. With the Arabic Mobile Game Heuristics, participants were able to find more usability issues and further severe ones than using the other two lists. Therefore, we believe the proposed usability heuristics will provide evaluators with an effective and efficient list to evaluate Arabic mobile games.
Although the research presented in this paper showed promising results, we believe there is an open window for improvement and future work directions. For example, we expect that further evaluation of the proposed heuristics might provide more insights for the list to be revised and evolved. Applying the heuristics on more games from several genres, more platforms, and including more Arabic mobile game evaluators in the study can be also identified as a future work direction. The focus of this research has been on the usability of Arabic mobile games. We consider this to be a limitation to our method but a necessity to concentrate on usability issues more than other design issues, such as playability and engagement of the game. Moreover, our research focused on Arabic, we expect that the new set of heuristics can be generalized to other platforms, but further work is needed to validate this generalization. We also believe that the new set can be further developed if it was used by evaluators who are not trained beforehand on the set and by those who have never carried out a heuristic evaluation before. Another interesting research direction might be in carrying out a heuristic evaluation of the new set on non-Arabic mobile games.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.