Abstract
As gestural interfaces emerged as a new type of user interface, their use has been vastly explored by the entertainment industry to better immerse the player in games. Despite being mainly used in dance and sports games, little use was made of gestural interaction in more slow-paced genres, such as board games. In this work, we present a Kinect-based gestural interface for an online and multiplayer chess game and describe a case study with users with different playing skill levels. Comparing the mouse/keyboard interaction with the gesture-based interaction, the results of the activity were synthesized into lessons learned regarding general usability and design of game control mechanisms. These results could be applied to slow-paced board games like chess. Our findings indicate that gestural interfaces may not be suitable for competitive chess matches, yet it can be fun to play while using them in casual matches.
1. Introduction
As computational systems advance in processing power and become more present in people’s lives, new types of interaction arise. Touch screen and voice user interfaces, for example, are being widely used as interaction interfaces for smartphones, as they provide more usability than onscreen keyboards. Electronic entertainment systems, for example, video games, are also experimenting new forms of interaction to better immerse the player in the game, such as Virtual Reality (VR) and gestural interfaces. Regarding the latter, many current video games already utilize this type of interaction to control games, for example, Microsoft Xbox and Nintendo Wii.
Apart from video games, which explore a wide range of gestural interaction possibilities in game genres such as dance and sports, this technology is also starting to be used to control other systems, such as a PC itself. Regarding video games, as gestural interfaces are a relatively new kind of interaction paradigm, their usability has not yet been deeply investigated for many other genres such as board or turn-based games.
Gestural-based games began to increase in number and popularity with the advent of modern video games’ built-in gesture technologies, such as the Kinect for Xbox One (http://www.xbox.com/en-US/xbox-one/accessories/kinect-for-xbox-one). At first, sports [1] and the dance [2] genres were dominant due to their straightforward mapping of gestures or body movements to game controls, but more games of different styles began to appear as this new form of interaction became more popular. However, one particular genre of game that has not gained popularity in gesture-based systems is board games, such as chess or checkers. A possible reason for the gaming industry’s and developers’ lack of interest in board games might be that, as a gaming platform that uses the whole body as a controller, it is expected that the player uses a lot of movement from their body to perform an action or complete an objective.
In typical board games, the only physical action that a player needs to perform is to move a piece, which requires much less effort than, for example, fighting an enemy with a sword. Nevertheless, this does not exclude the possibility that a gestural board game could be designed and still be fun to play, as it adds more interaction and movement to the game. Even if this type of control could possibly increase fatigue or discomfort, players could feel more motivated to play due to the enjoyment that this interface may bring [3]. Moreover, it could arouse the interest of users that do not like slow-paced board games due to their lack of body movement. Although there is no official gestural chess game available for the Kinect platform, some researches and developers are already exploring this particular genre.
This gestural interface is not easy to either conceive or implement, as this kind of interface brings many usability and gameplay issues [4, 5]. To help in this task, an online and multiplayer chess game was adapted to use gestural input based on the investigation of related gestural board games and the current available gestural technologies. To test the implementation, a case study was performed with chess players using the principles of experimental research [6, 7], and the findings were discussed.
There are a very few implemented board games that use gestural interaction. More importantly, there is no actual evaluation of this kind of control in this specific type of game, making it hard to define whether a gestural control fits or not into the context of the game and how it performs in comparison to the traditional mouse and keyboard input. These factors highlight the importance of this evaluation.
This paper is organized as follows: we briefly present related work of gesture-based computer board games in Section 2, proceed with detailing our Kinect-based gestural interface and its implementation in Section 3, showcase a case study using the gestural interface with chess players and its results in Section 4, and synthetize and discuss the results in Section 5. Finally, conclusions and future work are outlined in Section 6.
2. Related Work
We performed a search for gesture-based computer board games within the digital libraries of ACM, IEEE, and Google Scholar as well as in Kinect project communities, such as KineckHacks (http://www.kinecthacks.com/) and Develop Kinect (http://www.developkinect.com/). Based on the related games that were found, it is clear that there are a very limited number of implemented and reported on-computer board games using gestural interaction, both academic and commercial/hobbyist.
Although the use of gestural input for board games is not very popular, there are a few researches that explore this direction. Sriboonruang et al. [8], for example, developed a visual gestural interface for board games, which measures the distance between the player’s thumb and index finger to determine whether the player is in a state of holding a piece. If so, the system then tracks the hand until the action of holding the piece is no longer executed and puts the previously selected piece on the destination square. Li and Hong [9], using a stereo camera, developed a real-time hand gesture recognition system. This system can recognize up to five different hand gestures and was used as a controller for a Korean chess game, in which the player was able to move the pieces using those gestures.
Kalpakas et al. [10] developed a system that recognizes 2D gestures using a web camera on the top of a wood chess board. The user can select a piece and choose a destination spot by pointing the location on the board using the index finger. The system tracks the position of the user’s fingertip on the board to determine where to move the piece.
We also found projects in different Kinect developer communities. In [11], for example, the user controls the pieces by using the hands as in real life, that is, taking a piece with the hand and putting it on the destination square. The difference is that the computer’s artificial intelligence responds by actually moving a physical piece in a board using a mechanical hand. In [12], a gestural-based game of checkers was developed, where the user can select, drag, and drop the pieces using hand gestures. Even though it does not use direct gestures to control the pieces, in [13] a Kinect-based accessible system was developed to allow visually impaired users to play chess against the computer. The Kinect, positioned on top of the board, recognizes the hand of the user as s/he performs a movement. Then, the computer gives audible feedback, for the player to know to what square the piece has moved and what the next movement of the computer is.
In Robot Chess (http://apps.leapmotion.com/apps/robot-chess/windows), the most recent gesture-based chess computer game when the search was performed, the player uses a Leap Motion (https://www.leapmotion.com/) to control a robotic hand in a virtual environment, against a robot. The pieces are controlled by hovering the hand-cursor over them and grabbing using the fingers. All menus are also controlled using exclusively the hand and the Leap Motion device. The system is also fully operational with the Oculus Rift (https://www.oculus.com/en-us/rift/), making the user capable of playing the game in a VR environment, also using the Leap Motion to control moves.
It appears that no further significant investment was made in the research, development, and/or evaluation of this kind of game in the past few years. It also appears that there are a very small number of researches with actual users in this particular genre; therefore it may be hard to have conclusive results regarding the application of gestural technology in slow-paced board games, such as chess. Being so, it is clear that researches using actual players, in order to evaluate the use of gestural interface in this specific genre—especially in comparison to the traditional platforms—are needed to investigate this application.
3. MasterMind Chess
Chess is one of the most famous and most-played board games in the world. Its popularity makes it a valuable candidate to test this kind of interaction in turn-based board games and to evaluate whether it makes sense to adapt this kind of game for gestural input. To this purpose, the MasterMind Chess game (available for free download at http://www.mastermindchess.pairg.ufrn.br) [14], an online, multiplayer, and multilanguage chess game, was adapted to handle gestural input and enable users to play chess using only the movement of one hand. This adaptation was possible due to the development of a gestural interface for the MasterMind Chess game, that is, the MasterMind Chess Kinect Module. The programming language used for this implementation was C#, due to the integration with the Kinect platform.
The MasterMind Chess game supports two more game modes, apart from the traditional, classic game, composed of the classic pieces in an 8 × 8 board. The first is Capablanca chess, where the traditional 8 × 8 board is exchanged for a larger 8 × 10 board, with two additional types of pieces. The other game mode is Fischer Random Chess. In this mode, the game still occurs in an 8 × 8 board, but the starting positions of the pieces of the same color, apart from the pawns, are permuted with each other. The Kinect Module gestural interface for the MasterMind Chess also allows the users to play these game modes using gestural input.
Many factors led to the choice of using the Kinect for Windows (https://developer.microsoft.com/en-us/windows/kinect) over other candidates—for example, Leap Motion and Asus Xtion (http://www.asus.com/3D-Sensor/Xtion/)—of gestural devices in the implementation of a nonperceptual gesture-based interface [15]. It is a well-grounded technology that has been on the market for years, with both theoretical and practical background supported by a great number of scientific researches, not only for entertainment, but also in the medical field [16, 17]. Although it is not as accurate as recent technologies, it is more accessible, in terms of ease of acquisition in comparison to the other available devices, and already has all the necessary drivers and programming libraries to fully support its functionalities. This decision was as a result of an investigation of the available gestural technologies that exist nowadays and the availability of these technologies to the authors.
The main purpose of the gestural module is to map the position of the player’s hand, in Kinect’s skeleton coordinates, to the screen of the MasterMind Chess game. The module offers the option to the players to choose if they want to use the right or left hand to control the game. After tracking the chosen hand’s position, the gestural module sends the information in packets to the MasterMind Chess application. The Kinect device was positioned below the monitor, facing towards the player. Other approaches use the Kinect above the monitor, facing down, as in [18]; however, for our particular application, it was found that the Kinect performed better in this position and require less calibration to recognize the players’ hands.
These packets containing the mapped position of the user’s hand to the screen are organized in a seven-bit array and sent to the game through the UDP protocol. This packet is then read and translated to an action in the game, such as moving the cursor or selecting a piece, for example. The gestural interaction is not limited to the game itself; that is, it can also be used to navigate through the main screen of the software and the other menus. We found that using UDP for the communication between the application and the gestural module was easier to implement than using interprocess communication and shared-memory algorithms.
The action of clicking is represented by “holding the hand steady” at a position for a certain amount of time. In implementation terms, this means that if the user moves the hand in a radius of what corresponds to at most five pixels on the screen (or approximately 0.13 cm in space) during a period of 800 ms, this will be interpreted as a click. Furthermore, there is a 600 ms delay between two clicks, to try to avoid unintentionally clicking two times in a row.
A technical problem regarding Kinect’s field of view occurs when the mapped position of the hands stands near the bounds of the game screen, as the Kinect starts to fail at recognizing a precise location for the hand and the cursor starts to tremble, making it hard to select or move a piece, since the clicking action requires a steady position of the cursor. To avoid this problem, the mapping function that translates the skeleton coordinates of the Kinect into screen coordinates in the MasterMind Chess application was adapted to fit the view, so the boundaries of Kinect’s field of view remain out of the MasterMind Chess board range. This approach might cause a loss of precision regarding capturing the speed of the hand but avoids some of the imprecision caused by the hardware.
4. Case Study
In order to evaluate the user feedback regarding the new gestural interface, an activity was made using chess players with different levels of skill in the game. The users were asked to play the game using both the gestural and the mouse/keyboard interface and gave their opinions about the flaws and qualities of each one.
4.1. Research Questions
The general research question that this study presents and tries to answer is whether a gestural user interface is suitable for slow-paced board games. The specific research questions include the following:(i)Is gestural interaction suitable for serious chess-playing?(ii)What usability issues adding a gestural interface brings to the game? These issues can be overcome?(iii)How the gestural interaction performs while compared to the mouse/keyboard interaction?The evaluation process was composed of questionnaires, interviews, observation, and video analysis.
4.2. Methodology
For the application of the activity, there were two facilitators, one interviewer, two cameramen, and one background observer who overviewed the activity and only intervened when the facilitators required additional assistance. The interviewer performed the interview, recorded the audio, and made additional annotations to the answers. The cameramen filmed the entire sections of activities with users in different rooms. The facilitators resolved questions and problems of the chess players during the activities. They were also responsible for observing notable events and user reactions, in order to support the interviewer by indicating additional questions that should be posed to specific users.
4.2.1. Participants
A total of eight subjects participated in the activities, seven male and one female, with ages ranging from 19 to 25. The subjects were divided into three groups of four or two subjects each (), according to their skill levels in chess, that is, beginner, intermediate, and advanced. The beginner group (, and ) was formed by the subjects with only basic knowledge of chess, sufficient enough to know how to move the pieces and end the game. The intermediate group ( and ) was composed of the subjects that had a solid knowledge of basic chess strategy, such as famous opening and ending moves. The advanced group ( and ) was formed by players who have a deep knowledge of chess, including advanced strategies and counterstrategies. These players had sound ELO (the ELO rating is a mathematical method to calculate the relative skill of a player in competitive games) and FIDE (the World Chess Federation, FIDE (Fédération Internationale des Échecs), is an international organization that regularizes and connects national chess federations all over the world) ratings and have prospects of a professional career in chess, such as participations in big tournaments. Only two volunteers were available for participating in the advanced and intermediate group each, so that there was a difference in the size of the groups.
All users are undergraduate or graduate students. None have any kind of special need. Table 1 summarizes the demographic data from all subjects.
4.2.2. Procedure
The activity was explained to the participants as the first step. The background observer explained each step and what the participants would have to do. The main features of the game were described and the users were given instructions on how to use the controls. To ensure that all participants received a comparable level of help and instructions, the same instructions were given to all participants, and each was aware that they could ask questions regarding the game and the experiment at any time. The experiment only began once the user stated that s/he comprehended the instructions.
Four questionnaires were used. The first questionnaire gathered essential demographic information about the subjects, such as the age, gender, if the user is right-handed or left-handed, their level of skill in chess, and how often they play the game. The second questionnaire addressed questions about the mouse/keyboard interaction with the game and was applied after the user played the first match using this interaction. The questions included whether the players used the keyboard to play (and if not, why), whether they tried to use any hotkey, whether they suggested any hotkey, and which was the preferred control device (mouse or keyboard).
The third questionnaire considered the gestural interaction and was applied after the users played the match using the gestural interface. Some of the questions included whether the users already used a gestural technology, such as the Kinect, whether they tried to use another body part to interact with the game other than a single hand, whether they felt that the performance in the game was affected by the gestural interaction, whether this mode of interaction fit in the context of the chess game, whether they would use this type of interaction to play chess again, and whether they could satisfactorily perform all the desired movements. The fourth and last questionnaire was about the game in general. This included questions that regarded overall aspects of the game, such as if the visual elements of the game were presented in an intuitive way, if the user interface was enjoyable, if the current sounds of the game were sufficient, if the user missed some feature, and what kind of interaction, between mouse/keyboard and gestural, best fitted in the context of the game. This questionnaire was applied after the users had played both matches.
After the questionnaires, each user had a particular interview to probe any information left unanswered. The common questions of the interview were about how the users enjoyed playing the MasterMind Chess, whether they had any difficulties in interacting with the game and what these were, what interaction mode they liked most and why, whether they felt inhibited to play due to the fact of being filmed, how they were feeling during the match, and overall suggestions. Other questions were asked depending on the answers given by the users in the questionnaires. If an answer was too vague or required explanation, it was reinforced in the interview.
The activity’s overall process is illustrated in Figure 1.
The facilitators observed the users in their activity and made annotations about their reaction and extraordinary events. Some examples of these reactions and events include whether the user talks during the match, if s/he makes comments or complains about the gameplay, if s/he shows any signs of disappointment or frustration, and whether s/he is having any kind of difficulties in playing the game.
Each group was divided into pairs of subjects, making a total of four pairs of equal level of skill between themselves. Each of these pairs was put to play for a total of two matches of 15 minutes for each player (i.e., 30-minute maximum time) against each other, in separate rooms, that is, one match using mouse/keyboard and another using only the gestural module. The second pair of beginner users played in the inverse order (i.e., first gestural interaction, then mouse/keyboard). For the users in intermediate and advanced levels, one of the matches (the first match in the advanced level and the second in the intermediate level) was in Capablanca mode, since its board is larger and the gameplay could be slightly different.
In the mouse/keyboard match, the users were sitting on a chair and had only the mouse, keyboard, screen, and speakers (to hear the sound feedback of the game) available to interact. In the gestural match, the users were standing in front of the screen at an arbitrary distance. As the Kinect has a limited field of view, the sensor was adjusted to the person’s height, for it to be able to capture all of the user’s hands movements. The users were allowed to speak freely and make any commentary or observation during the match. After each match, they answered a questionnaire about the gameplay and usability of the type of interaction used and then a questionnaire about the game itself and then participated in a particular interview, to probe information left unanswered in the questionnaires. Figure 2 represents the configuration of the rooms and subjects for each match.
4.2.3. Design
In this case study, there were two independent variables:(i)Game interaction: mouse/keyboard and gestural.(ii)User expertise in chess: beginner, intermediate, and advanced.
Each user was tested on each interaction; therefore the study is a 3 × 2 within-subject design [19]. A 2 × 2 balanced Latin square [20] was used as a counterbalance in the beginners group to alternate the order of the interaction (i.e., users and played first with the gestural interaction mode). No counterbalance was used in the intermediate and advanced groups, as only two matches were played by each group.
4.3. Results
The activity revealed some important details about the gestural interaction and how the participants understood the game. By observing how the participants behaved while playing with the gestural module and given their feedback on the questionnaires and the interview, it became clear what points should be changed and improved both in the game in general and in the Kinect interface.
Regarding the mouse/keyboard interaction, no user claimed to have difficulties using it. The mouse/keyboard questionnaire had seven questions that the users answered after playing the game using only this kind of interaction. The first one regarded the use of the keyboard to play the game, and seven users had not used the keyboard at all. Only one user () used the keyboard instead of the mouse. This particular user said that the keyboard was more practical and accessible. All the others claimed that the mouse was faster and more precise, that they were used to the mouse to interact with user interfaces, and that they did not feel the need to use the keyboard.
Only one user () found it to necessary to use hotkeys. He suggested using the Escape key to deselect a piece. No user had tried to use a hotkey while playing. This was somewhat expected, as there are almost no special commands that could be used in a regular chess game, except for chatting, for example. If the same command that is used to move the pieces is also used to control the other functions of the game and if this command is satisfactory to the users, the handling of these options tends to also be of easy use.
Only one user () felt mental fatigue while playing, but no user felt physical fatigue. This particular user said that he felt “only a little” mental fatigue, in his words. As one user () commented, for a player used to playing computer games and/or chess very frequently, remaining seated for half an hour to play hardly causes any type of physical fatigue. All users stated that the way of selecting and moving the pieces was adequate.
Regarding the gestural and Kinect technology, only two users ( and ) had never used the Kinect before. No user tried to use any other part of the body than the hands to interact with the game. Seven users felt that their performance was impaired because of the gestural interface. From these users, four (, , , and ) said it was because the interface was slower than the mouse/keyboard. The others found it difficult to move the piece using the gestures.
Only one user () felt mental fatigue while using the gestural interface. Three users (, , and ) had reported feeling pain or discomfort in a part of the body while playing. and felt pain or discomfort in the legs, in the shoulders, and in the hand.
Regarding the movements, four users (, , , and ) stated that they did not accomplish all the movements they wanted in a satisfactory way. Only one user () said that he made an erroneous movement because of the interface; the others said that the precision in the corners of the board declined. Four users (, , , and ) said that the gestural interaction disturbed the playability of the game; three users (, , and ) stated that the gestural interaction fit in the context of a chess game.
In relation to whether the users would use the gestural module to play chess again, only two users ( and ) would do so. The others stated that the reaction time of the game increased during gestural interaction and that the game would be best played while sitting but would consider using the gestural interaction again if those changes were made. Nevertheless, it is important to notice that these statements may have come from a good-subject effect or a novelty effect.
Figure 3 compares the match durations of different interaction types for each player pair. Reaching the 30-minute mark—gestural matches of versus and versus —means that one of the players lost by timeout.
The great majority of the users had already used the Kinect technology, usually to play games on the Xbox 360, so they were habituated to using gestures to control the interface. In relation to other Kinect-based games, the MasterMind Chess gestural interface should be intuitive, as the user only uses a hand to control the cursor and click. Consistently with this conjecture, no user tried to use another body part to control the game. However, the majority of users felt that they had a lower-than-expected game performance, mainly because of the way used to select the pieces.
Two users ( and ), the advanced ones, said that the time it took to select or deselect the piece consumed too much of the user’s time. It takes around one and a half seconds to select and deselect a piece when the cursor is steady at the position, but if a user selects a piece and stays with the cursor at the same position for another 1,400 ms (800 ms standard time plus 600 ms of cooldown), the piece is selected again; that is, the user accidentally deselects the piece. This points towards the problem of the timing: as chess is a time-critical game to competitive players, the time that a player loses by selecting a piece impacts their performance. This could perhaps be overcome by adding a time bonus to the gestural user’s clock corresponding to the additional time it takes to move the piece in this mode or using an additional body movement to instantly click, for example, using the height of the other arm as reference; that is, the user would select the location of a piece with one arm and rise the other above the previous to select the piece.
Another issue caused by the timing for selecting a piece is that the users might accidentally move a piece to an undesired location, if they move the cursor too slowly. This caused some cases of the player moving the cursor much slower than expected, leading to unintentionally “clicking” in the middle of the path to the desired square and thus losing a piece or ruining the player’s game strategy.
Yet another problem that the users reported and that influenced their performance was that it was hard to control the cursor in the borders of the screen, making it difficult to select a piece in the corners of the board that was close to those borders. However, this is a technical problem of the Kinect platform, which becomes imprecise on the limits of the field of view of Kinect’s infrared camera. As stated before, one possible solution would be to adjust the calculation of the conversion of coordinates between the Kinect and the MasterMind Chess in a way in which a small movement of the hand causes larger displacement of the cursor on the screen. That way, the user would not require going to the borders of Kinect’s field of view to reach the outer borders of the board. This occurs because the gestural interface converts the exact size of Kinect’s view to the exact size of MasterMind Chess’s view; that is, to move from one corner of the screen to another, the user has to move the hand all the way across Kinect’s field of vision. However, as already mentioned, decreasing the necessary movement to move the cursor along the screen also decreases the precision; that is, the player must make more cautious movements.
One issue that most of the users reported was that it was at times difficult to recognize the pieces, since they needed to maintain a certain distance to the screen due to Kinect’s minimum required distance for recognizing the person’s body. As MasterMind Chess pieces have a 3D design that is seen from above, from a distance the pieces might look similar, especially the black ones. This forced most of the users to get closer to the screen to be able to distinguish the pieces, as shown in Figure 4.
While some of the users only approached to see the pieces and then got back to their original position to make a move, others tended to remain at a closer position to the screen as the game progressed, as shown in Figure 5. The figure also shows the time of each image.
Related to the fact that some users complained that they could not remain seated to play, one possible solution would be to develop a method in which the users could use the gestural module in the same way they use the mouse/keyboard input, that is, while sitting in front of the screen. This is virtually impossible using the Kinect platform, due to the minimum distance to recognize the person and the fact that the position of a sitting person is harder to identify. This could be solved by using another gestural technology that does not require seeing the entire player, but only the hands. A better approach in this case would be to use the Leap Motion technology, as it can be placed in front or at the side of the screen and recognizes the hands and fingers of the player in a volumetric field of view on top of it. Therefore, the player would be able to play the game while sitting, using the fingers to select the square and pushing it forward to click. A similar approach was used in Robot Chess (see Section 2), but no evaluation with users was yet made to compare the results.
When asked whether the gestural interaction fit into the context of the game, the opinions were divided. Some users said that the gestural interaction did not fit in because chess is the type of game that requires the players to stay seated while thinking and that it is unnecessary to stand while playing chess. When asked if they could have the option to play with the gestural module while sitting, the great majority of these players said that it would be much better.
Even not considering the gestural interface to be the best type of control for a chess game, some users stated that they would play again with the Kinect, for it is an interesting type of control that adds more interaction and movement to the game. This seems to be consistent with literature about tangible and natural interaction. Despite lower accuracy/precision and lower comfort/higher fatigue, users seem to be more motivated and to enjoy the interaction more. Others said that they would rather use the mouse, for it is more direct and precise and requires less time to move a piece. The players that would rather use the gestural interface were the more casual players, while the users that preferred the mouse for its precise and faster input were more competitive players, that is, cared more about their game performance.
When asked for suggestions to improve the gestural interface, many opinions were put forward as a result of the mentioned issues. Most of the users recommended improving the precision of the clicking action and the borders of the screen. One user suggested that it should require less movement of the arm to move the cursor, that is, to increase the sensibility of the movement. Three users recommended that, instead of waiting for the gestural interface to recognize the click gesture just by holding the position steady, there should be a specific click command, such as using a specific gesture of staying in a pose, to avoid clicking in an undesired position. Two users suggested playing while sitting. Comparing the two forms of interaction, all users found that the mouse/keyboard best fit to the game context.
Although the players have complained that the reaction time was increased, no significant difference was noted in the total time of the matches, as seen in the chart of Figure 3. However, given the low number of participants, this result cannot be generalized. It is also important to notice that this slight difference in time can be more perceived by experienced players, as they want to spend the minimum amount of time to make their moves.
4.4. Lessons Learned
In the following, we present the main lessons we learned from the case study. These lessons are mainly related to MasterMind Chess for Kinect. However, some lessons might be generalized to gestural interaction for chess-like board games or to Kinect-like devices for chess-like board games (the difference between these two is that Kinect enables bodily interaction beyond gestures).
(i) Interaction Mode-Independent Usability Problems. During gestural interaction it was possible to detect usability problems that were not detected previously but that also affect the interaction with keyboard/mouse. The poor visibility and recognizability of certain chess pieces due to poor contrast and realistic icons were only detected during gestural interaction when users were more distant from the screen. Fixing this problem also benefits users of MasterMind Chess on smaller screens or in environments with bright lighting.
(ii) Kinect-Like Devices Are Not Very Adequate for Point-and-Click Interaction. The Kinect version of MasterMind Chess essentially emulates point-and-click actions of the mouse using gestures of the hand. These actions require a considerable accuracy of motor movements. Due to the limited accuracy of the Kinect sensors, users had a lower performance than when using the mouse. Furthermore some users experienced fatigue or discomfort. These two facts probably explain why only two of eight users stated that they would use the interaction mode again.
(iii) Kinect-Like Devices Might Not Be Adequate for “Serious” Chess-Playing. MasterMind Chess is essentially a chess game for serious players playing tournament-like matches using time control. It does not use playful design elements such as Battle Chess (Battle Chess is a classic computer chess game where the pieces are animated, both in moving and in capturing others; each piece has its own animation) nor does it target novices who might need tutorials or other resources for learning chess. As a result, some players complained about the low accuracy of Kinect which resulted in moves taking more time. Other users suggested that the game should be played while sitting. Thus it seems that gestural interaction using Kinect-like devices might not be the best choice for tournament-like chess-playing. However, it is not clear whether this is also true for other variants of serious chess-playing [21] such as a simulation of open-air chess (open-air chess—or outdoor chess—is like a common game of chess but played with human-sized pieces on a very large board, usually located in an open space (thus its name)). Furthermore, it is also not clear to what extent this insight is generalizable to other slow-paced board games.
(iv) Gestural Interaction Is Not an “Add-On.” The previous lessons made it clear that gestural interaction is not simply an additional input mode that can be added to an existing application. The direct mapping of mouse actions to hand gestures proved problematic due to technical limitations but also due to ergonomic considerations. Even if we had used the Leap Motion it would not have been clear whether users would have yielded higher performance or higher levels of satisfaction. Thus the game controls in the gestural mode should be redesigned to cope with the limited accuracy and to minimize fatigue and discomfort. For example, the second arm might be used for certain controls in order to provide more balance to body movements. Instead of point-and-click actions, the goal-crossing paradigm [22] could be used. Furthermore, when selecting the home square, only the target squares that constitute valid moves could be highlighted and enabled for selection. Goal-crossing might also benefit players with motor impairments. Highlighting and enabling only valid squares might benefit novices.
5. Discussion
One of the main results was that our activity revealed usability and design problems that were not detected during mouse/keyboard interaction but that also would affect users of these devices. Another important result was that gestural interaction is not just another input mode that can be easily mapped to existing controls. In order to leverage its benefits, application controls need to be adapted. Furthermore, we found that user satisfaction and perceived performance, and therefore overall acceptance of the gestural interaction mode, were lower than when interacting with mouse/keyboard. This result could be explained with technical limitations on the one hand and the framing of MasterMind Chess for Kinect as a serious chess game on the other hand. This also relates and brings the question of why a purely gestural interface may not be a natural choice for board games.
These games usually are almost always discrete, since it does not matter where a token was placed within a square, as long as the correct square was selected. Games that are commonly used in gestural interfaces, such as the ones of dance and sports genre, are usually more continuous; that is, specific locations are taken into account when placing a token. As gestures provide good access to spatial information, it feels more natural to move or place tokens in such games rather than in discrete ones. However, the discrete part of the action in such games, such as picking up a token, may not be very suitable with gestures (as the activity showed), and this might explain why hybrid techniques such as the Wii controller perform well in these domains, as the gesture is used spatially and the button indicates an action. Other specialized controllers, like a dance mat, can be used to indicate discrete events. It is not clear whether this kind of hybrid control would perform in a casual or competitive chess game, and it is an interesting topic for research.
Since half of the participants reported a satisfactory interaction and since one of the two expert chess players stated that he would use the gestural interaction mode again, we are positive that gestural interaction with slow-paced board games such as chess provides an interesting topic area for future research, assuming that no good-subject or novelty effects took place.
Most of the lessons learned are related to the fact that the gestural interaction module was used as an add-on to an existing game that had not been redesigned to leverage the interaction potential of Kinect. As a result, players that focused on their chess performance expressed a somewhat lower satisfaction. Reframing the design problem as “designing a (playful) chess-like game that makes use of the interaction potential of Kinect-like devices” might yield different results. Two simple sources of inspiration might be the previously cited simulation of open-air chess or chess boxing, that is, the alternation between rounds of chess and boxing. Such a redesign would probably result in the user’s reframing of the game as one the primary purpose of which is more playful and less focused on the chess game itself. This might lower expectations regarding accuracy and timing and increase overall satisfaction and acceptance, especially in more experienced users.
With these results, the research questions can be addressed. Regarding the first specific question “Is gestural interaction suitable for serious chess-playing?” it is clear from the case study that Kinect-like devices do not perform well for serious players, as their precision does not allow the fast movement that the players require. However, as some users reported that they liked the Kinect more as they had more fun playing with this interface, it is possible that, in a specifically designed playful chess game, the use of a gestural interface could be suitable.
As for the second question “What usability issues adding a gestural interface brings to the game? These issues can be overcome?” the study revealed that many usability issues that were not present in the mouse/keyboard interaction arise with the gestural interface. However, most of these issues can be addressed to factors that are not related to the gestural interaction itself; the chosen device has inline problems, such as low accuracy and the need to be standing, which affects the gameplay, and the game was originally designed for mouse/keyboard interaction. As in the first question, it is possible that most of these usability problems would not arise with a chess game designed specifically for this interface or using other kinds of gestural interfaces.
The answer for the third question “How the gestural interaction performs while compared to the mouse/keyboard interaction?” is also clear from the results. The mouse/keyboard interaction performed better in precision, accuracy, and preference of the users. Nevertheless, these results are valid for the context of Kinect-like devices and cannot be generalized to other gestural devices, such as the Leap Motion, which uses a different gesture recognition paradigm, or “wearable” gestural devices (e.g., [23–26]). Also, despite being used for a slow-paced board game, the result on the user preference could be different if a more playful game was used, such as an instance of open-air chess. Therefore, one should be careful while designing the game for using gestural interface for slow-paced board games using Kinect-like devices, as accuracy problems will arise and should be handled accordingly, as the playfulness of the game may not be enough to compensate this issue.
6. Conclusion
In this work, a case study of the MasterMind Chess computer game with a gestural interaction mode enabled by a Microsoft Kinect sensor was presented. We described an activity with eight users of three different chess-playing skill levels, comparing mouse/keyboard interaction with gesture-based interaction. The results of the activity allowed us to synthesize lessons learned. These results apply to the specific case of MasterMind Chess for Kinect but can also partly be generalized to gestural interaction with slow-paced board games and to slow-paced board games for Kinect-like devices.
Future work includes investigating other gestural technologies, such as the Leap Motion and Asus Xtion, for gestural interaction with “traditional” slow-paced board games, as well as designing slow-paced board games with gestural interaction enabled by Kinect-like devices.
Competing Interests
The authors declare that there are no competing interests regarding the publication of this paper.
Acknowledgments
This work was partially supported by the Brazilian National Council of Scientific and Technological Development (CNPq Grant no. 130158/2015-1) and by the Physical Artifacts of Interaction Research Group (PAIRG) at the Federal University of Rio Grande do Norte (UFRN), Brazil.