Abstract
We present SONRIE, a serious game based on virtual reality and comprising four games which act as tests where children must perform gestures in order to progress through several screens (raising eyebrows, kissing, blowing, and smiling). The aims of this pilot study were to evaluate the overall acceptance of the game and the capacity for detecting anomalies in motor execution and, lastly, to establish motor control benchmarks in orofacial muscles. For this purpose, tests were performed in school settings with 96 typically developing children aged between five and seven years. Regarding the different games, in the kissing game, children were able to execute the correct movement at six years of age and a precise movement at the age of seven years. Blowing actions required more maturity, starting from the age of five and achievable by the age of six years. The smiling game was performed correctly among all ages evaluated. The percentage of children who mastered this gesture with both precision and speed was progressively greater reaching more than 75% of values above 100 for children aged seven years. SONRIE was accepted enthusiastically among the population under study. In the future, SONRIE could be used as a tool for detecting difficulties regarding self-control and for influencing performance and the ability to produce fine-tuned facial movements.
1. Background
Currently, children are surrounded by technology, whether this being at home, at school, or in entertainment areas. Starting in kindergarten, at the age of five, children are beginning to learn via the use of technological gadgets such as iPads and computers [1]. Society today continuously introduces technology into as many aspects of life as possible, even at elementary schools, with the target of faster access to information, enhancing learning, and providing mechanisms for interaction that are motivating for children [2]. The book published in 1997 by Sandholtz et al. stated that “if technology is used appropriately, it has been shown to have many positive benefits and to create student-centered environments” [3]. Over recent years, after the appearance of the smart phone, it is especially striking to observe how children are interested in the use of technology at increasingly early ages and are able to learn to use these devices with comparative ease.
This paper presents SONRIE as a serious game based on virtual reality and which aims to facilitate the process of detection and rehabilitation of difficulties affecting the orofacial muscles in children aged between 4 and 12 years. A multidisciplinary team of professionals (comprising occupational therapists, physiotherapists, engineers, and child educators) designed and developed SONRIE using four games based on a medieval theme. SONRIE presents the child with tests (in the form of games) that require the execution of different facial movements which seek to detect the correct activation of the orofacial muscles during the performance of the same. SONRIE uses the 360 Kinect sensor and is based on virtual reality (VR) games; thus it is a low-cost solution.
Recently, the researchers involved in the design and development of SONRIE presented a study on the application of SONRIE to a group of 10 children with cerebral palsy (CP) aged between 4 and 12 years [4].
The current paper presents the results of a pilot study performed using SONRIE on a group of 96 typically developing children, attending the Amanecer School (Madrid, Spain). The research team established the working hypothesis that SONRIE would be accepted by children with typical development within the context of their own school. Thus, as stated in the work published by Druin in 1999, it was important to include the participation of children for the development of technology designed to be used by children [5, 6]. SONRIE is well-suited to the context of schoolchildren as it is based on a game. Also, the chosen theme is appropriate for the intended age of the participants and the form of use is both novel and appealing.
2. Methods
2.1. Participants
A total of 96 children (of both sexes) participated in this pilot study. The age of participants ranged between five and seven years and they attended the Amanecer School in Madrid, Spain. The mean age of participants was 5.84 and the standard deviation was 0.65. The Amanecer School is a private center attended by middle-class schoolchildren (although this sociocultural aspect has no implication in the results obtained). Before commencing the process of validation at Amanecer School, we had to consider ethical issues related to an intervention with a highly vulnerable population. This study was approved by the Ethics Committee of the Rey Juan Carlos University. The trial was conducted in accordance with the Declaration of Helsinki, amended in 2013 by the 64th World Medical Association (WMA) General Assembly (October 2013, Fortaleza, Brazil) [7].
This study evaluates the appearance of any type of difficulty during the performance of the movements, as requested by the different game interfaces, as well as the existence of correlations between better game skills according to the age or the sex of users. Furthermore, this study aims to establish motor control milestones for the orofacial muscles involved in the requested movements (execution with precision and speed).
At the onset of the study, a test was performed with a group of five children aged four years and children over the age of eight. This revealed that the younger children were lacking the sufficient attentional capacity to enable them to play the proposed games independently. This meant that the results would be invalid. For this reason, we decided to exclude four-year-old children from the group of tests performed. Moreover, in the children over the age of eight (with typical development) the performance of the different movements almost always provided high scores. Therefore, we considered that this study, when performed on healthy children over the age of seven, would not be relevant for the purpose of this research.
2.2. SONRIE Serious Game
Figure 1 presents the features of SONRIE, which shows all the screenshots of the SONRIE game. The SONRIE serious game and its framework have been developed by a multidisciplinary team of professionals. The game’s design has considered the experience gained by occupational therapists while working with children with special needs.
2.3. Description of the Interfaces Featured in the SONRIE Game
Play motivates children and makes them want to participate in a particular activity, as long as it is designed within the parameters regulating games. Additionally, if it is interactive and technological, its capacity to motivate children increases and adhesion to treatment improves.
Physical therapists working with children may configure the SONRIE game according to the needs and progress of each child in order to design an effective therapy intervention. To achieve this, the therapist must set up a series of specific configurations [4]. For instance, the therapist can configure the number of repetitions for each game and establish a time limit for games.
The results obtained by each respective child for each game provide valuable information; thus, SONRIE continuously stores information regarding the performance of each child during the game, with data concerning whether the child has performed the movement or not, the time it took, and whether the child needed to use other muscles in order to accomplish the movement.
Thus, the gaming platform works by executing the stored values associated with each child and ensuring an appropriate and dynamic game. Once the game is over, the results are stored for subsequent review by a therapist.
The SONRIE’s games cover each of the movements in a simple, guided way, using a motivating interface for the child.
When the game begins, the introductory screen appears (shown in Figure 2); next, the Kinect performs full-body skeleton detection and tracking. Subsequently, the eyebrow raising game begins.
This game runs until movement is detected or until the time limit for execution of the exercise has been reached. Figure 2 shows the start and end interfaces of the eyebrow raising and blowing games. As shown in Figure 2, we designed stimulating starting and ending screens for the children for each of the games.
If this screen is still on display after the half time limit, because the child has not yet performed the movement, an audible sound will be heard to provide the child with positive reinforcement (see Figure 2).
Once the gesture is considered appropriate, the number of times the game has been played is counted. If it is the first or second time, the eyebrow video will repeat, but if the count is higher, the program will continue to the next game. In other words, each game must be played the same number of times as the assessed repetitions, with a time limit that comes with each corresponding exercise [4].
The kissing and the smiling games follow the same procedures as the previous one: a wait until the movement is recognized or the time limit is reached, which is followed by the final video for each game.
2.4. Description of the Values Returned by SONRIE
(i)Degree of acceptance: during the test with SONRIE in a real scenario, the children were asked about their satisfaction with the game after they played with the serious game. Up to 98.0% of children responded positively, showing great enthusiasm and acceptance towards the SONRIE serious game.(ii)Accuracy and speed of movement: with the Kinect 360 sensor it is possible to measure the accuracy of the requested movements. This has been done for each of the movements by comparing the distances between the points involved in each movement before, and after, the execution of the same. This difference between distances enables the calculation of a threshold value that indicates the movement accuracy. Besides, the speed of movement is obtained as the difference between the time when the child is requested to perform the movement and the exact moment that the movement takes place.(iii)Description of the algorithm used: during the execution of each game featured in SONRIE, the child obtains a numerical value (between −1 and 400) per game. This number indicates the precision and the execution speed of the child’s facial movement as prompted by SONRIE. Some possible values for the facial recognition algorithm are as follows:(a)Negative values: the child performs the opposite movement to that proposed by SONRIE.(b)Values between 0 and 99: the child executes a correct facial movement; nevertheless he/she does not reach the minimum threshold which measures the precision of the facial movement performed.(c)Values between 100 and 400: the child successfully carries out the facial movement prompted by the SONRIE serious game. The return value increases depending on the seconds required and the precision of the facial movement.
2.5. Stages Carried Out in This Study
Figure 3 shows the main stages carried out by SONRIE serious game and the final study necessary in order to evaluate the results.
2.6. Occupational Performance in Children
The performance components constitute the skills needed to achieve each task. Without these, daily activities cannot be carried out effectively. These components [8], which may be classified into sensorimotor, cognitive, psychosocial, and psychological components, are strongly influenced by the context. Aspects such as the age of development, the chronological age, the health status, or the moment in the life cycle clearly influence the acquisition and successful engagement with each component.
Fisher et al. (2006) defined these skills as being small and measurable units in a chain of observed actions that occur while the person performs meaningful tasks [9]. These skills are learnt and developed over time and take place in specific contexts and environments. Rogers and Holm (2008) proposed that, during the development of these concrete skills during a task, several functions and structures of the body are united in unique combinations that may affect performance in daily life [10]. Therefore, no mathematical law or other law can be established to regulate the appearance of one or another isolated ability within the framework of each individual’s behavior. To better understand this, we propose the graph featured in Figure 4, featuring a scenario where a child is asked to say her name in class when prompted to do so by a teacher.
During the game, in order for the child’s occupational performance to be considered appropriate, two requirements must be initially fulfilled: these are motivation and suitability of the proposed activity according to the preferences and interests of the child [6]. When these requirements are fulfilled and considering the activity analysis of the components of performance proposed by Máximo-Bocanegra in 2011 [11], different and varied components are required which are gathered in Figure 5. SONRIE favors the practice of the following performance components: visual, muscle tone, praxis, oral-motor control, attention, self-expression, and self-control.
Regarding the sensorimotor component, the SONRIE sensor is able to detect changes that occur during the movement of certain muscles. Table 1 lists the facial muscles involved in each of the movements performed in SONRIE.
2.7. Statistical Analysis
To carry out the statistical analysis we decided to use a box and whisker plot, as these diagrams are one of the most commonly used in statistical analysis and are appropriate for the purpose of this study.
A box and whisker plot is especially useful for indicating whether a distribution is skewed and whether there are potential unusual observations (outliers) in the data set. Box and whisker plots are also very useful when large numbers of observations are involved and when two or more data sets are being compared, as applies to this experimental study.
A box and whisker plot is a way of summarizing a set of data measured on an interval scale. These diagrams are often used in explanatory data analysis by showing the shape of the distribution, its central value, and its variability.
3. Results
This study was performed over five sessions. Figure 6 represents the distribution of the sample by age. The most frequent age of participating children was six years. The age of participants is related to the fact that the tests took place in the month of May, and therefore many children in their final year of nursery school had already reached the age of six. Likewise, many first-grade students still had not reached seven years of age. The mean age of children participating in this study was 5.8 years.
One of the first objectives of the present pilot study was to analyze the acceptance of the game in children with typical development within the aforementioned ages. All the children who were invited to play with SONRIE were subsequently interviewed using an unstructured survey regarding their experience with the game. They were asked whether they liked it and whether they wanted to play again. Up to 98.9% of children responded positively, showing great enthusiasm and acceptance towards SONRIE.
Regarding the objective of detecting possible execution anomalies, we were able to observe that 3% of the children under study displayed difficulties following the proposed games, as they were unable to remain still and attentive during the instructions provided, which resulted in subsequent execution errors. The researchers informed the school’s educational psychology team of these results to enable them to evaluate the students and establish a follow-up process, if necessary.
On the other hand, this study aimed to evaluate whether differences existed in the execution of the movements evaluated in SONRIE and whether motor control milestones for the orofacial muscles could be established. To exemplify this, Table 2 displays the mean and the standard deviation provided by SONRIE for each game.
During the tests, observations of the children were made which were gathered in a register designed for this purpose. These observations helped clarify some of the difficulties displayed by the children. In the eyebrow raising game, 9% of the cases with a threshold of zero did not know how to raise their eyebrows and 2% were unable to stay still. In the smiling game, 1% moved a lot during the execution of the game. In one of the cases, a girl did not score well on the smiling game as she only raised one side of her mouth.
For the overall data analysis, we designed a graph with the complete data. Figure 7 graphically presents the distribution of the execution values for each of the games.
An analysis of the results shows that, in the first place, in the smiling game (purple) some data is above the threshold of 400. These values reflect an incorrect execution of the movement, as previously mentioned: that is, the researchers observed that these were children who were unable to stay still without moving other parts of the body. Due to the sensitivity of Kinect, if the child moves too much, the game produces values above 400 (indicative of a possible error).
On the other hand, analyzing the results based on the tests included in the four games, we can conclude that the behavior of the blowing (red) and kissing games (green) was similar with regard to the distribution of the data. During the performance of these games, most participants obtained performance threshold values between the first quartile (Q1) and the second quartile (Q2-median). This demonstrates that both these games are homogeneously sensitive in their relation with the developmental evolution of the child.
The game with the worst execution was that of raising the eyebrows (blue), due to the fact that 51% of the time the game produced a threshold value of zero. However, this did not always mean that the child was unable to raise the eyebrows. This may be due to several reasons:(i)The child did not perform the movement with the necessary precision.(ii)There may have been a lack of understanding of the movement required.(iii)The order of the game in the series makes the execution more difficult, as the eyebrow raising game is the first movement that SONRIE displays.(iv)The children have not yet automated the rules and content of the game.
Smiling is the game with the greatest dispersion of data. The inferior part of the box is greater than the superior half, as displayed in Figure 7. However, this does not necessarily mean that the thresholds returned by this game (between 25% and 50% of the sample) are more dispersed than those between 50% and 75%.
Figure 8 shows three box and whisker charts which establish the dispersion of the values obtained for each of the games, according to the child’s age.
It is interesting to analyze the data obtained according to the age of the children. This analysis enables us to establish possible thresholds of motor control development for the orofacial muscles. The muscles involved in each movement are summarized in Table 1. Thus, in the kissing game (in green) a correct movement is obtained at the age of six, whereas a precise movement is obtained by seven years of age. Prior to these ages, the values are below 0, which is indicative of greater difficulties in the execution of this movement and, of course, a lack of precision.
For the blowing game, the data indicated that all children performed it correctly, without age distinctions; however, many five-year-old children obtained values below 100, whereas, after the age of six, approximately 75% of the values were above 100. This means that this movement is in the process of maturation from the age of five and is only attained by the age of six.
The smiling game was correctly executed among all the ages evaluated. The percentage of children who dominated the game with both precision and speed was progressively greater, reaching more than 75% of the values above 100, among the seven-year-old children.
The results of the eyebrow raising movement were similar among all age groups; the lowest value was 0 and there were some values registered above 300 among all age groups. However, as the number of six-year-old children was significantly higher (56% of the children were six years old) the median was 0, which indicated that a considerable number of children performed the correct movement without considering it to be precise and fast. This is due to the fact they only performed a slight eyebrow raising movement and failed to reach the minimum movement required.
Figure 9 graphically displays the median obtained by all the children in each game, grouped by age. We can see that, in all the games, except for the eyebrow raising game, an evolution of the motor execution of the requested movements took place. The seven-year-olds scored better than the six-year-olds, and these were better than the five-year-olds. Of the four games in SONRIE, the smiling game is where all children achieved the best execution. The eyebrow raising game is where there was the greater variability of data, showing a different tendency compared to that previously explained for the other games. There were no differences between values according to the participants’ sex.
4. Discussion
Chen et al. [12] stated that the treatment of motor speech dysfunctions in children with CP requires an understanding of the mechanisms underlying motor control. However, there is a lack of literature on quantitative measures of motor control. The present findings, which are based on the SONRIE game, enable us to have a greater understanding of the oral-motor mechanisms underlying language, together with nonverbal communication and, even, the emotional expression of children without pathologies.
Games based on VR have been employed as therapeutic tools to circumvent possible difficulties in the rehabilitation set for children with CP, as well as for other neurological disorders [13, 14]. The last decade has produced many reports relating to the use of Kinect in processes of body rehabilitation, balance, motor control, upper limb rehabilitation, and even cognitive rehabilitation. Over the last year, several studies have been published focusing on the rehabilitation process or the detection of pathologies which affect the orofacial muscles in adults using VR. Among these, we can highlight the following: () studies based on healthy adults who present a prototype for the identification and scoring of asymmetries in facial palsy [15]; (2) also, a study proposing the creation of a database regarding facial expressions studied in people of different ethnicities aged between 7 and 80 years [16]. Furthermore, in support of our findings, the 19 expressions analyzed by Cao et al. include kissing, smiling, and raising the eyebrows [16]; lastly, (3) a study that was published using Kinect as a tool for the detection and rehabilitation of people with Parkinson’s, which was considered as a highly valid tool due to its low cost and ease of use in the home [17]. It is difficult to compare studies with the same type of technology or similar to those that have been done, to date, among adults. The objectives and results of the studies found are very different from those established by SONRIE. Table 3 summarizes the relevant information concerning the studies performed to date on adults regarding the study of facial muscles.
One of the common aspects shared by SONRIE and these studies is the interest in the orofacial muscles. In addition, as a serious game, SONRIE fulfills several functions: it is a useful game among the population under study (completely unique providing data for use in the child population) and it may be used as a tool for the detection of possible motor disorders affecting the orofacial muscles, difficulties understanding instructions, attention difficulties, or lack of self-control in children [4]. We found that smiling is the game with the greatest dispersion of data. The explanation provided by the researchers is that smiling is the movement which allows the most variability in its execution, ranging from a small smirk to a broad smile. Furthermore, it depends closely on the expressiveness of each person, which explains the dispersion in the data.
Lastly, SONRIE is also a therapeutic tool which may be used to help detect cases or to help treat children with an established diagnosis affecting the functions and structures involved in this game. As reported by Gunel et al. in their study in 2014, VR provides new opportunities for rehabilitation professionals, even though these authors stress the need for further studies to be performed in order to prove its effectiveness [18]. In addition, it is worth noting that SONRIE is an educational tool as it grants an excellent opportunity to work on facial expressions in children as a way to improve their nonverbal communication and thus improve the expression of feelings and emotions and, as a result, enhance social relations.
The effect that learning has on the improvement of the motor execution of these muscles has already been demonstrated by some authors [19] and so this mechanism must be the basis of future projects using this tool. The study by Smith demonstrates the existing variability between children compared with adults in the execution of facial movements [19]. These differences, together with the previously mentioned learning factor, support the first explanation regarding the errors related to the eyebrow raising game: that is, as this is the first game, the children cannot rely on previous learning experience, which is further influenced by their condition as children. In future experiments, the order of the eyebrow raising exercise will be changed in order to verify whether this reduces the number of zeros obtained with its execution.
5. Conclusions
SONRIE is a valid tool for use in school contexts among a healthy population of children aged between five and seven years, which shows a high level of acceptance on behalf of the same.
Concerning the movements proposed by SONRIE, the eyebrow raising movement appears to be the most complex to perform with precision and speed for the age group under study. As commented previously, many thresholds were zero in the complete age range; therefore, this movement is not valid for measuring the maturity of the motor control of orofacial muscles. There is no evolution in the acquisition of precision and speed, but rather variability exists regarding the expressiveness of each child, as greater or lesser ranges of movement of the involved muscles were observed. Future research will adapt the game settings in order to modify the position of the same in the series or by including a previous preparatory game to serve as a “demo” in order to improve the children’s attention, learning, and, hence, performance.
On the other hand, the blowing and kissing movements displayed similar behavior regarding the sensitivity of the changes in the children’s development. Smiling is the movement which is performed the earliest but also the one with the most subtleties during its execution.
Difficulties regarding self-control influence performance and the ability to produce fine-tuned movements. Therefore, in the future, SONRIE could be used as a tool for detecting these problems. For this purpose, subsequent studies should involve larger samples.
With Kinect 360, users with a fringe or wearing glasses may suffer from undermined scores due to problems related to the sensor. Future programs developed by SONRIE should use a more sophisticated sensor in order to avoid these problems, as is the case of Kinect 2.0.
Ethical Approval
The protocol was approved by the Local Ethical Committee of the Rey Juan Carlos University.
Consent
All participants signed informed consent for the use of all experimental data in scientific publications and in accordance with the policies about trials with human subjects before starting experimental and rehabilitation sessions.
Disclosure
Nuria Máximo-Bocanegra holds a Ph.D. degree at the University Rey Juan Carlos (2010) and a Master’s degree in Neurological Pathologies. She is an occupational therapist and a specialist in early care. Nuria Máximo-Bocanegra is currently involved in several research projects in the field of occupational therapy for the improvement of children’s occupational performance and participation. María-Luisa Martín-Ruiz received her Ph.D. from the Universidad Politécnica de Madrid (Spain) in 2014 and her Laurea (Graduate Diploma) Degree in Engineering Informatics in 2006 from the Universidad Carlos III, Madrid (Spain). Since December 2014, she is a Postdoc researcher in Telecommunications at the Universidad Politécnica de Madrid, where she is now an Assistant Professor in Telecommunications. At present, María-Luisa Martín-Ruiz works on intelligent systems, developing and validating the same for e-health and telemedicine scenarios. She is a Research Member of a research innovation group at the Universidad Politécnica de Madrid (T > SIC), in which she is working in applied research projects on AAL and home building automation, supported by regional and national funds.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Authors’ Contributions
María-Luisa Martín-Ruiz and Nuria Máximo-Bocanegra have contributed equally to the manuscript.
Acknowledgments
The authors extend their gratitude to the professors and children from the Amanecer School for their participation in this study. Thanks are due also to Laura Luna, for her insightful help with the identification of the orofacial muscles targeted by the SONRIE program for the rehabilitation of children with cerebral palsy.