Abstract
Augmented reality (AR) technology, with its unique immersive interactive experience of virtual reality integration, can show children abstract concepts and theories in an intuitive way. In this paper, we improve the high-precision position and low delay interaction of AR recognition area through edge calculation, so as to enhance the immersion of children’s chord knowledge learning. Meanwhile, multisource fusion perception of audio and touch is used, and the AI algorithm model is used to monitor physical events (cloud AI training and edge execution) and effectively realize intelligent services such as chord combination, melody perception, and control. Chord knowledge in music theory is presented to children through multichannels (kinesthetic, visual, auditory, etc.) situationally, to improve children’s learning efficiency and experience. We invited 12 children to participate in user experiments to test usability of ChordAR. The test results showed that children can master the method of playing the game through simple learning, ultimately correctly imitating C, G, and E chords, and experience full immersion in the game. Finally, we discuss the positive effects of ChordAR on children’s learning and creativity and make suggestions for future AR games.
1. Introduction
In today’s rapidly developing information society, augmented reality (AR) technology has become popular in all aspects of our lives. AR, as an extension of virtual reality (VR), can provide a more immersive experience, and its application in education has increasingly shown benefits. Research shows that, through the integration of virtual and real teaching aids, highly immersive learning experience can provide better learning effect, as it can enhance children’s focus, engagement, and creativity during the learning process [1]. AR technology, which integrates real and virtual world environments, can thus be used to develop educational aids that provide highly immersive learning experiences, especially for children [2].
Therefore, we designed an innovative tool that teaches preschool children introductory music theory in an immersive way. Our prototype design includes a game based on AR recognition and supporting hardware and software. We used AR to develop an interesting, easy to learn, fun tool that provides positive feedback, with the goal of teaching children basic chord knowledge.
2. Cases Study
AR changes the way children learn by introducing learning through play. However, AR is rarely used in music education, and the current AR exploration in music education mostly focuses on how to promote learning to play musical instruments, such as the development of VR musical instruments. That approach aids in visual feedback and enhances the entertainment experience, but ignores children’s perception of sound to some extent. In basic education, the perception of sound characteristics is more important than learning to play a musical instrument.
2.1. Influence of Music on Children’s Development
A study of 6-to-11-year-olds showed that the period of music lessons was positively correlated with children’s IQ, and music training improved children’s cognitive abilities [3]. Music can make learning easier by having a positive effect on the two hemispheres of the brain; moreover, because language and music have similar cognitive processing systems, active involvement in music activities enhances language skills [4]. According to neurological research, music training can enhance abstract reasoning ability and the conceptual understanding of time and space. At the same time, music helps people memorize what they learn so the stored information can be retrieved, a phenomenon people often experience when a specific memory is triggered by a certain song [3]. In short, music education has a positive impact on children’s learning and cognitive development.
2.2. Importance of AR in Education and Music Education
According to AR pioneer Ronald Azuma, “AR allows the user to see the real world of virtual objects superimposed or composited with the real world.” AR can be used to construct a learning and entertainment environment in which VR and reality are combined, encouraging children to think proactively and gain new knowledge in the real world through interactions with the virtual world. Immersion learning can improve children’s in-depth understanding and long-term memory of gained knowledge more than traditional teaching methods [5]. AR provides a contextual experience, enhances interest and participation, and allows children to fully engage in their education and have a deeper understanding of the material being taught. Traditionally, the study of music theory focuses mainly on learning to convey notes and musical scores on paper. This can be extremely monotonous as it ignores the influence of music on children’s sensory organs, whereas the intuitive feeling of sound information strengthens understanding of music theory. It is a complicated process for the human brain to translate musical notes into music, and children’s ability to process such information is limited. This is where AR comes in: it is possible to improve children’s learning experience of music theory and improve learning efficiency through AR.
2.3. Educational Game Interactive Design
We have extensively studied the global intelligent products related to children’s education and found that most products focus on cultivating children’s ability in basic disciplines. For example, in the international market, products such as the AR mathematical games to promote the development of children’s mathematical ability [2] and products such as SpeechBlocks facilitate English learning [6]. In the Chinese market, Marvelous Circuit [7] is a teaching aid product for teaching circuit. However, focusing on music education, we know that the traditional music theory learning is mainly to learn paper notes and music scores, but it ignores the impact of music itself on children’s senses. This kind of learning is extremely monotonous and boring for children. It is easy for children to lose interest in learning, resulting in low efficiency of final learning. It is a complicated process for the human brain to translate musical notes into music, and children’s ability to process such information is limited. Therefore, simple book materials may be a burden for children, and they need more attractive tools to guide them to learn. Research has found that VR instruments (VMRIs) can help children practice rhythm and reaction speed. However, VMRIs can only play music and do not teach basic music knowledge. Many AR learning tools focus on letting users learn how to place their hands and when to move their fingers. For example, ARPiano uses MIDI keyboard and three-dimensional virtual interface to create MR environment, so that abstract piano notes are superimposed on the keyboard in the form of color patches, which is convenient for users to find the position of piano keys [8]. Similarly, Fretlight® guitar, an AR-based guitar learning system, shows the notes that need to be played on the guitar panel (neck) in the form of small light beams using LEDs [9]. However, these pay attention to visual presentation and feedback and do not reflect the real sense of music knowledge learning. An educational product that teaches about tone, rhythm, volume, and timbre (other basic musical elements that constitute music theory, important prerequisites for learning music well) is Music-AR, an application that trains children to perceive the nature of music. It teaches children the concepts of pitch, intensity, and timbre in the form of short games, and children can obtain visual and auditory feedback by clicking on the screen, which is easy for beginners to master [10]. However, it only converts the note information in traditional books into a playful interface. Although playful settings can cultivate children’s learning motivation and interest, children only operate through click and touch and do not break away from learning in the traditional sense. The constructivism theory of learning posits that students should actively explore learning, form their own thinking through interaction with the outside world, and construct their own knowledge system through actions [11]. According to the embodied theory, human cognition is deeply rooted in the interaction between the body and the physical world. We pay attention to children aged 4–7. At this stage, children’s cognitive abilities have been significantly developed, such as executive ability and spatial perception ability. Our project aims to promote children’s physical interaction with learning tools. And our goal is not to let children keep up with the rhythm, but to understand the concepts of music theory, rhythm, and melody. Based on the embodied theory, we have developed AR learning tools to teach music theory knowledge, changed the traditional paper learning method, and let children actively explore and learn music theory knowledge through hands-on practice. Our topic mainly solves the following three problems of current AR/VR music products: (1) a focus on the performance experience, ignoring the learning of music theory knowledge; (2) a lack of products suitable for preschool children and head-mounted displays and handles that are not suitable for children; and (3) a focus on visual feedback, ignoring the perception of sound.
3. Experimental Details
3.1. Design Process
To understand the current needs of preschool children, we conducted a survey of users and identified four problems they faced: (1) preschool education is too early to become “primary education”; (2) the way of completely instilling knowledge is not conducive to the overall development of children; (3) children are not easy to concentrate in class; and (4) even though parents attach great importance to their children’s education, they do not have enough time and energy to teach them knowledge. We put forward the following design factors as guidelines: (1) pay attention to the fun of the game and let children relax in learning; (2) avoid simply instilling boring written knowledge and transform simple knowledge input into pleasant knowledge experience; (3) transform knowledge input into children’s active exploration; and (4) cultivate children’s autonomous learning ability so they can complete learning tasks independently. We invited three experts in child psychology to four formal workshop discussions, and following our research, we designed and developed an AR recognition-based game, ChordAR, as well as supporting hardware and software, to help preschool children learn music theory. ChordAR solves the aversion and passive learning problems of children when learning music theory and presents the difficult music theory knowledge in the form of game, so as to strengthen children’s understanding. The user evaluation shows that our game can help children play simple chords in the game while interacting and learning happily through the game interface.
3.2. Hardware Design
3.2.1. Selection of the Recognition Mode
The game depends on mobile camera recognition. Therefore, it involves two recognition methods, front and rear camera recognition. Based on that requirement, we conducted two camera experiments, turning the mobile device once and twice, respectively. We chose iPhone 11 as the experimental equipment (as shown in Figure 1). For Experiment 1, to capture the image of the front operating area of the screen using the rear camera, the recognition area image needs to be turned twice given the principle of mirror reflection. As shown in Figure 2, when the vertical angle remains unchanged, when the included angle of the mirror is an obtuse angle, the path of the reflected light will become wider, which is conducive to recognition.
For Experiment 2, the forward camera of the mobile device was used for recognition. The front camera needs to identify the building blocks placed by children in the identification area. Therefore, a mirror is installed to reflect the image of the recognition area. Comparing the two schemes, it is found that, in scheme 1, the focus will be unstable due to the need for continuous reflection, so we finally chose scheme 2 front camera for recognition.
3.2.2. Supporting Product for the Mobile Terminal
As shown in Figure 3, in front of the supporting products of the mobile terminal is a plane recognition panel and a block assembly panel, which aims to promote the progress of various games. Built-in drawers on both sides of the product store teaching aids. The details of the drawer buckle ensure that they are not easy to separate. In addition, the design of recognition area reduces the cognitive cost of children when using the equipment. Specular reflection makes the identification technology of the equipment more cost-effective. Supporting products can also be used as iPad or mobile phone bracket to improve product utilization.
The most important object identified in this project is LEGO®. The building blocks can be assembled at will, with each building block representing a sound that can be combined into different chords. In addition, LEGO® are well-known children’s toys, so children understand the operation of the game quickly and are attracted to it. Because piano can help beginners learn basic skills and is easy to be recognized by learners [12], when designing the game sticker, we refer to the seven keys of the piano, which corresponds to the notes of CDEFGAB. A gray scale located at the bottom of the interface showed falling notes on the left and rising notes on the right, within a 49-key piano range. This left enough room for the design of more challenging games in the future.
3.3. Game Interface
We also designed the interface and story background of ChordAR, including detailed yet simplified music knowledge content and explanations, specific gameplay, and difficulty levels (shown in Figure 4).
The initial main game steps are explaining the story background and inviting the user to participate in creating chords (as shown in Figure 5). We included simplified material on chords, which is easily overlooked in children’s musical education. To achieve this, we had to research, understand, and summarize information about chords and then simplify it. The third step involves game levels (as shown in Figure 6): Level 1 taught single notes, which led to teaching the entire chord. We also included a blank space in the design of the chord progression interface to encourage children to think independently. Levels 2 and 3 followed the same path as Level 1. All game processes included simple and more complex levels, with each basic level used to teach different chords. In the advanced level, children played along with the complete music.
(a)
(b)
(c)
(d)
3.4. Interaction and Feedback Mechanism
The specific operation process of ChordAR is shown in Figure 7. When children saw the game instructions on the iPad or mobile interface, they put the corresponding blocks or combinations of blocks in the recognition area to progress in the learning interaction and advance through the levels according to the feedback shown on the interface. In addition, spare blocks that were assembled could be arranged on the block-assembling panel for easy access. The music theory game involved various types of interaction and feedback mechanisms.
Tangible interaction: when interacting with the game, children were required to follow the voice prompts to take out the blocks, assemble the blocks, and put the blocks into the recognition area where the system recognized the blocks. After correct identification, the children put the assembled blocks aside for subsequent assembly.
Touchscreen interaction: the game steps are as follows: “start the game - play the background story - level selection - play the chord teaching course - assemble the building blocks - AR recognition.” To enter the game, click the “play” button. When the background story can be skipped by clicking the screen at any time, click anywhere at the end of the screen to switch to the level selection page. After entering the level, the game will first play a video explaining the chord concept, and then the button of “Retry” or “Continue game” appears on the screen. Similarly, participants click anywhere on the screen to continue.
Feedback mechanism: when the building block is successfully recognized, the system will play the sound of the corresponding note of the building block. When several building blocks are put together, the system will play the sound of the corresponding chord. Voice feedback enhances children’s sense of achievement and arouses their interest in learning. When all chords are successfully assembled, the sound of completing the reward will be played. Virtual gifts also appeared, which visually enriched the game and triggered children’s emotional changes, making them more willing to continue to play. In the advanced stage, the music melody is played continuously. Only when the correct chord is recognized will the sound of the chord be played. Successful chord playing produces bonus points, while wrong or blank chords do not produce any bonus points. At the end of the game, scores and rankings are displayed according to the child’s wishes.
3.5. Prototype
We built a working prototype of ChordAR. The software was developed with Unity and can be adapted to multiple hardware platforms (Windows, Android, and iOS). We implemented two AR controllers based on Vuforia and ARKit engines to process the data collected from the camera of the target device. We rendered the game animation based on WorldTracking data from AR engines.
4. Discussion
4.1. Participants
We recruited volunteers through the Internet. Our selection criteria included being between the ages of 4 and 7 years, with or without foundational knowledge of music. Given the limitations of COVID-19, only 12 volunteers were recruited (6 boys, 6 girls), two of whom had acquired knowledge of music theory to some extent and had mastered performance skills. Parental consent to collect their children’s basic information (Table 1) was obtained prior to the study and test phase. The basic information was collected through a questionnaire. It was important to determine musical background to analyze the experimental results.
We conducted a test in a room of 3.5 × 3.5 meters, including a table, three chairs, and a tablet computer. We used two Huawei P30 mobile phones for audio and video recording (see Figure 8).
4.2. Methodology
We used observation, interview, and questionnaire analysis to verify whether games are good for children to learn and master music knowledge.
Test procedure was as follows. First, test controllers briefly told the children stories about the musical theory game to stimulate their interest in participating; researchers made a brief presentation to those who did not understand the procedure. Next, the children were shown videos about chords to teach them basic chord concepts. Finally, children were guided to play the game at primary Level 1 before advancing to Level 2. Their behaviors while playing the game were observed, and assistive explanations were provided to them when necessary. The children’s views on the game were obtained mainly based on the observation results of test controllers, semistructured interviews, and questionnaires. The observations contributed to information collection from an intuitive perspective and verified the results of the interviews. The interviews, which lasted 3–5 minutes, helped us understand how the children really felt. The two methods were used to answer the following key questions: (1) What problems did the children experience when playing the game? (2) What made the children feel most satisfied while playing the game? (3) What are the children’s expectations of the game in the future?
Before the game, in order to understand children’s understanding of music and their interest in music, we first asked open-ended questions. The main contents of the questions include “have you studied music?,” “do you know what chord is?,” “have you played games or toys related to music?,” and “if there is another toy that can play Lego and learn music, are you willing to play?”; in order to determine the change of children’s attitude after the test, we issued a scoring questionnaire after the test to investigate children’s overall satisfaction with the product.
4.3. Data Analysis
Data were collected manually from direct observation (quantitative), semistructured interviews, and the evaluation questionnaire (qualitative).
Observation: one researcher was responsible for assisting in children’s play, while the other researcher observed and recorded the whole process, including the number of questions and specific links, the number of times each child misspelled chords, and the seriousness and excitement of children playing games, and then sorted out all the data. The observation results show that all children can actively participate in the whole game process and finally correctly build chords C, G, and E. A 4-year-old child without learning music can correctly play chords after playing two games. The following problems were found by the researchers in the observation process. (1) The children often used the LEGO® blocks to construct the shapes they wanted instead of the shapes the game instructed them to build. (2) Some of younger children appeared to lack patience during the game interaction, suggesting that the recognition area and feedback mechanism require promotion. This could be a technical issue, or we could consider increasing the frequency of interaction between the animated characters and children in the game or allow the animated characters to sing. (3) The colors of the building blocks enhanced the children’s memory during the construction of chords. (4) The steps for the advanced level of the game were displayed too fast for the children to keep up and understand the form of each chord. This enlightens us to design different game difficulties according to age and optimize the design of game process.
Semistructured interview: the interview was conducted one-on-one by a researcher in a separate room after the children’s game, and the whole conversation was recorded for subsequent analysis. “Do you like this game?” “What is your favorite part of this game?” “What other playing methods do you expect in this game?” and “Do you want to learn music with it in the future?” The interview lasted 3–5 minutes; on the whole, children have a positive attitude towards ChordAR, and all children are willing to use it to learn music in the future. The children enjoyed the hints and audible feedback and were willing to interact with the game’s animated characters. When asked about their preferences for Levels 1 or 2, seven children reported liking both, while one child preferred Level 2 because he could hear the final chords that he had played. Four children hoped that additional building block toys would be provided to construct different chords, and two children expected the virtual characters to sing songs and hold more interactive conversations. One child disliked the color of the interface and wanted to be able to change the color himself. To a certain extent, these show that children are more interested in games. At the same time, they also show that the richness of game content can be further strengthened and give children more choices and creative space in the design of game links.
Evaluation questionnaire: the questionnaires were collected after all the children were tested, and subsequently the data analysis was carried out by the researchers. The participants completed the questionnaire with the assistance of the researchers. When they had questions, the researchers provided necessary explanations. ChordAR’s usability was verified by comparing children’s expectations with the variables of children’s interest, overall satisfaction, and usefulness after the experiment. The children’s interest in the game was scored on a 5-point Likert scale: 1 represents very dissatisfied, 2 represents dissatisfied, 3 represents average, 4 represents satisfied, and 5 represents very satisfied. The test results showed that half of the children held a neutral attitude in their overall expectations of the ChordAR before the experiment, while ten children were highly satisfied with the game after the experiment. All the children found the game interesting, except one child who had previous music and chord knowledge, who felt the game was too easy and had a neutral attitude toward it (ten children gave 5 points, one gave 3 points, and the other gave 4 points; see Figures 9 and 10). This further validates our observations, and the setting of difficulty level should be considered in future iterations.
During the test, the researchers found that the color of the building blocks seems to enhance children’s memory in the process of chord construction. In future work, we will consider making a comparative experiment between the colored building blocks and the white building blocks to further verify the usability of the game. ChordAR provided the beginner-level children who participated in its testing with an immersive experience and more excitement than expected from traditional teaching methods. As a music theory knowledge learning scheme based on Internet plus, the design uses building blocks to help children learn music elements and theories and guides them to complete music performance through building blocks, so as to improve their hands-on ability, physical coordination ability, and spatial perception ability while learning knowledge.
5. Conclusions
The main focus of this paper is immersive children’s intelligent music education. Through the analysis of preschool children’s needs for music knowledge learning, this paper reviews the application of AR technology of children’s AR products in previous relevant studies. Finally, ChordAR is developed and an AR prototype is built for user testing. The test results showed that children can master the method of playing the game through simple learning, ultimately correctly imitating C, G, and E chords and experience full immersion in the game. It solves the problems of boredom and passive learning when children learn traditional music theory knowledge, combines complex music theory knowledge with games, and promotes children’s understanding of music knowledge.
The study found that actively following the game tips, ChordAR can stimulate children’s interest in learning basic music theory knowledge, but there are still the following problems that need iterative design: (1) the data show that children of different ages and different music knowledge backgrounds have different degrees of acceptance and understanding of games. Older children prefer to play advanced levels, which inspires us to consider adding different difficulty levels in future game design. (2) We hope to give children more clear guidance through voice prompts in the game. (3) Children’s creativity and independent exploration ability have inspired us to consider providing children with more independent choices in future interface design.
Future studies on this or similar AR-based teaching devices should further explore game designs that not only improve children’s learning experiences but also increase the degree of difficulty and enrich interaction, so as to maintain children’s enthusiasm for learning. The development of dynamic effects and voice interactions and a collection of big data for individual learning is also required. In future research, we will focus on these fields and design a more versatile gameplay, interaction, and feedback mechanism to effectively provide children with an even more immersive and stimulating learning experience.
Data Availability
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Conflicts of Interest
The authors declare no conflicts of interest.