Abstract
This study explains the case study to develop a metaverse-based training system for the management of the physical fitness of youth. The study targeted developing a smart mirror-based AR physical fitness system and a bone age-based physical fitness management service to enhance exercise capability in youth. Accordingly, the data on youth’s bone age, physical strength, and physique were collected. Based on the collected data, the stage of the human development in youth was categorized, and the physical fitness management system providing the exercise program customized for the phases of physical development was implemented. Furthermore, a solution was devised so that youth with less physical exercise experience could practice physical fitness training without a coach using the smart mirror-based AR physical fitness system. This system uses an RGB-D camera and a three-dimensional posture estimation algorithm to recognize a user’s movement, determine movement accuracy, and provide feedback for the improved exercise effect. The XR Physical Fitness, using a smart mirror implemented in this article, features two features of metaverse’s augmented reality and lifelogging, enabling the youth to enjoy more immersive and enjoyable exercise experiences.
1. Introduction
Youth is a significant time in the process of physical growth and mental maturity [1]. However, recently, the obesity problem among children has become a significant social concern. Obesity in childhood is more likely to continue and lead to adult obesity. Moreover, obese children experience social and mental stress due to abnormal body shape formation. Another problem with healthcare for youth is their physical strength which is decreasing. As the diet level improves, children’s nutritional status also improves, and their physique increases, but their physical strength decreases. Obesity and physical decline, caused by increased mental stress and a lack of physical activity, are resulting in various diseases in children. Therefore, it is essential to analyze whether the body is appropriately growing according to age and managing it properly [2]. These situations show the importance of research for youth’s physical strength improvement and health care programs.
Especially, the COVID-19 pandemic has confused the world ever since 2020 and change our daily lives. Limited indoor activities, such as telecommuting and online classes, have rapidly reduced the number of our daily activities to cause a health crisis and well-being. A new word such as “quarantine-15” has been coined, and healthcare for children has also become more difficult. As COVID-19 forced most outdoor activities to stop, many have quenched their thirst for exercise with various applications, such as wearable devices, YouTube, and home workouts. Ring Fit Adventure, released by Nintendo, has become an excellent solution for these people. “Exergames” like Ring Fit Adventure allowed men and women of all ages to exercise in a limited space. Even the parents who were usually worried that their children would be too fixated on gaming were willing to buy the game, enjoy exercising with their children, relieve depression, and frustration from a lack of in-person activities and keep their bodies’ minds healthy.
The paradigm changes in the convergence of the real world and virtual reality, referred to as metaverses, are already quite noticeable in Korea at various points of our society, such as culture, education, and entertainment. As of 2022, the world has begun to think about how to enjoy a healthy “third life” in metaverses during the post-corona era beyond the COVID-19 pandemic. The “Third Life” is a phenomenon in which virtual space is organically connected to the real world to activate human senses and maximize experiences [3]. The world of the metaverse has begun to become a reality in our everyday lives, where one can enjoy a variety of sports activities without meeting friends in person and watch realistic games without having to go to the stadium. In other words, we now have to adapt to a new “Third Life” in which real and virtual spaces interact through a physical “first life” and a fictitious “second life” in virtual space.
This study aimed at developing a metaverse-based physical fitness services and management systems for effective youth athletic ability development. One factor that we concentrated on the most was preparing a plan for the young children to perform exercise programs without the trainers managing them in a non-face-to-face environment. Our conclusions can be summarized as twofold: the need for augmented reality technology and metaverse technology. A theoretical background was established by collecting research on extended reality technology and metaverse technology in the XR fitness and sports fields. Based on this, a smart mirror-based AR physical fitness system was developed.
2. Designing the Youth Exercise Enhancing System
2.1. The Importance of Physical Fitness Management for Youth
Adolescence is an important time when physical and psychological changes occur the most as a part of human development, so various studies are required for healthy and stable physical development. In particular, young children should be treated differently in comparison to sports and welfare services for adults, as there are many individual differences in development speed and pattern [4, 5]. Therefore, research and implementation for health care services and systems to efficiently provide appropriate physical activity services to children are needed.
We planned an integrated management system for physical fitness in youth as shown in Figure 1. In particular, we have developed a strategy that focuses on bone age. Because young children in the developmental process show outstanding exercise performance based on their weight and height, which are superior to those of immature young children [6]. These results show that youth health care should focus on bone age rather than chronological age [7]. In other words, since there are individual differences in bone age, which measures biological maturity and chronological age, the current evaluation of exercise performance based on chronological age is not appropriate for young people with slow developmental growth [8].
Thus, this research project established a new Korean development stage index by measuring bone age and young children’s ability to perform exercises, and based on this, we carried out clustering to derive characteristics of each development stage in young children. In addition, a management system was developed to improve exercise performance ability tailored to the characteristics of each cluster.
2.2. Bone Age-Based Physical Fitness Management System for Youth
The bone age-based physical fitness management system for youth was divided into four stages: (1) Data measurement and development stage classification, (2) cluster analysis and feature derivation, (3) fitness program development, and (4) management system development.
2.2.1. Measurement of Bone Age-Based Stages of Human Development
In order to develop physical fitness standard indicators according to bone age, we conducted data measurements on a total of 2,931 people (1,469 males and 1,462 females) between October 2019 and February 2022. The ages were distributed so that there were 361 people aged 6 to 7, 7, 529 people aged 8 to 9, 783 people aged 10 to 11, 931 people aged 12 to 13, and 327 people aged 14 to 16. The measurement items were bone age, physical fitness, and physique data. As far as the bone age is concerned, we used the TW3 method to measure bone maturity after X-raying the left hand and wrist of the participant to be measured [9]. For physique, we used inbody equipment to measure height and weight. To measure physical fitness, we measured a total of 7 physical fitness indicators: muscular strength (hand grip strength), balance (bass stick test), agility (plate tapping), power (standing long jump), flexibility (sit and reach), muscular endurance (sit-up), and cardiovascular endurance (suttle un). After that, individual stages of human development were classified based on bone age based on the measurement data. The development stages were categorized into five stages (early childhood, middle childhood, late childhood, early adolescence, and middle adolescence) following the study by analyzing the theories suggested by existing experts, such as Piaget, Kephart, and Stratz [10–12].
The analysis results of our target cluster group in the study (the 11–13–year-old) group show that there was a significant difference in men in the physical fitness category of muscular strength, power, muscular endurance, and cardiovascular endurance, and in women in muscular strength, balance, agility, power, flexibility, muscular endurance, and cardiovascular endurance [6]. Table 1 compares the physical fitness measurements of bone age of 11 years old and the chronological age of 11 years old. Here, as far as a man was concerned, values were higher in bone age than chronological age on the five categories (muscular strength, agility, power, muscular endurance, and cardiovascular endurance), showing statistically significant differences. As far as women were concerned, the values were higher in bone age than chronological age in the six categories (muscular strength, agility, power, flexibility, muscular endurance, and cardiovascular endurance), showing statistically significant differences [6]. It was confirmed that there was a clear difference between the evaluation of exercise ability based on chronological age and the evaluation of exercise ability based on bone age.
2.2.2. Cluster Analysis and Feature Derivation through Big Data Analysis
This step shows that the critical factors for recommending personalized exercise were extracted using the data collected through the previous step’s physique and physical fitness measurements (Table 2). K-means clustering was conducted to identify the data characteristics of the measurement targets and derive characteristics between groups. Although the number of clusters is planned to increase up to K = 5 in the future, clustering was conducted with K = 3 to derive meaningful features as sufficient data has not been accumulated yet. Finally, these data are again clustered into three groups through the three filters (gender, bone age-based stages of human development, and physical fitness measurement value).
The clustering results were represented as a graph (Figures 2 and 3) making it easy to grasp the characteristics of the physical ability of the same group and the heterogeneity between groups. After the cluster analysis, the data is classified once again. This process checks the current bone maturity by comparing an individual’s chronological age and bone age. Bone maturity is classified into three stages (immature, normal, and mature). The bone maturity classification is to know the growth and development of young children. Particularly, we were interested in an immature group of young children. Their bone age is lower than their chronological age, and they are at risk of being guided to perform excessive exercises compared to their abilities since their chronological age is higher. Since excessive exercise might adversely affect growth and development because of their immature bones, these young children should set their exercise performance goals more carefully as per their abilities.
As such, the big data analysis program recommends the type, intensity, time, and method of exercise programs appropriate for individuals based on the analysis of the characteristics and bone maturity of physical fitness by means of clustering as well as physique and physical fitness data. Then, the director observes the individual physical fitness of the young children more closely in the field and trains them with the recommended exercise program.
2.2.3. Development of Physical Fitness Programs
We constructed a total of 20 youth-tailored exercise programs, 5 of which were newly developed exercise programs considering the main characteristics of each stage of human development (Table 3). The exercise program analyzed the validity and reliability of the draft developed through preliminary research and invited external experts (professors, physical education teachers, and youth sports center instructors) to develop the final draft through expert meetings. Finally, the developed exercise program is applied to the big data analysis program described above. According to the data analysis result, an exercise schedule is generated that is suitable for the individual and recommended for the user.
2.3. System Design and Prototype
User service and management programs were developed as a system that considered individual bone age, physical strength, and physique measurement data. This system is configured as shown in Figure 4. Users can select and use the application and the web service according to their goals. Youth users can check attendance, recommended exercise programs, and physical fitness measurement results through the application (Figures 5 and 6). Coaches use the application and the web service as per the situation. First, the coach checks youth attendance using the application in the exercise field, checks each youth’s customized recommended exercise program, and teaches them whether they are performing it correctly.
(a)
(b)
(c)
(d)
The web service was developed to easily manage all youth members, exercise management, and exercise recommendation management (Figures 7 and 8). The application has a small screen, making it difficult to compare information between each member and to grasp much data at once. Moreover, the customized recommended exercise results can be checked in the program, determined, and modified from the position of coaches.
(a)
(b)
(c)
3. Developing Metaverse-Based Youth Exercise System
3.1. The Need for Metaverse-Based Physical Fitness Services for Youth
Before the COVID-19 pandemic, fitness programs and services with Extended Reality and XR technology such as smart mirrors, VR, and AR glasses were also sometimes released. However, the COVID-19 pandemic brought the need for exercise to become more prominent and began to emerge more competitively. Extended reality exercise programs may be recognized as replacing face-to-face exercise activities for a while only due to COVID-19 situations. However, in reality, they can collect and analyze a lot of valuable data for increasing exercise effects, such as customized exercise content or posture feedback. It also has the potential to provide an evolved exercise experience that cannot be obtained from existing exercise methods. However, quality problems such as insufficient content, inaccurate feedback, and incorrect technology continue to be pointed out, and many of these programs are often ignored after briefly attracting attention from users in the early stages of release. Therefore, if the upcoming extended reality exercise program fails to solve the current problem, no matter how good the technology is applied, it will not satisfy users’ desire to participate in exercise and exercise satisfaction. Augmented reality affordance can be a great way to solve quality problems that may arise in mixed reality exercise programs. Augmented reality affordance is a concept consisting of a technology that mixes virtual objects into real space to interact [13], and an affordance called behavioral induction is an essential attribute that induces an action in the user [14]. In other words, this can be understood as a method of inducing behavior for a natural learning process in the user using augmented reality. If this augmented reality affordance is applied to exercise performance, it is considered an effective tool to supplement or replace existing learning methods [15].
In addition, extended reality exercise programs and services have continuously been chiefly developed for adults, so young children were not the primary target. However, the modern youth generation lives in and out of the real world through various digital technologies. In the future society, the real world and the virtual world will be closely connected. Accordingly, it is becoming common for humans to interact with and influence the virtual world and their experiences in the real world. For that reason, in the future educational environment, there is a need for educational content that develops future capabilities to understand new metadata and freely perform social and economic activities and various hands-on activities in the world. Teaching the ability to live independently while actively responding to digital technology and the metaverse for new extended realities will be an important mission for future education. This makes it necessary to research and implement future education actively. Exercise activities are mostly difficult to deal with in a metaverse environment. However, a metaverse-based training platform needs to expand the boundaries of limited experiences in the virtual world to affect real users’ experiences directly.
3.2. Understanding the Metaverse
When Marc Zuckerberg renamed the company Meta in 2021, many people began exploring what the Metaverse was [16]. However, various types of metaverses are already emerging in various fields. In addition to technology development for metadata such as VR, AR, and various IoT devices, various research on content and service planning, storytelling, production and design, immersion, and reality are being conducted.
The American Foundation for Future Acceleration Studies (ASF) defined “metaverse” as “a complex concept, a fusion of virtually enhanced physical reality and physically permanent virtual space [17].” Therefore, the metaverse is not just a 3D space but a space where virtual space and reality actively interact and can be understood. In order to embody the concept of the metaverse, the metaverse roadmap is divided into two axes and provides four key elements (Figure 9). The longitudinal axis is the spectrum of “augmentation and simulation.” Augmentation means the ability to add new realities to existing systems, and simulation means the reality of providing new environments such as virtual space. The horizontal axis is the spectrum of “intimate and external.” External information is about the surrounding world, while intimate focuses on individuals and individuals and focuses on users [17].
Metaverses can be classified into augmented reality, lifelogging, virtual worlds, and mirror worlds depending on the implementation space and type of information. Augmented reality is a technology that enhances virtual information in a real environment and combines virtual three-dimensional images into reality to increase a sense of reality further [18]. Pokemon Go, which was released in 2016, has become a craze among the public and is a representative example of being used as an augmented reality game. Lifelogging refers to the overall activity of recording, storing, and sometimes sharing things and people’s daily experiences and information for their purposes [19]. Examples include Facebook, Instagram, and Twitter, such as SNS and social media. Mirror Worlds imitate the real world as it is, as if it were a mirror, and express the real world in digital form in a metaverse [20]. Unlike augmented reality, Mirror World focuses on “realistically reproducing” information related to the location of the actual area. Google Earth, Airbnb, and Earth 2 are examples of Mirror World. Google Earth is building a database of satellite images, maps, topography, and 3D building information around the world using satellite image maps. Virtual Worlds have already been used for a long time in computer games. It is built into a virtual world created based on 2D and 3D engines that are inherent and simulated [21]. The virtual world can be divided into three types (game-type virtual world, community-type virtual world, and work-type virtual world). The game-type virtual world aims to achieve common or individual goals of players while cooperating or competing with each other with specific rules. In the community-type virtual world, activities such as setting avatars to explore the virtual world and communicate with other users are carried out. The working virtual world enables remote work by reproducing the working environment of companies or public institutions as it is.
These metaverse platforms enable differentiated experience values with new convergence technologies, and the 4I, immersion, interaction, imagination, and intelligence are achieved, enabling new experience designs beyond time and space [22]. In other words, metaverses provide modern people with differentiated experience value, which allows modern people to create the same appearance as themselves in the real world, or a completely new created self, as a meta-human (Avatar). Not only can it be designed in the virtual world by cloning the physical space of the real world into an entirely new imaginary space, but it can also implement or recreate the real past in the virtual space in terms of time and provide new experience value to explore the predictive future.
3.3. Understanding the XR Physical Fitness
The non-face-to-face fitness environment has several limitations. The critical problem is that it is impossible to meet with a trainer. In fitness, coaching by trainers plays a significant role. First of all, one needs to figure out what kind of exercise one needs based on their current physical condition. However, one needs a trainer because it is pretty challenging to do it yourself without professional knowledge [23]. In addition, maintaining correct posture while exercising can prevent injuries and increase exercise effects. However, most people are not used to controlling their movements and posture accurately and do not know how to exercise, so they have to learn from their trainers. Finally, leadership is necessary to provide continuous motivation. Trainers play an important role in presenting exercise goals and steadily providing motivation to not give up continuously. However, due to the absence of a trainer, these three points—physical conditioning, exercise guidance, and motivation—are bound to weaken in a non-face-to-face fitness environment.
For that reason, XR physical fitness has recently attracted attention. XR physical fitness provides fitness services with a coaching system that sufficiently analyzes the condition of athletes, monitors their movements through XR technology using cameras and sensors, explains the how and effects of exercise, and provides exercise content to report and follow. Many YouTube channels and fitness applications provide various exercise content, but they have limited available streaming content. In addition, some fitness applications check the user’s exercise time and running distance, but they cannot correct and motivate posture or determine the current athlete’s physical condition. As a result, various XR physical fitness systems and services are being released to compensate for these shortcomings.
MIRROR is a digital healthcare service using smart mirrors. A smart mirror is a device with an interactive display that looks like a mirror and a built-in camera and speaker. MIRROR helps one do fitness exercises easily with accurate movements at home. The professional coaching system of MIRROR optimizes the exercise course to obtain the maximum exercise effect according to exercise goals and preferences by tracking and managing the user’s profile and biometric data. More than 50 exercise programs can be trained in real-time by professional instructors or exercise with friends [24].
Peloton provides connected fitness equipment (bike, treadmill, and various exercise tools) and subscription-type online fitness streaming services. Users can use online trainer lessons and training programs in real-time using connected fitness devices. A large tablet installed on a bike or treadmill allows one to view training programs and enjoy exercise with other users in real-time. Data measured by sensors mounted on fitness equipment are transmitted to coaches conducting real-time classes. Feedback can be received from an expert trainer through the measured data. And one can compare his or her training with that of other users or be recommended a training program that fits your body data [25].
One of the most popular XR physical fitness services is Zwift in recent years. Zwift is a simulation program that connects with Smart Bike Roller for door cycling and allows virtual riding through a computer or mobile. Various training programs are provided to help improve one’s athletic ability. As an event, one can ride together with several people, such as in group riding or a race, to compare their skills through competition with others. Group riding has a different difficulty level so that one can participate according to his or her skills. If one achieves a specific mission while riding, one can receive various jerseys to decorate virtual characters. Virtual characters can change not only clothes but also bicycles, helmets, goggles, gloves, and socks. Zwift also provides training programs for running. When a device called Runpod is worn, the speed and cadence (SPM, number of steps per minute) are measured. These two data allow virtual characters to run at the same pace as real users run on the treadmill [26].
XR Fitness is attempting to provide the same exercise environment and effectiveness as offline in combination with various high-tech fields such as wearable devices, motion sensors, and AI technology [27]. In particular, for this purpose, smart mirrors are actively used in XR Fitness as an intelligent health management system [28]. Smart mirrors have the most important feature of being able to easily understand and follow expert movements compared to other XR Fitness methods because users can see their exercise movements through the mirror while exercising [29]. The currently released exercise coaching system using smart mirrors is currently divided into three types. First, it is a form of measuring the number of exercises and the accuracy of the posture by recognizing the user’s movement through the camera. Second, it is a form in which exercise equipment is combined with a smart mirror. This type can also perform lifting exercises. Third, it is a form of measuring exercise in conjunction with wearable devices such as smartwatches and heart rate sensors. These three forms may be configured in combination for each service or may be configured alone. The MIRROR introduced above measures the user’s motion through such a camera and also measures the amount of exercise with a wearable device. TONAL and TEMPO combine with exercise equipment to provide content for various muscle exercises. Users must select smart mirror-based XR Fitness devices and services according to their desired exercise purpose. However, no matter how smart a mirror-based XR Fitness is, there are common essential requirements to provide a satisfactory exercise experience for users. First, it is the easy accessibility and composition of exercise programs that can produce sufficient exercise effects. The following is a stimulus to the belief that you can receive accurate exercise coaching even in a non-face-to-face environment and have a steady desire for exercise. The last is having a pleasant exercise experience and obtaining effects through it. Therefore, a feature analysis that can accommodate these requirements should precede the development of the XR Fitness system.
3.3.1. Analysis of Features of the XR Physical Fitness
Since XR Physical Fitness is different from existing offline fitness programs, sustainable elements must be identified and included in the program composition. Analyzing the XR Physical Fitness service discussed above, factors necessary for successful program development are derived. There are six of these elements, which can be divided into three groups. The first group is “usability” and “professionality.” Professional fitness clubs already have an environment for exercising, and users can start exercising as soon as they go to the fitness club. However, it takes much time to prepare for XR Physical Fitness at home, and if a complex preparation process is required, it hinders the continuous participation of users. Therefore, it should be conveniently configured to start an exercise program as soon as possible when you decide to exercise. In other words, it should be highly useable, which means convenience. Convenience should not be misunderstood as easy or simple. Since the XR Physical Fitness program requires the user to perform exercise alone, the provided exercise program must have expertise. As pointed out earlier, if sufficient information about exercise programs that must be performed without a trainer is not provided, it adversely affects the user’s motivation to exercise. Therefore, professional exercise programs must be organized as much as possible. Information necessary for exercise must be provided to users promptly during the exercise process so that they can be immersed in exercise without being disturbed as much as possible [30].
The second group is “feedback” and “motivation.” Users can focus on exercise with confidence only when they receive feedback on whether they are continuously performing accurate movements. Although it is quite difficult for the system to accurately feedback in situations where user stamina and physique characteristics are different, it is a necessary factor in XR Physical Fitness. To this end, the XR Physical Fitness system is focusing its efforts on developing technologies to collect users’ exercise performance data as much as possible using cameras and various sensors, analyze them in real-time, and give feedback. Accurate data analysis increases trust in exercise results, and this trust provides a “motivation” for the user to exercise continuously. Another effective way to “motivate” is to provide a variety of exercise goals and help monitor the user’s progress. Achieving a goal gives users a sense of accomplishment. This sense of accomplishment is positive energy and greatly influences continuing to do something. If one exercises continuously through motivation, one will be able to get the effect he or she wants [31].
The third group is “joy” and “exercise effect.” Most people start exercise programs for health-related reasons, but interest and social experience are essential in maintaining them continuously [32]. It is pretty challenging to make exercise enjoyable. This is because exercise itself is a complicated activity. One of the ways to make exercise enjoyable is to compete with others. Working out alone makes one feel more lonely than happy. Therefore, many people try to enjoy exercising with others at fitness clubs. However, XR physical fitness is an environment where one exercises alone in an independent space. Therefore, many XR physical fitness services provide connectivity between users and induce them to enjoy exercising through competition with each other. If a user exercises happily like this, the exercise effect naturally increases. No matter how much time one exercises, the exercise effect decreases if one cannot concentrate on oneself and exercise. However, if one focuses on exercising through competition with others, the exercise effect will inevitably improve. Therefore, competition with others in exercise can be a cooperative process that increases the effectiveness of exercise.
The six elements analyzed as such play a crucial role in the development of XR physical fitness, and each element is not independent but strongly interrelated. These elements work for the goal of increasing sustainability to secure the user’s exercise effect in XR physical fitness. The elements of these three groups relate to 3I of virtual reality. Burdea and Coiffet describe 3I, immersion, imagination, and interaction as important components of virtual reality [33]. Imagination relates to joy and the exercise effect. This element is necessary to feel the effects of exercise while exercising joyfully and competing with other remote users. Users with weak imaginations cannot interact sufficiently with the XR Fitness system. Interaction relates to the second group, feedback, and motivation. Through interaction, users explore the XR Fitness environment and select and manipulate objects. Users must receive accurate feedback on their actions to increase their reliability in the system. This belief and trust provide the user with an incentive to continuously perform XR Fitness. Finally, immersion relates to usability and professionalism. Users exercise through XR Fitness based on imagination and interaction and are fully immersed in the exercise without any interference. As such, 3I is a key element for providing users with a sufficiently meaningful experience in a virtual environment, and the six elements of XR Fitness derived in this paper are consistently understood as important factors in enhancing users’ experience.
The concept and characteristics of the metaverse and the core elements of XR Physical Fitness have been explored to develop a metaverse-based training platform to enhance the physical fitness of youth. As a result, we decided to develop a smart mirror-based AR physical fitness system for youth (Figure 10). The smart mirror was selected because the most optimal environment where the youth can engage in exercise without coaching another person can be created with the smart mirror. Furthermore, it was deemed that a user can safely perform an exercise by following the movements directed by a professionally created exercise program shown on display. An RGB-D camera such as the Kinect can be used to sense the movements in real-time and verify the accuracy of the movements before finally giving precise feedback to the user.
This system features the application of lifelogging and augmented reality of the metaverse (Figure 11). A user can get real-time feedback on the accuracy of their movements. The exercise data will be consistently recorded and used to analyze the exercise effect. The technologies derived during the development process are expected to be significantly instrumental in applying the experiential elements into metaverse-based training platforms.
3.4. Developing a Smart Mirror-Based AR Fitness System for Youth
The smart mirror, RGB-D camera, and server operate in connection with the smart mirror-based AR physical fitness system for youth. When a user stands in front of the smart mirror, the RGB-D camera uses the three-dimensional body posture estimation algorithm to extract the skeleton data of the user. The camera with a depth sensor finds information on a body part corresponding to a user’s skeleton. Then, based on the image data from the camera, the positions of each joint of the user are estimated through the open-source library of OpenPose. OpenPose is the first real-time multi-person human pose detection library [34]. Figure 12 is the result of detecting the user’s skeleton using OpenPose. The user’s motion is determined by measuring the angle of bending of the corresponding body part. For example, the movement of the left arm uses joint pointers on the left shoulder, elbow, and wrist. First, the system calculates the angle of the elbow by calculating the distance between each pointer. And we measure the arm’s movement through the angle of the elbow. The skeleton data collected accordingly are used to determine the accuracy of the movement during the exercise.
The process of examining the movement accuracy begins with pre-processing the joint position data. Then, the angle of the skeleton of a user is compared against that of the skeleton of the professional in the video to evaluate the similarity between the two. The movement data of the professional is created with the skeleton extracted every two frames out of the video of the professional and the angles between a total of 16 skeletons recorded. When a user performs the exercise, the angles will be compared against the prerecorded movement data of the professional. Suppose the skeleton of the user exceeds the threshold set in the movement data of the professional. In that case, it is determined that the user makes an inaccurate movement or posture, and the visual feedback will be provided accordingly. If there is a difference between the expert motion data and the angle range of the user’s skeleton by 15 to 20 degrees or more, the system recognizes it as an incorrect posture (Figure 13(a)). The danger range between 10 and 15 degrees (Figure 13(b)), and if it matches within 10 degrees, it is judged as the correct action (Figure 13(c)). Feedback on the user’s motion accuracy is displayed in a visual graphic. Visual graphics help users intuitively recognize their motion accuracy. In order to reduce the range of determining operation accuracy, a plan is being developed to increase the accuracy of user motion measurement in the system by using machine learning.
(a)
(b)
(c)
Yoga, stretching, step box, tae bo, and dance were selected for exercise content appropriate for the environment where a smart mirror is used based on the consultation with experts. The exercise programs were planned and designed to conduct the selected sports within the range in front of a smart mirror. The exercise program video was filmed in 4K resolution in a Chromakey environment to create expert data and exercise content (Figure 14). The difficulty in producing exercise videos was to organize a program so that all movements could be made in the limited space of a smart mirror. Therefore, the movement to the left and right was reduced as much as possible, and the motion was carried out in the front-rear or vertical direction. Exercise may change the static while reducing left and right movement, but it is configured to provide the same exercise effect even in this situation. For example, the movement had to move left and right, allowing the user to turn sideways and move. However, at this time, since the user cannot check the information through the smart mirror, it was instructed to see the front again quickly after a short operation. Configuring an exercise program so that dynamic movements can be performed in such a limited space is expected to be a continuing challenge in the future. The level of difficulty was categorized depending on the capabilities of the user. For example, the step box has three levels: beginner, intermediate, and advanced, while the K-pop dance has levels of lecture per part, checking per part, and full version.
(a)
(b)
(c)
The program flow is shown in Figure 15. First, a user selects the desired exercise content. If the dance is selected, the user selects the music and the difficulty level (Step 2). As below, the themes of the lecture per part, checking per part, and the full version can be selected (Step 3). When the difficulty level is selected, the content will be played. Next, the user performs the exercise according to the video on the smart mirror (Step 4). The program recognizes the user’s movement in real-time. At the same time, the exercise is being performed, which compares the movement against the data of a professional, calculates the accuracy of the movement, and then provides feedback to the user. The user can see the feedback information to recognize the accuracy of one’s movement and understand one’s exercise capability. In addition to the feedback, a user can check and adjust one’s movement intuitively through the smart mirror. Once the content is finished, the result window will appear, displaying information such as exercise time, calorie consumption, and heart rate (Step 5).
A user interface designed for the interaction optimized for the smart mirror was also developed (Figure 16). When the content is played, unnecessary icons are disabled to improve the user’s engagement with the content. Furthermore, the transparency rate and font color were selected in consideration of the half-mirror transmissivity so that a user could obtain the information quickly while practicing the exercise. It required an intuitive and attractive design of UI elements such as visual icons to increase the system’s usability. Therefore, icons such as home, shut down, and skip adopted a form commonly used in digital devices (Figure 16(c)). In addition, menus and UI elements were consistently configured so that users would not be confused when using the program. The distance between the device and the user is an important consideration for the UI design of the program. When selecting an exercise program, the user operates directly in front of the device. All components are concentrated in the center so that all components can enter the user’s field of view. The size of the letters and icons is designed to be relatively small. However, when the program is all over, the letters are enlarged, and it is designed to check the exercise results through various effects (Figure 16(d)). In the future, further research on UI design according to user age and physical conditions will be needed. This is because it has been confirmed that there is a big difference in usability during system operation depending on the user’s vision and height.
(a)
(b)
(c)
(d)
4. Discussion
The system currently has certain limitations. First, the movement details of a professional should be prepared and optimized to create new content, which consumes a considerable amount of time. If the optimization is not processed sufficiently, the standard threshold value setting for analyzing the accuracy of the movement will not be accurate. Consequently, the body posture feedback information cannot be reliable. Moreover, when the feedback is inaccurate, a user cannot check whether they perform the movement accurately. Hence, the user will not trust the exercise result, and consequently, there will be an adverse effect on performing the exercise continuously. Difficulty with preparing the movement data of professionals will also influence quick content expansion. Inducing interest in continuous exercise will need a variety of sports programs. A user will become easily bored after repeating a couple of sports programs a few times. The difficult process of preparing and optimizing the movement data of the professional will interfere with fast content updates. It is expected to take less time as the know-how is accumulated after the repetition of program development.
Another limitation is that the exercise cannot be conducted with other users. As mentioned in the characteristics of XR physical fitness, exercising with others is more enjoyable and enables more extended exercise than engaging in exercise alone. However, the current system enables only one user to exercise in front of a smart mirror. When it comes to the youth who usually would not concentrate for a long span may well exercise for a relatively short period. Therefore, the communications feature enabling a group exercise can bring many advantages. A user can find an exercise partner or learn from more experienced users with the communications feature. Even if the system provides the sports program, voluntary coaching and learning among the users should be one of the strong points of a metaverse-based training platform.
The last limitation is regarding the exercise space. A smart mirror limits the area where the exercise can be performed. The exercise should be performed within the specific area, right in front of the smart mirror, so that feedback on the movement accuracy can be provided through the camera. A user can receive intuitive feedback by checking on one’s image in the mirror. The limitation in terms of the area serves as a strong constraint in terms of content design, because the movement that goes out of the area in front of the smart mirror cannot be recognized. Another problem due to the useage of a smart mirror is that it is complicated to secure the exercise space in a small area. The RGB-D camera requires a minimum distance of 2.5 M to recognize a user’s movement accurately. The distance between the smart mirror and the user can cause space problems and reduce the level of user engagement with the content. The fisheye lens and the image compensation processing technology were used to reduce the distance to less than 2 M, but the space problem limited by the smart mirror remains.
Nevertheless, the smart mirror-based AR physical fitness system demonstrates the possibility of creating new experiences and highlighting the importance of the heuristic elements on the metaverse-based training platform.
5. Conclusions
In this study, a bone age-based physical fitness management service was proposed to improve the exercise capability of youth. Developing the smart mirror-based AR physical fitness system was demonstrated as the metaverse-based XR physical fitness offering physical fitness programs. This study was initiated to provide a physical fitness service for stable physical development after realizing that the youth shows individual differences in terms of speed and pattern of physical development. Through the physical fitness data analysis of youth, it has been confirmed that the categorization of growth development based on bone age rather than chronological age is the most accurate and reliable for estimating the growth status of youth. Continuous data accumulation is underway to establish the Korean standard indexes for growth development based on bone age. When the indexes are established, it is expected that the characteristics related to the physical strength of youth can be analyzed in more detail.
Furthermore, the smart mirror-based AR physical fitness system provides an exciting exercise experience for youth. The system collects the movement data and provides posture accuracy feedback using the RGB-D camera and body posture estimation algorithm. Accordingly, it offers sufficient exercise in an environment without a coach or instructor. There are still issues concerning the content, communications features, and exercise space. However, improvement plans are already prepared, and technology development is in progress. Collecting a user’s movement data is essential for the research of interactive metaverse services. To this end, continuous efforts will be made to research the methods of collecting actual human data, processing, analyzing, and utilizing the collected data, and finally substantiating the methods in the metaverse content.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest.
Acknowledgments
This work was supported by the Soongsil University Research Fund of 2019 and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea Government (MSIT) (Grant no. 2022-0-00218).