Abstract

There are increasing new advances in virtual reality technologies as well as a rise in learning virtual environments for which several studies highlighted the pedagogical value, knowledge transfer, and learners’ engaged-behaviors. Moreover, the notion of user experience is now abundant in the scientific literature without the fact that there are specific models for immersive environments. This paper aims at proposing and validating a model of User eXperience in Immersive Virtual Environment, including virtual learning environments. The model is composed of 10 components extracted from existing models (i.e., presence, engagement, immersion, flow, usability, skill, emotion, experience consequence, judgement, and technology adoption). It was validated in a user study involving 152 participants who were asked to use the edutainment application Think and Shoot and to complete an immersive virtual environment questionnaire. The findings lead us to a modified user experience model questioning new paths between user experience components (e.g., the influence of experience consequence on flow).

1. Introduction

The digital age has led to the emergence of digital games for learning [1]. These games are now too limited to conventional techniques such as Web, e-mail, or video conferencing [2]. So, new types of games known as “virtual learning environments” have succeeded in fully exploiting advanced technologies such as virtual reality. These virtual environments have opened up new perspectives in the field of education, increasing the number of stimuli by an immersive experience and then improving motivation [3].

Immersion should increase the motivation of users, and immersive virtual environments should provide a good User eXperience (UX) [4]. UX has been defined as “the user’s perceptions and responses resulting from the use of a system or a service […]” by the ISO 9241-210 (2009) norm. However, only few UX models are identified for immersive virtual environments (IVE) in the scientific literature. The UX approaches for virtual environments discussed in literature focus on a single or a couple of components, particularly presence and immersion (i.e., [58]). Indeed, all of the UX components in IVE are not dealt with in the literature. The IVE should be useful, usable, and acceptable, provide a high level of presence, immersion, and flow, and cause little cybersickness. This led us to propose holistic model of the UX in IVE that considers the multiple components of the UX.

The aim of the present study is to propose and validate a conceptual model of the UX in IVE named User eXperience in Immersive Virtual Environment Model (UXIVE Model) in the field of edutainment. In Section 2, the UXIVE conceptual model is described in relation to UX models referenced in literature. An empirical study aiming at validating the UXIVE Model is detailed in Section 3. In Section 4, the validity of our model and component relationships with regard to the literature are discussed. In conclusion, suggestions on how to use our model to improve the UX in IVE, particularly in the field of edutainment, are evoked.

2. Proposition of a Model Measuring the UX in IVE

The UXIVE Model is based on four UX models [912]. These models are reviewed in Section 2.1. The hypothesized model was designed based on these four models and presented in Section 2.2. We chose these four precursor models for two reasons: they are—to our knowledge—the only ones to precisely identify and describe the relationships between components of the UX; they are suitable or adaptable immersive virtual environments in the field of edutainment. These models are complementary to more recent works but not specific to virtual environments as, for example, [13, 14].

2.1. User Experience Models Review
2.1.1. User Experience Based on Flow

The flow is defined as “holistic sensation that people feel when they act with total involvement” ([15], p. 36) as “an enjoyable experience in which a participant feels an important level of behavioral control, happiness and enjoyment” ([9], p. 174) when interacting within the virtual environment. These authors propose a model designed in a context of a virtual entertainment environment and its components are structured around the concept of flow in a virtual environment. Their model comprises 10 components identified as interactivity, involvement, vividness, skill, challenge, focused attention, flow, telepresence, positive effect, and loyalty. They consider that the user enters in a flow state when he interacts and perceives that the whole virtual environment is in his grip, when there is a balance between his skills and the challenge given, when he focuses on the current activity, and when he feels “telepresent”. Moreover, this model suggests that vividness stimulates the sensory perceptions of the user improving his interactivity and enhancing the sense of telepresence. The model also reveals that higher level of involvement (expert/novice distinction, importance) increases the likelihood of forming positive telepresence, creates greater user challenge, and yields the user to develop stronger skills. The authors also found that mediated environment with higher interaction increases user attention (i.e., concentration) on the current activity, increases the perceived challenge for even the most experienced, and improves user perceptions of control skills. They observed that the user who focuses on stimuli in the virtual environment perceives a higher level of telepresence and that strong flow creates an emotional state in users and positively affects users’ attitude and future usage desirability, thus, loyalty.

2.1.2. User Experience Based on Acceptance and Continuance

The technology acceptance is defined as the actions and decisions taken by the user for a future use of the virtual environment. The continuance intention is the intention of the user to continue using such system. A second model created in the virtual learning environment context is built on acceptance and continuance theories [10]. This model includes 9 components known as immersion, presence, flow, confirmation, satisfaction, perceived usefulness, perceived ease of use, previous experience, and intention. The authors stated that the three constructs of flow, presence, and immersion coinfluence each other. They also revealed that users perceived immersion, flow, and presence as having impact on confirmation of what they expected regarding the technology. They observed that satisfaction, perceived usefulness, and perceived ease of use are influenced by the users’ confirmation of the level of technology services and that user satisfaction is impacted by perceived usefulness, perceived ease of use, and confirmation. The author found that users’ continuance intention is primarily determined by their satisfaction with prior technology use and that previous learning experience with technologies is a factor influencing intention directly.

2.1.3. User Experience Influenced by Virtual Environment Features

Lin and Parker [12] proposed a model designed in a context of virtual environment for entertainment which aims at identifying the virtual environment features (i.e., field-of-view, stereopsis, visual motion frequency, level of interactivity, and visual interventions predictability to visual motion) leading to an optimal UX. The authors define the UX through three components: presence, enjoyment, and simulator sickness. According to their model, there is a positive correlation between presence and enjoyment, a negative correlation between enjoyment and simulator sickness, no specific correlation between presence and simulator sickness, and a decline in presence or a growth in simulator sickness results from a decrease in enjoyment.

2.1.4. User Experience and Interaction

Mahlke [11] proposed a model which introduces the UX resulting from the interaction with a technical system. This model comprises 8 components defined as system properties, user characteristics, context parameters, human-technology interaction, perceptions of instrumental qualities, perception of noninstrumental qualities, emotional user reaction, and consequences of the UX. System properties, user characteristics, and context parameters are defined as categories of factors that determine the UX. The author observed that system properties lead to differences in objective measures of the interaction (e.g., number of accomplished tasks and time on task) as well as differences in UX (i.e., instrumental and noninstrumental quality perceptions as well as emotional user reactions) and in consequences of the experience (i.e., overall judgements and alternative choices). Their model suggests that user characteristics influence subjective feelings and context parameters influence overall judgement. The model also shows that emotional user reactions are assumed to be influenced by instrumental and noninstrumental quality perceptions. This suggests that consequences of UX are based on instrumental and noninstrumental quality perceptions as well as emotional user reactions.

2.2. Towards the UXIVE Model
2.2.1. Presence, Flow, and Experience Consequence

Presence is a component defined as the user’s “sense of being there” in the virtual environment. Presence is achieved as soon as another reality is evoked, e.g., reading a book, watching TV, and experiencing virtual reality [16]. The concept of presence can be divided into three categories: presence or telepresence, copresence, and social presence in a collective or collaborative virtual environment [9, 17, 18]. Experience consequence is a component we defined as the symptoms (e.g., the “simulator sickness”, stress, dizziness, and headache) the user can experience in the virtual environment. As we saw previously in the user experience models review, for [9], the user enters in a flow state when he feels “telepresent”. Shin, Biocca, and Choo [10] have investigated the role of flow and presence and found that all two coinfluence each other; it can be said that they play enhancing roles for each other. As for [12], they investigated the degree of presence, enjoyment, and simulator sickness users experience in a virtual environment; they found that there is a positive correlation between presence and enjoyment, a negative correlation between enjoyment, and simulator sickness. Zhou and Lu [19] noted that flow experience includes perceived enjoyment and attention focus. Consequently, we may hypothesize that a low experience consequence and a high sense of presence greatly enhance the state of flow which influences back the two components (Hypothesis 1).

2.2.2. Presence, Immersion, and Engagement

Immersion is a component defined as the “objective level of sensory fidelity a virtual reality system provides” ([20], p. 38). The immersive dimension in a virtual environment is created by “complex technologies that replace real-world sensory information with synthetic stimuli” ([20], p. 36). Engagement is a component defined as “a psychological state experienced because of focusing one’s energy and attention on a coherent set of stimuli or meaningfully related activities and events” ([21], p. 227). Following the Witmer and Singer’s approach, immersion and engagement are two elements that contribute to the idea of presence [21]. Shin et al. [10] consider immersion as clearly related to the widely research concept of presence. Hence, we may hypothesize that the sense of presence enhances the degree of engagement and the feeling of immersion; the same way the two components enhance presence (Hypothesis 2).

2.2.3. Flow, Usability, Skill, and Emotion

Usability is a component defined as the ease of using (i.e., efficiency, effectiveness, and satisfaction) the virtual environment. Skill is a component defined as the knowledge the user gain in mastering his activity in the virtual environment. Emotion is a component defined as the subjective feelings (i.e., joy, pleasure, satisfaction, frustration, disappointment, and anxiety) of the user in the virtual environment that varies with his prior experience with virtual reality [22]. According to Shin et al. [10] users feeling present and in a state of flow may want to perceive what is useful and easy to use and thus feel satisfied. Given the widely accepted factors, perceived usefulness, perceived ease of use, and satisfaction can be indicators of perceived usability [10]. Cheng et al. [9] asserted that the user can derive emotions from the human-machine interaction experience and that strong flow creates this emotional state. They also suggest that skill affects flow. In fact, the state of flow results from an equilibrium between the user’s perceived skill and the challenge given. Thus, we may hypothesize that the perceived skills influence the state of flow which in turn influences the perceived usability and the emotion (Hypothesis 3).

2.2.4. Experience Consequence, Judgement, and Technology Adoption

Judgement is a component defined as the overall judgement (i.e., positive, indifferent or negative) of the virtual environment. Technology adoption is a component defined as the actions and decisions taken by the user for a future use or intention to use of the virtual environment. According to Shin et al. [10], experience consequence can alter the user’s prior use and thus the intention to use the virtual environment. Indeed, as revealed by Lin and Parker [10], simulator sickness (i.e., a UX after effect) severity could make the users feel uncomfortable and even withdraw from a virtual environment exposure. Thus, simulator sickness could contribute negatively to UX. According to Mahlke [11] the consequence of the experience incorporates the acceptance of the system (overall judgement of the system), intention to use the system, and usage behavior. Consequently, we may hypothesize that the experience consequence influences the overall virtual environment judgement and the propensity of adopting the technology (Hypothesis 4).

These four hypotheses, justified by the literature, lead to a hypothesized model named User eXperience in Immersive Virtual Environment Model or UXIVE Model (Figure 1). This model has been validated through an empirical study.

3. Validation of the User eXperience in Immersive Virtual Environment Model

After explaining the purpose of the study, methodology detailing participants, material, measures and the procedure, and results of the structural equation modelling analysis of the UXIVE Model are exposed.

3.1. Aim of the Study

This study aims at validating the theoretical UXIVE Model defining relationships between ten UX components specific to IVE: presence, immersion, engagement, skill, emotion, flow, usability, technology adoption, judgement, and experience consequence (Figure 1). In other words, this aim was to validate the following four hypotheses:(i)Hypothesis 1: a low experience consequence and a high sense of presence greatly enhance the state of flow which influences back the two components.(ii)Hypothesis 2: the sense of presence enhances the degree of engagement and the feeling of immersion; the same way the two components enhance presence.(iii)Hypothesis 3: the perceived skills influence the state of flow which in turn influences the perceived usability and the emotion.(iv)Hypothesis 4: the experience consequence influences the overall virtual environment judgement and the propensity of adopting the virtual environment technology.

3.2. Method
3.2.1. Participants

One hundred fifty-two participants (28 women and 124 men; 37 inexperienced and 115 experienced in virtual reality; students or professionals) had volunteered to take part in this study. The sample mean age was 23.96 years and the standard deviation was 6.93 (ranging from 18 to 63).

3.2.2. Material and Measures

Think and Shoot on Oculus DK2. Think and Shoot is an edutainment virtual environment aiming at familiarizing the participants with the notions of function and parameters. It consists of two main actions which are collecting balls and shoot on evil creatures. The participants are given instructions in a pseudo programming language where the action of shooting is represented by a function and the three parameters of the function are represented by the type of evil creature to shoot on (i.e., two different types of creatures), the type of ball to shoot (i.e., three different types of balls), and the remaining balls that can be used. An example of instruction could be “Shoot (fire creature, ice ball, 0)” (Figure 2), the participant should understand that there is no ice ball left to shoot on the fire creature, and he must collect more.

This edutainment virtual environment was designed with the development tool UNITY © and ran on a Dell 64bits with 4 GB of RAM computer, an Intel® Xeon® processor, CPU E5-16030 2.80 GHz. A Logitech wireless gamepad and an Oculus development kit 2 (DK2) allowed the participant to collect balls and to shoot on three different sphere targets in the training session and on two different evil creatures in the regular session according the instructions given on a panel in the application. 3D spatialized sound was rendered in a Tritton AX 180 audio headset.

Questionnaires. Participants had to answer two questionnaires before and after the task.

The questionnaire filled in before the task contained 11 questions: four demographic items (age, gender, marital status, and occupation) and seven items on prior knowledge about programming and familiarity with virtual reality (e.g., “Among the following devices (such as virtual reality headsets, data glove...), which ones have you used and how often (never, little, sometimes, often, very often)?”).

The subjective questionnaire completed after the task was a UX questionnaire consisting of 84 items rated on 10-point Likert scales. It contained 12 items to measure presence taken from [21] (Cronbach’s α = 0.76; e.g., “The virtual environment was responsive to actions that I initiated”), 3 items to measure engagement translated from [21] (Cronbach’s α = 0.74; e.g., “The sense of moving around inside the virtual environment was compelling”), 7 items to measure immersion inspired by [21] (Cronbach’s α = 0.74; e.g., “I felt stimulated by the virtual environment”), 11 items to measure flow taken from [23] (Cronbach’s α = 0.82; e.g., “I felt I could perfectly control my actions”), 6 items to measure skill translated from [24] (Cronbach’s α = 0.80; e.g., “I felt confident selecting objects in the virtual environment”), 15 items to measure emotion taken from [25] (Cronbach’s α = 0.72; e.g., “I enjoyed being in this virtual environment”), 9 items to measure experience consequence inspired by [26] (Cronbach’s α = 0.91; e.g., “I suffered from fatigue during my interaction with the virtual environment”), 12 items (grouped in 4) to measure judgement taken from [27] (Cronbach’s α = 0.82; e.g., “Personally, I would say the virtual environment is impractical/practical”), and 9 items to measure technology adoption inspired by [28] (Cronbach’s α = 0.75; e.g., “If I use again the same virtual environment, my interaction with the environment would be clear and understandable for me”).

Usability Measures. Usability was measured through level completion time, number of errors, total levels score, and level reached. Participants evolved from one level to the next after having eliminated all the creatures in the level. Level completion time is the time the users took to complete the level n°2 (i.e., level reached by all participants). The number of errors are the points the user had lost when he collided with a creature (i.e., one point was lost for every collision with a creature) during the whole session. The total levels score is the total score gained in all levels, and the level reached is the level the users reached at the end of the session.

3.2.3. Procedure

The experiment took place in a research laboratory in computer sciences. Once participant had filled in the pretask questionnaire (personal details), the experimenter introduced and gave the participant instructions about the experiment. The first one was an orally guided training session of about 5 minutes (the participants could ask for more or less training time if they felt more or less comfortable in the IVE). The second session was a regular session of 5 minutes. For each session, participants followed the instructions in a pseudo programming language written on a panel in the application. A questionnaire on UX completed the session. Each participant spent between 30 to 45 minutes in the experiment room.

3.2.4. Statistical Analyses

A Structural Equation Modelling (SEM) analysis was used to validate the hypothesized UXIVE Model. This statistical method has long been associated with the idea that a high sample size was necessary [29], recent studies show that the association between sample size and result is much more complex. For example, Wolf, Harrington, Clark, and Miller (2013) indicate that the recommended minimum sample size may decrease if the number of components to be measured and the number of items per component is high (e.g., 8 items per component) [30]. Moreover, the reality on the field demonstrates the difficulty in obtaining the recommended minimum sample size. This gives rise to SEM studies with small samples [31]. According to these authors, our sample size (i.e., N = 152) and the complexity of the UXIVE Model (i.e., 10 components) are suitable for SEM analysis.

Scale scores of the UX questionnaire and usability measures (i.e., level completion time, the number of errors, the levels score, and the level reached) for all participants were both taken into account in the structural model designed on the IBM® SPSS® AMOS 24 software. To assess how well the model represented the data, four goodness-of-fit recommended indices were evaluated in a structural model: Comparative Fit Index (CFI), Normed Fit Index (NFI), Root Mean Square Error of Approximation (RMSEA), and the ratio Chi-square and the degree of freedom. The recommended fit indices of these analyses are presented in Table 1.

3.3. Results: From a Hypothesized UXIVE Model to a Modified UXIVE Model

Structural Model of the Hypothesized UXIVE Model. The structural model analysis—based on a SEM analysis—should reveal significance and fit of the components relations with the data collected. The test from the structural model (Figure 3) revealed three statistically significant relations and nine nonsignificant relations. We rejected these nine relations (Table 2).

Furthermore, the model fit according to the SEM fit indices indicated that CFI is 0.539, NFI is 0.390, and RMSEA is 0.086 which also indicate poor fit of our hypothesized UXIVE Model. However, χ2/df falls into the acceptable range (i.e., 2.098). Indeed, Kline (1998) suggested that there should be a minimum of four indices that are acceptable and compatible with the model fit, we only have one acceptable index fit for this tested structural model [32]. The hypothesized UXIVE Model is then poorly acceptable.

Modified UXIVE Model. To improve the structural model of the hypothesized UXIVE Model, a covariance analysis was conducted. The analysis revealed 32 hypothesized covariances within which 15 statistically significant covariances matched with the tested structural model (i.e., the statistically significant relations from the tested structural model were preserved). According to the covariance analysis, nine hypothesized new relations were added to the conceptual model and six relations were dropped. All the 15 relations of the modified UXIVE Model were statistically significant (Table 3).

So, in the modified UXIVE Model (Figure 4), the hypothesis 4 is fully validated, and the hypotheses 1, 2, and 3 are partially validated.

4. Discussion

This present research set out to integrate the fragmented theories and research on UX into a unified UX in IVE model named UXIVE Model. We analyzed our UXIVE Model and questioned the relationships between UX components.

Our first hypothesis stating that a low experience consequence and a high sense of presence greatly enhance the state of flow which influences back the two components is partially validated in the modified model. The results rather show influence of experience consequence on flow and influence of the latter on presence. According to several studies there is indeed a correlation between presence and flow and this relationship is of great interest for game fields (e.g., [9, 10, 33]) due to the immersive experience provided by the two components. However, no known studies focused on the relationship between experience consequence and flow.

Furthermore, the modified model revealed a significant path estimation from flow to technology adoption and from experience consequence to emotion. Studies show interest in the relationship between flow and technology adoption (e.g., [34, 35]) in online games or e-learning fields; studies in this area wish to predict user technology acceptance by measuring the flow. Experience consequences such as motion sickness are known to be affected by emotional factors such as fear, anxiety, and nervousness [36].

Our second hypothesis stating that the sense of presence enhances the degree of engagement and the feeling of immersion, and the same way the two components enhance presence, is partially validated in our modified model. The results rather show influence of engagement on both presence and immersion and no correlation between presence and immersion contrary to other studies (e.g., [5, 10, 37]). In addition, the modified model shows correlation between engagement and technology adoption, presence and emotion, and presence and judgement. The relationship between presence and engagement is indeed well known and studied in several works (i.e., [12, 3739]), other studies in the game field investigated engagement as being the first state of immersion (i.e., [40, 41]), and more studies in the education field and technology adoption issues investigated engagement through technology adoption (i.e., [42, 43]). Emotional reactions and presence are related according to studies in virtual environment and game fields (i.e., [44, 45]).

Our third hypothesis stating that the perceived skills influence the state of flow which in turn influences the perceived usability and the emotion is again partially validated in our modified model. The results rather show skill influence on flow which in turn influences emotion. As shown by [46], flow is highly correlated with the user’s emotions and just as [47]; he relates flow and skill in their studies.

The modified model revealed a significant correlation between usability and technology adoption, skill and experience consequence, and skill and usability. Perceived usability as enhancing technology adoption is indeed confirmed by other studies (i.e., [10, 48]). Several other studies confirmed that usability perception depends on user skill (i.e., [5, 49]), whereas no studies are known that report on the effects of perceived skill on experience consequence.

Our fourth hypothesis stating that the experience consequence influences the overall judgement and the propensity of adopting the VE technology is validated in the modified model. These findings are in line with Mahlke’s model [11] and specify that the judgement is a moderator between the experience consequence and the technology adoption.

Finally, this study revealed new UX components relationships to investigate. Indeed, the modified model states that flow and engagement are the two components influencing presence. Experience consequence and skill are the two components influencing flow. Presence, flow, and experience consequence are the three components influencing emotion. Engagement is the unique component influencing immersion. Skill is the unique component influencing experience consequence and usability. Presence and experience consequence are the two components influencing judgement. Flow, experience consequence, engagement, and usability are the four components influencing technology adoption.

5. Conclusion

This paper examines new relationships among UX components in an IVE framework and confirms others through two steps. First, a review of UX models led us to propose the hypothesized UXIVE Model. Second, we assessed the whole theoretical structure with analyses based on SEM and correlations. Results do not allow us to validate hypothesized UXIVE Model but allowed us to propose a modified UXIVE Model validated in the field of edutainment.

The most significant limitation of this study is the sample size, even if this sample is sufficient according to [30, 31]. To resolve this limitation, the experimental protocol could be pursued in the same conditions to gather a larger participation rate. Another solution to gather a larger participation rate would be to conduct an online experimental protocol. Tools such as “webvr” websites (see, for example, webvr.info) allow the user to experience virtual reality in his browser with any type of virtual reality device. Online experiments usually gather many users (e.g., 1000). Indeed, the SEM analysis recommend that the more variables are included in an SEM analysis; the more participants are needed to guarantee the stability of the results. In addition, more experiments are needed to corroborate the findings in this study as some relationships revealed by the study are relatively new and have not been investigated in previous studies.

The second limitation of this study is the sample profile. 48% of the participants share the same characteristics (e.g., age, sex, studies or work area, and experience with 3D technologies). Although, the target population of virtual edutainment environment is young, it could be interesting to apply these findings to other population because virtual reality devices are more and more reaching every type of profile. This limitation is directly linked to the third limitation which involves the device and content types.

The third limitation of this study is that other types of immersive virtual devices and other types of content are excluded. Indeed, the experiment is conducted with a Head Mounted Display (HMD) with an edutainment application. So, our modified UXIVE Model is only valid for Think and Shoot on HMD. First, this model should be validated for other immersive edutainment applications such as REARTH [50]). Second, this framework could be extended to different fields such as therapeutic, industrial, or collaborative applications. Third, the UXIVE Model can be experienced through different types of devices such as 6 axes virtual simulators, 6 walls CAVEs, and Z-spaces. Indeed, using our model in other studies from several fields and with different immersive virtual technologies may contribute to generalize our modified UXIVE Model.

Data Availability

The data used to support the findings of this study are included within the Katy Tcha-Tokey’s thesis manuscript.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

The authors acknowledge Laval Agglomération and the Mayenne Department for their financial support and the participants who took time to engage and provide feedback on this project.