Abstract
The objective of this study is to investigate the effect of the gap between two different mental models on interaction performance through a quantitative way. To achieve that, an index called mental model similarity and a new method called path diagram to elicit mental models were introduced. There are two kinds of similarity: directionless similarity calculated from card sorting and directional similarity calculated from path diagram. An experiment was designed to test their influence. A total of 32 college students participated and their performance was recorded. Through mathematical analysis of the results, three findings were derived. Frist, the more complex the information structures, the lower the directional similarity. Second, directional similarity (rather than directionless similarity) had significant influence on user performance, indicating that it is more effective in eliciting mental models using path diagram than card sorting. Third, the relationship between information structures and user performance was partially mediated by directional similarity. Our findings provide practitioners with a new perspective of bridging the gap between users’ and designers’ mental models.
1. Introduction
Originating from psychology, mental models are applied to Human-Computer Interaction (HCI) to explain people’s understanding about how computers work [1]. Information technology (IT) products are developed based on designers’ mental models, but what designers believed to be easy to understand is not necessarily true for users. Users interact with IT products in a different perspective from designers. They form their own understanding and predict feedback of IT products [2]. Users whose mental models are different from those of designers encounter interaction difficulties, but certain users with wrong/incomplete mental models can also successfully use IT products [3]. Although actions (e.g., training) have been taken to reduce the gap between users’ and designers’ mental models, few studies examine whether and to what extent the gap is reduced.
To address the above problem, this study quantified the gap between mental models and investigated its impact on user performance. However, it is challenging to elicit and quantify the gap between mental models, because they are in the users’ head and are not directly observable. Many researchers have explored mental models using IT products with a well-defined information structure (e.g., web pages and menus) that clearly reflects designers’ mental models. Then, users’ mental models of information structures can be elicited. Thus, the extent to which users’ mental models differed from those of the designers can be quantified.
Traditional method for eliciting mental models of information structures was card sorting, but mental models elicited by card sorting cannot represent users’ understanding of specific directional relationship between elements of information structures. This study proposed a new method, path diagram, to elicit mental models of information structures. The mental models elicited by path diagram can represent users’ understanding of directional relationship between elements of information structures. To further quantify the gap between the mental models, this study introduced an index called mental model similarity. Two kinds of mental model similarity are distinguished: directionless similarity, calculated from the card sorting, mainly represents the directionless relationship between elements; directional similarity, calculated from the path diagram, mainly represents the directional relationship between elements. Therefore, two research questions were considered: (i) Which method is more effective in eliciting mental models of information structures? (ii) How influential are directional similarity and directionless similarity on interaction performance?
In this study, websites with three information structures (i.e., net, tree, and linear) were developed to elicit mental models through card sorting and path diagram. Then, a method to quantify the degree of match between users’ and designers’ mental models was proposed. Based on that, the impact of mental models on performance was analyzed.
2. Literature Review
2.1. Mental Models
The theory of mental models has obscure origins [4, 5], but the notion of mental models first appeared in a book written by the psychologist Craik. Craik [6] believed that a brain could translate an external process into a model of the world, which is “a small-scale model of external reality and of its own possible actions within the head.” Since then, it has attracted much attention from researchers, particularly psychologists.
Forty years later, two researchers used the term “mental model.” Norman stated that “in interacting with the environment, with others, and with the artifacts of technology, people form internal, mental models of themselves and of the things with which they are interacting. These models provide predictive and explanatory power for understanding the interaction” [3]; Johnson-Laird [7] believed that people could create mental models that were structural analogs of the world, and their ability to construct, manipulate, and evaluate mental models had a hidden strong influence on rational thought.
Subsequent research on mental models can be approximately divided into two branches. The first mainly focused on internal mental processes and cognitive phenomena within the long-standing field of psychology. Typical research interests included the role of mental models in comprehension [8, 9], reasoning [9, 10], and deduction [11]. The second branch stepped out of psychology and applied mental models to support better interaction between people and the external world. Typical research interests include the role of mental models in learning and training [12–15] and using computers and appliances [16–19].
Among the second branch of research, one noticeable trend is a surge in the application of mental models in Human-Computer Interaction (HCI). Norman [20] showed that the root cause of problems in using technology products was the gap between users’ and designers’ mental models. This highlights a new perspective for HCI, and workers have tried various ways to deliver a design that matches users’ mental models [1, 21].
Originally used to explain team effectiveness, Team Mental Models (TMMs) refer to the extent to which team members shared organized understanding and mental representation of knowledge or beliefs relevant to the key content of the team’s tasks [22, 23]. Mathieu et al. [24] proposed that team members’ mental models consist of four parts: technology; job or task; team interaction; and other teammates’ knowledge and attitudes.
Since mental models are within the head, they cannot be directly detected. People’s mental models had to be indirectly inferred from observing and analyzing their elements. Many early researchers followed this focus, finding it challenging because mental models had the following five characteristics: (1) incompleteness: mental models are constrained by users’ background, expertise, and so forth; (2) vague boundaries: they can be confused with similar/related operations and systems; (3) being unstable over time: they evolve as people forget and learn; (4) they contained aspects of superstitions [20]. (5) Tendency to parsimony: people tend to construct a limited model of the relevant parts of a system [21].
Despite these challenges, researchers have identified various techniques to elicit mental models. General techniques to elicit individuals’ mental models and team members’ shared mental models were summarized in three comprehensive review papers [25–27]. Specifically, in the field of HCI, techniques to elicit mental models have four major categories: (1) verbalization through interview, thinking aloud, laddering and so forth: verbalization is the most widely used elicitation technique [5]. However, people’s verbalization was inconsistent and tended to evolve as they spoke [28, 29]. One solution to this problem is laddering, a modified interview technique, in which people were asked to identify multiple aspects of a problem and explore the relationship between their answers [30]. (2) Rating: people were asked to rate using questionnaires [30, 31], but this technique is not frequently used because relatively few questionnaires are well established. (3) Drawing sketches: people were asked to visually draw how they thought of a concept or the pathways from the start to a specific point in a system [18, 32]. (4) Card sorting: this technique is widely used in eliciting mental models of hierarchical systems [18, 33]. In most cases, the method is effective in eliciting mental models. This is especially true when dealing with information structures without considering the directional relationship. However, card sorting is not adequate for eliciting complete mental models if we consider the directional relationship of information structures.
2.2. Relation between Information Structures and Mental Model Similarity
Mental models are nowadays widely applied to analyze user performance on web pages, which are the best carrier of information structures. Many aspects of human interaction with hierarchical systems involve complex processes; thus, people who interact with hierarchical systems must have some type of mental models. Since mental models represent users’ understanding about a system including web pages, we argue that simplicity of information structures has an impact on mental model similarity. Previous studies have indicated that the card sorting is widely used in eliciting mental models of hierarchical systems [18, 33]. Based on this, we proposed Hypothesis (H1a).(H1a)The more complex the information structures, the lower the directionless similarity.
The directional similarity is calculated from path diagram, which is used to elicit mental models of information structures with directional relationship. We add more details (e.g., directional relationship) to mental models. Based on this, we proposed Hypothesis (H1b).(H1b)The more complex the information structures, the lower the directional similarity.
2.3. Mental Models and User Performance
Many previous studies have shown that mental models are correlated with user performance. On the one hand, mental models have a positive effect on user performance. Ziefle and Bay [33] pointed out that the better the mental models of navigations menus, the better the performance using the devices. Young [34] thought that mental models could explain user performance with the systems with which they interact. Dimitroff [35] found that students with more complete mental models made significantly fewer errors when they used the University of Michigan’s website. Sasse [36] noted the significant effects of mental models on user performance using Excel. Slone [37] found that users’ mental models affected their performance on websites. Brandt and Uden [38] pointed out that novices without strong mental models for information retrieval could not gather information successfully.
Numerous studies have also shown the relationship between TMMs and team performance. For example, Mathieu et al. [24] considered shared mental models as two categories: task and team, finding that both team-based and task-based mental models related positively to subsequent team process and performance. Mathieu et al. [39] demonstrated in a PC-based flight simulator that both task and team models had an impact on performance; the team process was supported by shared mental models and task-work mental model similarity, but not by teamwork mental model similarity, which was significantly related to both team processes and team performance.
On the other hand, workers have found that mental models had either an adverse effect or no significant effect on user performance. Halasz and Moran [40] and Borgman [41] found that, whether users formed a mental model of a system or not, they performed no differently on routine, simple tasks. Norman [3] found that users with wrong/incomplete mental models could use technology products successfully. Payne [42] noted that even wrong mental models did not necessarily result in the bad usage of devices. Schmettow and Sommer [43] found that the degree of match between the mental model and website structure had no effect on users’ browsing performance. As for TMMs, Webber et al. [44] found that team members sharing a common mental model with poor quality did not likely perform well.
It is obvious that the research consequences were contradictory. It seems that researchers cannot get the unified cognition of the effects of mental models on user performance. One possible reason is that most researchers do their studies without considering mental model similarity between users and designers. It is necessary for us to investigate mental models’ effects on interaction performance involving the mental model similarity. The only study of mental model similarity on interaction performance is seen in Schmettow and Sommer [43]. They found that mental model similarity had no effect on interaction performance. However, the way they elicited mental models was card sorting and they did not consider the directional relationship of information structures. In this paper, we considered more details such as directional relationship to elicit a mental model of information structure and get the directional similarity from path diagram and the directionless similarity from card sorting, which may get different results from what Schmettow and Sommer [43] found. Considering that there are more positive effects than adverse effects in the existing studies, in this paper, we are partial to supporting the positive effects. Therefore, we proposed Hypothesis (H2a) and Hypothesis (H2b).(H2a)Directional similarity predicts the task completion time; the higher the directional similarity, the less the task completion time.(H2b)Directional similarity predicts the number of clicks; the higher the directional similarity, the less the click times.
The effects of information structures on user performance have been widely investigated: since mental models are closely related to users’ behavior using various devices, a good mental model of information structure will likely enhance user performance. However, there are many other factors influencing user performance besides mental models, for example, users’ age and gender. Ziefle and Bay [33] pointed out that younger users were more effective in using a cell phone menu than the older users. Mathieu et al. [24] found that team processes fully mediated the relationship between team mental models and team effectiveness. Mathieu et al. [39] found that team performance was partially mediated by teammates’ mental models. Based on these findings, we have assumed that mental model similarity mediated the relationship between information structures and user performance, which is Hypothesis (H3a) and Hypothesis (H3b) (see Figure 1).(H3a)Directionless similarity will mediate the relationship between information structures and user performance.(H3b)Directional similarity will mediate the relationship between information structures and user performance.
2.4. Quantitative Methods of Mental Model Similarity
Elicited mental models are widely used in two ways. First, the difference between elicited mental models can be qualitatively and quantitatively compared. One common quantitative method is to compare the depth and width of the elicited structures. Users use an interface with a well-defined information structure such as phone menus and web pages.
Another quantitative method is to calculate the team mental model (TMM) similarity score, which indicates the percentage of team members’ shared, organized understanding and mental representation of knowledge or beliefs relevant to the key content of the team’s task [22, 23]. Specifically, team members usually rated the relatedness of pairs of statements describing team interaction processes and characteristics of team members on the Likert scale. The response could be analyzed using multidimensional techniques (e.g., the Pathfinder technique) to calculate the similarity score [26, 45, 46].
However, none of these methods can accurately elicit mental models of information structures with directional relationship; also these methods cannot quantitatively calculate the degree of similarity or dissimilarity of the mental models between users and designers of systems. Previous studies [43] have shown that the widely used method of eliciting mental models of hierarchical systems is card sorting, but it is inadequate to elicit mental models without considering the directional relationship of information structures. Also, the previous studies did not introduce a suitable method to quantify the mental model similarity. Wu and Liu [47] proposed a new computational modeling approach, which was composed of a simulation model of a queuing network architecture and a set of mathematical equations implemented in the simulation model to quantify mental workload, to model the mental workload in drive and driving performance. However, the approach proposed was not the quantitative method for calculating the mental model similarity. It was applied to analyze the mental workload in driver information system. Thus, here, we elicit mental models through card sorting and path diagram. Then, we conducted a new method to quantify the degree of mental model similarity.
3. Methodology
3.1. Equipment and Materials
A notebook computer with a touch screen (ThinkPad YogaS1) was used to present web pages, and a whiteboard was used to draw path diagrams. A camera (Sony, HDR-PJ610E) was used to record participants’ results of card sorting and drawing the path diagrams. A Morae Recorder was used for counting task completion time and the number of clicks and was also used to present the task specification for the participants. Web pages with different information structures were developed. The web pages were first considered about the navigation structures, which were net, tree, and linear. The depth of tree and net structure was three levels, which was seen common in daily web pages. The width of the bottom level of tree and net structure was nine items, which was also seen in common web pages.
To avoid the effects of familiar knowledge, topics about ancient inventions, ancient books, and ancient historical characters of different Chinese dynasties, which were not widely known, were presented in web pages as the content. There were totally nine different Chinese dynasties and each information structure was made of three different Chinese dynasties. Ancient inventions, ancient books, and ancient historical characters were corresponding to their own dynasties. Two pretests were carried out by two college students, and the interaction forms and content layout were adjusted according to the results. The memory capacity that influences user performance [33] was tested by a KJ-I spatial location memory span tester. Spatial ability was tested through the paper folding test [48]. The original paper folding test was translated into Chinese.
3.2. Participants
A total of 32 students from Chongqing University were recruited. The age of the participants ranged from 21 to 26 years (mean = 23.13, SD = 1.7). The gender was balanced. The experience of using laptops, tablets, and smartphones (see Table 1) was investigated; the results indicated that handheld devices (i.e., smartphones and tablets) were used intensively.
3.3. Task
All participants completed tasks in three different web pages with different information structures: net, tree, and linear. To avoid learning effects and make participants get a full understanding of the information structures of web pages, each participant completed eight tasks in a web page. The order of web pages in which participants completed the tasks is random.
The tasks are searching tasks. Participants found a target and read its content. The target is an item hidden in web pages, which is not widely known for participants. To check whether they read the content carefully, a single-choice test was conducted. Participants wrote down answers on a piece of paper. The test has two questions: one is about what dynasty the target belonged to, and the other one was about which the target is about. Taking “Huang dao you yi” as an example, the task is “Please find “Huang dao you yi” and read its description and then answer the following two questions: Q1: which dynasty “Huang dao you yi” belongs to? Q2: what “Huang dao you yi” is about?”
3.4. Dependent Variables
The two dependent variables were task completion time and the number of clicks. Task completion time was the average of eight tasks’ completion time under each information structure and the number of clicks was the average of the click times to complete all of the eight tasks. They were both measured by the Morae Recorder; only when the participant obtained the correct answers for both questions were the missions completed.
3.5. Independent Variable
The independent variables were the directionless similarity and directional similarity. It is a new quantitative measure proposed here (see Section 3.7), the directionless similarity was calculated from card sorting, and the directional similarity was calculated from path diagram. The mental model similarity was computed for each information structure.
Demographic variables included age, experience of using technology product, spatial ability, and memory capacity. A questionnaire was used to collect basic information about experience of using technology product
The mental model similarity was a between-subject variable, while the information structure was a within-subject variable. That is, each participant used three prototypes. In an effort to avoid learning effects, the order of using prototypes was random.
3.6. Procedure
The experiment took each participant about one hour to complete. It consisted of four tests: a questionnaire test, a memory test, a paper folding test, and a card sorting test.
First, each participant began the experiment by filling out a consent form and a general questionnaire about his/her demographic information and using experience with technology products.
Secondly, a memory test was conducted using the KJ-I spatial position memory span tester. After completing the memory test, the test scores were recorded on paper and were not disclosed to the participants.
Thirdly, a paper folding test was conducted to test the spatial ability of the participants. This test is divided into two parts. The time limit of each part was three minutes to avoid participants losing their patience. The participants needed to find the correct answer independently. The final score of the test is the correct number minus the incorrect number on the test paper. The higher the score, the better the spatial memory ability.
Fourthly, a brief introduction and practice about the experiment were given to each participant. Finally, participants completed tasks on each web page and then went on with the card sorting. The cards were the titles of each node in the information structures. Then, the participants were required to draw a path structure of the experimental web page navigation with a whiteboard stroke. During the whole process, participants were left alone, and questions related to the path were not answered in a relevant way, which aimed at avoiding the subjective impacts of the experimental designer. At the end, the experimenter conducted a five-minute exploratory interview with the participants to understand their thoughts and feelings about using the three web pages. The questions in the interview included “Q1: Please score the three web pages in this experiment considering information searching, ease of use and user experience. Q2: Please sequence the three web pages according to your experience.”
3.7. Quantifying Mental Model Similarity
The method proposed to quantify mental model similarity consists of two parts: one is the method which can be used to elicit mental models of information structures with directional relationship, and the other one is the mathematical equations which can be used to calculate the mental model similarity.
The first part is the method of eliciting mental models. In order to quantify mental model similarity, a method which can be used to elicit mental models with more details such as directional relationship has to be used. Such a method should represent the understanding of elements and directional relationship of a hierarchical system, which are the key factors in eliciting mental model of a hierarchical system. The literature provides several methods to elicit mental models, such as card sorting, which is widely used in eliciting mental models of hierarchical systems [18, 33]. However, card sorting cannot elicit complete mental models of hierarchical systems, particularly the directional aspects of hierarchies.
A new method is needed to reflect the directional information of mental models of hierarchy structures. Web navigation is similar to the real-world navigation. In real-world navigation, people usually take three strategies to find a destination. They remember properties of landmarks such as shape and structure (i.e., landmark knowledge) [49, 50], or the sequential order of landmarks encountered and directional relationship between these landmarks (i.e., route knowledge) [51], or an overview of the environment like a map showing spatial relationships between routes and landmarks (i.e., survey knowledge) [52] to find a destination. This knowledge is also involved in web navigation. Landmark knowledge is mainly represented through card sorting. Inspired by route knowledge, we proposed the path diagram to elicit mental models of hierarchical systems with more details (e.g., directional relationship)
The second part is about quantifying the mental model similarity. The limited research in quantifying mental model similarity was reported by Sinreich et al. [53]. They introduced process chart into eliciting mental models of Emergency Department Management and quantified similarity through four formulae. However, the method was not applied to analyze the mental models of information structures.
The method of quantifying the degree of mental model similarity aims to reflect the extent to users’ understanding of the system: it can present causality and logic. In addition, this method should be easy to understand, so that it can be mastered by nonprofessional persons. Thus, in this study, we extended the work of Sinreich et al. [53] as our proposed quantitative method. We calculated mental model similarity in two ways: calculating the directionless similarity from card sorting and calculating the directional similarity from path diagram.
The first component represents the elements (nodes) of the different content, for example, “Tang dynasty,” “Invention,” “Book,” and “Historical character” in Figure 2. The directionless similarity measure can be obtained usingwhere denotes the number of identical elements in card sortings and and denotes the number of elements that exist in that do not exist in (see that ). It is clear that . In the case that both card sortings are identical in terms of their elements (not necessarily their relationships); then we have while if no common elements exist, and by definition .
The second component represents the relationship between elements (arcs) in the path diagram. A relationship is defined by the elements it connects (there may be more than one connection between elements) and by the direction of the connecting arc, for example, the arcs that connect elements “Tang dynasty-Invention,” “Invention-Book,” and “Invention-Diao ban yin shua” in path diagram ① in Figure 4. The first step in calculating the directional similarity is to obtain the adjacency matrices. The element of adjacency matrix was the numbers of directed segment between two nodes (e.g., if there were one directed segment from “Tang dynasty” to “Invention,” the element is 1).
Based on the adjacency matrix, the sum of all the common arcs and the sum of all exclusive arcs between any two path diagrams and can be calculated, as shown in (2) and (3), respectively. Finally, the directional similarity measure can be obtained using (4).
It is clear that . In the case that both path diagrams are identical in terms of their relationship (arcs), then we have while if no common relationship exists between the two path diagrams. By definition .
These formulae, adapted from Sinreich et al. [53], are validated and used to calculate the similarity between process charts. In this study, we used them to analyze hierarchies and extended their work by adding directions. To analyze directional relationship, an adjacency matrix was used to indicate relationship according to the graph theory. Then, relationship similarity of hierarchical systems was calculated by using formulas (2), (3), and (4).
The process of calculating the mental model similarity can be seen in Figure 3.
In order to illustrate the calculation procedure of the similarity measure, the following example is given (see Figure 4).
From Figure 4, the following values are obtained: and .
Using these values in (1) results in the directionless similarity measure of . Using the graph theory, the adjacency matrix can be calculated as follows:
Using these values with (2) and (3) the common and exclusive path vector can be calculated as follows:Based on these values and (4), the directional similarity measure can be calculated as follows:
The directionless similarity is 1 while the directional similarity is 0.36. The calculation results show that the mental model similarity was totally different when we considered the directional relationship of information structures.
4. Results and Discussion
The following sections first examine the effects of the information structures on mental model similarity and then examine the relationship between mental model similarity and user performance. Finally, we examine whether the mediation effect was found.
4.1. The Influence of Information Structures on Mental Model Similarity
Simplicity/complexity generally refers to the level of intricacy or detail in a stimulus [54, 55]. Specifically, the detail could be the number of closed figures, open figures, letters, horizontal lines, vertical lines, and so forth [54].
Complexity of web pages with different information structures consists of two high-level notions: content complexity which is mainly measured through the number and types of objects to load a web page and service complexity which is mainly measured through “the number and contributions of the various servers and administrative origins” [56]. Specifically, if the interaction semantics and UI design are not considered, the structure of website is the focus of complexity. It could be computed through a function which calculated the number of outgoing links and controls such as buttons and checkboxes [57].
Regarding these studies, objective metrics of complexity of information structures were the number of nodes and links. As shown in Table 2, the numbers of nodes of all three information structures (i.e., net, tree, and linear) are the same, but the net structure has more directional links between any two nodes than the tree structure, which in turn has more links than the linear structure. Therefore, complexity of the information structure has three levels termed low complexity, medium complexity, and high complexity.
Subjective metrics of complexity were consistent with results of objective metrics. Participants rated the simplicity of three websites on a 5-point Likert scale anchored from “easy” to “complex.” The average rating for web net, tree, and linear was 4.1 (SD = 0.58), 3.1 (SD = 0.76), and 1.9 (SD = 0.94).
To further examine the influence of complexity of information structures on two different kinds of similarity, one-way repeated ANOVA analysis was conducted. As shown in Figure 5, the results indicated that complexity of the information structures had significant influence on directional similarity (, ). Specifically, results of multiple comparison with the Bonferroni corrections indicated that the complex net structure resulted in wider gap between the users’ and designers’ mental models than the three structure (, ) and the linear structure (, ). The complexity of information structures resulted in no differences in directionless similarity (, ). Therefore, (H1a) was rejected, and (H1b) was supported.
4.2. The Influence of Mental Model Similarity on User Performance
Linear regression was conducted. The dependent variables were the task completion time and the number of clicks, and there were seven independent variables: directional similarity, directionless similarity, information structures, age, technology product experience, spatial ability, and memory capacity. The results of regression analysis were shown in Table 3.
According to Table 3, the directional similarity was linearly related to the task completion time. Participants take less time to find the target item hidden in the information structures when the directional similarity was lower. This may explain the phenomenon that users who only remember the paths to a special item of web pages take less time to find a target in web pages than those who get a full understanding of elements and relationship of the web pages.
Table 3 also indicates that the directional similarity was linearly related to the number of clicks. Participants clicked less to complete the tasks when the directional similarity was lower and the web pages had more paths to obtain targets such as net structure. Although mental model similarity between the participant and designer was lower in the net structure than in the tree structure and the linear structure, the number of clicks was smaller with the lowest similarity. One possible reason for this is that the net-structure web page has many paths and the elements are connected to each other, while the tree structure and the linear structure are not. This means that users cannot return to the previous page when they need to reach other pages in the net-structure web page, unlike the tree structure and the linear structure.
It can also be inferred that using path diagram to elicit mental models is more effective than using card sorting. It can also verify that the directional relationship of information structures is important, which cannot be ignored in eliciting mental models of information structures.
The mental model similarity was calculated from the information structure. Under each information structure, how does the mental model similarity affect user performance? Correlation analysis and one-way ANOVA analysis were conducted.
The results indicated that only the directional similarity was positively correlated with the task completion time in the net-structure web page. However, either directional similarity or directionless similarity had no significant impact on user performance under each information structure.
In addition, the user performance was not affected by their age, spatial ability, memory capacity, or technology product experience. Moreover, no matter what kind of information structure, the user performance was still not affected by demographic variables. One possible reason is that the participants chosen for this study were all college students: the differences among them were too small. In other words, the selection of participants has limitations. It may be different when expanding the scope of participants, especially to the elderly, children, and those with lower levels of education.
4.3. Mediation Effects of Mental Model Similarity
According to MacKinnon et al. [58], three modes were used to estimate the basic intervening variable model which were shown in Mod 1, Mod 2, and Mod 3.
In these equations, is the independent variable, is the dependent variable, and is the intervening variable. , , and are the population regression intercepts in (8), (9), and (10), respectively, represents the relation between the independent and dependent variables in (8), represents the relation between the independent and dependent variables adjusted for the effects of the intervening variable in (9), represents the relation between the independent and intervening variables in (10), and represents the relation between the intervening and the dependent variables adjusted for the effect of the independent variable in (9) [58]. According to Sobel’s [59] test, the mediation effect was investigated. The results of statistical analysis were shown in Tables 4 and 5.
For both the number of clicks and the task completion time, the directional similarity was a significant mediator. However, the effects are only partial mediation because the direct effect is still significant (Tables 4 and 5). The simplicity of information structures had significant effects on the task completion time and the number of clicks. A part of the effect was achieved by directional similarity, which means that user performance was partially influenced by the degree of match between users’ mental models and information structures.
Tables 6 and 7 indicate that the directionless similarity was not a mediator; there was no significant mediation effect for directionless similarity on task completion time or the number of clicks. One possible reason is that the directionless similarity was calculated from the card sorting, in which the details about the directional relationship of information structures were not considered, and this would lose the accuracy when using card sorting to elicit mental models.
Both the task completion time and the number of clicks showed partial mediation by the directional similarity. The information structures had a direct effect on user performance. The directionless similarity did not account for the relationship between information structures and user performance, while the directional similarity, as the new index to measure the degree of match between mental models, was a significant mediator. Thus, (H3b) was partially supported and (H3a) was rejected.
4.4. Discussion
Users’ activities provide objective information of web navigation behaviors and thus could complement users’ subjective understanding of web pages. The subjective understanding of web pages is usually elicited through card sorting. However, card sorting is not adequate for eliciting complete mental models if we consider the directional relationship of information structures. To get more objective information from users’ activities, we proposed a new method called path diagram to elicit mental models with directional information of hierarchical systems. To further quantify the difference between mental models, mental model similarity was calculated through the mathematical equations. It might be a quick and dirty way to predict user performance. In addition, designers can get more precise information about how users think about the system by applying path diagram into the two phases of interaction process: the designer-to-user communication phase and the user-system interaction phase [60].
The mental model similarity has two major theoretical and practical implications: (1) the mental model similarity provides an index to check if the designers’ improvement on their websites is effective, which is quite different from when designer can only check if their improvement is working by means of their feelings; (2) the mental model similarity also provides an index of measuring the usability of websites. People can get a more precise understanding of which kind of website was preferred by comparing mental model similarity of various websites.
Hypothesis (H1a) was rejected and Hypothesis (H1b) was supported. Many studies have shown that information structures are correlated with mental models. For example, Gregor and Dickinson [61] thought that good mental models could design good information structures. Roth et al. [62] pointed out that different information structures had different mental models. However, hardly any studies have investigated the relationship between information structures and mental model similarity between users and designers. Here, we have explored this relationship; the results showed that information structures had a significant effect on the directional similarity. The more complex the information structures, the lower the directional similarity.
The results also indicated that information structures had no significant effect on directionless similarity. Previous studies also indicated that card sorting lost its validity when dealing with the complex websites such as municipal websites which are complex information structures [43].
Hypothesis (H2a) and Hypothesis (H2b) were rejected. The results indicated that the directional similarity was positively correlated with the task completion time and the number of clicks. When the directional similarity was lower, the participants took less time and smaller number of clicks to find the target. This is different from the findings of Schmettow and Sommer [43]: they discovered that mental model similarity between users and designers had no effect on users’ browsing performance of municipal websites. The possible reasons for this may be as follows: (1) path diagram involves specific directional relationship between various elements of an information structure, while card sorting mainly represents a user’s understanding of the directionless relationship; (2) culture difference might influence the way users are navigating in websites. Chinese users will benefit from a thematically organized information structure of a GUI system, whereas American users will benefit from a functionally organized structure [63]. Specifically in the card sorting tasks, Chinese subjects were more likely to stress the category by identifying the relationship between different entities, while the Danish subjects preferred to stress the category name by its physical attributes [64]. The participants in the study of Schmettow and Sommer were from Netherland and the web pages used were functionally organized structure, while the participants of this study were Chinese and the websites used were thematically organized structure. Possible influence of cultural difference might be considered in future work. However, the navigation strategy is systematic, focused, and directed when individuals have specific targets or goals [65]. Card sorting cannot describe the users’ mental models completely when it relates to information structures with directional relationship. In other words, a better way to elicit mental models was using path diagram rather than card sorting. In addition, the method we proposed to elicit the mental model is predictive.
Demographic variables, such as age, spatial ability, memory capacity and technology product experience had no correlation with user performance. This is quite different to the findings of Arning and Ziefle [66], who indicated that user age and spatial ability were major factors affecting user performance. The possible reasons for this may be as follows: (1) participants in this study were all younger students, while participants in the study of Arning and Ziefle [66] were younger and older adults. (2) This study focused on web navigation on computers, while the study of Arning and Ziefle [66] focused on menu navigation on Personal Digital Assistants, whose small screen makes navigation more challenging and thus set higher requirements for spatial ability.
Hypothesis (H3b) was partially supported and Hypothesis (H3a) was rejected. The mediation effect of directionless similarity on user performance was not found, but a partially mediated effect was found between directional similarity and user performance. The results are different from those of Ziefle and Bay, who thought that the more similar the mental models between the user and designer, the better the performance using the device [33]. One possible reason is that the tasks in the study of Ziefle and Bay [33] were about browsing tasks, while they were searching tasks in this study. It is necessary to distinguish browsing without specific goals and searching specific goals in future studies. Anyway, results implied that designing hierarchical systems according to users’ mental models was not the only way to solve problems caused by the gap of mental models between users and designers. Alternatives such as providing navigation aids in complex websites and training may be considered [67].
This study only considered individuals, and the results may not apply to groups where interaction with other people influences constructions of mental models. Mathieu et al. [24] found team processes fully mediating the relationship between team mental models and team effectiveness. However, their subsequent study [39] indicated that team performance was partially mediated by teammates’ mental models.
In addition, the results showed that information structures had a direct effect on user performance. They seem to testify that “a meaningful information structure will promote efficient navigation, to ensure that information is organized in a way that is meaningful to its target users is essential when designing websites” [68].
5. Conclusion and Future Research
We have discussed the impact of the mental model similarity between users and designers. A new method, path diagram, was applied to elicit mental models and calculate the similarity and comparably tested to the traditional method.
Path diagram is more effective than card sorting in eliciting mental models of hierarchical systems, particularly considering the directional relationship of hierarchical systems. For general information structures, directionless similarity cannot predict user performance, while directional similarity can predict both task completion time and the number of clicks. Users will take less time and fewer clicks when the directional similarity is lower. However, for a specific information structure, neither the directionless similarity nor the directional similarity has a significant impact on user performance.
In addition, user performance is not affected by their age, spatial ability, memory capacity, or technology product experience. The results have also shown that it is more generally effective in eliciting the mental model using path diagram compared to card sorting.
Limitations of this study should be noted: (i) participants were sampling from young students, who were not representative. Future studies may consider older adults and those with lower education level; (ii) web pages with mixed information structures and various content were not considered; (iii) multiple tasks (e.g., browsing tasks) were not involved; (iv) possible impact of cultural difference was not examined; (v) changes of mental models were not tracked over time; (vi) similarity calculation required additional human efforts, so future work may explore ways to automatically calculate it.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grants nos. 71401018 and 71661167006 and Chongqing Municipal Natural Science Foundation under Grant no. cstc2016jcyjA0406.