Appraisals of Salient Visual Elements in Web Page Design
Visual elements in user interfaces elicit emotions in users and are, therefore, essential to users interacting with different software. Although there is research on the relationship between emotional experience and visual user interface design, the focus has been on the overall visual impression and not on visual elements. Additionally, often in a software development process, programming and general usability guidelines are considered as the most important parts of the process. Therefore, knowledge of programmers’ appraisals of visual elements can be utilized to understand the web page designs we interact with. In this study, appraisal theory of emotion is utilized to elaborate the relationship of emotional experience and visual elements from programmers’ perspective. Participants () used 3E-templates to express their visual and emotional experiences of web page designs. Content analysis of textual data illustrates how emotional experiences are elicited by salient visual elements. Eight hierarchical visual element categories were found and connected to various emotions, such as frustration, boredom, and calmness, via relational emotion themes. The emotional emphasis was on centered, symmetrical, and balanced composition, which was experienced as pleasant and calming. The results benefit user-centered visual interface design and researchers of visual aesthetics in human-computer interaction.
Visual design of user interfaces is significant to users interacting with different software. Through visual user interfaces, constructed of visual elements (i.e., color, size, and shape), interfaces communicate to the users and are expected to be both visually and emotionally appealing. However, visual user interface design is rather complex, because there is no universal formula of visual design to be applied in all user interface design contexts to elicit positive emotional experiences. Visual elements can be designed in countless different combinations and therefore, the same information content can be designed to appear in numerous visual forms which affect the users in different ways. For instance, different user interface genres, such as online banking web pages, are expected to visually represent the context in an appropriate way. If users’ expectations and visual design solutions are not considered in the design process, interaction can lead to negative emotional outcomes, such as frustration. Thus, numerous perspectives are required to be considered when designing visual user interfaces. One approach is to empirically examine how visual elements are emotionally experienced and how this knowledge informs user interface design decisions. This information is especially important when considering programmers’ appraisals of visual elements. This is because often in a software development process programming and general usability guidelines are considered as the most important parts of the process, while neglecting experiential aspects of the interaction conveyed with visual user interfaces.
Therefore, in addition to the traditional efforts in human-computer interaction, current and future user interfaces need to be developed in a novel way to match the potentials offered by the newest computing technologies and the users’ requirements based on their visual and emotional experiences. From a user-centered perspective, the rapid development of user interface technologies demands clarification of how visual elements should be utilized in user interface design to promote positive user experience. The improvement of visual aesthetics in user interface design enhances understandability of the product, by improving visual organization, clarity, and conciseness of user interfaces , and more profound understanding of how visual elements are experienced contributes to enhanced visual usability of user interfaces [2, 3], that is, how people represent visual elements and how the visual aspects of objects relate to the functions afforded by them. Moreover, in the current era of visual user interfaces, usable designs need to highlight aesthetic expression as meaningful presence for users instead of just providing designs as functional tools .
However, the snowballing research of user interface design has, until now, largely left aside the study of how visual elements in user interface design elicit emotional experiences. Current research of experiencing visual user interface designs, in the research area of visual aesthetics in human-computer interaction, has mainly focused on the overall impression of visual user interfaces (e.g., [5–7]), as a means of enhancing user experience with aesthetic pleasantness (e.g., ). In addition, more detailed approaches have focused, for example, on typography  and on high-level attributes (e.g., ). These high-level attributes include, for example, unity and prototypicality , novelty , and typicality and novelty . Therefore, research of visual aesthetics in human-computer interaction lacks knowledge of emotional responses in experiencing low-level attributes, that is, visual elements (e.g., ), such as color, size, and balance . In this paper, appraisal theory of emotion [16–18] is utilized to elaborate the relationship of emotional user experience and visual elements. Therefore, this study adopts an interactionist approach to human-technology experience; that is, it does not merely focus on either user interface design properties or users’ impressions and preferences of visual user interface designs, but analyses screen-based visual elements and their appraised dimensions together. Screen-design based approach focuses on detecting the visual properties of design components and their spatial organization in user interfaces , which affect user experience. How these identified visual elements are appraised is the key to understanding their role in human-computer interaction and in web page design.
According to Tractinsky , usability experts and designers have come to the conclusion that these two aspects of design, visual aesthetics and usability, could and should coexist in the same context of use. Before this shift in the first decade of the 21st century, visual aesthetics and usability were often seen as having a contradictory relation in that when one was emphasized, the other one was automatically omitted. The shift has emerged mostly because recent research corroborates a positive correlation between aesthetic and usability principles (e.g., [5, 6, 19–22]). Due to this shift, aesthetic qualities are currently emphasized in designing software for human-computer interaction. In addition, advancements in computer graphics have opened a variety of new design possibilities, which have increased the importance of aesthetic design of user interfaces providing emotional experiences. Yet, often in software development process, only the implementation of software (e.g., programming) and the traditional, general usability design  are considered as the most important parts . In reality, users’ experiential needs are not taken into account in the software development process . The essential deficit in user interface design lies in the absence of visual user interface design specialists in the process. Therefore, in this study, appraisals of salient visual elements are addressed from the point of view of future programmers and engineers. Focusing on programmers’ emotional experiences of visual elements in web page designs provides information on the visual elements that are considered essential and the way they are appraised. Understanding the appraisal process of the most salient visual elements among future programmers enriches the visual user interface design practice in software development and provides insights into visual design solutions applied in current user interface designs that we interact with.
The rest of the paper is organized as follows. First, the background concerning the visual elements and their capability in eliciting emotions in human-computer interaction is described with the appraisal theory of emotion. This is then concluded with presenting the research questions of the study. Second, the method is described including participant information, research procedure, stimuli, and data analysis. Third, the results are presented and discussed. Finally, the conclusions are presented with future research and limitations of the study.
2. Visual Elements and the Emotion Process
Establishing the connection between visual elements and the emotion response is one of the key elements in developing theory for describing visual elements in human-computer interaction. Visual elements in user interfaces are important factors in experiencing technological products due to their potential of eliciting attributed emotions and the ability to elicit actually felt emotions . Additionally, visual elements are capable of evoking strong emotional responses in technology interaction and thus affect the overall experience . While the psychological research on emotion has already posited the cognitive process, which connects an external event to emotional experience [16–18], this process has not yet received a satisfactorily discussion in user experience literature. However, research connecting the psychological theory of appraisal with emotional experience in human-technology interaction has started to produce promising examples concerning this [28–30].
Visual elements as the constructing units in user interfaces are the components through which the emotional process of experiencing visual user interfaces can be understood and approached in detail. By exploring users’ emotional experiences of visual elements, an in-depth view on visual aesthetics in human-computer interaction can be approached. Emotional aspects of visual user interface design have been studied in relation to trust. For instance, beauty of simplicity , visual appeal , and color appeal  have been stated to affect feelings of trust in interacting with user interfaces. However, more profound and detailed knowledge is needed in order to understand how and which visual elements in user interfaces elicit emotions and in which contexts.
In this study, visual representations that draw on the theories of two-dimensional pictorial elements are utilized as theoretical vehicles to create novel insight into experiencing visual user interface design. The dispensation of visual theory is in that it provides knowledge of exact visual elements, such as size, value, hue, orientation, contrast, texture , shape, proportion, and position  and, for instance, form and expression , which are seen as visual elements inherent in contemporary user interfaces. Mullet and Sano  refer to visual language in designing visual user interfaces. Through visual language, based on visual elements, user interfaces communicate to the users. Visual language of user interface design is divided into visual design factors which are visual characteristics (shape, color, position, texture, size, orientation, etc.) in a specific set of design elements (point, line, volume, plane, etc.) and the factors by which they relate to each other, such as balance, structure, proportion, and rhythm. These elements facilitate exploring the salient visual elements that draw attention to website design, elicit emotions in users, and affect the interaction between users and user interfaces. Further, visual theory serves as a starting point in the study because aesthetic impressions conveyed by visual representations influence users’ experiences of user interface qualities  that may promote positive feelings on user interfaces  due to the emotional nature of visual experience [37–39].
The cognitive process in which emotion episodes occur is called appraisal [16–18]. Appraisal is the evaluation of the personal significance of an event and its consequences and consists of multiple levels, such as motivational changes, physiological responses, and subjectively categorized emotions [18, 39]. There are various models of appraisal, which differ in the details of the emotion process, but most agree that the event is appraised at least from the following dimensions: implications to personal goals, pleasantness, causality, coping, and conformity [18, 28, 40]. Each is discussed here briefly, and their relation to visual user interface design is considered.
Events which satisfy one’s personal goals are usually appraised as positive, while goal-incongruent events are appraised as negative . The appraisal dimension of goal-relevance is understandably very salient in human-computer interaction research and design: usability problems, including visual usability problems, are usually taken to cause goal-incongruent events, which elicit negative emotions. Obstructions to efficient, goal-congruent use of systems result in frustration and anger, while successful mediation of the goal-oriented action results in feeling of competence [29, 30, 41]. However, pleasantness of an event is also appraised independently of the personal goals and current motivations, especially with the so-called aesthetic emotions, which can be contrasted with utilitarian emotions . This notion is central in user experience research, where the focus is more on the noninstrumental side of interaction . Positive experience of a user interface is not necessarily related to the immediate goals of the user.
The causality dimension of the appraisal process considers responsibility, that is, who caused the event and what was the motive behind these actions . The agency can be attributed to oneself, such as when a user feels competent after being successful in demanding tasks [29, 41], or to other actors, for example, when evaluating visual user interface design choices made by designers. It is also possible to attribute the cause of the event to nonhuman agent. An example of this appraisal would be the aggression towards an inanimate object, such as a computer, when the goals of the actor are obstructed .
The coping dimension refers to the potential of the subject to manage the event or the emotion which results from the event. Coping is an individual’s adaptation effort to events and as such is not associated only with negative events [44, 45]. Therefore, a response to a goal-incongruent computing event may result in either enthusiasm or anxiety, depending on how much the individual expects to be able to exert control over the event. There are two main coping strategies, problem-centered and emotion-centered. For example, in human-computer interaction, individual differences in how people are able to solve interaction problems and how well they are able to cope with their emotions play a significant role in the user experience process . The conformity dimension of appraisal refers to the compatibility of the event in relation to self-concept and social norms and values . Anger, for example, is the result of an event in which causality is attributed to someone who is appraised to act in a deliberate norm violation. In user interface design, violating the norms of established design practices may, therefore, cause negative emotions even if the norm-violating aspects do not cause goal-incongruent use events.
Of the all possible appraisal dimensions, perhaps the most frequently used scheme for describing emotion is the combination of valence and arousal . Valence refers to the pleasantness of the experience and arousal to how much the emotion is associated with activation. For example, feeling calm is pleasant but not an active emotion, while feeling energetic is a pleasant and active emotion. Sadness is a negative and deactivating emotion, and anger is a negative and arousing emotion. While these two are not the only relevant dimensions of emotional experience, they are often the most salient and can be used for rich descriptions of emotion in human-computer interaction . Regarding emotional experiences of pictorial representations, for instance, works of art can differ in their potential to cause arousal. Works of art which possess ability to evoke high arousal are most likely perceived as dramatic and dynamic, and works of art with low potential on eliciting arousal are generally perceived as static and harmonious .
This paper focuses on studying the most salient visual elements in user interfaces and their relation to emotions attributed and elicited by them from the perspective of future programmers, in order to provide usable insight for the evaluation and design of visual elements in user interfaces that promote user experience and thus to benefit user-centered visual user interface design. This study focuses on the following research questions: What are the most salient visual elements in web pages from programmers’ point of view? What kind of emotions do the salient visual elements elicit? How are the salient visual elements evaluated in the appraisal process?
Participants () expressed their impressions and emotions regarding visual website designs with 3E-templates. Of the participants, 17.5% were female and 82.5% male. The average age of the participants was 25.4 years (SD = 5.4 and range 20–46). The participants were allowed to return the templates anonymously without information that could be used to identify them, because the data collection was organized as a part of a university course. Thus, the reported age and gender information is from 40 participants, while ten participants answered without identification information. However, the average age of the 40 participants did not differ much from the average age of the participants in the course: the average age of the participants in the course was 25.0 years (SD = 5.8 and range 19–50).
All the participants were university students, mainly from the Faculty of Information Technology. 67.5% of the participants who returned the templates with identification information were students of computer science, 27.5% were students of information systems, and 5% of the students were from other majors. From the students in the course, 66.7% had a major in computer science, 26% were students of information systems, and 7.3% were from other majors. The circumstances of the data collection were designed similarly for all the respondents. The stimuli of the study and the data collection template were presented to the participants at the same time with the same advice in an auditorium, and the participants were allotted the same amount of time for answering the template.
3.2. Research Procedure
The data collection was conducted as a part of a University course about user-centered design. The data was collected from the lecture dealing with layout design. The lecture introduced the participants with relevant terms to describe the user interfaces. From the beginning of the lecture, the participants were familiarized with the design process in general, the evaluation methods of visual user interfaces, and visual elements, such as visual rhythm, dynamics, balance, tension, symmetry, contrast, Gestalt laws, and other various different design principles. People who are experienced in creating visual representations are more capable of analyzing them and more aware of their responses to visual phenomena . Therefore, the introduction of the visual design terminology was conducted to provide the participants a starting point for utilizing verbal vocabulary to express their visual experiences. The numerous terms were equally emphasized and the participants were not forbidden to use other terms to express their impressions. If the data would have been collected with questionnaires, with predetermined concepts of visual elements and emotion terms, the data would have been a result of a conception of the researchers’ understanding of emotions attributed to visual elements.
Therefore, the data was collected with the 3E- (expressing emotions and experiences) template [48, 49]. The 3E-template was selected as a data collection method to allow respondents to express their thoughts both verbally and nonverbally: by writing and drawing (Figure 1).
After the introductory part of the lecture, participants were introduced to the 3E-template and asked to write and draw their thoughts and impressions of two still pictures of web pages, with a focus on the compositional elements of the web pages. The participants were not aware during the introductory part of the lecture that they would need to report their impressions of user interface designs with visual design terminology. In the templates, participants did not describe their impressions only with the concepts presented in the lecture, and they did not mention some of the presented concepts at all.
The participants took approximately 20 minutes in answering the two templates. The research data is comprised of the written and drawn reflections and interpretations on the compositional aspects of the example web pages. A total of 100 templates were returned, as each participant evaluated two web pages. One template was returned without any written or drawn reflections and was therefore excluded from the analysis. The templates that comprise the data were mainly used by the participants to express their impressions of the UIs in written form. Examples of written expressions in the speech and think bubbles are presented in Figure 2.
The objects for the compositional reflection were two web pages (Figures 3 and 4) from the CSS Zen Garden web page gallery, web pages created with CSS-based design . All the gallery web pages have the exact same content but altering visual appearances. The web pages were selected to serve as stimuli, because the altering visual designs with the same content enable reflecting on the visual elements of the designs without concentrating on the content. In addition, the CSS Zen Garden web pages were selected as the stimuli in order not to bias the respondents with product expectations , such as brand experience.
The CSS Zen Garden web pages were surveyed for two example layouts that would differ from each other, especially with regard to the amount of elements that divide the surface such as lines, shapes, and the overall use of space. The web pages were first divided into two categories and then compared step by step, finally resulting in two example web pages.
Emphasis on a choice of web pages according to their differentiating elements was made in order to provoke participants towards a comparative analysis between web page layouts. Bell  also emphasizes that evaluations with content analysis are often comparative. The web pages were, therefore, selected with a comparative setting, regarding the pages’ differences in constructing visual elements. This was achieved in the study by using two different web pages with similar content but differing visual appearances. In addition, these two web pages were asked to be reflected in detail. Therefore, the two selected web pages were considered sufficient for the present study with a comparative setting, since all pictorial representations are constructed with visual elements. A broader sample of stimuli could be studied in future research. But first, it is essential to understand in detail which elements are considered important and what kind of emotions is attributed to them.
3.4. Data Analysis
When people experience pictorial representations, such as visual user interfaces, numerous visual elements are encountered . People may experience visual elements as both explicit and simple but also as elements involving interpretation. Therefore, the research method should facilitate the combination of the analysis of qualitative and quantified issues. As Collier  points out, a studied phenomenon must first be examined without premature analyses of the data, maintaining a focus on preexistent structures and points of interest. First, a data-driven content analysis was conducted to detect and describe the user interface elements depicted by the informants after having looked at the selected user interfaces. Second, the elements found and defined were quantified in order to identify the visual elements that are considered most salient in visual user interface design, particularly in layout design from the programmers’ perspective. The aim was to find the visual elements that have drawn the most attention and can be seen to have importance due their frequent emergence and significance of the content. The quantification of the explicitly written words representing specific objects is important, but it also relates to the qualitative procedures. Krippendorf  emphasizes the meaning of context in content analysis. Texts and images are always produced in some specific cultural context and they also refer to wider cultural context. This aspect was considered by first deploying the interpretative viewpoint as an independent phase before quantifying the found elements.
The methodological decisions for the study are influenced by the nature of the visual viewpoint: instead of directly analyzing visual user interfaces, the data is comprised of participants’ descriptions of the two example web pages. Therefore, many methods, for example, semiotics and iconography , as well as social semiotic visual analysis , are not applicable because they assume that the data along with the object of analysis are visual images. Furthermore, even though the data collected by other methods (such as eye tracking) could enable the extraction of specific points of attention, it would be impossible to analyze which particular element draws attention, color, form or, for instance, a visual tension between the elements. The analysis of the visual elements was conducted with these two procedures, which supported the analysis from two different viewpoints. The emotions elicited by salient visual elements were analyzed with same procedure.
The data analysis proceeded as follows. First, the data was observed as a whole by reading the templates. The purpose was to first focus on the visual elements in a neutral context in order to gain an understanding of the visual elements that are seen as important in visual user interface design despite the emotions they might evoke. The second phase was to create categories of relations and then to critically combine several different categories of relations between elements into main categories. For instance, from the example illustrated in Figure 2, the first observation about left side alignment is related to the spatial organization as is the second notion of diagonal lines. Stability refers to symmetry and the role of the arrow is related to perceived functionality. The human figure in the templates was only used in a few templates to illustrate emotions by drawing facial expressions. Almost none of the drawings emphasized the written content or the facial emotions of the figure but brought to attention something without a clear connection to the written content. Due to these characteristics of data, the focus of the analysis was on the written texts describing the visual elements and emotions attached to them. After the researchers had acquainted themselves with the data, an interpretation framework was developed and used to assist in the analysis. The interpretation framework included the items, which directed the focus on a conceptual level during the data analysis, in interpreting and comparing interesting insights within the data. The interpretation framework consisted of compositional interpretation  and visual elements in user interface design .
Compositional interpretation refers to describing the appearance of images with a detailed terminology. This form of visual analysis requires contextual knowledge of pictorial representations and a particular way of looking at images (“the good eye”), which is not methodologically explicit but functions as visual connoisseurship and is a specific way of describing images. Compositional interpretation focuses on the image itself by trying to comprehend its significance mostly by focusing on its compositionality. Interpretation does not focus on “external factors” such as the kind of messages the image sends and whether it has some functional meaning. The terminology of compositional interpretation includes several components. The first component is the content, that is, what the image actually shows. The second component is color, which is more specifically defined with concepts of hue, saturation value, and the harmony of color combinations. The third component is spatial organization, which includes volume, lines, static and dynamic rhythm, geometrical perspective, logic of figuration (how the elements of a picture offer particular viewing position outside the photo), and focalisers (the visual organization of looks and gazes inside the picture and in relation to the viewer’s gaze). The fourth is light: the type of light that is represented and its sources. The last component is expressive content, which describes the “feel” of the image combining the effect of the subject matter and visual form. Compositional interpretation approach is established in Art History and is usually used in studying paintings . Visual user interfaces can be comprehended as paintings or, in a more general viewpoint, as any two-dimensional pictorial representation that is constructed with visual elements, such as lines and shapes. Compositional interpretation can, therefore, also be extended to analyzing visual user interfaces.
Mullet and Sano  present visual user interface design factors, which are shape, color, position, texture, size, orientation, point, line, volume, balance, symmetry, scale, contrast, structure, proportion, rhythm, and position. The emphasis on visual factors in user interface design is to provide insight into designing good visual usability and effective visual communication. They also point out that negative space (i.e., empty or white space between visual elements and objects, opposite to active space) and grouping are important components in visual user interface design. The interpretation framework combined the discussed visual elements and guided the detection of the concepts related to visual elements in the coding phase of the analysis. The elements were derived from the two approaches described above and created the content of the interpretation framework. The interpretation framework included, but was not restricted to, the following components: color (hue, saturation, value, and harmony of color combinations), spatial organization, geometrical perspective, volume, lines, points, size, texture, shape, static and dynamic rhythm, orientation, balance, symmetry, scale, structure, proportion, negative space, grouping, position, figuration logic, focalisers, contrast, and light. From a compositional interpretation, the expressive content component was excluded from the interpretation framework in the analysis of salient visual elements, because of its emphasis on subject matter evoking emotionality which was not in focus in the first analysis phase of this study. The interpretation framework functioned as a “theoretical lens” in the analysis. The analysis required accuracy and concentration in detecting tiny nuances in finding the relations between the different visual elements, which required close attention in detecting how many altering ways there are, for instance, to describe the use of space in user interfaces. The goal was to create a model that illustrates the hierarchical order of visual elements. The structure of the model is data-driven but the logic is validated using theoretical framework of visual language.
The analysis of the emotions elicited by visual elements followed a procedure similar to the analysis of the salient visual elements. First, all templates were regarded as a whole with the focus on finding the emotions that had been expressed constantly and drawn most attention in relation to the salient visual elements. The construction of the interpretation framework for emotions was based on the appraisal theory of emotion, in which the appraisal of the subjective significance of an event results in subjective emotional experience. Overall, the participants used emotion words or categories clearly less often than visual elements: in total, 26 emotion words or categories in relation to visual elements were observed. Here, the interest was especially in the relationship between the visual elements of a user interface and the subjective emotional experience, that is, the feelings which arise as a response to visual elements. The interpretation framework for analyzing the emotion responses was derived from the appraisal dimensions described above. For the observed emotions, a relational theme which refers to the narrative explanation of the emotion was constructed . The narrative can be constructed with the appraisal dimensions. For example, frustration is an unpleasant (valence) and activating (arousal) emotion, but this is not enough to understand frustration. Frustration results when there is an obstruction preventing the subject from reaching her goals, and the subject still feels some power over the situation [18, 29]. Therefore, in order to understand an emotional response, such as when a user is frustrated at a computer program, a thematic explanation relating the goals of the user to the events of the use, as well as a reference to the coping possibilities of the user, is required .
A template was included into the emotion analysis if it contained a common emotion word that is included in lists of emotions, (e.g., ) or was an appraisal dimension, for example, pleasant . The following words relating to emotions were expressed: frustration, anxiety, calmness, apprehension, boredom, disgust, energetic, pleasant, disturbing, threatening, and confusing. From these, five thematic groups were created: frustration (including “frustration” and “disturbing” as the latter is part of the appraisal profile for frustration), calmness (which included “calmness,” “apprehension,” and “energetic” because they belong to the same appraisal dimension), “confusion,” “boredom,” and “pleasantness.” Other words were discarded from the analysis, because they were mentioned in connection with the visual elements, which were not analyzed here due to their low frequency. Each thematic group was given a relational theme and was then connected to the visual elements in order to establish the explanatory logic between the user interface design and the emotional response. Examples of expressed emotions in relation to visual elements are presented in Figure 5.
All the 100 templates were analyzed by writing down all the described elements in the templates and by counting the frequency of their occurrence. The number of times each visual element was mentioned in a template and the number of participants who at least once mentioned the visual element are presented in Table 1. The table illustrates the second level classification of visual elements. Different utterances were used to describe, for instance, guiding the gaze, such as “the picture directs the gaze to the left side menu,” “the gaze moves from up right to left down along with the title, and in addition, the gaze is guided with a separate arrow,” and “…directs the gaze outside of the user interface.” All these different notions were coded in the first phase of the analysis and were classified to the group of guiding the gaze in the second phase of the analysis. Visual elements that had been mentioned only three times or less were excluded from the analysis due to the low frequency.
The interpretation framework guided the detection of written reflections of the described elements. For instance, the interpretation framework did not include usability, which emerged from the data as a connective factor between relations of different elements, especially in terms of visual usability of the user interfaces. In many templates, think bubbles were used to express additional reflections, mostly about the supposed functionality of the user interface. The results of the study are data-driven; the most salient visual elements emerged from the data through content analysis, in which the theoretical framework was used to assist in detecting the utterances describing different elements. Therefore, the background theory of the visual language  and compositional interpretation  aided the analysis as a theoretical lens in detecting the visual elements as well as organizing the categories to hierarchical relations. For instance, the classification of the two main categories of visual elements, spatial organization and color and contrast, was conducted by reflecting on the background theory.
The salient visual elements are presented in Figure 6. The hierarchical structure of the results illustrates the different levels of visual elements from the lowest-level elements (e.g., diagonal lines) through the visual elements of organizing the lowest-level elements (e.g., alignment and asymmetry) to the main design dimensions (e.g., spatial organization), resulting as positioning of the viewer in appraising the most salient visual elements. Positioning of the viewer gathers all the results under one definition and functions as the main process outcome in experiencing visual user interfaces. Positioning of the viewer (including figuration logic and focalisers) was seen as a visual strategy that guides the viewer’s gaze in the user interface, which functions through visual elements guiding the gaze. Overall, positioning of the viewer refers to the user interface’s ability to communicate the whole content in a visual manner that is quick and easy to grasp.
Positioning the viewer was discussed in relation to visual elements contributing to spatial organization and color and contrast. Spatial organization and color and contrast were seen to apply both to communicability and to visual usability and interaction. Spatial organization and color and contrast are the main dimensions in visual user interface design language (e.g., ), which can be further described in detail with lower-level visual elements contributing to these higher level design dimensions. Spatial organization was emphasized by focusing on grouping and negative space. Imbalance, balance, asymmetry, and symmetry were seen as the primary visual elements affecting grouping and the use of negative space. In addition, grouping and the use of negative space through imbalance, balance, symmetry, and asymmetry were seen as contributing to the spatial organization of visual user interface elements in creating an impression of the user interface as a consistent totality, in which visual elements contribute to the impression of the user interface as a whole. Grouping of similar contents and functions were seen as substantive factors creating clarity. Spatial organization through balance and symmetry was seen as important regarding the overall use of space. Grouping and negative space were further discussed in terms of aligning or centering the content. In the data, spatial organization was reflected in detail and visual elements referring to this category were often mentioned.
Grouping with similar content and functions and negative space were seen in relation to the composition of the user interface in general and impression of overall use of space. Observations of balance and imbalance were seen in relation to symmetry and asymmetry. Alignment to the left or right and centering the content were seen to affect impressions of balance, imbalance, symmetry, and asymmetry. An important relation was often found between diagonal, horizontal, and vertical lines in creating impression of overall use of space and guiding the gaze. Diagonal, horizontal, and vertical lines were especially emphasized and seen as visual elements that guide the gaze forward. A strong emphasis was placed on describing how the visual elements function and how they support visual usability and interaction, especially the ways by which visual elements direct attention in the user interface towards the most important areas.
Even though the participants were instructed to only use the template for reflecting the impressions of the compositional elements of the layout, attention was paid towards contrasting colors and their role in visual usability. In relation to contrast, most of the remarks were about color contrasts, especially between the colors of the texts and background colors, which were, moreover, attached to legibility. Size was only discussed in the context of the font size.
The relations of the five thematic groups to the salient visual elements are presented in Figure 7. The figure is based on the first phase of the analysis concerning salient visual elements. The structure of the figure has not been affected by the second part of the analysis due to the order of the analysis phases. The salient visual elements in all categories elicited and were attributed with emotion responses. However, emotions with negative valence were expressed more frequently than positive emotions, and emotions with higher activation, such as frustration, were more frequently expressed than emotions with lower activation, such as boredom.
Frustration was reflected regarding legibility, spatial organization, imbalance, asymmetry, diagonal lines, color and contrast, background picture, and font. Emotional utterances relating to calmness or unease were frequently expressed in reference to spatial organization, grouping, negative space, horizontal and diagonal lines, color, and user interface as a whole. Confusion was reported in relation to how colors in the user interface design were experienced. Boredom was connected to spatial organization, color, and background color. Pleasantness was reflected regarding the user interface as a whole, spatial organization, and centering the content.
Positioning of the viewer functions as the main appraisal process outcome, in which the interactionist view combining the salient visual elements’ ability to guide the interaction and the appraisals of these events is experienced. Positioning of the viewer functions through evaluations and experiences of communicability and visual usability. Visual usability includes the ways of how different visual elements guide the interaction and usability. User interfaces that fluently communicate the content in an understandable visual form are appreciated. Clarity, legibility, impression of the user interface as a whole, and understandability were considered as the most important factors in contributing to communicability and visual usability of the user interface. Thus, in designing visual user interfaces, the power of the visual elements to guide users’ attention needs to be taken into account. Besides visual usability, the communicative ability of the visual elements , in relation to the content suitability of the visual user interface design, affects positioning of the viewer. The overall visual impression of the user interface’s appearance should be suitable for the context for which it is designed.
Alignment and centering the content were considered as constitutive factors influencing grouping, balance, and symmetry. Diagonal, horizontal, and vertical lines were highly emphasized as visual elements that strongly affect the positioning of the viewer and lead the interaction as the main low-level elements. In addition to spatial organization, color and contrast were seen as influential design dimensions. Functional and communicative impressions of the visual elements were implicated in terms of understandability and legibility of the content. Contrast between figures, fonts, and background picture and color was conveyed through different font sizes and color combinations. These ways of creating contrast between visual elements in user interfaces were often expressed, especially the contrast between background color and the size and the font colors. Designing contrast between these elements contributes to efficient visual usability, which promotes fluent user experience in interacting with visual user interfaces.
The visual elements are appraised with attributed potential for creating different emotions and eliciting emotional states. Frustration was expressed in connection with legibility, spatial organization, imbalance, asymmetry, diagonal lines, color and contrast, background picture, and font. Frustration is the response to a goal-incongruent event, that is, an event that frustrates, disturbs, or obstructs the subject [41, 43]. In particular, if the disturbing event is appraised as goal-incongruent and unfair and resulting from a deliberate norm violation, the feeling of frustration is strong and may result in anger or even aggression [18, 43]. When interacting with computers, the user may often appraise that an obstruction to her goals is the result of bad user interface design, which results in frustration response with an implicit causal attribution: “It’s disturbing that [the UI] is not fitted to the browser. This creates an impression of imbalance.” Likewise, poor readability of font obstructs grasping the user interface and frustrates the user, as expressed by one participant: “The composition is frustrating, grey fonts disappear to the background.”
Emotion words relating to calmness or unease were common in the data and referred to spatial organization, grouping, negative space, horizontal/diagonal lines, color, and user interface as a whole. Being calm (or, on the opposite, apprehended) is dependent on the appraisal of the amount of control and power that the subject has over the event [18, 40]. The appraisal process evaluates certain visual elements in relation to the coping dimension, which results in feeling of control or loss of control. If the user of a user interface feels that she is not in control of the situation, she becomes uneasy, and if the situation is incongruent to the goals of the user and she would need to exert control to remedy this, the user may even feel anxious or threatened . Participants emphasized calm, centered, and balanced composition: “Only aligned to the left side. Diagonal shapeot very calm.” Diagonal lines were considered apprehensive and horizontal calming: “Diagonal lines cause apprehension.” and “Text boxes […] are horizontal, reading them is calming.”
It is noteworthy that not all visual elements that are related to coping and feeling of control are necessarily related to the actual control functions of the system; that is, they do not signify affordances, but they are nevertheless associated with coping. This means that the appraisal process integrates low-level visual elements into a complete experience of the user interface, and some connections between the emotional experience and the details of the interface may not be explicitly clear. Further exploration of this issue could, for example, reveal that shapes and lines are associated with emotional coping, whereas the problem-focused coping would be connected to different visual elements.
Confusion was connected to colors in the participants’ responses. Confusion is related to not being in control of the situation and is, therefore, closely related to both frustration and apprehension . Avoiding confusion is critical in the context of human-computer interaction, where the user interface does not necessary reveal the inner logic of the software. Often, the user has norm-based expectations concerning the composition and functionality of the user interface, and deviations from this norm are confusing. However, conventional design is not necessarily good design. The explication of the connection between visual elements and emotional responses, such as confusion, should help the formulation of research-based design principles.
Boredom was expressed in relation to spatial organization, color, and background color. Boredom is an unpleasant feeling, which is associated with low effort and decreased attentional activity . It is distinguished from other emotions in that a person feeling bored ignores what is happening; this often happens in a situation where the situation itself is uninteresting and there is nothing to occupy and stimulate the subject. If the composition of the user interface, for example, is not judged as interesting, the user may feel bored: “The page is just a layer of things. Boring.”
Pleasantness was related to the user interface as a whole, spatial organization, and centering the content. As discussed above, pleasantness-related appraisals have two dimensions: satisfaction of goals and intrinsic pleasantness . Pleasantness associated with the former dimension reflects the visual usability of the user interface, relating to goal-congruence in possible use cases, and the latter dimension is associated with subjective aesthetic preferences, relating to the intrinsic pleasantness appraisal dimension: “[…] More pleasant to watch, and probably also to use. Everything is nicely centered.”
In general, negative emotions were used more frequently when evaluating the user interfaces. This observation is explainable by hedonic asymmetry, which means that adaptation to pleasant emotions happens more quickly than adaptation to unpleasant ones . The participants were, therefore, hastier to report negative than positive emotions. Although negative emotions were more salient in the participants’ appraisals, there were also salient visual elements that elicited positive emotions. These were restricted to the spatial organization elements. Color and contrast did not provoke positive emotions. In addition to imbalance with valence, emotions with higher activation in the arousal dimension, such as frustration, were more frequently expressed than emotions with lower activation, such as boredom. It seems that the threshold for expressing emotions with higher activation is lower than expressing emotions with lower activation. However, unlike with valance, the arousal dimension was not observed to be distributed unevenly between different visual elements.
While all of the salient visual element categories were associated with some emotion words or dimensions, spatial organization clearly evoked emotions most frequently and the expressed emotions were also the most diverse. This is understandable as spatial organization is an umbrella term for lower-level elements constructing the user interface composition, and the visual elements belonging to spatial organization were the most frequently observed content in the analysis of salient visual elements.
Grouping with similar contents and functions was reflected to be calming and contributing to grasping the user interface as a whole and to understandability. Also, appropriate amount of negative space was experienced as calming. Both observations are in line with the narrative of calmness, especially, as understood with the help of the coping dimension of appraisal: the listed visual elements helped the participants maintain overall control over the user interfaces, which was reflected as calmness. Imbalance and asymmetry were experienced as frustrating. On the contrary, balance and symmetry were connected to pleasantness. In addition, centering the content was considered pleasant. As the lowest-level visual elements, diagonal lines were emphasized in general and in relation to emotions evoked by them. Diagonal lines were seen as important as constructing elements of the overall impression of the layout. However, the emotions attributed to diagonal lines were controversial. They were experienced as frustrating and apprehensive but also as contributing to elegant impression with dynamic tension.
Color and contrast were discussed frequently, but they were seldom connected to emotions. When color and contrast were discussed in relation to emotions, the connotation was negative (frustration, confusion, and boredom). This is explicable with color’s essential role among visual elements. Color is considered as a primary visual element evoking aesthetic preferences  and strongly influencing legibility and content understandability.
The focus of the study was on future programmers and engineers appraisals of salient visual elements. The results of this study indicate which visual elements are considered as the most salient ones in web page design that contribute to communicability and visual usability and what kind of emotions they elicit or were attributed with. The appraisal process integrates the salient visual elements into an emotional experience. This results in positioning of the viewer, in which the user’s experience of the user interface can be explained in terms of how the subjective relevance of the user interface is appraised. This connects salient visual elements with the pleasantness or unpleasantness of the experience, as well as to how well the user is able to cope during the interaction.
Spatial organization and color and contrast, with the lower-level elements which they are based on, such as diagonal lines and colors contrasts, are essential to users in web page designs that promote fluent interaction and visual usability. Characteristics of lines have an important role in emphasizing visual usability and interaction with user interfaces. Spatial organization through grouping with similar content and functions, appropriate use of negative space, and balance and symmetry contribute to the overall impression of the user interface as a clear and understandable totality. Contrast between figures, texts, and backgrounds enhances visual usability and interaction with the user interface. In addition, through contrast, user interfaces can be designed to communicate the content in an effortless manner by grasping the content with one glance. Communicability of the user interface also refers to the content’s suitability in relation to the context. Without contrast between visual elements, user interfaces could not communicate the content to the users interacting with them. This notion emphasizes the essence of visual elements contributing to fluent human-computer interaction. User interfaces’ ability to position the viewer through visual elements and emotions elicited by them is essential in interacting with different user interfaces.
The emotional responses to user interfaces were analyzed with relational themes created from appraisal dimensions [18, 40]. Connecting these themes to the salient visual elements created a coherent narrative of the process in which emotional experience occurs as a response to visual elements in web page design. Using this framework, future studies should focus on detailing the preliminary findings made here. For example, are all typical emotions encountered in human-computer interaction observed in a connection to all of the salient visual elements so that only the theme of the emotion is different between them? These findings are critical in design, which sets certain experiences as design goals and operationalizes concepts from the explanations which connect designable elements with the user’s emotions.
Overall, the emotional emphasis was on centered, symmetrical, and balanced composition, which was experienced as pleasant and calming. Diagonal lines received controversial emotional expressions, as frustrating salient visual element, but also an element which creates a subtle dynamic tension. According to the results, it can be stated that if the goal of the design, from a programmers’ perspective, is a pleasurable and controllable experience, the overall composition needs to provide an impression of static balance and symmetry by grouping, centering content, and appropriate use of negative space. This finding emphasizes how attaining feeling of control, another important design goal, is attained by focusing on these elements . Designing with contrasting colors and diagonal lines needs to be considered in relation to the content the user interface is designed to communicate and to whom.
The salient visual elements experienced as pleasant and calming are appraisals of programming students. Visual elements in web page design might gain differing appraisals in different user groups. For example, coping with more various and dynamic combinations of visual elements can be influenced by the level of design expertise and visual literacy abilities . Therefore, future research focuses on resolving differences between different user groups with varying design backgrounds. For more culturally varied results, future research should also be conducted with participants of different backgrounds: information on the cultural variation in the importance of different visual elements would be significant in today’s user interface design.
Proceeding from specific contexts towards the discussion of a more general understanding of the studied phenomena is another suggested line for future research. For example, color’s powerful position among other visual elements raises questions of its status in emotional interaction. Participants were asked to express their impressions regarding composition and layout design of the web pages, yet color design was frequently discussed in the templates. Therefore, future research should focus on studying the impact of color in user interface design and the emotions elicited by them in order to provide more profound insights into visual elements eliciting emotions in user interface design. Here, the color elements were mostly associated with frustration, which is connected to the goals of the user , but the relation of such appraisal dimensions as intrinsic pleasantness should also be investigated. In addition, the emotions triggered by visual elements could be approached in studying symbolist and abstract characteristics attached to the elements and how these meaning making processes influence emotional responses.
In addition, the validity of 3E-templates should be tested by triangulating data with various methods, such as emotion questionnaires, and with various stimuli. It is possible, for example, that with open-ended questions, not all emotions actually elicited by the user interfaces were recorded. Standardized emotion questionnaire would force the participants to rate their emotional experiences, and these results could then be correlated with similar ratings of the visual elements. Moreover, visual design of user interfaces is not universally understood and appreciated similarly in different cultures. Different meanings attached to, for instance, dimensions of visual space are highly influenced by the writing and reading direction . Therefore, due to the context of this study, the results can be applied to Western culture. In addition, this study was conducted in relation to visual web site design and, therefore, might not be applicable to other visual user interface design contexts, such as mobile user interface design. Visual elements as the construction elements of visual mobile user interfaces might be appraised differently, for instance, due to the size and shape of the screen.
The authors declare that they have no competing interests.
The research has been supported by COMAS (Doctoral School of the Faculty of Information Technology, University of Jyväskylä).
K. Mullet and D. Sano, Designing Visual Interfaces: Communication Oriented Techniques, Prentice Hall, Upper Saddle River, NJ, USA, 1995.
T. Schlatter and D. Levinson, Visual Usability, Principles and Practices for Designing Digital Applications, Morgan Kaufmann, Boston, Mass, USA, 2013.
J. Silvennoinen, M. Vogel, and S. Kujala, “Experiencing visual usability and aesthetics in two mobile application contexts,” Journal of Usability Studies, vol. 10, no. 1, pp. 46–62, 2014.View at: Google Scholar
N. Tractinsky, “Visual aesthetics: in human-computer interaction and interaction design,” in Encyclopedia of Human-Computer Interaction, M. Soegaard and D. Rikke Friis, Eds., The Interaction Design Foundation, Aarhus, Denmark, 2012, http://www.interaction-design.org/encyclopedia/visual_aesthetics.html.View at: Google Scholar
W.-K. Hung and L.-L. Chen, “Effects of novelty and its dimensions on aesthetic preference in product design,” International Journal of Design, vol. 6, no. 2, pp. 81–90, 2012.View at: Google Scholar
J. Silvennoinen, “Towards essential visual variables in user interface design,” in Proceedings of the 7th International Conference on Advances in Human-Computer Interactions, pp. 229–234, Barcelona, Spain, March 2014.View at: Google Scholar
R. S. Lazarus, “Relational meaning and discrete emotions,” in Appraisal Processes in Emotion, K. L. Scherer, A. Schorr, and T. Johnstone, Eds., pp. 37–67, Oxford University Press, New York, NY, USA, 2001.View at: Google Scholar
J. Nielsen, Usability Engineering, Morgan Kaufman, San Francisco, Calif, USA, 1993.
D. Cyr, “Emotion and website design,” in Encyclopedia of Human-Computer Interaction, M. Soegaard and D. Rikke Friis, Eds., The Interaction-Design Foundation, Aarhus, Denmark, 2013, http://www.interaction-design.org/encyclopedia/emotion_and_website_design.html.View at: Google Scholar
E. Demir, P. M. A. Desmet, and P. Hekkert, “Appraisal patterns of emotions in human-product interaction,” International Journal of Design, vol. 3, no. 2, pp. 41–51, 2009.View at: Google Scholar
K. Karvonen, “The beauty of simplicity,” in Proceedings of the ACM Conference on Universal Usability (CUU '00), pp. 85–90, Washington, DC, USA, November 2000.View at: Google Scholar
M. Wiberg and E. Robles, “Computational compositions: aesthetics, materials, and interaction design,” International Journal of Design, vol. 4, no. 2, pp. 65–76, 2010.View at: Google Scholar
L. Hallnäs, “On the foundations of interaction design aesthetics: revisiting the notions of form and expression,” International Journal of Design, vol. 5, no. 1, pp. 73–84, 2011.View at: Google Scholar
D. Norman, Emotional Design, Basic Books, New York, NY, USA, 2004.
J. Dewey, Art as Experience, Capricorn Books, New York, NY, USA, 1958.
P. Hekkert, “Design aesthetics: principles of pleasure in design,” Psychology Science, vol. 48, no. 2, pp. 157–172, 2006.View at: Google Scholar
R. S. Lazarus and B. N. Lazarus, Passion and Reason, Oxford University Press, New York, NY, USA, 1994.
D. E. Berlyne, Aesthetics and Psychobiology, Appleton-Century-Crofts, New York, NY, USA, 1971.
D. Curtiss, Introduction to Visual Literacy: A Guide to the Visual Arts and Communication, Prentice Hall, Englewood Cliffs, NJ, USA, 1987.
M. Tähti and L. Arhippainen, “A proposal of collecting emotions and experiences,” in Interactive Experiences in HCI, vol. 2, pp. 195–198, 2004.View at: Google Scholar
M. Tähti and M. Niemelä, “3E—expressing emotions and experiences,” in Proceedings of the WP9 Workshop on Innovative Approaches for Evaluating Affective Systems. HUMAINE (Human-Machine Interaction Network on Emotion), Stockholm, Sweden, February 2006.View at: Google Scholar
CSS Zen Garden Gallery, http://www.csszengarden.com.
P. Bell, “Content analysis of visual images,” in Handbook of Visual Analysis, T. van Leeuwen and C. Jewitt, Eds., Sage, London, UK, 4th edition, 2004.View at: Google Scholar
H. Zettl, Sight, Sound, Motion: Applied Media Aesthetics, Wadsworth Publishing Company, Boston, Mass, USA, 1999.
M. Collier, “Approaches to analysis in visual anthropology,” in Handbook of Visual Analysis, T. van Leeuwen and C. Jewitt, Eds., Sage, London, UK, 4th edition, 2004.View at: Google Scholar
K. Krippendorf, Content Analysis: An Introduction to Its Methodologies, Sage, London, UK, 1980.
T. van Leeuwen, “Semiotics and iconography,” in Handbook of Visual Analysis, T. van Leeuwen and C. Jewitt, Eds., Sage, London, UK, 4th edition, 2004.View at: Google Scholar
C. Jewitt and R. Oyama, “Visual meaning: a social semiotic approach,” in Handbook of Visual Analysis, T. van Leeuwen and C. Jewitt, Eds., Sage, London, UK, 4th edition, 2004.View at: Google Scholar
G. Rose, Visual Methodologies: An Introduction to the Interpretation of Visual Materials, Sage, London, UK, 2005.
R. Poulin, The Language of Graphic Design: An Illustrated Handbook of Understanding Fundamental Design Principles, Rockport Publishers, Beverly, Mass, USA, 2011.
T. van Leeuwen and G. Kress, Reading Images. The Grammar of Visual Design, Taylor & Francis, 2nd edition, 2006.