Abstract
In human-computer interaction, the visual interaction of user experience (UX) and user interface (UI) plays an important role in enriching the quality of daily life. The purpose of our study analyzes the use of brain-computer interface (BCI), wearable technology, and functional magnetic resonance imaging (fMRI) to explore the aesthetic processing of visual neural response to UI and UX designs. Specifically, this review aims to understand neuroaesthetic processing knowledge, aesthetic appreciation models, and the ways in which visual brain studies can improve the quality of current and future UI and UX designs. Recent research has found that subjective evaluations of aesthetic appreciation produce different results for objective evaluations of brain research analysis. We applied SWOT analysis and examined the advantages and disadvantages of both evaluation methods. Furthermore, we conducted a traditional literature review on topics pertaining to the use of aesthetic processing knowledge in the visual interaction field in terms of art therapy, information visualization, website or mobile applications, and other interactive platforms. Our main research findings from current studies have helped and motivated researchers and designers to use convincing scientific knowledge of brain event-related potential, electroencephalography, and fMRI to understand aesthetic judgment. The key trend finds that many designers, artists, and engineers use artistic BCI technology in the visual interaction experience. Herein, the scientific methods applied in the aesthetic appreciation to human-computer interface are summarized, and the influence of the latest wearable brain technology on visual interaction design is discussed. Furthermore, current possible research entry points for aesthetics, usability, and creativity in UI and UX designs are explicated. The study results have implications for the visual user experience research domain as well as for interaction industries, which produce interactive projects to improve people’s daily lives.
1. Introduction
A part of human nature is liking for beautiful things [1]. This idea covers a wide range of disciplines, including human-computer interaction (HCI), which includes hardware and software design and development. HCI is concerned with the design, evaluation, and implementation of interaction and computing systems for human use and the study of the major phenomena surrounding them, such as behavior and psychology demands [2]. Human-computer interface design is more specific, is considered the contact interface between the application and the end user ([3], p.1402), and concerns the user interface.
The main parts of HCI or interface design are the user experience (UX) and user interface (UI) designs, which are related to the neuroaesthetic and aesthetic processing research. With the development of technology innovation globally, people have started to pay attention to user experience in enriching their quality of life. UX [4] is defined as “a person’s perceptions and responses resulting from the use or anticipated use of a product, system, or service.” Usually, UX is correlated with human emotion, perception, preferences, reflections, feelings, and psychological variations.
The motivation of this study is to help designers and researchers use brain science technology to understand aesthetic judgment from visual interaction projects. This review article aims to provide an overview of design challenges and their solutions and of problems of discussing subjective aesthetic judgments in the daily design process and visual interaction markets. Moreover, this review summarizes the objective methods of measuring aesthetic appreciation and methods of solving scientific aesthetic judgment problems in visual user experience research. Therefore, the present review aims to analyze human neuroaesthetic processing of UX and UI designs and explore these developments in the visual interaction field.
There are some differences between UX and UI designs, and UI is one of the interaction factors of UX; UX focuses on the entire experience concept or user journey towards a project, whereas UI is related to the visual format, such as animation and color selection [5]. For example, creative artistic therapy brain painting is a type of UX journey [6], but website interface aesthetic design is a type of UI design. The scope of the UX in the present study is focused on visual interaction experience. It covers many fields, such as virtual reality, augmented reality, art therapy, education, game design, public art, UI design, and information visualization. These fields strongly rely on visual content, and aesthetic processing and neuroaesthetics play essential roles here. Furthermore, neuroaesthetics of UX is broader than that of UI; it also involves spatial and temporal effects, environment, virtual space, and social interaction, and it covers more about rhetorical beauty than early first impression visions.
Visual interaction is a critical part of UX, and therefore, neuroaesthetics can contribute to the development of current UX biofeedback. A past brain research review reported that 40% of stimuli concern visual content, 43% of stimuli combine visual and audio, 15% of stimuli contain audio content, and 2% contain others [7]. Neuroaesthetics has been described as a study investigating the neural substrates of aesthetic appreciation [8], and it comprises three aspects: “cognitive neuroscience of aesthetics,” “cognitive neuroscience of art,” and “cognitive neuroscience of beauty.” However, the sense of “beauty” or “aesthetic” is primarily subjective and is a part of neuroaesthetics. As a result, more researchers have started to study neuroaesthetics and analyze the possible methods to determine human aesthetic processing when perceiving everyday interactive things. To test visual biofeedback on beauty, visual stimuli are the first possible method in neuroscience to study the aesthetic feeling biofeedback through event-related potential analysis. Visual stimuli are regarded as the picture or color stimulus presented on a screen, which usually is used to evoke a visual evoked potential [9], containing static and dynamic stimuli. Many researchers utilize visual stimuli to test participants’ direct brain activity: for example, P100 is evoked with high color contrast [10] and P200 is sensitive to ugly images [11]. The second method uses time-frequency and topography analysis to study the activation area using electrodes when users observe visual elements. Usually, there are five bands, alpha, beta, theta, gamma, and delta; lower theta is associated with better UX design, and alpha is active with beautiful images [12]. These experimental methods can be implemented using brain-computer interfaces (BCIs), which are communication systems where people interact with external devices solely using brain activity [13]. The third professional method is to use functional magnetic resonance imaging (fMRI) to study brain activation when looking at beautiful objects; the bilateral insular cortex shows a strong response in facially and morally ugly conditions [14]. Event-related potential, time-frequency, and topography methods have been focused on evaluating visual aesthetic processing and can contribute to UX and UI, as discussed in this review article.
The main significance and contributions of this study are as follows:(i)Summary of the scientific aesthetic judgment through event-related potential, time-frequency, topography, and fMRI technology(ii)Finding the potential interactive methods to stimulate visual nerve and enhance the quality of user experience through artistic BCI(iii)Understanding and mastery of the brain data analysis and technology of aesthetic judgment on visual interaction projects(iv)Description of the contributions of aesthetic appreciation models related to the visual UX and UI designs(v)Discussion of the advantages and disadvantages, limitations, and SWOT (strengths, weaknesses, opportunities, and threats) analysis of subjective and objective evaluations on visual interaction projects
The novelty of this review is that it makes the connection between neuroaesthetics, user experience design, and visual interaction develop the quality of future interactive projects through scientific aesthetic appreciation approaches. Therefore, the present review has two main research questions. What aspects of aesthetic processing affect user interaction and user interface design? How do current researchers use noninvasive brain neurophysiology technology to understand visual biofeedback on UX and UI designs? The following sections explore the research review around these research questions: Section 2 describes the methods for finding relevant papers on aesthetic processing with UI and UX designs. Section 2.8.3 describes two neuroaesthetic models, top-down and bottom-up analysis. Section 3 discusses the comprehensive current studies on the uses of event-related potential (ERP), electroencephalograph (EEG), and fMRI for brain region analysis and the data that can aid in understanding the current aesthetic processing of UX and UI designs and in forecasting future developments. Section 3.11 summarizes all related research papers that developed UX and UI designs, including BCI paradigms, descriptions, participant information, and apparatus. It also involves the SWOT analysis. Section 4 discusses the advantages and disadvantages, literature comparisons with previous studies, and appropriate usage of brain technology in the visual interaction field. Section 5 contains conclusions.
2. Methodology
The research method of this review is similar to the traditional literature review process. Although the systematic literature review is well defined, it was not used in our research because systematic literature reviews rely on a clear purpose and objective at the beginning of the research [15]. For this review, we read many articles and explored the relationship between brain aesthetic appreciation and human-computer interaction, which requires comprehensive understanding and wide reading knowledge to determine a meaningful topic. The traditional literature review process is creative and appropriate to explore a new topic in a review study and is suitable for a study that has not been strictly defined [16]. After wide reading, we used XMind software to draw a mind mapping figure and find the relationship within the research materials. However, future researchers can use a systematic literature review to explore higher research quality based on this review’s research findings.
The electronic search for research material in this state-of-the-art review was conducted as follows: First, we typed the keywords to collect papers from the following search engines [17]: IEEE Explore, Springer, Elsevier, Frontiers, Taylor & Francis Online, Hindawi, Google Scholar, and ScienceDirect. The keywords used were “neuroaesthetic”/“aesthetic processing”/“brain aesthetic signal processing” + “visual interaction”/“human–computer interaction”/“human–computer interface”/“user experience design”/“user interface design” + “EEG or ERP or fMRI.” The publication year range selected was 2016 to 2021. Second, we also used the same keywords to search several target and top journals: Nature Publishing Group, ACM Transactions on Computer-Human Interaction, Computational Intelligence and Neuroscience, and Interactional Journal of Human-Computer Studies/Interaction. Third, we also searched the References section of retrieved papers because some useful journal articles had cited similar or important papers on visual interaction and aesthetic judgment [16]. The overall literature review research progress is shown in Figure 1. Some representative papers related to aesthetic processing, aesthetic appreciation, and aesthetic judgments published before 2016 were considered, such as those on the topics of classical aesthetic models and neuroaesthetic theories.
2.1. Aesthetic Processing in Neuroaesthetics towards UX and UI Designs
The present review article discusses the two most related visual aesthetics models: Chatterjee’s model of the neural underpinnings of visual aesthetics and the model of aesthetic appreciation and aesthetic judgment [18]. These two models are the most suitable and related to aesthetic processing on UX design and clearly show the initial, the intermediate, and the overall aesthetic process for designing visual interaction projects.
2.1.1. Model 1: Chatterjee’s Model of the Neural Underpinnings of Visual Aesthetics
The first model considers the early vision aspect and concerns the first impression of the UX design. The visual interface activates visual neurons from the frontal lobe to the occipital lobe. The aesthetic experience of a visual artwork begins with a visual analysis of the stimulus, which then undergoes further levels of processing [19]. Meanwhile, attention is normally connected to the frontal lobe associated with cognition, decision-making, and working memory of UX design. In contrast, visual attention has applied visual processing differentially to a focused subset of a scene [20]. It contains overt (gaze attention) and covert visual attention. Visual and cognitive attention is in charge of the complete visual process in Chatterjee’s model. Chapman also defined the intermediate vision as “the mechanisms that connect bottom-up early vision with later, task-specific processing.” Intermediate visual reduced retinotopic maps to compact encodings. The intermediate vision can group the selected task-specific early vision components to reflect some preferred representational domain, such as UI, to decide on liking for or wanting of the interface design. The first model is related to UI design, and it concerns early and intermediate vision on aesthetic interfaces and makes a fast reflection of judgment of liking for or wanting of the UI design. Experts and laypersons always consider the following initial visual design principles [21]: affordance, consistency, feedback, visibility, constraints, and mapping. The model also considers the primary step of emotion reflection, but emotional EEG feedback is out of the scope of the present review.
2.1.2. Model 2: Model of Aesthetic Appreciation and Aesthetic Judgment
The second aesthetic appreciation model is well known and contains five main steps of aesthetic processing: perception, implicit memory integration, explicit classification, cognitive mastering, and evaluation [22]. The model also includes supplementary factors influencing aesthetic appreciation, such as environment and social interaction. The second model is more concerned with UX because it relates to space or environment, experience consideration, interactive fields, and so on. It includes the overall aesthetic experience of projects. It entails continuous evaluation of the UX design. Figure 2 summarizes the UX components related to the aesthetic appreciation model (Model 2) [18, 22].
2.2. Perception
Visual perception is concerned with the neuro-biofeedback elicited by certain design products, artworks, or art-like stimuli. Visual perception is vivid and full of detailed information, making visual stimuli descriptive, convincing, and persuasive [23]. In the model of aesthetic experience, perception is typically concerned with size, intensity, color stimuli, brightness, and saturation [22]. The initial visual selection follows the salience of objects in the visual field, which can be regarded as a bottom-up approach [24]. Pearson [23] also depicted the bottom-up visual perception as primarily influenced by the eye’s recognition of the world.
For example, current research uses the ERP to test aesthetic perception. Aesthetic perception research tried to utilize P100, P200, N200, P300, and LPP to analyze affordance perception and aesthetic neurofeedback [25]. The study attempted to use the high- and low-attractiveness tools to present the oddball process. Especially, P200 represents some aspects of higher order perceptual processing. P100 and P200 show the enhanced amplitude for perceiving highly attractive objects, N200 latency seems to reflect response time, and LPP shows enhanced conditions for unattractive perceptions.
2.3. Implicit Memory Integration
UX and visual aesthetic processing rely on certain implicit memory effects. This implicit memory integration is normally in the unconscious state to influence a person’s emotional processing. Prototypicality is challenging to measure according to implicit memory integration testing because it depends on a person’s individual experience, evaluation, and testing [22]. Implicit memory concerns symmetry and familiarity aspects of the aesthetic stimuli to test the memory inside people’s minds. These implicit memory integrations might be related to episodic memory, which influences people’s likelihood, attitude, and preference towards objects.
2.4. Explicit Classification
Processing is usually affected by the perceiver’s profession and knowledge, and explicit classification is meaningful and can be expressed. Interestingly, one recent study found that visual biofeedback differed between expert and layperson perspectives. In one study, the researchers invited ten experts and ten laypersons to judge web page stimuli in an EEG experiment [26]. They found that aesthetic experts needed more time for website evaluation than laypersons. A possible reason is that aesthetic experts have professional knowledge, and they require time for considering evaluation methods and considerations.
2.5. Cognitive Mastering
Meanwhile, the ability to reflect further biofeedback and the following stage of cognitive mastering is based on a person’s past learning and knowledge. The cognitive mastering results are normally evaluated based on the successful confidence in either revealing a satisfying comprehension, successful cognitive mastering, or changes in the level of ambiguity. “Less is more” mental workload should be considered. In the UX design process, some designers aim to capture user attention and employ very complicated concept designs. However, such a product is less interesting to the end-users because they do not know how and where they should focus their attention owing to the extent of required cognitive information processing. The human body usually delivers 11 million bits per second to the brain for information processing, but the conscious mind seems to be able to execute only 50 bits per second [27]. As a result, designers should put the most important information on the UX design interfaces. For example, researchers revealed that website designers only have 50 milliseconds to capture the users’ attention in the website design process [28]. Therefore, more designers follow the design principle of simplicity or design profound concepts in an easy-to-understand method because users will not stay for a long time to understand the interfaces or interactions.
2.6. Evaluation
The evaluation stage completes the aesthetic processing through a somewhat more complicated process. In fact, after the explicit stage, participants have already had different notions in their minds. As a result, researchers always invite suitable participants to participate in their studies, such as laypersons, experts, engineers, artists, and designers. Selecting the right participants for brain data analyses directly influences the research outcomes and results. To conduct a user evaluation process on an interactive prototype or project, professional test methods, such as heuristic evaluation, usability testing, and expert evaluations, are used.
2.7. Time, Environment, and Social Interaction
Research has found that spatial (environmental) elements have a potential and profound influence on aesthetic processing. Researchers [29] observed that participants spent more time looking at artwork when they viewed it in the context of museums rather than in the laboratory. Researchers have also mentioned “aesthetic attitude” and studied how distance and disinterestedness influence the aesthetic stimuli [30]. This phenomenon highlights the spatial factors that also affect the UX in aesthetic appreciation, affective processing, and usability testing. The research of Herrera-Arcos et al. observed people in a museum to find that beta-band suppression is present during engagement and aesthetic appreciation towards their favorite paintings [31]. Moreover, past research on UX focused on delay and subjectively experienced time. Past studies used fMRI and found that frequent delay causes fewer expectations of future interactive behavior and can activate brain regions such as the posterior medial frontal cortex [32]. The latest research found that subjectively experienced time was in accordance with the perception of interactive temporal characteristics and enhanced UX design [33]. These three supplementary considerations can improve the UX in future interactive research and provide a link to neuroaesthetic analysis.
These two models can contribute to the following discussions and analysis towards UX and UI designs. For example, early human vision is associated with the initial aesthetic network, visual novelty, and first impression when users perceive the UI design. The evaluation stage contains heuristic evaluation user testing for UX quality. Understanding the aesthetic steps from both models would offer research entry points and clarify how aesthetic processing can be involved in the following visual interaction design process.
2.8. UX and UI Designs
2.8.1. Creativity
Visual novelty involves visually brand new things that users do not expect and have not experienced before [34]. Kim’s research emphasized that visual novelty is an essential factor in pleasure, arousal, expressive aesthetics, classical aesthetics, and perceived usability when the project is perceived initially. There are two words related to visual novelty, creativity and originality. Creative and original works are pieces of work, which are, first, to a significant extent new, original, and distinct and, second, show a high degree of success in the domain [35]. Both creativity and originality can enhance the visual novelty feeling because people like experiencing highly creative projects to improve their “Wow” sense. The frontal lobe and prefrontal cortex play a key role in creativity and help invent more distinctive interactive products [36]. Meanwhile, the past research has found that creative inspiration evokes higher alpha indices than creative elaboration [37]. Therefore, brain status analysis can help further understand the visual novelty of the early vision or perception on UI and UX designs.
2.8.2. Aesthetics and Usability
In usability testing, the functions, features, interactive methods, and perceived usability are the main considerations for evaluating the success of the interactive products. Meanwhile, there are two usability factors: inherent and apparent usability [38]. Inherent usability focuses on the ease of use, and apparent usability is related to aesthetics. In the UX design, usability refers to evaluation by psychological and behavior experience “during and after use (or behavior)” and influences other experience aspects. Another critical term called perceived usability, formed by impression “before use (or behavior),” is affected by quickly formed subjective decisions such as aesthetic judgments and functionalities [39]. On the aspect of UI design, first impression is a part of aesthetic processing. Most user experience contains both elements of aesthetic and usability testing. On the aspect of mobile application, one research utilized N200 and LPP to observe utilization and aesthetic biofeedback of first impressions of mobile applications [40]. This research utilized the brain data analysis to judge the product in four dimensions containing “High usability & High aesthetic,” “High usability & Low aesthetic,” “Low usability & Low aesthetic,” and “Low usability & High aesthetic.” The research found that N200 was higher on the higher aesthetic visual interface and had a lower response to low usability and aesthetic. Contrarily, the low aesthetic can enhance LPP for mobile applications. This research study focuses more on utilizing implicit evaluation. Past research has promoted belief that a pleasant initial aesthetic experience induces the halo effect, mitigating unsatisfying usability problems to affect the overall UX and leading to acceptable design [41]. However, the focus should remain on addressing usability problems. Both UX and UI designs are different from graphic design because UX and UI also need to consider the logical thinking of the design progress.
2.8.3. Top-Down and Bottom-Up Processing on UI and UX Designs
Visual perception, mentioned in both models, is vivid and full of detailed information, making visual stimuli informative and persuasive [23]. It is a process that combines top-down cognition and bottom-up information reception [42]. For the top-down approach, the designer should consider the overall macro design elements, such as the whole design concept of the information architecture. After the initial decision, designers should consider the micro aspects in terms of icon style, design modifications, color enhancement, and other design details. By contrast, bottom-up visual perception is primarily influenced by the eye’s recognition of the world [23]. People construct their world view from what they see and observe. On the aspect of UI and UX designs, the bottom-up strategy is used in some uncertain design tasks or unclear main directions. First, designers or researchers find and classify the users’ needs in detail and then analyze the corresponding user-centered design directions. Second, designers start to fill up the design gaps and to integrate the overall design situations before finalizing the whole design concept. Designers and researchers are concerned with user behavior and expect to satisfy users’ needs, and top-down and bottom-up approaches should complement each other to explore the integrated UI and UX design concepts.
Top-down and bottom-up approaches also play a role in the selection of visual stimuli and visual information [24]. The initial visual selection is based on the salience of objects present in the visual field. After massive recurrent feedback processing, volitional control based on expectancy and goal setting will bias visual selection in a top-down manner. Top-down knowledge of nonspatial features of the objects in the visual field (e.g., color, shape, and luminance) cannot easily change after the initial selection priority. Most visual information has been determined during the initial aesthetic appreciation, but profound or meaningful content or concepts can alter human perceptions and interpretations. Furthermore, there is an ideal and appropriate model for future HCI invention and creation called “the unifying model of visual aesthetic experience” [43]. The model combines visual and context stimuli to detect human aesthetic experience through bottom-up aesthetic perception and top-down aesthetics of cognition. The model starts with the visual stimulus and content background to construct the basic perception and cognitive thoughts, which is helpful for project evaluation or creation process. Figure 3 is constructed based on the abovementioned model and involves UI and UX design factors. In general, both top-down and bottom-up pathways are significant methods to explore human visual interaction behavior and the aesthetic appreciation process.
3. Current Brain Research on Aesthetic Processing
3.1. Experiment Apparatus and Wearable Technology
In the wearable technology area, several companies have invented relatively reliable algorithms to help users know their daily experience through BCI devices such as Muse band, Neurosky Mindwave, and Emotiv EPOC+. These devices have calculated algorithms to output meditation, attention, and emotive reflections when interacting with daily installations, platforms, interfaces, smart vehicles, and other interactive computer applications. The Neurosky Mindwave headset provides meditation and attention values, and Emotiv EPOC+ provides interest, excitement, engagement, stress, relaxation, and focus values. Several studies have used both devices to study visual brain biofeedback on user experience. Past research used a brainwave headset to practice meditation on seven chakras and visualize their minds through Neurosky Mindwave [44]. Antonijevic’s team used EPOC+ to test users’ visual memory performance [45].
Wearable BCI technology can use data from users’ brains and has helped invent many interactive research projects. These reactive and active BCI projects combine artistic design and interactive projects by helping people using art therapy, game design, and education control [46]. For example, the current technology utilizes the P300 BCI to control the preferred game object [47].
Some researchers used 16/32/64/128 electrodes and fMRI to study UX design and aesthetic biofeedback through brain region analysis on the level of clinic apparatus. This apparatus is more advanced and can analyze detailed ERP, topography, time-frequency, and brain region activation. Most EEG apparatus employs the international 10-20 system and has electrodes placed at 10% and 20% points along the lines of latitude and longitude on the head [48]. Meanwhile, past research demonstrated that fMRI could evaluate the topography of the human primary visual cortex [49].
3.2. Four Brain Lobes
To understand the aesthetic reflections and visual stimuli in the context of UI and UX, it is essential to understand the optical functions of brain lobes. The cerebrum contains the frontal, temporal, occipital, and parietal lobes. Each lobe region plays a different role in aesthetic appreciation. Figure 4 summarizes the areas related to aesthetic processing; these will be updated with future research findings.
3.2.1. Frontal Lobe
The frontal lobe contains inferior, middle, and superior frontal gyri. The frontal lobe [50] relates to attention mechanisms and expression of emotional feelings [51]. The inferior and middle frontal gyri are associated with perceptual, cognitive, emotional, and reward processing feelings. The inferior frontal gyrus and left medial superior frontal gyrus relate to the aesthetic judgments of positive social meanings and beautiful objects. Specifically, the inferior frontal gyrus is also active in the aesthetic senses of rhetorical beauty and moral beauty. The superior frontal gyrus is related to aesthetic judgment and cognitive processing. The orbitofrontal cortex is associated with affective processing. In the spatial environment, it has been found that the medial orbitofrontal cortex and prefrontal cortex have more robust activation in the context of art gallery appreciation than in the digital computer context [52]. The frontal lobe and prefrontal cortex play a key role in creativity and are helpful in the invention of more distinctive interactive products [36]. The frontal area also oversees the formation and manipulation of the mental images and coordinates in the spatial and sensory regions.
3.2.2. Temporal Lobe
The temporal lobe is located between the central sulcus and the parietal fissure. Neurons in the temporal lobe are tuned for 3D surface orientations [53]. The inferior temporal gyrus relates to visual imagery and the representation of object shapes, and the superior temporal gyrus is involved in abstract aesthetic processing [54]. The bilateral middle temporal gyrus and super temporal lobe show reaction to moral ugliness, whereas the inferior temporal gyrus is associated with visual processing and moral goodness [14]. Visual neurons in the temporal lobes have different brain functions.
3.2.3. Occipital Lobe
The occipital regions mainly influence the initial aesthetic network, such as the perception of shape and color [54]. The bilateral inferior occipital gyrus and the right middle occipital gyrus have a more robust reaction to aspects of negative social meaning. The bilateral inferior occipital gyrus especially reflects both ugly objects and negative social meaning. Zhang’s team also found that the middle occipital gyrus oversees the appreciation of facial beauty and moral beauty. Occipital lobe visual neurons are more comprehensive in aesthetic processing.
3.2.4. Parietal Lobe
The fourth brain lobe is the parietal lobe. The superior parietal lobule is related to visuospatial exploration [54]. Parietal areas of the scale are also in charge of attractive faces. Moreover, the right parietal cortex [52] shows increased neural activation in response to artistic paintings or images. Past research found that the parietal regions significantly reveal the differences in aesthetic judgment between the sexes [55].
In general, the frontal lobe is the center of reasoning. The parietal lobe is the center of recognition and the perception of stimuli, the temporal lobe is that of visual memories and emotional association, and the occipital lobe is that of visual processing [56]. Researchers [57] have found that the reward circuit in the brain system influences aesthetic appreciation involving the ventral striatum, the hypothalamus, and the orbitofrontal cortex. The reward circuit primarily affects the emotional effects of the UX design, which determines the liking and wanting in the aesthetic models.
3.3. Contribution of EEG to UX/UI Design and Aesthetic Processing
The EEG contains the alpha, theta, beta, gamma, and delta bands. The use of EEG can support the observation of a more extended period or epoch than the use of ERP because it supports continuous variation, so it is more suitable in the UX design experiment. Wojciech Salabun described [58] EEG as a measurement of electrical brain activity and state. Azzazy et al. and Bell and Cuevas summarized the EEG as providing direct measurements of electrocortical activity with millisecond precision with high sensitivity to signal changes in arousal, perception, and cognitive process [59, 60]. Table 1 summarizes the contribution of EEG research towards aesthetic processing in the visual interaction.
3.4. Contribution of ERP to UX/UI Design and Aesthetic Processing
The event-related potential can detect and record the fast response of the user to the visual, emotional, and usability testing. The event-related potential (ERP) was defined as “a series of voltage oscillations or components recorded from the scalp to indicate the brain’s electrical response to discrete stimulus events” [71]. Normally, researchers utilize rapid serial visual presentation to evoke ERP visual biofeedback. Rapid serial visual presentation presents the process of sequential images at the same spatial location at high display rates with multiple stimuli per second [72]. Some researchers applied the oddball paradigm in the experiments. The oddball-paradigm stimuli are classified into two classes: targets and nontargets, and nontargets appear more frequently than targets. The oddball paradigm often contains participants’ behavioral responses. As for the specific visual interaction design research, some researchers set aesthetic and usability response questions following every visual stimulus [26, 73]. Table 2 summarizes the contribution of ERP research towards possible aesthetic processing in UI and UX designs.
3.5. Contribution of fMRI to Initial and Delayed Aesthetic Appreciation
The human brain contains crucial parts, such as the occipital cortex, which deal with visual information or signals from visual stimulus inputs. These parts of the neuromechanism respond to aesthetic appreciation and are helpful in the interactive visual context. Aesthetic appreciation is related to the activation of two different networks: initial aesthetic network and delayed aesthetic network [86]. People will initially observe the appearance, external morphology, color, shape, and other general appraisals of objects; these are regarded as the initial aesthetic network. Meanwhile, the human sense of beauty is also influenced by other components, such as culture, value, experience, supplementary knowledge, and social appreciation. After deeply analyzing the beauty meaning by consciousness, the delayed aesthetic network emerges in people’s minds. For example, in the area of text design, in ancient Chinese culture, abstract art and concrete art have been applied to Chinese text. Research by Wei Zhang’s team (2017) focused on the pictographs and oracle bone scripts, which represented abstract social beauty and concrete object beauty, respectively. The study demonstrated that human sense of beauty, ugly, positive, and negative emotions was affected by both object parameters and social values. The research used fMRI to record and analyze the activated brain regions under morphological beauty and social-context positive and negative situations. The research helps explore the direct reflection of both initial and delayed aesthetic considerations from people’s minds. These data demonstrate that analyses of neuroscientists can help improve the understanding of both initial and delayed, perceptual and cognitive, aesthetic appreciation for UX improvement. UI design involves the initial aesthetic network, whereas UX design considers both initial and delayed aesthetic networks. In this context, Table 3 summarizes the initial and delayed aesthetic appreciation influenced by object beauty, social environment, culture, background, and context meaning in the human brain regions.
3.6. Subjective Evaluations and Brain Analysis towards UX and UI Designs
Some studies show that, in aesthetic preferences, there are significant differences between subjective evaluations and brain data analysis. For example, a UX study on smartphone user experience compared the subjective evaluations and five brain bands of relative power on UX, flow, immersion, controllability, responsiveness, functionality, and pleasure [12]. It showed the significance of correlation of pleasure with beta, responsiveness with alpha, UX with gamma, and immersion and pleasure with theta, which demonstrated that subjective evaluations had some correlations with brain frequency bands but not all of them. In aesthetic evaluation research, Ding’s team utilized the ERP to test the visual aesthetic biofeedback on mobile phone product appearance through N100, P200, and N200. Their research found no correlation relationship between P200 and subjective evaluation, which means that brain analysis can reflect more accurate visual preference through electrophysiological data.
Most subjective evaluations of UX and UI designs include heuristic evaluations, probing questions, and user tests. A well-known evaluation model consists of ten principles of heuristics evaluations [87]:(1)The visibility of the system status(2)User control and freedom(3)Error prevention(4)The match between the system and the real world(5)Consistency and standards(6)Recognition rather than recall(7)Flexibility and efficiency of use(8)Aesthetic and minimalist design(9)Help for users to recognize and diagnose errors and to recover from them(10)Help and documentation
Many interactive research experiments have used these ten principles to test users’ evaluations on the development of HCI installations and prototypes. These evaluations are quite useful on 2D program applications, but psychophysiological data could be more appropriate for UX or UI designs in the virtual environment [88]. Meanwhile, psychophysiological data are more appropriate for a fast reflection brain system without any user control [89].
In contrast, professional aesthetic judgment should follow the design principles [21], consisting of affordance, consistency, feedback, visibility, constraints, and mapping. These design principles have been applied in many interactive platforms, interface designs, information visualizations, and product designs. However, traditional interview techniques cannot uncover people’s unconscious states, and some users even offer unreliable answers that are inconsistent with their consciousness, which influences the quality of research data and results. Daniel Kahneman [89] identified two system models of brain processes. System 1 is fast, automatic, and outside users’ control, while System 2 is slow, voluntary, and completely under users’ control. Neuroscience brain imaging can detect people’s fast reflection system (System 1) to reveal the nonvolitional qualitative data needed to improve visual study reflection and human behavior research in the prototype-testing process. Brain analysis linked to heuristic evaluations and design principles applied in everyday activities should be investigated in future HCI research and applied in the prototype development process. The chart in Figure 5 is inspired by a concept-driven interaction paper [90]. The chart combines the brain analysis and the HCI workflow through the accumulated experience on interaction design implementation. The graph describes the potential BCI utilization during the HCI-developing process. The variety of BCI data categories is beneficial for improving design idea creation, extensions, user evaluations, and prototype development. The variety of brain data categories and status analysis helps improve HCI idea creation, extensions, user evaluations, and prototype development [91].
3.7. Neuroaesthetic and Artistic BCI UX Projects
Excluding experimental research studies on aesthetic processing and UX, many past and recent projects have started to connect neuroaesthetics and UX to develop interactive visual installations, projects, and products. Meanwhile, artistic BCI can help record people’s brain data and understand users’ affective states through creative forms [46]. Visual interaction of artistic BCI consists of passive, active, and reactive BCIs. Active and reactive BCIs are especially applicable because they control the interactive outputs such as artwork independently or dependently of external events through users' biofeedback on the interactive interface. This technique can be utilized to combine with mindfulness, health benefits, education, game, market promotion, and many interdisciplinary fields.
Currently, the BCI can record users’ meditation and attention values to analyze their current mental status. The meditation value can test the mindfulness effect of the corresponding art therapy [44]. Neurosky Company developed one wearable technology called the Mindwave headset, and the company invented meditation algorithms. The meditation algorithm indicates the level of mental relaxation, and the value ranges from 0 to 100, which helps users know their inner state of mindfulness. In contrast, attention algorithms indicate mental focus and familiarity and measure learning and understanding with educational tasks [92]. The latest review described that the meditation or relaxation state would cause stronger theta and alpha and weaker beta waves [93]. Furthermore, stronger beta rhythms and weaker alpha and theta rhythms were observed when the user was in a concentration or attention state. Both attention and familiarity data help manage and recognize people’s learning effectiveness through BCI technology.
Mindfulness can help people reduce stress and enhance cognition. The concept of mindfulness originated in Buddhist psychology and has been well documented in clinical interventions [94]. Medical doctor Jon Kabat-Zinn from the Massachusetts Institute of Technology created the Mindfulness-Based Stress Reduction program [95], which targets stress reduction, improves heart/body/mind health, and enhances immunity. Current wearable BCI technology can link with mindfulness and user meditation data to create a brain-painting artwork. Mindfulness art therapy is combined with seven-chakra meditation to improve the mindfulness effect through digital visual stimuli technology [44]. Pauline Wills [96] classified chakra concepts into seven colors: red-base chakra, orange-sacral chakra, yellow-solar plexus chakra, green-heart chakra, blue-throat chakra, indigo-brow chakra, and violet-crown chakra. Past research used abstract brain wave patterns to depict the seven-chakra situations to help users visualize their current meditation mind status [44]. Abstract art brainwave patterns intensify people’s interest in practicing healthy mindfulness exercises. The combination of visual art and mindfulness can refine users’ emotions and feelings. Positive emotions can improve cognitive processing and enhance memory ability and decision-making [25]. Therefore, the combination of visual stimuli and mindfulness has been expected to reduce people’s stress, recover emotions, and enhance cognitive abilities. This research utilizes the BCI to combine the visual aesthetic and user experience designs.
Both concrete and abstract art can contribute to UX and UI research of artistic BCI interaction. Abstract art breaks away from the traditional representation of physical objects [97]. In contrast, concrete art makes people think directly about real things without fabrication [98]. Visual interaction derives from the graphical UI (GUI) in the BCI. With digital and computer technology development, visual artists can draw and paint their artworks using the GUI by linking it with a drawing board. Visual interaction also gives rise to digital innovation in abstract and concrete art through the BCI. The University of Sydney Design Lab invented BCI public art projects on the digital canvas in Central Park [99]. Master students presented their digital drawings, videos, and interactive animations on the digital wall by inputting users’ meditation, attention, delta, theta, alpha, beta, and gamma values. This project considered the overall environment immersion and presented people with creative ideas on the digital interface. In contrast, recent research [100] created a human-machine interface platform to draw public painting through robotic machines and P300 ERP values. The current wearable BCI technology and human-machine interface have been explored and have demonstrated how the BCI supports HCI, neurophysiology, art, and computing.
Some UX/UI designers used the concept of describing abstract presentations to the public. If an ambiguous interactive abstract art idea is presented, people may not understand the abstract user interface promotion. In research experiments, the BCI also helps push the interpretation of abstract art. The ERP N400 was used to study understanding and meaning in the abstract art context, and the research found that N400 can test “related” or “unrelated” meanings in abstract art [84]. The study revealed that users could perceive and link related context with the corresponding abstract artwork through N400 data analysis. Unrelated contexts elicited more negative N400 amplitudes than related ones. This finding encourages and motivates UX/UI designers to create and invent more interactive abstract art installations and products. Sometimes, abstract content can leave distinct impressions on customers.
3.8. Visual Memory Research
To investigate visual memory on UI design, many researchers considered the affordance and consistency of the design progress. In the study of brain, researchers found that the ventral and dorsal pathways oversee the perception of objects and spaces formed in the episodic memory, which is defined as explicit memory of specific items or events tied to a particular spatiotemporal context [53]. Recent UX research of a navigation system [101] in the virtual environment investigated environmental visual stimuli. Users are needed to recognize and memorize the spatial content and structure in virtual game, learning, and exercising domains. The study of dorsal and ventral pathways can help understand brain data used to enhance spatial and temporal HCI experiences. According to Connor and Knierim, the hippocampus should also be considered in this research [53]. They stated that the binding of objects to locations takes place in the hippocampus representing spatial and temporal information, which builds human episodic memory. It oversees the object-space and spatial-temporal navigation systems. Moreover, the hippocampus organizes some milestones and landmarks in people’s minds, and it works with ventral and dorsal pathways. Future BCI electrodes and research experiments should concentrate on the hippocampus to study episodic context reflections. The current technology had explored the method to synchronize the GUI with EEG signal measurements to test intentional and incidental memory evaluation on visual memory [102].
3.9. Interactive Information Visualization
User experience and user interface receive information and data from GUI, and many designers and artists started to contribute their creative ideas of designing information or data visualization interfaces to improve the UX design in terms of multiblock data analysis, pattern animations, interactive platforms, virtual reality, finger movements, and so on [103, 104]. Information visualization presents interactive visual representation of data to amplify cognition and enhance information understanding [105]. It creates meaningful interfaces to communicate ideas or facts about data and to explore the discoveries.
Recent research has found that dynamic visual stimuli are considered more beautiful and visually pleasing than static visual stimuli [1]. Animated visual stimuli are more interesting to read and interact with than a static UI. Animated data or information visualizations can enhance the acceptance and interest in reading data or information. Research has found that animated visualization can engage users and facilitate learning [106]. Furthermore, the current UI animation time research of Skytskyi [107] found that the most optimal animation time is around 200–500 ms. It suggests that future research can use the animated UI to improve event-related potential studies because the duration of visual stimuli is always less than 1 second in ERP experiments.
In their recent research, Nuamah’s team used the EEG to analyze people’s frontal and parietal areas to determine the cognitive reflection of information visualization [108]. They found that visualization of inherently spatial (intuitive) tasks reduces cognitive mental work and effort when it is in a spatial or graphical condition than symbolic or numerical conditions. This finding can help UX design of memory and cognitive effects in reading data information. The latest research has also demonstrated that the use of EEG, eye movement, and visualization logs showed the significance for current and future visualization evaluation [109].
Meanwhile, BCI data and information have been used in the interdisciplinary research fields of health, gaming, education, and entertainment. Information and data designs apply to BCI information presentation. When designing BCI information for healthcare, it has been suggested that the design should follow rules that are thoughtful and clear [110]. The rules produce a specific characteristic or pattern of action and encourage moral, physical, and mental development in a particular direction. The visual stimuli of the information design should be acceptable and convincing.
In other interactive fields, pictographic symbols have become essential components of the systems. These symbols are used to indicate information to people through visual design hints. Pictographic symbols appear in people’s daily interactions: in mobile applications, web pages, interactive game installations, shopping mall navigation systems, and so on. Illustrative and legible icon designs allow people to quickly comprehend the meaning of the interface and navigate their interactions accordingly. More specifically, color usage is widely applied in users’ interactions. BCI research [10] reported that color selection can play an essential role in pictographic design. P100 is extremely sensitive to high-preference color combinations (blue and white), which enhances the level of legibility. P300 can test the attention and reaction times for icon design legibility, and the mean amplitude for shorter icon display times was significantly higher than that for longer icon display times. Brain research can contribute to the color and legibility context in the design of pictographic symbols in daily applications.
The colors also connect with people’s emotions and become the designers’ inspiration to create a color interactive installation to refine people’s daily feelings. Ambient BCI color design should focus on the users’ moods and consider the use of artistic patterns to improve people’s everyday environment. Research has found that a combination of colors and emotions resulted in significantly higher classification accuracy and evoked stronger P300 and N400 reflections than a gray face pattern and color ball pattern [111]. This research demonstrates that affective, meaningful patterns and color can enhance people’s visual impressions and evoke higher ERP values.
Current data visualization is also a part of UX and UI in future research development. The professional data visualization book classifies the information into Data Process, Data Blocks, Data Circles, Data Log, Data Nets, Data Maps, and Data Aesthetic [112], which improves the UX and UI designs in many interdisciplinary data categories. Future neuroaesthetic research towards UX and UI improvements on information visualization could refer and obtain inspiration from Data Flow book on understanding data and further aesthetic processing improvement through EEG and ERP analysis.
3.10. Influence of Brain Impairment Conditions on UX and UI Designs
Many researchers recruit participants to join their brain or eye test experiments. However, the conditions in Table 4 should be considered when testing user interaction on UX and UI designs using aesthetic processing and visual biofeedback. This summary of brain impairment conditions can inspire UX designers to invent more interactive installations or products to help visually impaired people understand this world. For example, current studies utilized visual-to-auditory sensory substitution devices to convert visual information to auditory information [113], and Mymemory mobile memory application helps people with traumatic brain injury [114]. Table 4 summarizes brain impairment conditions that influence aesthetic appreciation and UX and UI designs.
3.11. Comprehensive Summary of Current Trends and Opportunities in UI, UX, and Aesthetic Processing
Some environment considerations are not directly related to human-computer interaction but are suitable for considering future user experience design in human-computer installations. Table 5 summarizes the recent research studies contributing to the UI and UX designs.
Table 5 summarizes the UI and UX visual stimuli/content related to brain research. The table is ordered alphabetically based on visual stimuli/content. By understanding Table 5, researchers can understand current and future research developments in the area.
After reviewing current brain region analyses on aesthetic processing, it is clear that researchers have used different types of visual stimuli (interface, interactive product shape, animation, information visualization, etc.) and brain neurophysiological methods to analyze the aesthetic biofeedback in the UI, UX, and interactive outputs. The SWOT analysis in Table 6 describes brain research on aesthetic processing contributing to UX and UI designs, which is summarized according to noninvasive neurophysiology SWOT research [151].
4. Discussion
The main reason for conducting this review was the lack of research review concerning brain aesthetic processing of the visual interaction towards user experience design. Our paper can contribute to better understanding of the ways researchers and designers can use EEG and fMRI to analyze the aesthetic judgment in user interface and user experience designs through ERP, time-frequency, topographies, and fMRI data analysis. In this study, the research findings and results demonstrated that every ERP amplitude could contribute to visual interaction, projects, and analysis. Then, the activated or inactivated status of time-frequency, topography, and fMRI can provide understanding of users’ aesthetic biofeedback on the visual interface. Finally, many designers and engineers started to import EEG data into the daily visual interaction applications.
4.1. Advantages and Disadvantages
An objective evaluation has some essential advantages and disadvantages when developing visual interaction design. The main advantage of an objective assessment of aesthetic appreciation it that it is convincing and supported by scientific knowledge. The objective evaluation covered users’ unconscious status without any user control. It is connected to visual neurology of human physiological development and evolution. The advantages of the review study also can support the emotional brain systems to develop current and future intelligent visual interaction systems. Other than the limitations and weaknesses described in the SWOT analysis, the main disadvantage of recent objective evaluation is the difficulty of working with open-ended questions in aesthetic preference. All the visual stimuli reported on were designed in a restricted way by researchers and designers. At this time, subjective evaluation should be included in studies. The user experience researcher should probe and ask open-ended questions about developing an interface design and preferred direction by recording participants’ answers. It is reasonable to combine both subjective and objective evaluations for future visual interaction in user experience research.
4.2. Visual and Emotional Orientations
Recently, review research has primarily focused on the relationship between emotions and EEG studies [7, 17]. They discussed how valence and arousal influence the user experience quality and summarized the corresponding activated brain regions for anger, disgust, fear, happiness, and sadness. Meanwhile, emotion-related studies used the wearable Emotiv EPOC+ headset to understand emotion status during the user experience process. Furthermore, an interesting review study presented similar findings to user experience research from EEG and EMG studies [16]. The work discussed professional studies on emotional biofeedback in terms of game context, usability testing, and applications through EEG study. It also discussed wearable technology in the gaming environment and collected emotional EEG data and information. However, these studies did not mention aesthetic appreciation of visual user experience design. The key difference from our review study is that our study shows a clear influence of aesthetic processing on visual interaction research. Our study created a new connection between aesthetic appreciation, brain data analysis, and visual interaction design, which helps researchers and designers with the use of scientific aesthetic judgments for daily interactive product or interface design. Furthermore, this review considered the ways wearable EEG headsets can enhance the visual interactive user experience in people’s everyday lives, such as brain painting and visual memory research. The study discussed the input/output brain data in daily interactive art therapy platforms. Through active or reactive BCI data input/output, people can visually interact with an intelligent system to enhance user experience design. Meanwhile, our study can support emotion-related applications and allow more users to accept daily BCI interactive applications. Past research stated that positive emotions improve cognitive processing, visual sensory perception, attentional resources, memory ability, and decision-making [25]. Our research results can help researchers combine visual and emotional research orientations in visual interaction and the user experience market. The two aforementioned aesthetic models can combine both visual and emotional orientations for future user experience studies.
4.3. Appropriate Use of Brain Technology in Visual Interaction
In this review, researchers recommended the ERP, EEG, and fMRI to measure aesthetic judgment of daily interactive projects. First, researchers can use the ERP to present the picture stimuli or short film (1 to 2 seconds) and observe fast visual biofeedback towards these visual stimuli, such as visual animation and website user interface [26]. Second, researchers can use time-frequency to observe a longer period of user interaction behavior and compare these brain data with the rest-mind status [152]. Third, the topography and fMRI can detect higher or lower activation for specific brain regions. Past research found that relative power of EEG analysis (delta, theta, alpha, beta, and gamma) was quite useful in analyzing activation of brain regions during smartphone user experience [12]. Fourth, researchers can allow participants to use wearable headsets to record data outside the laboratory [31]. Meanwhile, artistic active or reactive BCI can import users’ EEG data and present creative forms on the visual interaction application. These inventions can enhance the original user experience quality, such as mindfulness brain painting [44]. Finally, future researchers and designers can combine computational intelligence (machine learning, deep learning, etc.) and brain neuroscience technology to discriminate visual interface images. For example, the relevant review described that feeding calculated network measures into support vector machines and using P300-based brain network analysis could improve the classification accuracy of visual stimuli [118]. Through the appropriate analysis of brain data, researchers can improve more interaction platforms.
4.4. Limitations
The primary limitations of the psychophysiological measurements of price, complexity, environment consideration, device format, time consumption, data dependency, physical movements, noise interference, and knowledge accumulation should also be considered [16] before the physiological experiments are executed in this visual user experience research field.
Our study also found that using objective brain evaluations poses a difficulty in exploring open questions from users’ assessments in the visual interaction context. Users’ communications and opinions towards aesthetic appreciation are limited in the experimental physiology resources and processes. Moreover, owing to the high cost of physiology experiments and complex process of data analysis compared with experiments involving subjective evaluation, the number of participants must always be considered for EEG and fMRI experiments.
Finally, the limitation of this study is the lack of analysis connecting aesthetic appreciation and emotion. The research focuses on the visual interaction with UX and UI designs and does not contain other sensory interactions. Thus, a limitation of the study is that it only focuses on brain science and data analysis, and future research should also involve detailed analysis of combinations of other psychology measurements to study aesthetic judgment, for example, with the use of an eye tracker.
5. Conclusions
In this review, we have described the methods that use EEG and fMRI to analyze aesthetic processing of visual interactions in user experience research. Many researchers have analyzed brain data to obtain neuroaesthetic biofeedback on UX design in terms of website, mobile application interface, art therapy, brain painting, vehicle human-machine interface, virtual or augmented reality, information visualization, and so on. The critical importance of the study is that it has summarized the corresponding event-related potential and brain region activations when perceiving the visual interaction projects. Another critical point is to motivate people to apply artistic BCI applications in daily visual interactive experience. Designers and engineers can combine both creative and logical ideas to invent more active BCI interactive applications. Understanding the aesthetic appreciation models will clarify the appropriate methods to analyze the aesthetics and usability in the visual interaction development process. Therefore, the study has benefits for researchers, designers, and engineers who use brain scientific research results to support aesthetic appreciation preference. Furthermore, this review encourages UX and UI researchers to combine subjective and objective evaluations to demonstrate the research results for the appropriate interaction context. The implications of the review are mainly for visual interactions in the user experience research domain and users who want to pursue a better aesthetic user experience to enrich their life quality. In summary, this paper provided a helpful review updated with the current research in aesthetic appreciation, models, EEG, and fMRI in the visual interaction area; these offer a new neural aesthetic research base for future user experience research.
Data Availability
No data were used to support this study.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
The authors would like to thank their Ph.D. supervisor, Prof. Dr. Zhenyu Gu, who guided him in writing review articles. Moreover, Prof. Dr. Hongde Li recommended specific writing methods to write a qualified review article. The encouragement and guidance provided by them greatly improved this review. The authors would like to thank the Shanghai Jiao Tong University of the School of Design for their help in this review study. This study was supported by the National Natural Science Foundation of China (Grant no. 71802132).