Abstract
A lot of efforts have been directed to enriching human-computer interaction to make the user experience more pleasing or efficient. In this paper, we briefly present work in the fields of subliminal perception and affective computing, before we outline a new approach to add analog communication channels to the human-computer interaction experience. In this approach, in addition to symbolic predefined mappings of input to output, a subliminal feedback loop is used that provides feedback in evolutionary subliminal steps. In two studies involving concentration-intensive games, we investigated the impact of this approach. In a first study evolutionary feedback loops adjusted the user interface of a memory game whereas in the second study the lighting of the test room was adjusted dynamically. The results show that in settings with an evolutionary feedback loop test participants were able to reach significantly higher scores compared to the static counterparts. Finally, we discuss the impact that such subliminally working applications might have on the user's acceptance.
1. Introduction
Would it not be great if working with our computers would somehow magically make us feel better about ourselves and increase our performance, computers that feel a little bit more emotional than these cold static subjects we are used to? Could computer systems make us more productive by subliminally influencing us?
The idea of influencing human beings by means of subliminal stimuli has been prevalent since a long time. The famous experiment by Vicary in 1957, where he claimed to have subliminally persuaded visitors of a cinema to buy more Coke and Popcorn, drew big attention to the topic of subliminal persuasion. Even governments were concerned about the impact that such subliminal persuasion might have on the public and therefore prohibited its use. However, this experiment has been identified as fake a few years later [1]. Still, many seem to take the fake Vicary experiment as real and as a successful demonstration of the power of subliminal persuasion—even the laws remained, although many scientists doubt that subliminal persuasion actually works [2].
In this regard, it is important to make a differentiation between the terms subliminal persuasion and subliminal perception [3]. Scientific evidence for subliminal perception is well established [4–6] whereas scientific support for subliminal persuasion is seen critically [3].
Kunst-Wilson and Zajonc [4] found out in their experiments that the effect of subliminal perception highly depends on the query directed to the subjects afterwards. People were presented with subliminal presentations of geometrical figures. Afterwards, when they had to identify the figures that were presented to them, they could not remember—their answers seemed to be by chance. However, when asked about which figures they liked more, they more often than by chance selected figures that were subliminally presented before. This seems to clearly indicate that subliminally presented information can have an effect on the feelings of persons. The familiarity aspect might be the cause for this [5].
Closely related to subliminal perception is the matter of unconscious information processing. If we can perceive information without being aware of it, can this information also be processed unconsciously? Furthermore, could it be that information already stored somewhere in our brain might be processed further and help us for example in the process of making decisions while we sleep? Perrig et al. [5] mention the example of Poincare, a French mathematician, who came to great insights during overnight. Also the experiments performed by Dijksterhuis support this theory. He suggests that the unconscious is a much more powerful processor than the conscious [7]. Gigerenzer also stresses the power of the unconscious in decision making [8].
Bargh et al. [6] found out in their study that behavior of persons can be influenced by subliminal or unconsciously processed information. They found out that by activating an elderly stereotype in persons by presenting them words closely related to the elderly, they walked more slowly after the experiment than persons that were shown other words. Kay et al. [9] showed that also the presence of physical objects can influence person's behavior even though it was not consciously registered. Dijksterhuis [10] found evidence that self-esteem of persons can be enhanced by subliminal conditioning. Custers and Aarts conclude in their review and analysis of research regarding pursuit of goals outside conscious awareness that basic processes required for goal pursuit can take place without conscious awareness [11].
All these findings show that information outside the context of conscious awareness can influence the feeling and behavior of a person. While this is still different from being able to actually persuade a person to do something subliminally, these insights might be of benefit in the realm of human-computer interaction (HCI). Having a computer system that actively uses these findings might help to enrich HCI and thus positively influence the performance of a user.
Before using this in a practical application, however, it is important to clarify the notion of subliminal stimuli. Merikle and Joordens [12] investigated whether or not perception without awareness (unconscious perception) and perception without attention (i.e., stimuli outside the focus of attention are perceived) share the same underlying concept. They concluded that these concepts might indeed be closely related. In the realm of HCI, this could mean that it might not be necessary to have stimuli below the absolute threshold level of a person, which would be different for each individual person. It might actually be sufficient if users concentrated on a task do not notice given stimuli, that is, stimuli can not be identified verbally. In our work, whenever we use the word subliminal, we therefore refer to the threshold of conscious awareness, also referred to as subjective threshold [13].
Another important aspect in HCI is that of communication itself. Communication regarding computer systems currently is mostly limited to symbolic communication. Everything we enter into a computer is mapped to a specific meaning, LIKE every keypress, every menu selection, every mouse movement. HCI is mostly reduced to the content aspect of communication. However, as research on human communication has shown, there are more levels in communication. von Thun [14] describes a four-earmodel where each ear is directed to another aspect in communication (content, appeal, relationship, self-disclosure). Watzlawick et al. [15] propose that every communication besides the content aspect also features a relation aspect. Norman [16] describes three levels of information processing: the visceral level (mostly unconscious, automated), the behavioral level (learned skills—mostly subconscious), and the reflective level (conscious processing, self-awareness), also suggesting that communication involves more than the content-aspect. Picard [17] points out that humans engage in interactions with computers in very much the same way as in face-to-face interactions between persons, attributing emotional awareness to machines. However, on the computer side human-computer interaction is still mostly limited to explicit symbolic communication. Kempter et al. [18] propose that human-computer interaction should be enriched by a nonsymbolic form of analog communication.
Symbolic communication in HCI mostly consists of using a combination of icons, gestures, and textual or even verbal commands [19]. Metaphors are introduced helping to translate between the computer and human worlds [20]. Still, the meaning of these components might depend on everyday experiences, culture, and prior knowledge. Often meaning is defined in style guides or even assigned arbitrarily by developers, requiring the user to learn new paradigms. However, there have already been efforts to enrich symbolic user interfaces with aspects of analog communication, like in operating and servicing assistants [21] or multimedia enhanced learning environments [18, 21]. Kempter et al. [18] point out that analog communication in the HCI context is about integration of spontaneous interaction mechanisms. The main challenge therefore is the question of how to integrate analog communication channels into applications without defining explicit mappings between user input and system responses.
In interpersonal communication, visceral recognition and impulsive reactions are explained by subliminal stimulus- and reaction patterns that arose from evolution and are independent of conscious cognition [22]. It is seen as evolutionary adjustment of the expression to the impression [18], meaning that a communicative impulse can evolve because of the effects it was able to invoke in the recipients and where these effects have been proven to be of advantage in the course of evolution. Kempter and Ritter [23] therefore propose that in HCI, for example, graphical user interfaces could be adjusted by similar evolutionary processes. As a basis for these evolutionary processes, continuous information about physiological reactions of the human body could be evaluated to determine whether or not a certain reaction had the desired effect. These evolved cause-effect relationships, however, should not be communicated to the user to avoid symbolic interpretation of them.
In this paper, we are going to extend on the ideas presented in [23] and evaluate how we can enrich the human-computer interaction by analog communication channels. In the next section, we will briefly look at related work, before we show two different examples where we used analog communication channels in an evolutionary way to enhance the performance of users.
2. Related Work
The idea or desire for computer systems that are emotional is probably as old as computers themselves. Even HAL, the protagonist from Arthur C. Clarke's Space Odyssey was able to recognize and express emotions [24]. In 1997, the topic gained new interest with Picard's proposal for Affective Computing, encompassing computing that relates to, arises from, or deliberately influences emotion or other affective phenomena [17, page 3]. The idea is to make computers more humanlike in making them aware of the emotional state of a user. This will help them in decision making and should therefore enhance the overall experience of the interaction with computers. This might also enable completely new fields of use as outlined by Picard. Norman [25] even goes so far as to say that in future intelligent systems need to be able to have emotions to be successful.
One major aspect of making computers aware of the user's emotion is in analyzing the channels over which such emotions are transported. There are different approaches on how to access these. Psychophysiological signals play a major role in looking inside a user [26], a prominent example might be that of lie detectors. But also from other channels, emotional state can be deduced. Batliner et al. [27], for example, propose a system for automatic processing and classification of emotional features from speech analysis. Neviarouskaya et al. [28] showed that deducing affect from text messages delivers valuable information about the emotional state and can be used to give users a more explicit feedback about the emotions of the other part in computer-mediated communication between persons like, for example, in virtual world games like Second Life.
The benefit of emotional awareness of machines is not only seen in enhancing decision making processes as described by Picard. Also media-based applications could be enhanced by taking emotional state information into account as pointed out in [29]. Hui et al. [30] present a model to estimate the level of disruption as a basis to adjust the user interface to the user's current needs. The idea to adapt user interfaces to the emotional state of a user, however, also bears some risks. Paymans [31] discusses the usability tradeoffs regarding learnability in adaptive environments.
Work is also done in the area of standardizing the format of exchanging emotional state information to share such information among different applications and platforms, making it easier to create new emotion-aware applications [32].
So is making computers emotion aware the key to making them easier to use? The works listed above seem to support this thesis. However, R. S. Lazarus and B. N. Lazarus [33] point out that emotions are something intrinsically personal. To understand our own or another's emotions is to understand the ways people interpret the significance of daily events in their lives, and how these events affect their personal well-being [33, page 5]. Norman [25] describes this problem as the lack of common ground between human and machines. How could computer systems ever become aware—even only in a very small way—of all the things that contribute to our emotions? An interesting question therefore arises: do computers have to be aware of our emotions, or would awareness of some of the underlying processes (or indications thereof) be sufficient?
The work presented in this paper differs in a major aspect from the works described above. Instead of finding mappings of certain features to specific meanings (i.e., emotional states), we propose a model based on real analog communication. By this, we mean strictly nonsymbolic communication. In this approach, no explicit mappings between features and meanings are defined in advance by the developer and no discrete emotions are interpreted by the system.
3. An Approach to Evolutionary Feedback Loops
As pointed out above, typical human-computer interaction is forced into symbolic communication. Only predefined vocabulary can be used to communicate (see Figure 1). Everything users want to communicate to the computer has to be encoded in a tunnel of symbolic communication to pass the virtual wall between human and computers. Analog signals emitted by the person are not seen by the computer.
One way to enrich the communication between users and their computers could be to actually look into the symbolic communication stream for hints about emotional content (like done, e.g., in [28]). Another way to enrich communication would be to simply tear another hole into the wall between humans and computers, for allowing more channels, many of them being analog and unconscious by nature, to pass through (one example would be [27]). Here the question arises, how this newly gained analog information should be processed. Typically, this information is classified and mapped to an explicit meaning. We propose to introduce an evolutionary feedback loop between users and computers, without explicit mappings of inputs to outputs. The computer would receive additional analog (not explicitly encoded) input and would also provide analog (not explicitly encoded) output back to the user, resulting in an analog feedback loop as illustrated in Figure 2. Users and machines could then be able to develop an implicit feeling about the interaction process, without being aware of it or its meaning, just like persons mostly can not pinpoint what actually influenced their impression about another person but have learned it implicitly over the course of evolution.
In the following sections, we present two studies where we tested this approach. We made use of evolutionary user environment adaptations based on psychophysiological responses of users in the hope to increase their mental performance while playing certain games. In the first study, we adjusted the user interface of a memory game during gameplay. In the second study, we adjusted the lighting situation of the room during gameplay.
The games for both studies share the requirement for high levels of attentiveness and memory capabilities on the user's part. According to studies, these two states are supported by a high level of psychophysiological activation (arousal) [34]. The concept of psychophysiological arousal has been introduced by Duffy [35] and has been further referenced in many studies: e.g., [26, 36–38]. Skin conductance has proven to be an efficient indicator for arousal and became popular not the least because of its unproblematic measurement. Furthermore, a relationship between stimulus-dependent skin conductance reactions and memory capabilities has already been proven [39]. Therefore, we decided to use skin conductance reactions as input source for the adaptive feedback system in our studies.
The feedback system of our approach is based on a genetic algorithm. Genetic algorithms fit well into the idea of resembling evolution for adjusting certain interface features, as they are modeled after natural evolution [40–42]. In genetic algorithms, certain properties (genes) are defined in a chromosome, most often encoded as a bit pattern. A collection of such chromosomes (also referred to as individuals) forms one generation of a population. Among the population, a breeding process creates new offspring. Hereby individuals that proved to be successful (determined by a fitness function) are more likely to take part in the breeding process than weaker ones. This breeding process is done by recombination of individuals. Offspring (new individuals) is created by taking one part of one parent and another part of another parent. In our feedback system, two parents generate two new individuals (children). A crossover point is determined randomly based on a Gaussian distribution around the center point of the chromosomes. This crossover point splits the chromosomes of the parents into two parts. The first part of parent one and the second part of parent two form the first child. The second part of parent one plus the first part of parent two form the second child. After this recombination, mutation can occur to the bit patterns of the new children. This mutation flips individual bits of the encoding with a given probability ( in both of our studies). The newly generated children then form a new population whereas the parents are singled out. The population size from generation to generation is kept constant in our system. In both studies, we chose a relatively small population size of six due to the limited duration of the games and the requirement to evaluate all individuals based on the physiological reactions of the participants.
3.1. Adaptive User Interface of a Memory Game
For the first study, we implemented a memory game, where users would have to find matching card pairs. Once a card pair was found, both cards could no longer be uncovered (i.e., they were out of the game). In this memory game, interface elements like background color, text size and color, and border size and color were encoded into 15 bit string chromosomes (see Figure 3). The first three bits represent eight variations of border color, followed by three bits representing the variations of background color, one bit for two different sizes of the text font (40 or 50 points), two bits for four variations of border width (0–3 pixels), three bits for eight variations of text color, and three bits representing the variations of the background color of the cards. All color variations were shades of gray between 0 and 100%. The initial set of interface configurations was created randomly but held constant among the test participants. During the game, these interface configurations were evolved by the genetic algorithm. The transition from one interface configuration to the next was performed in hardly recognizable (i.e., subliminal) steps by the system. Each transition between two interface states was animated over a period of five seconds in 20 steps. This interface transition was triggered by the first click on a card. After uncovering the second card, the current interface configuration was evaluated by the system. The skin conductance level at the moment of uncovering the second card formed the baseline while the average of all increases from this point on during the evaluation period of five seconds determined the fitness value for the current configuration. Once all interface configurations were applied, a new set of chromosomes was created by the genetic algorithm. Figure 4 shows the various scenarios for skin conductance reactions (SCRs) parametrization used as fitness function for the genetic algorithm as well as the evaluation of periods of arousal.
(a)
(b)
3.1.1. Method
To test the effects of the adaptive user interface version of our memory game, we used two different setups. In the first setup, we compared static user interfaces to predefined interface changes whereas in the second setup, we compared static user interfaces to adaptive ones. In the first setup, 17 persons (ten female, seven male) participated. 14 of them (eight female, six male) then also participated in the second setup. The order of static and changing interfaces has been randomized in both setups to avoid succession effects. The participants' age ranged between 21 and 52 years. Participants did not know that the interface of the memory game would change during the game to avoid them paying extra attention to changing interface properties.
In both setups, the memory game featured 50 card pairs (100 cards), showing photographs of landscapes of northern Europe. One game session lasted for six minutes. The number of discovered card pairs within this time frame was recorded as performance indicator.
During the game, skin conductance was measured using a Varioport biosignal recorder of Becker Meditec (Karlsruhe, D). Ag/AgCl electrodes were attached to the inside of the palm of the nondominant hand of the test participants to measure skin conductance.
Our hypothesis was that the version with evolutionary user interface adaptations should show a higher count of discovered card pairs than the static variant. Also the level of arousal should be higher in this setting than the static one according to the basic theoretical model for our system.
3.1.2. Results
The results of the first setup show that during the static interface phase participants solved significantly more card pairs on average ( solved card pairs) than during the predefined user interface-changes phase ( solved card pairs; ). At first, we did not find any significant changes regarding the arousal level between the two settings by comparing the average skin conductance responses within the evaluation periods. However, when we compared the time for which positive SCRs occurred during the evaluation windows, we did find significant differences between the two settings (). The average SCR time for the static user interface was whereas the adjusting user interface only accounted for of arousal time within the evaluation window (see Figure 5). These results are in accordance to the results of a study by Lewis and Critchley [39] that connects higher arousal levels to better memory performance.
(a)
(b)
In the second setup, participants who were playing the static version of the memory game solved significantly less card pairs on average ( solved card pairs) than those who played the variant with evolutionary interface changes ( solved card pairs; ). This equals a 44.9 increase of solved card pairs from the static to the adaptive version of the game. Also the parameters of average SCR time during the evaluation periods showed significant differences. Players of the static interface variant showed of arousal time whereas the group playing the variant with evolutionary interface changes showed ().
In both settings, a training effect between the two setups can be seen. However, the increase of performance between the setting with predefined user interface changes of the first setup versus the setting with evolved user interface changes of the second setup is much more pronounced than in the static variant between the two setups. This clearly seems to support our hypothesis that evolutionary interfaces indeed can have positive effects on a user's memory performance.
During the final debriefing test, persons were asked if they recognized any changes to the memory interface during the gameplay. None of them were aware of any interface changes, even though the final appearance of the game was vastly different (see Figure 6).
(a)
(b)
(c)
(d)
3.2. Adaptive Lighting Supported Concentration Games
In this study, we wanted to test if we could enhance the performance of users in concentration games by adjusting the light setting in the room where they were playing.
Besides the memory game from our previous study we introduced four more concentration games for selection: In a simple calculation game, users had to decide if a presented result of additions and subtractions was wrong or right. In another game, a collection of black and white dots was presented and users had to determine without counting whether there were more black or white dots. Another game, showed a list of letters where users had to determine if a given word could be constructed out of the presented letters. Finally, a text-based memory game showed a list of words for two minutes. After this time, the users were asked if certain words had been included in the list shown before. The duration of all games was set to 10 minutes (see Figure 7 for an illustration).
(a)
(b)
(c)
(d)
Besides the gaming computer, the test system featured eight light-control circuits based on the Digital Addressable Lighting Interface (DALI; (see http://www.dali-ag.org/ for more information)) protocol, each attached to indirect 80 W fluorescent lighting devices. One half of the circuits featured yellow/red light (2700 K), the other half white/blue light (8000 K) (see Figure 8). By varying the light intensity of each circuit, the system could produce light situations in an almost continuous gradient between the two color extremes, and also vary the light distribution in the room. To control this light system, we developed a lighting control system that featured the same genetic algorithm as in the previous experiment, but instead of encoding user interface properties in the bit string, we encoded different lighting setups. For this, we used a representation with 32 bits. Each control circuit was represented by four bits giving 16 variations of brightness levels between 3 and 100%.
(a)
(b)
The starting set of light parameters (the initial population) was generated randomly but held constant among the test participants. These light parameters were then evolved during the gameplay. The transition from one light setup to the next was kept at a level where persons would not recognize the change while playing the game. For this, a transition from one lighting setup to the next was animated in dynamically calculated increments of one to five steps per update (depending on the distance between the current and target state), with updates for all lighting devices being issued each second. Light levels in DALI are represented with integer values from 0 to 254 (0 to 100%). Once a transition from one setup to the other completed, the evaluation period (20 seconds) was invoked. The evaluation used the same time-of-activation-based parametrization as described in the memory experiment. After evaluation of the current light setup, the next transition was invoked. Once all of the individuals of a population had been evaluated, a new generation was evolved.
3.2.1. Method
To test the effectiveness of our system, we tested two setups. One setup featured static lighting whereas the other featured adaptive lighting. A total of six participants (four male, two female) played games in both setups. The order of the setups as well as the games was randomized to prevent succession effects. Test participants in total played 17 games in the static setting and 16 in the adaptive setting. The test participants' age ranged from 24 years to 45 years.
The test participants were not informed about the different setups and the adaptive lighting. They were told to play a concentration game and should go for the highest possible score. Questioned afterwards, the test persons did not notice the lighting changes during the game sessions.
During the game, test participants wore a sensor glove that continuously measured skin conductance and transmitted it wirelessly to the gaming computer that used this information as basis for fitness assignments of the genetic algorithm (see Figure 9).
Our hypothesis was that in games played under the evolutionary light adaptations setting a higher total score should be achieved than in the games performed under static lighting.
3.2.2. Results
The comparison of the mean values of the normalized game scores (M) for the two settings shows a significant increase of performance for the adaptive session (, ) versus the static one (). This considerable 20 increase of the participants' performance in the concentration games indicates that evolutionary adapted light situations might indeed be able to improve performance for such games. One issue we encountered in this study was the relatively slow performance of DALI, limiting our maximum possible update rate for the light levels. With eight individually controllable lights, the system was limited to one update per lighting device per second (a total of 8 updates per second). This caused relatively long transition times between one setup to the next, thus limiting the performance of the genetic algorithm.
4. Discussion
The results of both studies indicate that there is benefit in using evolutionary feedback loops to support a user's performance. In both studies, we were limited to a relatively small number of participants. Therefore, the gained statistical significance levels have to be considered with a level of uncertainty. A future study with more participants will have to consolidate these results. Still, as both of these studies show the same trend of the participants' performances, the approach of evolutionary feedback loops in human-computer interaction seems promising.
On the technical side, at first we were reluctant to use genetic algorithms in our applications, as evolutionary algorithms usually are applied in a highly parallel manner, with parallel evaluation of individuals. In our experiments however, we had to serialize the evaluation of individuals. To enable the evolution of several generations, we were therefore forced to keep the population size low.
We only evaluated short-term effects of evolutionary feedback loops in these studies. It would therefore be important to see if a beneficiary effect can also be achieved in a long-term setting or if users would finally get distracted or even tired of the constant pursue for activation, even if not noticed consciously. Also it would be interesting to see if such feedback loops could also improve a user’s perception of a system. Do they feel comfortable in working with such systems or in such adaptive environments? Do they regard the system as more pleasing?
Even more questions arise. While our participants did not claim to have recognized any adjustment effects during the games, how would they react once they knew something was going on in the background. Would they spend more attention in trying to detect any changes, and thus being deviated from their primary task which would cause a negative effect? This risk could be reduced by further smoothing transitions between two scenarios. In the lighting setting, this would require an alternative lighting control circuit with higher bandwidth than DALI. Also great care has to be taken as to what is being manipulated by the feedback loop. To not confuse users, we propose to only adjust elements that are not directly involved in the explicit interaction process. Constantly changing menus or positions of vital elements would definitely irritate users and make the system as a whole unpredictable.
One example often mentioned in affective computing is that of a virtual agent adjusting its facial expressions to the emotional state of a user. However, if the classification of the user's emotion went wrong, that is, the system does not behave like the user expected, the user will loose faith in the system. One major point of our approach is that there is no such defined linkage between cause and action, therefore, such a situation should not happen with these feedback loops.
However, the approach of having something unknown and therefore uncontrollable evolve in a system that influences the user's performance might be frightening, especially when remembering how emotional computers like HAL from the 2001: A Space Odyssey finally wreaked havoc. This raises the question of how emotional machines in general and such evolutionary feedback loops as used in our approach in particular have to be designed to be accepted by users. We propose that it is essential to limit such evolving feedback loops to nonessential parts of a system. Users must always be in control—if they do not like what is happening they have to be able to deactivate the behavior, with the overall system still remaining intact. In our case, they might just turn of the switch of the sensor glove or stop wearing it in first place to stop the evolutionary feedback loop.
So, would it not be great if computers could automatically improve our self-esteem or our behavior by subliminally presenting stimuli like the ones used in the studies by Dijksterhuis [10] or Bargh et al. [6]? Actually products already exist that claim to do just that. The market for audio tapes or magical software programs that state to subliminally affect people in a positive way seems to be huge, despite the lack of scientific proof [2].
However, this raises another concern: if machines can impact humans without them noticing it, who is in control of what the machines actually do? Given the huge efforts taken against subliminal messages in advertising, how would people react to machines that try to do something to them, without knowing exactly what they do? As the reported success of the products mentioned above seems to indicate, many people appear to be less worried about this. Still, researchers must not forget their responsibility for the ethical impact of such work.
5. Conclusion
In this paper, we briefly covered current work on the topics of subliminal perception and affective computing. We then presented an approach for enriching human-computer interaction with analog communication channels. Instead of predefined mappings, this approach is based on an evolutionary feedback loop. In two studies, we were able to confirm significant positive effects of this approach regarding performance of users in concentration intensive games, where the interface or the environment has been adapted in subliminal steps based on physiological signals.
The studies only covered short-term effects (six and ten minutes game duration) so future work in this field should investigate the long-term effects of such evolutionary feedback loops. Also different algorithms for the feedback loop should be investigated. An interesting option for further research would be to add more communication channels to the system, like auditive feedback (for example, computer-generated music).