Abstract
We propose a technique to enhance emotional expressiveness in games and animations. Artists have used colors and painting techniques to convey emotions in their paintings for many years. Moreover, researchers have found that colors and line properties affect users' emotions. We propose using painterly rendering for character sequences in games and animations with a knowledge-based approach. This technique is especially useful for parametric facial sequences. We introduce two parametric authoring tools for animation and painterly rendering and a method to integrate them into a knowledge-based painterly rendering system. Furthermore, we present the results of a preliminary study on using this technique for facial expressions in still images. The results of the study show the effect of different color palettes on the intensity perceived for an emotion by users. The proposed technique can provide the animator with a depiction tool to enhance the emotional content of a character sequence in games and animations.
1. Introduction
The expressiveness of character-based animations heavily relies on their success in conveying emotions to their viewers. In order to achieve high levels of expressiveness, today's game character sequences not only rely on the animation techniques, but also on other elements such as lighting and scene composition. The latter not only enhances the emotional content of the scene, but also draws attention to the specific regions designed for conveying the message. In a similar way, painters use color, various brush strokes, and painting techniques to emphasize certain features of a scene while leaving out unnecessary details. Edward Munch and Vincent Van Gogh are two exemplary artists that used colors to depict emotions in their work [1]. The use of texture for inducing mood is also evident in different brush stroke styles used by artists [2].
The emotional effects of art and paintings on users point to the importance of visual style on users' perception and emotional responses. Some past psychological studies demonstrated that there are links between rendering style and one's perception and feelings about a rendered object [3–5]. Furthermore, some other studies investigated people's association between colors and emotion [1, 5, 6].
Painterly rendering, a subset of Nonphotorealistic Rendering (NPR) simulates the work of illustrators and portrait artists and is commonly reported as an expressive style [7–9]. We propose using the expressive style of painterly rendering to enhance the emotional content of facial character sequences in games and animations. Specifically, we discuss using semantic or knowledge-based painterly rendering systems whose painting process is informed by two sources: (1) knowledge about the contents of the scene and its animation as a sequence. As our focus is mainly on facial character sequences, thus facial features and emotions compose the content of the sequence. In this regard, parametric facial animation and available face standards make facial animation very suitable for knowledge-based painterly rendering. (2) Knowledge about the affective value of painting parameters such as the color palettes and the shape of brush strokes. Such affective knowledge can be obtained with carefully controlled user studies on face sequences. A preliminary study in Section 8 examines the affective value of painterly rendering and colors on facial expressions.
The main contribution of this paper is proposing knowledge-based painterly rendering with two sources of information and discussing its appropriateness for facial character sequences. In addition, we detail two parameterized facial animation and painterly rendering systems and introduce our method and extensions for integrating these two existing systems into a knowledge-based painterly rendering system.
A knowledge-based technique applies color and painting parameters depending on the content of the image and the effect desired in different parts of the image. Thus, it can lead to conveying emotions to viewers more effectively and providing more satisfactory emotional experience for them. Furthermore, using painterly rendering for facial character sequences improves expressiveness with regard to the phenomenon of the uncanny valley of the human believability. While many games and animations try to achieve higher expressiveness by adding to the reality of faces, this makes people notice even the slight flaws in the created faces. Moving to a completely synthetic painted world mitigates this issue and is less process-intensive compared to the algorithms used for the highly realistic outputs.
2. Related Work
Our work is based on past psychological studies on the affective value of colors and line properties [1, 3, 5, 6].
Valdez and Mehrabian [6] found that brightness and saturation significantly affect emotions. They used the Pleasure-Arousal-Dominance (PAD) space for emotions [10] and suggested linear relationships between the Hue, Saturation, and Brightness components and the PAD dimensions. According to their results, color brightness has a strong positive correlation to the pleasure dimension, and the saturation component is correlated with the arousal axis in the PAD space.
Recently, da Pos and Green-Armytage [1] asked European and Australian designers to choose first, a combination of three colors and then a single color that best fits each of the six basic facial expressions. They found that there is an overall consensus among designers in their color-emotion association. For instance, a great percentage of the participants used red and black for anger; for surprise and happiness, they mostly chose yellow and orange hues. Colors for fear and sadness were mostly desaturated. This study suggests that there might be specific colors congruent to each emotion. We used findings of this study to design color palettes for our preliminary study.
Some other studies demonstrated that various line styles can communicate affective messages. A study by Hevner [5] explored the affective value of colors and different shapes used in a painting. According to their results, blue is expressive of dignity, sadness, and tenderness, and red is indicative of happiness, restlessness, and agitation. Moreover, they realized that curves are tenderer and more sentimental compared to the straight lines that are sad, serious, vigorous, and robust. These findings point to the importance of the visual style on viewers’ perception.
Duke et al. [3] investigated the affective qualities of NPR images to provide suggestions for computer graphics. They conducted a series of experiments to show that different elements of NPR illustrations could affect users’ feelings. Some of the results of their experiments show that: (1) NPR can be used to induce perception of safety; people associated jagged outlines to danger, whereas they perceived straight outlines as safe, (2) NPR can affect interpersonal relationships; in another experiment, people associated line thickness and continuity to a character’s strength, and (3) NPR can influence both navigation and exploration behaviours; subjects in their experiments tended to choose paths with higher levels of visual details. They concluded that psychological theories and studies can be used to enhance the outputs of NPR applications; and moreover psychology and NPR can mutually benefit from such interaction.
Furthermore, recent studies in human vision demonstrated that texture properties of a painting can guide viewer’s gaze through a portrait painting. DiPaola et al. [11] conducted a series of eye-tracking studies and demonstrated that Rembrandt used lost and found edge techniques and a center of interest area with smaller brush sizes to guide users’ gaze in a portrait painting. Our system is partially based on this work and the findings of da Pos and Green-Armytage’s study [1].
3. Our Proposed Approach: Knowledge-Based Painterly Rendering
Having a high-level knowledge of a scene and its contents is critical for an artist to make a concrete painting with a strong unified message. In a similar way, having knowledge about the contents of an input image in a painting algorithm allows the painting system to adjust painting parameters based on the characteristics of the objects to be depicted towards a meaningful output.
This type of NPR techniques is called semantic or content-based. Semantic NPR techniques have been used in a few NPR systems. For example, Santella and DeCarlo [12] used data from an eye tracker to determine the regions of higher saliency in an image in order to paint those regions with more details. O’Regan and Kokaram [13] used skin and edge detection techniques to recognize people’s head within a video sequence and set the appropriate painting parameters for those areas.
We propose semantic or knowledge-based painterly rendering systems with two sources of information: (1) knowledge about the contents of the scene and its animation as a sequence and (2) knowledge about the affective value of painting parameters such as the color palettes and the shape of brush strokes.
Regarding the first source of information, a semantic painterly rendering system for sequences of faces requires knowledge about different parts of the face and the facial emotions expressed. Meanwhile, the parametric approach to faces and facial animation segments the face to meaningful, configurable units. The knowledge of these facial parameters is what the proposed painterly system requires. Thus, we believe facial sequences have great potential for knowledge-based rendering approach compared to the sequences of other contents such as landscapes.
There are two well-structured parametric systems for face and facial animation: (1) Ekman’s Facial Action Coding System (FACS) and (2) the MPEG-4 standard for faces. FACS was developed by Paul Ekman in the field of psychology to study the human face and its emotions. MPEG-4 is another standard commonly used by facial animation systems. The MPEG-4 standard includes Facial Definition Parameters (FDPs) and Facial Animation Parameters (FAPs). These two standards enable modular modification of facial sequences. Moreover, they enable transmission of the facial sequences between different systems and even from one head model to another. Thus, these standards allow us to communicate information about facial features and their movements in a sequence to a depiction system, which in turn allows more sophistication in the painting process and better visual outputs.
Moreover, we propose an additional layer (the second source of information) to the existing semantic systems, which utilizes the knowledge about the affective value of painting parameters, in order to make more expressive painterly outputs. The repository of such affective knowledge needs to be built by conducting controlled user studies. In Section 8, we present the results of a user study which examines the effect of color palette. The color palette is an important parameter in a painting; the appropriate choice of the colors can strengthen the emotional content of the painting (e.g., the color palette in scream by Edward Munch). Knowledge about the emotional impact of color palettes on facial expressions helps a painterly system to choose the colors based on the desired effect. Similar studies on other painting parameters such as the size and shape of brush strokes enhance the output of a painterly system.
As mentioned above, parametric authoring tools both in the animation and depiction side help in better control over the painting process and lead to more appealing results. In the next section, we describe a parameterized facial animation authoring tool, a parameterized painterly rendering system proposed by past researchers [14, 15], and a method to integrate them into a knowledge-based painterly animation system.
4. A Parameterized Facial Animation Authoring Tool
iFace is a parameterized facial animation authoring tool, which was developed as a set of .NET components in C# [14]. iFace is based on the definition of Face Multimedia Object (FMO) which encapsulates all the functionality and data related to facial actions. It simplifies and streamlines the creation of facial animations and provides programming interfaces and authoring tools for defining a face object, its behaviours, and animating it in static or interactive situations. Four spaces of geometry, mood, personality, and knowledge in the framework cover the appearance and the behavior of a face object.
Geometry
This parameter space is concerned with the physical appearance and movements of facial points. It is grouped into three levels of increasing abstraction: vertices, facial features/parameters, and facial regions. A user can manipulate the face on all three levels. Higher levels in the hierarchy have access to the lower levels internally. This architecture helps in hiding the details from users and programmers.
Mood
This space is concerned with short-term emotions of a character and modifies how an action is animated.
Personality
This space deals with the long-term individual characteristics and is mainly based on the facial cues associated with a given personality type. The association between facial cues and personality types was derived from the past user studies [16].
Knowledge
This space is concerned with behavioural rules, stimulus-response association, and the required actions. This space uses an XML-based language called FML. FML is a Structured Content Description mechanism specifically designed for face animation. An FML file can incorporate a sequence of actions by a head model or a set of parallel and event-based actions, which can be played back in iFace. Thus, FML provides iFace with a built-in functionality for communicating the emotional content of a facial sequence.
Thus, FML is useful for(i)hierarchical representation of face animation from simple moves to stories, (ii)timeline definition of the relation between facial actions and external events. This includes parallel and sequential actions and the choice of one action from a set based on an external event.
In addition, iFace provides a keyframe editor toolkit for creating and manipulating a key frame facial animation sequence based on MPEG-4 facial parameters.
5. A Parameterized Semantic Depiction System
For an NPR system to be flexible and to generate a wide variety of outputs and styles, it should provide the user with a set of controls. In such a system, the user or algorithm can adjust the parameters based on the input image and the desired effect/style. The “Painterly” system by DiPaola [15] is a parameterized NPR toolkit developed for portraiture paintings. It takes advantage of portrait and facial knowledge in the NPR process.
The Painterly program has been developed based on the collected qualitative data from art books and interviews with portrait painters. The qualitative data about how portrait artists achieve the final painting was then translated into a parametric model and incorporated into the painting algorithm (see Figure 1).
The Painterly system can differentiate and recognize areas of character scene, thereby approximating the cognitive knowledge of a human painter. This functionality enables a technical user to produce sophisticated paintings by applying different painting techniques on different parts of an input image.
The parameters in the Painterly fall into functional types: (1) constant parameters such as brush size, (2) method parameters such as the type of palette to use, and (3) process method parameters which guide the process flow of the other parameter types.
The Painterly system receives the set of painting parameters in an XML format together with the name of the input image and a number of matte files. Then, the system renders the input image in an oil-painting style (see Figure 2).
6. Integration of the Animation and the Painterly Systems
In this section, we explain how an exemplary parameterized animation system such as iFace can be used along with a parameterized painterly system (e.g., Painterly) to generate an NPR output.
The animator creates a facial sequence by setting appropriate values for the facial parameters in the key frame editor of iFace. The resulting keyframe file from this step contains values for facial parameters such as eye and mouth parameters over time. In a similar way, the animator sets emotion parameters over the timeline of the sequence using iFace with the new emotion-based extensions to our FML language. The emotion information with timing is saved in an FML file. Algorithm 1 shows a sample of FML which describes the surprise emotion. According to the file, the emotion starts at frame 20 and ends at frame 48. The “timing” tag provides information about the onset, duration, and the offset of the emotion, as well as the highest intensity that the emotion reaches on a scale of 0–100.
| ||||||||||||||||||||
The Painterly rendering program for animations reads the information in the keyframe file and the FML file. Then, the Painterly program sets the appropriate painterly parameters to intensify the surprise emotion for frames 20 to 48. Knowing the onset, duration, and offset frames for the emotion, the Painterly program can change the painterly parameters for surprise gradually over the sequence of frames to achieve a more artistic output. As shown in Algorithm 1, the animator can optionally specify the configuration (.csv) file for the painterly parameters. If they do so, the specified frames are painted using the parameters in the configuration file. Figure 3 shows a diagram on the integration of the animation and painterly systems.
In this method, the animator determines the curve for emotions over time in iFace, which then will be used by the depiction system for painting the sequence. Furthermore, if the user wants to adjust the lower-level parameters in the depiction system, they can draw the curve for the values of any desired painting parameter such as the brush size over time. Then, the Painterly system will adjust other parameters based on the provided information and the emotional content of the frames.
7. Sample Images with Painterly Effects
We designed five-color palettes based on color combinations provided in da Pos and Green-Armytage's paper [1]. Three of the color palettes were designed to enhance the four basic emotions of anger, fear, joy, and surprise (see Figure 4). According to the results of that study, colors for fear are mainly desaturated, while the designers' choice of colors for anger was mainly red and black. Therefore, we designed the dark-red-color palette for the anger emotion and the cool-light-color palette for the fear emotions. Also, the result of da Pos and Green-Armytage's study [1] suggests similar color choices for joy and surprise by the designers. Thus, we designed one-color palette for these two emotions, the yellow-orange-color palette. The two other color palettes were added for the purpose of our user study. Cool-dark color palette was added to analyze the effect of brightness in the data. Finally, the natural-color palette uses the same colors as the original 3D frame and is considered as a baseline for our analysis. Figure 4 shows the color palettes with the corresponding emotions as captions.
Figure 5 shows one frame with the computer graphics style in the middle and two painterly renderings of that image in different color and brush types in both sides. The left image conveys sadness, while the right image demonstrates a more active emotion; a combination of pain and anger.
8. Preliminary Study and Applications
A knowledge-based painterly system for faces requires knowledge about the emotional effect of painterly parameters such as color palette and brush strokes applied in the context of face and facial expressions. Such knowledge is obtained by careful user studies.
Thus, here we describe a preliminary study that we conducted on the effect of color palette and painterly rendering on facial expressions. The question of interest is whether different color palettes can impact the emotional intensity perceived for a given expression or not. Although humans can express a wide range of facial emotions, six facial expressions are known to be universal. Also, there are well-defined facial configurations for these expressions. In addition, many other emotions are a combination of these basic emotions. Our study examines four of these six universal emotions: anger, joy, fear, and surprise.
Materials for the study were still images from animated facial sequences of the four mentioned universal expressions in medium- and high-intensity levels. The image for each of the facial expressions was painted by the Painterly program in all the five-color palettes (see Figure 4), totalling to 40 images (). We call the color palette designed for an emotion the congruent palette for that emotion. All other color palettes are considered incongruent for that emotion. 15 university students (6 female, 9 male, between 20–35 years old) from various cultural backgrounds (Asian, Middle East, and North America) participated in a 30-minute testing session. During the session, the participants watched all 40 images for three times (in random order) and chose the intensity perceived for each facial expression on a Likert scale of 0 to 6 (zero being the neutral position and six the highest intensity for the emotion).
Analysis of the results with a within-subject ANOVA test with three factors of emotion type (4 basic emotions), congruency (3 levels: congruent, incongruent, natural), and intensity level (medium and high) yielded a main effect of congruency (, ). The results of Bonferroni's post hoc test showed that for all four emotions, the mean rating for congruent color palette to each emotion (, ) was significantly higher than the mean intensity rating for the incongruent color palettes (, ). This result indicates that an appropriate color palette increases the intensity of the emotions perceived by the viewer. These preliminary results support the idea of using painterly rendering to affect the emotional content of facial expressions. Also confirmed is the importance of color palettes as substantial tool for emotional depiction. These results also provide information about the affective value of different color palettes for facial expressions.
The animation and movie industries usually use a variety of techniques such as lighting and music to boost or flavor the emotional impact of a scene. An exemplary feature of a soundtrack, which corresponds to emotions, is tempo. Music with fast tempo induces excitement, while a slow tempo conveys calmness or sadness [17]. Movie and animation sequences usually require carefully composed soundtracks to involve viewers in the story effectively [18]. Lighting is another effective tool for adding meaning and emotion to characters and scenes [19]. The number, colour, direction, and the quality of a scene's lightings have a direct impact on how the characters and scenes are perceived, and consequently on the viewers' emotional gain from the movie or animation sequence. As an example, well-lighted scenes induce a pleasant mood, while dark scenes are usually associated with sadness and mystery [20]. These techniques have been used for many years as effective tools for adding to the emotional impact of movie and animation sequences. We believe that the proposed technique provides the animators with an additional tool to intensify the emotional content of a scene.
Rapidly generated character sequences in games and commercials use different algorithms and artificial intelligence techniques for adding expressiveness to facial and body movements in those sequences. The proposed technique in this paper requires very little manual work by the game designer and is an efficient way for achieving better visual results in facial character sequences. In addition, the proposed technique modifies the visual aspect of the scene in the final stage of the production. Thus, it can accompany almost any other method to achieve higher quality for the animated sequence.
Lastly, having knowledge about the affective content of the NPR elements not only has advantages for the gaming and animation industry and any other application areas of the NPR systems, but also provides psychologists with a better picture of the human cognitive processes [4].
9. Conclusion
In this paper, we propose a knowledge-based painterly rendering technique to enhance emotional expressiveness of game and animation character sequences. This approach is especially useful for facial sequences because of the well-structured standards for faces. We detail our parameterized animation and painterly authoring tools and our extensions and method to integrate them. The Painterly system receives information about positions of facial features and the emotional content of the sequence from the animation system. Having this information, the system utilizes its affective knowledge of different lines and colors to apply painterly effects depending on the content of the sequence. Our preliminary study shows that such informed painterly rendering increases the intensity perceived for facial expressions in still images. We continue our studies on the effect of color on animated character sequences (dynamic stimuli). One anticipated direction for future research is using a continuous measurement for capturing users' responses. For instance, using a continuous emotion space such as the PAD space could lead to capturing slight variations in perceived emotion type and quality caused by changing colors. Examining the effect of different painterly textures by using various brushes is another possible direction for future studies. Similar types of studies on the effect of painterly rendering for 3D characters and avatars can provide better understanding of affective response to visual stimuli and thus enhance the proposed knowledge-based approach.