Abstract
Color is a main channel to express emotions of an image. Selecting a good color scheme with color harmony and proper emotional expression is very important in many fields, such as design. There are many color palette tools to help users search, create, edit, save, and share color themes, but few tools are from the perspective of emotional expression and explicitly address the emotional effects of colors. In this paper, a color hunt system with affective words called ColorEmo is developed. Multiple types of input, including affective words, affective categories in image-scale space, and main colors, are allowed for users with accurate emotional description. Based on the dataset with 428,924 color themes, the system provides rich candidates. The system is designed for user research experiments. The user-friendly interactions are offered for easy color modifications. The affective matching and color harmony are evaluated in real time while changing colors. This system can be used in many scenarios of designs and applications.
1. Introduction
The use of colors is everywhere. With the proliferation of Cyberphysical Systems (CPSs) [1–4], finding a set of colors for matching different scenarios of practical applications can be tricky. Much of the feelings about an image are expressed by colors even if the context is also important. Colors can make us feel happy or sad or nervous or relaxed. Based on the study of human color perception, there are many color harmony rules to describe the matching relationship among colors in a color theme, and some color schemes are provided. For ordinary users, even designers, it is difficult to find the proper color theme when using color palettes. So, there are many color palette tools to help users find color themes from the perspective of different user demands. But few color palette tools are from the perspective of emotional expression and explicitly address the emotional effects of colors.
A color theme is a template of colors to describe the color composition of an image [5]. The most commonly used color themes are the 3-color theme with three colors and 5-color theme with five colors. In this paper, a 5-color theme with richer expression is adopted. To quantify emotions and establish the relationship between emotions and color themes, the image-scale space with two dimensions warm-cool and soft-hard is adopted [6], which has been applied to many applications [7–9].
In this paper, a color hunt system called “ColorEmo” is built to find proper color themes with affective words. First, how to represent the user’s emotional expression as an accurate input is a tough question. The most intuitive input type is an affective word. Since it is sometimes difficult to find a word accurately, some emotional categories and affective words distributed in the image-scale space are provided to give instructions to users. Based on the affective input, abundant color themes are provided based on the dataset with 428,924 color themes and 180 affective words labeled by image-scale coordinates. The semantic similarity calculation is adopted to give the image-scale coordinates for any affective words. The interactions in ColorEmo are designed for easy operations, and the information is kept on one page as much as possible to avoid the cognitive load caused by page jumps. When users modify the colors, the affective matching and color harmony evaluation results based on the assessment algorithm are shown in real time. Furthermore, the recommendation and collection functions are provided in the system.
The main contributions include the following: (i)A complete interactive system called “ColorEmo” is developed based on user research to find proper color themes based on affective words(ii)An assessment algorithm is proposed to evaluate the affective matching and the color harmony of color themes, giving instructions when the users tune colors
The rest of the paper is organized as follows. Section 2 gives related work. Section 3 describes the user research for color hunt, and seven functions are obtained for system design. Section 4 shows the system design and dataset of ColorEmo. Section 5 presents the ColorEmo system and the assessment algorithm. Section 6 finally draws the conclusions.
2. Related Work
2.1. Color Palette Tools
There are many color palette tools to help users search, create, edit, save, and share color themes. Most of these tools provide popular colors and color palettes by the way of waterfall flow. The color palette contains five colors with HEX, HSV, or RGB representations, such as Color-hex [10], COLOURlovers [11], and Coolors [12]. Users can select suitable color themes to match with their application expectations, create their own palette based on the selected color theme, appreciate, share, and save other color themes.
Searching hundreds of thousands of color themes one by one is time-consuming. Some tools apply rules or provide semantics to narrow the search results. The color harmony rule is commonly used, which contains analogous, monochromatic, triad, complementary, compound, shades, and custom. One of the most authoritative tools Adobe Color [13] applies the color harmony rule to restrict the range of color adjustment. Color Hunt [14] provides color palettes arranged by new, hot, popular, and random. Color Palettes [15] divides color themes into four categories, warm, cool, pastel, and contrasting, in which each color theme is extracted from an image and contains a description. “I want hue” [16] provides the differentiation report and color vision deficiency by calculating the distances between all color pairs in the palette with some color difference metrics, such as Euclidean distance, Protanope distance, Deuteranope distance, and Tritanope distance.
Despite there being so many color themes related to different aspects, very few color themes explicitly address the emotional effects of colors. In this paper, a system is proposed to search color themes with affective words.
2.2. Color and Emotions
Colors can make us feel happy or sad or nervous or relaxed. How to quantitatively describe emotions is the key question in affective computing. The quantitative methods of emotions can be divided into two categories, the category method and the dimension method. In the category method, the emotions are divided into some categories, such as the six basic emotions proposed by Darwin et al. (happy, sad, surprise, fear, anger, and disgust) [17] and four positive emotions (amusement, awe, contentment, and excitement) and four negative emotions (anger, disgust, fear, and sad) proposed by Machajdik and Hanbury [18]. In the dimension method, the emotions are distributed in the emotional space, such as the PAD model with pleasure, arousal, and dominance and three orthogonal dimensions [19].
One of the most famous two-dimensional emotional systems widely used in art design is the image-scale space proposed by Kobayashi [6, 20]. It has two dimensions: warm-cool and hard-soft and contains some categories, such as romantic and clear. Based on the long-term psychophysics experiments, Kobayashi collected the 180 affective words and 490 color themes, which are distributed on the image-scale space, shown in Figure 1. The image-scale space is commonly used for emotional analysis in some applications [7–9].
2.3. Color Harmony
In art design, there are many color harmony rules and color harmony algorithms [21, 22]. Itten introduced a kind of color wheel to describe color harmony, especially on hue [5]. For example, complementary color schemes are that colors opposite each other on the color wheel, like red and green. Monochromatic color schemes are based on different tones of the same color. Analogous color schemes use colors that are next to each other on the color wheel. Triadic color schemes use colors that are evenly spaced around the color wheel. Rectangle or tetradic color schemes are four colors arranged into two complementary pairs. Matsuda summarized a set of 80 color schemes [23]. In this paper, we refer to these color schemes and quantify the color wheel to describe the unified degree in the assessment algorithm.
The hue channel mainly affects the color harmony [5]. Tokumaru et al. described how saturation and luminance channels affect color harmony [24]. In the assessment algorithm of this paper, the color harmony degree based on saturation and luminance channel is described as “contrast” and “saturation” by using rules as simple as possible.
The color harmony rules have been used in many color-related applications [25–27]. Cohen-Or et al. applied the harmonic models for image color harmonization [28]. Classic color harmony theory combined with a new deep learning method is used to evaluate the beauty, sentiment, and remembrance of art [29].
3. Method
As a designer, it is important to understand the psychological effects colors might have on an average person or your client’s target audience. In the user research, we use the in-depth interview method to collect opinions and suggestions from designers for the color hunt. To collect information from the perspective of different design fields, designers in four areas (graphics, UI, original painting, and photography) are our target subjects of our in-depth interview.
The content of the in-depth interview includes two aspects: first, the interviewer directly asks the subjects questions; second, the subjects provide their commonly used color palette tools and evaluate them. The questions are summarized into ten categories as follows. (1)How do you select the color theme in normal work(2)Do you think your color theme selection habits are good? What problems have you encountered when selecting color themes(3)How do you filter the candidates when selecting a color theme? Describe the factors in detail(4)Which color palette tools do you often use(5)Do you know the color harmony rules? Give the rules in detail(6)Do you consider the emotional factors when selecting color themes(7)What functions would you expect if there is an emotion-based color hunt tool(8)Do you usually collect some good color themes(9)Do you collect some beautiful pictures to extract color themes from these pictures(10)What is the interactive process when selecting color themes in the color palette tools
The in-depth interview results are summarized, shown in Figure 2.
The user demands and the expected interactions are extracted based on the results. Finally, the seven functions of the ColorEmo system are obtained, which are the basis of the system design. We introduce how to obtain each system function briefly as follows.
3.1. Input Types
The subjects mention that there are few tools to search color themes by emotions, but they want such a tool. Besides, they do not know how to give input. So, “emotion-based input” is obtained as the user demands. Multiple types of input are expected in the system interaction. Therefore, the input types should be well designed.
3.2. Color Theme Dataset
There are not enough candidate colors in some color palette tools. Some color themes lack science and authority. When designers select a favorite color theme, they do not know whether it meets the public’s preference. An authoritative color theme dataset is very important, which is the data foundation of the system. In our system, there should be abundant and popular color themes with accurate labels.
3.3. Candidate Color Theme Display Form
When using the color palette tools, browsing and selecting color themes are the most time-consuming. And getting a satisfactory color theme requires fussy interactions. The subjects expect simple interactions. The key to reducing the number of operations is the user-friendly candidate color theme display form.
3.4. Modify Colors
There are some inflexible ways to modify color themes. In some tools based on certain color harmony rules, candidates cannot be modified to arbitrary colors to ensure that the harmony rules are not broken. The flexible modification of candidates is needed, and there are no restrictions on color modification. Based on the proposed assessment algorithm, when the emotion of the modification is far from the input affective word, the assessment value will be very low to remind users. The need for interaction is that what you see is what you want. It is possible that an unintentional modification produces a very satisfactory result.
3.5. Obtain Color Theme from Uploaded Images
The subjects want to extract a color theme from their favorite images, which is also a function in many tools. In our system, we keep this function.
3.6. Assessment Algorithm
Most users, even designers, lack color scheme knowledge. They are hard to distinguish the good and bad candidate color themes, which are always mixed in most systems. Furthermore, after they modify the candidate color theme, they do not know whether it is in line with the emotion and color harmony rules. So, the evaluation criteria of color themes are needed. When users modify the colors, the evaluation results should be shown in real time. Therefore, the assessment algorithm to evaluate how well the color theme matches the emotion and color harmony rules is the key function of the system.
3.7. Discover and Collect
The subjects want the system to recommend good images and color themes and let them collect good candidates. To discover good images and color themes and provide a collection function is a must-have feature in a complete system. The interaction for discovering good candidates should be easy browsing. The interaction for collection is a one-click operation.
4. System Design
4.1. System Flow
According to the summarized functions based on the user research, we design the ColorEmo system, and the system flow is shown in Figure 3. The color theme generation flow is proposed to find the appropriate color theme with user demands.
The input of the system can be an affective word and a main color. The input affective word is selected from the image-scale space or directly entered. Based on our previous work [8], 428,924 color themes labeled by image-scale coordinates are collected in the dataset of ColorEmo. 180 affective words labeled by image-scale coordinates [6] are adopted. To give feedback for any user input, we use the semantic similarity calculation based on Hownet to obtain the coordinates of the input based on the 180 words. So, for any affective word, hundreds of color themes are filtered from the dataset. Besides, color themes can also be obtained from uploaded images like most color palette tools.
For simple interaction, hundreds of color themes are displayed by a slider instead of waterfalls to avoid the cognitive load caused by page jumps, which will reduce the number of operations. Furthermore, display color schemes by geometry for each color theme intuitively. To modify colors flexibly, we use the color wheel to adjust colors roughly and to modify color values for fine adjustment while visualizing their harmonious relationship in the wheel. While changing colors, the evaluation results of the affective matching and the color harmony are shown in real time based on our proposed assessment algorithm. Therefore, the color values, the distribution of five colors in the wheel, the color scheme result rendered on the geometry, and the evaluation results are all obtained in real time, which realize that what you see is what you want.
When the satisfied color theme is obtained after modification, you can collect it. Besides, the system recommends lots of good images and their color themes.
4.2. Dataset
In our previous work [8], we collect more than 420,000 color themes from Adobe Kuler [30] (44,986 themes) and COLOURLovers [11] (383,938 themes) which are famous online communities to create and share color designs. The mapping between these color themes and image-scale space is built [8]. In this paper, we use mapping to find abundant color themes based on affective words. So, in the ColorEmo system, there are more than 420,000 color themes labeled by the two-dimensional coordinates in the image-scale space.
When users input an affective word, the two-dimensional coordinates of the word in the image-scale space are calculated first. Then, the color themes are obtained based on the image-scale coordinates from the above dataset. When users select an affective word from the image-scale space, the image-scale coordinates are obtained directly from Kobayashi’s research [6]. If users input an arbitrary affective word which is not in the set of the 180 words collected by Kobayashi [6], the image-scale coordinates are calculated based on Hownet [31]. Hownet is a Chinese dictionary to reveal the relationship between concepts and the attributes of concepts, which is similar to WordNet in English [32]. We use Hownet to calculate the semantic similarity between the input word and the 180 words. The top most similar words are selected, and the coordinates of the input word are the weighted average of the coordinates of the words. After the experiments, is set to 3.
5. ColorEmo System
The homepage of ColorEmo is shown in Figure 4. In the navigation bar at the top of the homepage, there are four buttons corresponding to four pages. The default page is the “generation” page to find and generate the appropriate color theme to meet the emotional expression of the input. The “discover” page shows the recommended color themes and images. The “assessment” page evaluates how well the color theme matches the emotion and color harmony rules. The “my pick” page shows all the user’s collection while logged in.
In the “generation” page, there are two parts for user input: the affective word selection part and the main color selection part. In the affective word selection part, users can enter an affective word in the text box or select an affective category in the image-scale space. When describing emotions, it is sometimes difficult to find a word accurately. So, some affective categories [6] distributed in the image-scale space are provided to give users an intuitive concept to help them quickly describe the emotional expression they want. The color of the category is the main color distribution of this emotion, and the shape represents the location distribution. The main color can be selected on the right color table.
When a category is clicked, some affective words in this category are shown, taking the “Clear” category as an example shown in Figure 5. There are 180 words in total collected by Kobayashi [6].
After entering an affective word, press the “next” button in the bottom right corner of this page. The specific color theme generation page is shown in Figure 6. The layout of the color theme generation page focuses on the spatial and temporal relationship of the interaction [33].
For each function summarized from the user research, the interface of the system is carefully designed, highlighted by rectangles and numbers, which is consistent with the labels in Figure 2. Figures 4 and 5 are designed for system function 1 “input types.” The color theme dataset of function 2 is the data foundation of the system and is not directly displayed on the interface.
Instead of using a waterfall to show all the candidates, a slider is adopted to carry them shown in the box labeled by 3. Users slide the slider left and right to see all the candidates. If you like it, press the add (“+”) button to collect it temporarily. All the collections are displayed in the box under the slider. Having all the candidates hidden behind the slider is to keep the information on one page as much as possible to avoid the cognitive load caused by page jumps. Besides, to show the color scheme effect of each color theme intuitively, we adopt an abstract geometry with random coloring by the color theme.
The main area of the interface is used for color modification. The color wheel is the basic tool for combining colors. In the ColorEmo system, the color wheel has two functions, one is to adjust colors roughly, and the other is to distribute the five colors in the wheel to visualize their harmonious relationship. Besides, the color values of each color under the color wheel are used for fine adjustment. The values in different color spaces are provided. Modification results are displayed in real time, which achieves that what you see is what you want.
Similar to most color palette tools, color themes can be extracted from images. In the ColorEmo system, the entrance of reference picture uploading is shown in the upper left of the interface.
Based on the assessment algorithm introduced in detail in subsection “assessment algorithm,” the affective matching rate is shown in the assessment part in the right of the interface. Contrast, saturation, and unified evaluate how well it matches the color harmony rules. The color harmony value is quantified as five levels, namely, very low, low, medium, high, and very high. The evaluation results are shown in real time to give users instructions while changing colors.
If you like the color theme, you can click the “pick” button in the upper right of the interface. The color theme appears to the left of the button again for the user to confirm this color theme conveniently.
The “discover” page of the ColorEmo system is shown in Figure 7. There are lots of images, and their color themes are automatically recommended based on the user’s input.
The “my pick” page shows all the user’s collection while logged in. The interface is in the form of tiled favorite color themes and images, which is similar to the “discover” page.
5.1. Assessment Algorithm
The assessment algorithm includes two parts, the affective matching rate and the color harmony degree. Based on the color theme-image-scale coordinate relationship model in our previous work [8], the image-scale coordinates for any modified color themes can be calculated. The affective matching rate is the Euclidean distance of coordinates between the input affective word and the modified color theme, which is normalized to be between 0 and 1.
When applying color harmony theory to an application, a very specific theory is adopted. In this system, we adopt the color harmony theory as simple and versatile as possible to get the universal color themes, which can be subsequently applied to many applications. Contrast, saturation, and unified evaluate how well it matches the color harmony rules from luminance, saturation, and hue and three aspects in the HSL color space. The contrast degree describes the contrast of both luminance and saturation of the five colors in a color theme. The saturation degree reflects the average of the five saturations. The unified degree describes the distribution and contrast of the hue channels of the five colors. The color harmony value is quantified as five degrees, namely, very low, low, medium, high, and very high.
Based on Itten’s contrasts in color [5], the color contrasts include the pure color (hue) contrast, the light-dark contrast, the cold-warm contrast, the complementary contrast, the simultaneous contrast, the contrast of quality (color saturation), and the contrast of quantity. In this paper, the contrast is a combination of the light-dark contrast and the contrast of quality, which is quantified by luminance and saturation. Without considering black, white, and gray colors, the maximum absolute value of the difference in saturation and luminance between two of the five colors in a color theme is set as . If is less than 45, the contrast degree is very low. If is between 45 and 60, the contrast degree is low. If is between 60 and 75, the contrast degree is medium. If is between 75 and 90, the contrast degree is high. If is more than 90, the contrast degree is very high.
The saturation degree evaluates the average saturation of five colors. The calculation method is to extract the saturation channel of each color and to take the average value of the five saturations. If the average is less than 20, the saturation degree is very low. If the average is between 20 and 40, the saturation degree is low. If the average is between 40 and 60, the saturation degree is medium. If the average is between 60 and 80, the saturation degree is high. If the average is more than 80, the saturation degree is very high.
In the RYB (or subtractive) color model, red, yellow, and blue are the primary colors. Green, orange, and purple, created by mixing two primary colors, are three secondary colors. Six tertiary colors are created by mixing primary and secondary colors. If the colors in a color theme include three different primary colors, the unified degree is very low. The unified degree describes the distribution and contrast of the hue channels of the five colors, which is less intuitive than luminance and saturation. The largest angles (180 degrees) in the color wheel are divided into five parts. The maximum absolute value of the difference in hue between two of the five colors in a color theme is set as . If is less than 36, the unified degree is very high. If is between 36 and 72, the unified degree is high. If is between 72 and 108, the unified degree is medium. If is between 108 and 144, the unified degree is low. If is more than 144, the unified degree is very low.
6. Conclusion
In this paper, a complete system called “ColorEmo” is proposed to hunt colors based on affective words. More than 400,000 five-color themes with image-scale coordinates and 180 affective words are data support of this system. Based on the user research experiments, seven system functions are summarized to find the accurate color theme with user demand. In the interactions of the system, slider-based color theme display, geometry-based color scheme display, and color wheel-based modification are designed for user-friendly color adjustment to reduce the cognitive load. While modifying colors, the affective matching rate and color harmony degrees are shown in real time based on the proposed assessment algorithm. The recommendation and one-click collection functions are also provided in the system. In the future, we will design more applications by incorporating other fields [34–37] based on the system.
Data Availability
Data is not open.
Conflicts of Interest
The authors declare that they have no conflicts of interest.