Detecting Happiness Using Hyperspectral Imaging Technology
Hyperspectral imaging (HSI) technology can be used to detect human emotions based on the power of material discrimination from their faces. In this paper, HSI is used to remotely sense and distinguish blood chromophores in facial tissues and acquire an evaluation indicator (tissue oxygen saturation, StO2) using an optical absorption model. This study explored facial analysis while people were showing spontaneous expressions of happiness during social interaction. Happiness, as a psychological emotion, has been shown to be strongly linked to other activities such as physiological reaction and facial expression. Moreover, facial expression as a communicative motor behavior likely arises from musculoskeletal anatomy, neuromuscular activity, and individual personality. This paper quantified the neuromotor movements of tissues surrounding some regions of interest (ROIs) on smiling happily. Next, we selected six regions—the forehead, eye, nose, cheek, mouth, and chin—according to a facial action coding system (FACS). Nineteen segments were subsequently partitioned from the above ROIs. The affective data (StO2) of 23 young adults were acquired by HSI while the participants expressed emotions (calm or happy), and these were used to compare the significant differences in the variations of StO2 between the different ROIs through repeated measures analysis of variance. Results demonstrate that happiness causes different distributions in the variations of StO2 for the above ROIs; these are explained in depth in the article. This study establishes that facial tissue oxygen saturation is a valid and reliable physiological indicator of happiness and merits further research.
There is a growing interest in the more positive emotions such as happiness [1–3]. Moreover, a state of happiness can overcome negative emotions such as stress [4, 5]. When people are engaged in social experiences that make them feel happy, people may exhibit measurable physiological characteristics such as blushing, facial expression features such as smiling, and body behavior such as dancing. With the exception of facial expression, specifically, happiness presumably includes a greater involvement of, for example, human physiology, psychology, behavior, and other human factors. Comprising a complex mix of behavioral, facial, physiological, and psychological traits, happiness is suspected to play a key role in many fields, including task execution [6, 7], healthcare [8, 9], and teaching and learning [10, 11].
2. Review of the Literature
Various in-depth studies have explored human emotions in terms of facial expressions. Witt and Flores-Mir [12, 13] and Janson et al.  investigated the facial smile paradigm by observing the subjects’ lips and dentition. Arigbabu et al.  investigated smile detection from face images in unconstrained environments. A proposed framework provided a very competitive detection rate and exploited image alignment as an important stage for improving the performance of smile detection. Of course, the studies based on facial expressions mostly are based on the hypothesis that smile is often assumed to indicate happiness, which contradicts the fact that smile sometimes is pretended to deliberately adapt to a certain subjective understanding, which is actually not happiness.
Physiological signals is another significant entity that is studied in the field of emotional research. It is noteworthy that physiological signals are not influenced by human subjective consciousness. In addition, emotional reactions have been found to cause fluctuations in a variety of physiological indicators used to measure nervous activities. Kreibig  summarized 134 studies and found significant differentiation between the following physiological indices: heart rate (HR, the most commonly used indicator), skin conductance (SC), blood pressure (BP), and respiration rate (RR). Using images or videotapes as emotional stimuli, Codispoti et al.  and Bianchin and Angrilli  found that the heart rate (HR) was slower in positive emotional states than in neutral states. Gendolla et al.  and Neumann and Waldstein  reported that significantly higher systolic blood pressure (SBP) occurred in negative emotional states rather than in positive states. Using the multivariant correlation method, Wen et al.  analyzed the affective physiological changes in multisubject galvanic skin response (GSR), the first derivative of GSR (FD_GSR) and HR, and obtained an overall accuracy rate of 74% for quinary classification of amusement, anger, grief, fear, and the baseline state. Von Leupoldt and Dahme  verified that both positive and negative emotions can lead to respiratory resistance because of airway constriction when a picture stimulus pattern is introduced. Vianna and Tranel  found a higher positive correlation (r = 0.64) between the peak amplitude of the electrogastrogram (EGG) and the degree of arousal assessed subjectively by film-elicited methods. Pavlidis et al.  quantified the perspiration responses in the perinasal area—regarded as a physiological phenomenon. In addition, these authors introduced thermal imaging (TI) technology for the unobtrusive detection of perspiration signals.
Traditionally, various classic physiological indicators, including HR, BP, GSR, respiration, and EGG, require diverse contact sensors to measure changes in body responses. The inconvenience of applying these measuring techniques to certain fields in particular motivated researchers to investigate more reliable assessment methods using noninvasive technology. Thus, more attention has been paid to contactless detection technologies such as TI and hyperspectral imaging (HSI). These instruments are unobtrusive and require minimal interaction between subjects and examiners. The two imaging technologies, TI and HSI, allow measurement of physiological features such as blood flow, pulse rate, and breathing rate by spectral technologies. TI is directly related to tissue blood flow and eventually detects the changes in skin temperature and perspiration responses of the targeted regions. However, the technology becomes considerably less reliable when the ambient temperature changes suddenly. Consequently, we selected an alternative technique, HSI, to measure the affective features.
3. HSI Technique
HSI enables the imaging of a scene in hundreds of contiguous, narrow wavebands, with a bandwidth of approximately 10 nm, particularly in the visible and infrared regions of the electromagnetic spectrum, to form image cubes with both spatial and spectral dimensions [25, 26]. Due to the nature of light reflection by the object in the scene, HSI can capture the intensity of the reflected light within a narrow slice of wavebands across the whole spectrum and transform the extent in terms of pixels. Compared with conventional photography, HSI uses a narrow bandwidth for spectral sensing, enhancing the color discrimination ability. Its power of material discrimination is the reason why HSI is used as the primary technique in this research.
In this paper, HSI instrument employed records the imaging only for one row line of an object at a time—this only requires the scanning movement of HSI camera. Each of the recording process consists of the imaging under different wavebands within the whole spectrum. Next, it constructs a two-dimensional image (spatial and spectral axis). With a predetermined scanning frequency, it can obtain the total image slices of the object. Then, a 3D cube can be created by stacking all the 2D slices in sequence. Therefore, depending on the configuration, HSI can be introduced to remotely sense and discriminate blood chromophores from body tissues; the amount of oxygenation within the blood is subsequently quantified using an optical absorption model.
To recap, the HSI method is a noncontact detection technology for detecting physiological signals. Because of the power of material discrimination, HSI technology has increasingly been explored in various areas of research [27–29]. As a technology of longer spectrum width and higher imaging precision, HSI was utilized directly to remotely sense StO2 [30–32]. Thus, HSI is convenient to both subjects and testers. Additionally, this method can effectively counteract the aforementioned defects remaining in TI, and most important of all, it is not easily affected by environmental uncertainties.
4. Purpose of the Study
This study is based on the hypothesis that a happy state of mind results in measurable physiological changes that can ascertain whether someone is experiencing genuine happiness. The authors introduce facial tissue oxygen saturation (StO2), an innovative physiological measure to evaluate the effects of happiness on facial tissues by using HSI technology and test the reliability and repeatability of these measures.
The aims of this study were to examine (1) whether happiness causes fluctuations of affective signal (StO2) in facial regions, (2) the affective meanings of the changes in the neurophysiological movements of tissues using StO2 as a measure, and (3) the relative distribution pattern of StO2 for six facial regions of interest (ROIs)—forehead, eye, nose, cheek, mouth, and chin—when smiling happily.
This section describes the population and sample for the study, research design, and data analysis. The study used two distinct self-reporting questionnaires and an interview with the expert to arrive at the sample from the population for the study. The process is explained in the following sections.
5.1. Population and Sample
Twenty-six healthy young-adult volunteers (16 females) participated in this study. The sample size for determining the main and interactive effects of emotion and ROI was computed to achieve a priority power = 0.80 and detection of medium effect sizes Cohen’s f, calculated by GPower. None of the participants had known functional insufficiencies, and all participants were selected from Southwest University of Chongqing, China. After receiving a full description of the study, all volunteers gave written informed consent. They were paid a fixed amount (¥30) to participate. The inclusion criteria were as follows: (1) willingness to participate; (2) good dental health with functional dentitions; (3) no congenital orofacial abnormalities; (4) no prior history of neurological or psychiatric problems; (5) ability to follow study instructions; (6) normal or corrected-to-normal vision; and (7) no known facial impairment or orofacial pain that could interfere with facial expression or affective motor behavior. The exclusion criteria were predefined based on an earlier study by Kokich et al.  and additionally included the following: (1) an alexithymia score greater than 62 on the Toronto Alexithymia Scale (TAS) ; (2) decayed or missing teeth, sans 3rd molars; (3) orofacial defects or abnormalities; (4) prior history of organic or congenital diseases; (5) reported neuromotor or musculoskeletal impairments that would interfere with smile production (e.g., as described by ); and (6) reported use of medications with known motor side effects, e.g., abnormal involuntary movements or extrapyramidal symptoms. Sample characteristics are displayed in Table 1.
5.2. Research Protocol
The study protocol was approved by the Academic Committee of the Institute of Signal and Information Processing at Southwest University, which monitors the ethics of research involving human subjects. This study was conducted in accordance with the Declaration of Helsinki revised in 1989. Based on the protocol, data from three female participants were removed from the analyses (two persons are self-reported having the alexithymia scores of TAS greater than 62; another person reported with poor sleep at the night right before the experiment). All participants were instructed to read a brief description of the research and sign an informed consent form.
Each participant was informed about the three qualifier tests before the commencement of the study. The qualifier tests comprised of a learning session and a testing session. In the learning session, authors conducted face-to-face interviews with the participants to ensure that they felt comfortable with the procedure and had a full understanding of it.
The data from 23 participants (mean age ± SD = 21.6 ± 1.6), who met the inclusion and exclusion criteria described above, were used for further analysis.
5.3. Questionnaires and Interview
This study used two self-reporting questionnaires and one interview with an expert as qualifiers for the population. The two questionnaires are Toronto Alexithymia Scale (TAS) and Affect Questionnaire .
The TAS —a measure of difficulty in differentiating, describing, and expressing emotions—was utilized to screen and exclude subjects who could not vividly express their emotional states (TAS score > 62). The TAS score of 62 is often considered as a valid, conservative cutoff point estimate of alexithymia in nonclinical populations . Each item was scaled on a five-point Likert scale from 1 = strongly disagree to 5 = strongly agree; the TAS has a reliability of 0.82.
The Affect Questionnaire was used to test whether specific emotions are elicited effectively during arousal tasks. The questionnaire comprises the following 28 items of affect adjectives: happy, delighted, excited, astonished, aroused, tense, alarmed, angry, afraid, annoyed, distressed, frustrated, miserable, sad, gloomy, depressed, bored, droopy, tired, sleepy, calm, relaxed, satisfied, at ease, content, serene, glad, and pleased. Each subject was required to finish the questionnaire prior to, and following, each task. Each adjective item is then rated on a five-point scale (where 1 = very slightly or not at all, 2 = a little, 3 = moderately, 4 = quite a bit, and 5 = very much). The subjects who passed the two questionnaires were then interviewed by an expert trained in FACS.
The interview session was conducted by an expert trained in the facial action coding system (FACS) [38, 39]. The FACS is designed to exclude those subjects who do not show their facial expressions when feeling happiness.
Based on the results of the three tests, members of the population who scored at or above the required score participated in the study. To recap, this procedure was chiefly aimed at ensuring that subjects who participated in the study elicited smiles spontaneously when feeling happiness; subjects who displayed self-control in displaying emotional facial expression were excluded.
5.4. Stimulation Protocol
The emotion stimulation material is an important tool to induce experimental emotion. On the basis of the different channels, the existing emotion stimulation materials may be divided into visual stimulation materials, auditory stimulus materials, olfactory stimulation materials as well as the multimedia materials, and so on. Therefore, emotion stimulation research treated as an extraordinary issue has received more extensive attentions. Along with the deepening of emotion research, Gross and Levenson [40, 40] have found that film can induce the stronger emotional feelings and acquire more cognitive participation as an emotion elicitation way compared with other stimuli such as music, picture, and recall, etc. In this work, therefore, we choose short-length video clips for emotion elicitation.
The elicitation stimuli used in this work are some funny audiovisual video clips which are chosen by prior questionnaire surveys using the nine-point Likert scale. Meanwhile, validation results from 391 subjects also demonstrate that the materials could obviously elicit the human’s emotions in some sense and help them achieve the desired movements . During the selection of the film clips, the film plots which have aroused certain target emotion of the subjects are labeled as emotion-eliciting film plots. Once the experiment begins, the subject can only follow the experiment instructions which have been informed to the subject before. When the labeled film plots occur instantly (i.e., the emotion elicitation frames appear in time), it will trigger the capturing of HSI instrument by the start click operation of the experimenter. Then, in the process of emotion eliciting, the original data will be collected and transferred to a computer disk.
5.5. Experimental Setup
All subjects were seated comfortably in a brightly illuminated and acoustically and electrically restricted room. To capture affective data, an HSI camera (imaging spectrometer V10 E from SPECIM Inc, Finland; CCD from Lumenera Inc, Canada) was placed at about eye level 200 cm near the level of the subject’s Frankfort horizontal plane. Additionally, after being synchronized for videotaping the visual data, a visual camcorder was aligned with the subject’s midsagittal plane to provide a full frontal view of the face. Figure 1 shows the experimental setup. The specified resolution of this camera is 1392 × 1040 pixels with a spectrum range of 400–1000 nm.
The subjects who passed the three qualifying tests were invited to participate in the experiment, which comprised of three sessions: Calm (for baseline data), Happiness Session 1, and Happiness Session 2. Two happiness sessions were employed to evaluate the reliability of the test.
Before the test, each subject was required to rest for about 5 minutes. After the short rest, baseline data were collected while the subject remained “calm.” Then he/she would rest once again—this time for about four minutes. Subsequently, the test required the subject to be prompted to elicit spontaneous emotions of happiness by the stimulus material. Simultaneously, the instrument would collect and transfer the original data to a computer disk. The data transfer would last about four minutes depending on the amount of block data and the processing speed. Meanwhile, the subject was asked to rest quietly. After the data had been successfully saved, the subject would be instructed to embark on another happiness task, but this time with another stimulus material for the purposes of comparative analysis. Once again, the relevant data was stored on a computer disk. The whole experiment typically lasted for about 20 minutes per participant. The experimental procedure is illustrated in Figure 2.
6. Data Collection
6.1. Action Units
In this study, the selected smiles were based mainly on certain distinct facial movements defined as action units (AUs). Most smiles essentially include the following AUs: (1) AU6 (“Cheek Raiser and Lid Compressor”), contraction of the orbicularis oculi; (2) AU10 (“Upper Lip Raiser”), contraction of the zygomaticus minor and levator labii superioris; (3) AU12 (“Lip Corner Puller”), contraction of the zygomaticus major; (4) AU20 (“Lip Stretcher”), contraction of the risorius; and (5) AU25 (“Lips Part”), relaxation of the lips and orbicularis oris in the mouth area. Besides the above AUs, smiles can more or less naturally trigger the movements of other muscles such as the depressor labii inferioris, mentalis, buccinator, masseter, nasalis, procerus, and temporalis because the muscles of the face function as a whole rather than individually. The distribution of muscles implicated in a smile is illustrated in Figure 3. In particular, AU6 (usually referred to as “Duchenne’s marker”) has been highlighted as the primary unit to represent spontaneous smiles or genuine happiness . Indeed, many studies have revealed that AU6 was observed when subjects genuinely experience more positive emotions such as happiness, and these same subjects also generated concomitant changes in neuromuscular movements .
6.2. Regions of Interest
In previous studies, researchers have selected different regions of interest (ROIs) to conduct extensive investigations in their respective study fields, using various advanced technologies. The analytical methods used in these investigations have achieved good results.
Pavlidis et al.  studied stress by measuring transient perspiratory responses in the perinasal area through thermal imaging. The results showed that different responses genuinely existed in human movements resulting from the manifestation of latent neurophysiological mechanisms. Chen et al.  utilized an HSI technique to detect stress in the forehead area. The accuracy for stress recognition from baseline using a binary classifier was 76.19% and 88.1% for the automatic and manual selections of the classifier threshold, respectively. Fischer et al.  analyzed the differences in the muscles (frontalis, nasalis, and orbicularis oris) between the lower face and upper face using focal transcranial magnetic brain stimulation. Kim and Provost  investigated the temporal characteristics of specific ROIs such as eyebrow, cheek, forehead, and mouth. This led the authors to conclude that combining different ROIs enhanced the overall accuracy of the findings.
6.3. HSI Data
Based on the AUs associated with the smile, the authors considered both the neuromuscular movements and the previous research findings of the characteristic correlations between ROI and psychophysiological reactions. A detailed correlation among ROIs, muscle groups, and AUs is shown in Table 2. The last column of the table lists studies which have explored the psychophysiological responses associated with each ROI. According to the literature, we located the positions of the corresponding ROIs manually for each participant. For example, for forehead M1, we identified its location manually (i.e., from the center of the left eyebrow to the center of the right eyebrow and from the top of the eyebrows to 1/2 the distance from the top of the eyebrows to the top of the head). Other regions of ROIs can be similarly determined according to the associated literature. To this end, 19 ROIs were investigated in this study; these are depicted in Figure 4. Each ROI was marked with a black rectangle. The ROIs contained unequal pixels owing to individual and regional differences.
This study aimed to quantify the neuromotor-controlled movements relating to happiness for further analysis. First, the reflective digit number of the human face using an HSI camera was obtained. According to the literature [57, 58], StO2 is considered as a psychophysiological signature to evaluate the effectiveness of differentiating the emotion states. Then, owing to differences in molar absorptivities in the subjects, the affective data (StO2) was computed using an optical absorption model [30–32, 59], which is cited as a scientific rationale. This study primarily used the Beer–Lambert law to calculate StO2 from HSI raw data. Here, StO2 is defined as the ratio of the amount of oxy-hemoglobin (HbO2) to the total amount of HbO2 and deoxy-hemoglobin (Hb) in body tissues.
The StO2 was calculated for 23 subjects for each ROI, which generates a subset of 23 data points for each ROI. Table 3 shows the distributions of the StO2 variables.
7. Data Analysis
7.1. Descriptive Statistics
Based on a five-point Likert scale, the 23 participants completed their estimate of Affect Questionnaire by rating their arousal responses. In addition to the self-reporting scores of specific affect, we collected further data by calculating the composite scores to represent each of the four quadrants of the Circumplex Model  by summing the self-estimate ratings as follows: Quadrant 1: sadness (distressed, miserable, gloomy, bored, tired, sad, depressed, droopy, and sleepy); Quadrant 2: anger (alarmed, afraid, astonished, tense, angry, annoyed, and frustrated); Quadrant 3: relaxation (calm, satisfied, content, relaxed, at ease, and serene); and Quadrant 4: joy (happy, excited, aroused, glad, delighted, and pleased). The composite groups were mainly utilized to evaluate whether the targeted specific affect was aroused during each task session. Table 4 shows the means and standard deviations of the composite groups and specific affects. Analysis of variance (ANOVA) revealed that the participants displayed significantly more intense feelings on the calm adjectives during the calm session and on the happy adjectives during both happiness sessions than the other three groups (). However, for the “joy” groups, no significant differences were found between the two happiness sessions (). The authors therefore concluded that the stimuli could effectively elicit the subjects’ happiness. For the specific affects, further analysis showed that there was no significant difference in scores between the males and females () with homogeneity of variance (F(1, 21) = 1.652, ). A paired t-test revealed that the arousal responses did not differ significantly within the two happiness sessions () and were highly positively correlated (r(45) = 0.494, ). The authors also concluded that the arousal responses were seemingly only influenced by the stimulus alone and not by other additional factors such as the interval and memory load.
7.2. Inferential Statistics
To examine whether the StO2 differed among the different emotion tasks (Calm, Happiness Session 1, and Happiness Session 2) for each ROI, multivariate analysis of variance (MANOVA) was performed using the emotion types (calm and happiness) as independent variables. First, the ln(·) transformation was used to comply with analysis of variance assumptions.
Furthermore, in order to compare the changes in the subjects’ emotional responses within different sessions, a modified baseline correction was applied to compensate for the individual differences by transforming to comparable scales (changing rate) based on a calm state for a given subject. The ln(·) transformation was applied to comply with analysis of variance assumptions. As a result, two-way repeated measures ANOVA was conducted using ROIs as independent variables.
The results showed that the error variances of the dependent variables were equal across the groups, as evaluated by Levene’s test. A Greenhouse–Geisser correction for nonsphericity was applied if Mauchly’s test of sphericity was significant. For simplicity, the whole facial region was divided into three parts by sagittal plane: left region (LR); middle region (MR); and right region (RR). The LR comprised L1, L2, L3, L4, L5, L6, and L7; the MR included M1, M2, M3, M4, and M5; and the RR was made up of R1, R2, R3, R4, R5, R6, and R7. The results discuss these three parts: analysis for each ROI, within-region analysis, and between-region analysis.
8. Results and Discussion
First, MANOVA is used to analyze the main effect of the emotion factor for each ROI. Second, in the within-region analysis, the LR, MR, and RR are discussed separately to determine any significant correlations within them. Third, the between-region analysis explores the interrelationships between the LR, MR, and RR.
8.1. Analysis for Each ROI
For the ROIs, MANOVA was performed for three sessions (Calm, Happiness Session 1, and Happiness Session 2). The comparable distributions of their correlations and differences are shown in Tables 3 and 5. In Table 3, it can be seen that the average intensity of StO2 was a little greater in Session 1 or Session 2 than in the Calm state for each ROI. In addition, the proportion of StO2 increased from 2.74% to 9.03% in Session 1 and varied within the range of 2.81%–9.10% in Session 2. Thus, the increase in fluctuations observed was in a similar range for both happiness sessions. Furthermore, it was found that the average values in the two sessions were very close.
An intuitive explanation is that these results are indeed all about smile processing and may have an approximately equal emotional expression in terms of neuromotor mechanism.
However, this is only a subjective judgment; further analysis will be evaluated and discussed in the following section. Moreover, another aspect for consideration is that since the individuals each have their own distinct differences in personalities, they may vary in their affective reactions so that StO2 as a physiological signal differs according to a given emotional state. The experimental results in Table 3 show that the 19 different regions do not exhibit the same reactivity for a specific state. At the baseline, L7 has the minimal value of 37.24%, and R4 has the maximal score of 54.72%. While emotions are elicited, they would be correspondingly enhanced.
Figures 5(a)–5(c), respectively, describe the raincloud distributions of StO2 under three sessions (Calm, Session 1, and Session 2) for different ROIs such as LR, RR, and MR. These plots also specifically illustrate the individual differences of ROIs within each session group. They visually characterize the intuitive interrelations between individual and group distributions. Figure 5(d) shows the distributions of average StO2 during the calm state and the two sessions. Using a Bonferroni-adjusted significance level (α′ = 0.05/3 = 0.017), the analysis showed that the subjects displayed more happiness in the two task sessions than in the calm state (ANOVA, for both happiness sessions). However, no significant difference in arousing happy feelings was found between the two task sessions (ANOVA, ). This finding reflects not only the changes in the different ROIs for the same person for a given emotional state but also reveals the oscillation of StO2 during different states. Figure 5(e) shows a scatter plot of StO2 versus ROI distributions. The results clearly show that the happiness sessions could not be distinguished from each other, but that both of these sessions could be distinguished from the calm state.
Subsequently, the authors tested whether the subjects had equivalent mean responses across the three sessions. For each ROI, there is a family of tests. Hence, the significance level α = 0.05 is Bonferroni adjusted to α′ = 0.05/(3 ∗ 19) = 0.00088. The significant differences are shown in Table 3. It could be concluded that compared to the calm groups, the physiological indicators revealed significant happiness differences for most ROIs in both happiness sessions using a paired t-test (), while there are no significant differences between Session 1 and Session 2 for the ROIs ().
Additionally, another evaluation index (effect size, Cohen’s d) was used. Table 5 shows the correlations and divergences for the groups. The correlation coefficients between the two happiness sessions were greater than those between other sessions. Such intercorrelations indicate that if people smile happily, the range of increase in StO2 is proportional in each ROI.
After considering all of the above indicators, it was concluded that StO2 increased significantly in different periods for every ROI, with fewer overlapping regions between happiness and calm states and a similar increasing gradient for two individual changing stages from calm to happiness. Moreover, no significant variation in physiological signals was observed between the two happiness sessions with larger superposed areas. This suggests that arousal does not influence neuromuscular signals. Furthermore, for a specific emotional state, distinct differences in the physiological movements measured by StO2 were observed in all ROIs. These significant differences will be discussed in the following sections.
8.2. Within-Region Analysis
For the within-region analysis, the LR, MR, and RR were analyzed separately. In order to compare the subjects’ happiness responses with calm responses on an equal footing, it was necessary to apply a modified baseline correction for every subject to calibrate the individual differences. The ln(·) transformation was used to ensure analysis of variance assumptions.
For each segmented region, LR, MR, and RR, and the whole region abbreviated as AR, two-way repeated measures ANOVA was used for the task type; these results are shown in Table 6. The results show that the interrelation between ROI and session was insignificant. Furthermore, the effects between the sessions are also insignificant, but ROI is found to be highly significant in all the experiments.
First, for the LR, a post hoc analysis with a paired t-test for each session was performed and the results are displayed in Table 7. It was found that for both Sessions 1 and 2, L5 differed significantly from L1, L2, L6, and L7, and the difference was also significant between L4 and L7. The mean differences between the other tests had no statistical significance. Additionally, L3 was significantly different from the regions L1, L6, and L7, and a difference was also found between L1 and L4. The differences in other comparisons were not significant. These interrelationships are illustrated in Figure 6(a).
By observing the distribution patterns of StO2 for different sessions, no significant interaction effect () was noticeable between ROI and session.
Second, the same analysis was performed for the RR, and the results are shown in Tables 6 and 7. In Session 1, R1 and R6 were both found to be significantly greater than the other regions R3, R4, and R5. The mean differences in the other tests were not statistically significant. However, in Session 2, R5 was significantly different from the other regions R1, R2, R6, and R7. The differences in the other comparisons were not significant. Figure 6(b) illustrates these interrelationships. Once again, no interaction effects were observed between ROI and session. While minor differences between the two sessions in the ROIs were apparent (in R4, for example), none of these were significant for any of the ROIs ().
In terms of the symmetry characteristics of the LR and RR, the variation trends of the ROIs were identical which complied with an intuitive judgment. The intensity levels of the average StO2 of the corresponding ROI between the LR and RR—such as the StO2 of L4 in the LR versus the StO2 of R4 in the RR—were markedly different. One explanation of this is that the subjects’ neuromuscular responses did not involve the same mechanism of action because of the emotion mode. For example, smile intensities were stronger in the left regions than in the right regions when the participants elicited genuine happy smiles. These interesting findings of partial asymmetry (lateralization of affective processing) have also been addressed in many studies [61–64]. They concluded the hemiface differences in visual exploration patterns when displaying genuine emotions. Lindell  reviews research examining asymmetries in the expression of facial emotion in humans, representing the right hemisphere’s dominance for emotion processing. More specifically, it is the right hemisphere that innervates the lower left hemiface, resulting in more intense expressions. Najt et al.  reevaluated empirical evidence with respect to three competing yet partly conflicting models (the Right Hemisphere Hypothesis, the Valence-Specific Hypothesis, and the Approach/Withdrawal model). Results from their investigations showed that they did not fully support the models, demonstrating a left hemisphere advantage for the perception of happy expressions and a right hemisphere advantage only for a subset of negative emotions including anger, fear, and sadness (rather suggesting a “negative valence model”). Prete et al. [63, 64] concluded that the right hemisphere was more responsive to emotional faces than the left hemisphere. Additionally, the authors presented that there was no correspondence between behavioral and electrophysiological results concerning asymmetries for emotion processing, and that the Valence-Specific Hypothesis and the Right Hemisphere Hypothesis were not mutually exclusive. Therefore, it is very complicated to evaluate the hemifacial asymmetries in expressivity.
An analysis of the MR produced similar results: the difference between the two sessions and the interaction effects between ROI and session were not significant (Table 6). With a paired t-test, comparing Session 1 and Session 2 and adjusting for Bonferroni’s correction, the results were not significant. Subsequently, an ANOVA revealed significant differences between M5 and both M3 and M4 for Session 1 and Session 2, as shown in Table 7. The deviations of the mean differences in the other comparisons were not significant. Figure 6(c) shows the interactions between session and ROI for the MR.
Not surprisingly, other than the partial asymmetries of the left/right hemiface, upper and lower parts of the face would also express the “partial asymmetries” for the perception of emotions [65–67]. Ross et al. [66, 67] claimed overwhelmingly independent motor control of the upper and lower face in the studies. In addition, they found evidence that the right and left face may also exhibit independent motor control, thus supporting the concept that spontaneous facial expressions are organized predominantly across the horizontal facial axis and secondarily across the vertical axis. Unlike the cognitive control of facial expressions for social purposes in the lower face, the upper face may often display/leak a person’s true feeling state by producing a brief facial emotion. Meletti et al.  presented, by ERPs study, the response patterns of EEG with faces encoding happiness and fear in the eye region compared to those encoding emotions in the whole faces or in the mouth region. Zeng et al.  revealed that blocking the facial feedback of lower face significantly boosted the recognition accuracy of subtle and intense microexpressions under all duration conditions, highlighting the important role of applying the upper face in judging the subtle movements of microexpressions. As known from the above observations in the experiment, most of the upper ROIs demonstrate the significant signatures than the lower ROIs, which are almost consistent with the previous findings.
8.3. Between-Region Analysis
Having performed contrastive analysis for the three individual regions, we then needed to investigate the interactions among them. Following the above analysis, in order to effectively determine their deeper connections, we divided the regions into two groups according to the degree of significance, i.e., a high-StO2 correlation group (HCG) and a low-StO2 correlation group (LCG). Thus, L3, L4, and L5 of the LR and R3, R4, and R5 of the RR were grouped into the LCG, while the other subdivisions of the LR (L1, L2, L6, and L7) and the RR (R1, R2, R6, and R7) were included in the HCG. MR was not divided into groups. Owing to the specialty of the MR in the midsagittal plane and the symmetry of the LR and RR, we studied only the interrelations between the LR vs. the MR and the RR vs. the MR. Figure 6(d) shows the distribution of the average StO2 for all of the ROIs.
Firstly, we performed a comparison analysis between the LR and MR. In the contrastive analysis between the LCG and the MR, two-way repeated measures ANOVA revealed that the interaction effect was not significant () and the sessions had no effect on the variations in StO2 (). However, the main effect of the ROI was significantly different (). After adjusting for Bonferroni’s correction, the results of our post hoc analysis showed that L5 was significantly lower in StO2 than M1, M2, and M5 in Session 1 ( for M1, for M2, and for M5), with M1 significantly higher than both L3 and L4 ( for L3 and for L4). In Session 2, only M1 showed a significant difference with L3 or L5 ( for L3 and for L5). Next, we used ANOVA between the HCG and the MR, which demonstrated that the interaction effect was also insignificant (), with session insignificant () and ROI significant (). However, the differences in ROI between the two sessions were no longer significant after adjusting for Bonferroni’s correction. With no adjustment in the least significant difference (LSD), we found that both M1 and M5 showed more significant intensities in StO2 than L2 in Session 1 ( for M1 and for M5), with L7 more significant than M1 (), M3 (), and M4 () in Session 2.
We then performed a comparison analysis between the RR and MR. Firstly, we conducted an analysis on the LCG and the MR. ANOVA revealed that the interaction effect was insignificant (), with session also insignificant (). However, the main effect of ROI was significantly different (). In Session 1 after Bonferroni correction, M1 was found to have a significantly greater StO2 than either R3 or R4 ( for R3 and for R4). Also, StO2 was significantly greater in M5 than in R3 (), R4 (), and R5 (). Furthermore, in Session 2, after Bonferroni correction, R3 was significantly lower in StO2 than the ROIs (M1: , M2: , and M5: ). Moreover, the StO2 of R5 was also significantly lower than M1, M2, and M5 ( for M1, for M2, and for M5). Subsequently, the test study between the HCG and the MR verified an insignificant interaction effect () and session effect () and a significant ROI effect (). Post hoc analysis shows that the differences in both Session 1 and Session 2 were insignificant after adjusting for Bonferroni’s correction. We nevertheless applied the LSD test to compare each of the two groups. In Session 1, M1 and M5 both showed more significant happiness than R6 in terms of changing rate ( for M1 and for M5), with M5 more significant than R2 (). The differences in ROIs for Session 2 were all found to be insignificant.
8.4. Effects Size for Each ROI
The size of the selected ROI determined the number of pixels involved in computing the average StO2 and subsequently further analysis. Generally speaking, the smaller the ROI, the fewer the number of pixels; therefore, higher variations in emotional signals would be generated. By increasing the size of the ROI appropriately, the variable effect was reduced and variations were uniformly diminished. Figure 7 shows the distribution of StO2 for a random region L4. It illustrates that neither smaller nor larger ROIs could attain the maximum value of the average StO2. StO2 reaches an optimal average score for 13px-by-13px region. The average values obtained from the regions whose sizes were smaller than N = 13px almost reached the optimal value. This shows that the smaller regions had already included the most effective information. Only by significantly expanding the ranges could we acquire more extensive data; yet by continuing to enhance the ranges, more noises are introduced into the dataset. Moreover, the average range of values for the whole region only varied slightly—from 61.31% to 64.65%. Smaller ROIs mean higher oscillations which leads to an unstable distribution, while larger ROIs result in unnecessary noises becoming amplified in true signals. Both of the above scenarios are likely to lead to poor StO2 average values. Accurate results are achieved by using suitably sized ROIs. Regularities similar to those of L4 were observed with the other ROIs, and the observations were confirmed by validating the distribution of StO2.
9. Summary of Findings and Conclusions
Happiness as a psychological activity has attracted significant interest as a key area of research because it is associated with physical well-being. HSI technology, as a contact-free detection technique, is used to distinguish and quantify the amount of blood chromophores (Hb and HbO2). Subsequently, StO2, as a neurophysiological indicator, is considered to represent the affective response signals of neuromuscular activities.
In summary, this research shows that when people elicit genuine smiles, which indicate happiness, all of the facial regions are involved in displaying this emotion. However, there are significant variations in the degree of facial regions that show changes. The largest changes are seen in the upper eyelid, angulus oris, and mandible, which are controlled by the orbicularis oculi, risorius, and mentalis muscles, respectively. The forehead and ophryon exhibit the second largest variations in emotional intensities, managed by the frontalis and procerus muscles, respectively. The regions of the eye corner around the temple, oculonasal groove, nose, and perinasal area exhibit the second smallest variations in happiness measurements. The variations in the regions of the eye corner around the ophryon, lower eyelid (both controlled by the orbicularis oculi muscle), and cheek (controlled by the zygomaticus muscle) are least distinct. Thus, it may be concluded that although all the facial regions are more or less engaged in happiness and serve to create different effects, certain muscles, such as the orbicularis oculi, exert distinct effects on specific regions: the upper eyelid, lower eyelid, eye corner around the temple, and eye corner around the ophryon.
It was found that the regions involved in the observed expressions, as defined by action units (AU), produced statistically significant excitement based on the neurophysiological measurements. They include the regions of orbicularis oculi of AU6, zygomaticus minor and levator labii superioris of AU10, zygomaticus major of AU12, risorius of AU20, and orbicularis oris of AU25. Considering the above findings, it may be concluded that all of the above mentioned muscles show an increased degree of involvement when people display happiness. However, not all of them exhibit maximum variations.
For each ROI, the average intensities of StO2 increased significantly in both happiness sessions compared to the intensities found in the calm state as shown in the ANOVA results in Table 3. This indicated that the two arousal tasks prompted a statistically significant emotional response in the participants compared to the calm state (ANOVA, for both happiness sessions). Additionally, the happiness intensities in the two sessions were not significantly different (ANOVA, ). The results show that the physiological activities as a neuromotor mechanism have approximately equal emotional expression, and this is not affected by the time interval. Other evaluation indexes such as effect size (Cohen’s d) and correlation coefficient also support the above mentioned conclusions.
To facilitate subsequent analysis, the whole facial region was divided into three parts: LR, MR, and RR, according to the sagittal plane. Both within-region and between-region interaction effects were studied separately to determine significant correlations among them, using two-way repeated measures ANOVA. It was found that the interaction effects between ROI and task factors were not significant. Only the ROI was found to be significant in all the experiments.
For the within-region analysis, in the LR segment, it was concluded that L1, L2, L6, and L7 could be divided into a group with higher variations in the changing rate of StO2 and L3, L4, and L5 into another group with lower variations. In the RR segment, though the variable regularities were not the same as for LR, similar general trends emerged. It was concluded that R1, R2, R6, and R7 form one group while R3, R4, and R5 form another group. One explanation of this nonuniform distribution could be that the subjects’ neuromuscular responses did not involve the same mechanism of action because of the emotion mode. For example, by virtue of the hemiface differences caused by lateralization of affective processing, more neuropsychological responses may have occurred on the left side of the face than on the right side.
Lastly, in the MR, it was concluded that M5 showed a significantly greater changing rate than either M3 or M4 in both happiness sessions. Thus, M5 is a group in itself, while M1, M2, M3, and M4 form a second group.
For the between-region analysis, after adjusting for Bonferroni’s correction and a LSD test, considering correlation differences in the changing rate of StO2, and taking into account the conclusions drawn from the within-region analysis, the authors concluded that the groups should be organized as follows: (1) L1, L7, and M5; (2) M1 and M2; (3) L2, L6, M3, and M4; and (4) L3, L4, and L5, with a similar organization in the RR. It is proposed that these grouping strategies be investigated further and may serve as a basis for further research.
10. Recommendations for Further Research
Genuine happiness manifests distinct responses among individuals which are measurable using remote-sensing technology. The blood-supply system in the facial muscles shows significant changes when people smile, and is suggested to be an indicator of genuine happiness. The authors will continue to further study this topic with the aim of learning more about, and modeling, patterns of happiness. In this work, we do conduct the experimental investigation only focusing on eliciting the genuine happiness. The interaction effects of the single psychological emotion on physiological activities have not been investigated in this experiment. Also, the contribution of neuromuscular activity (i.e., making a facial expression without any real emotion) still needs to be investigated further. Therefore, the individual contribution of how psychological emotion and neuromuscular activity affect the interactive performance of physiological reactions will be explored in the future work.
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as potential conflicts of interest.
This research was supported by the National Natural Science Foundation of China (Grant no. 61872301) and Chongqing Key Laboratory of Non-Linear Circuit and Intelligent Information Processing. The authors are grateful to the colleagues from the Institute of Signal and Information Processing at Southwest University for their thoughtful suggestions. M. Hao acknowledges the help of Y. Xu and R. Chen for their reviews and feedback in the early versions of this manuscript. M. Hao also would like to appreciate the helpful comments and reviews of Professor G. Liu and Professor Anu Gokhale in particular.
R. L. Payne and C. L. Cooper, Emotions at Work: Theory, Research and Applications for Management, John Wiley & Sons, Hoboken, NJ, USA, 2003.
L. McNamara, J. A. McNamara, M. B. Ackerman, and T. Baccetti, “Hard- and soft-tissue contributions to the esthetics of the posed smile in growing patients seeking orthodontic treatment,” American Journal of Orthodontics and Dentofacial Orthopedics, vol. 133, no. 4, pp. 491–499, 2008.View at: Publisher Site | Google Scholar
G. A. Shaw and H. K. Burke, “Spectral imaging for remote sensing,” Lincoln Laboratory Journal, vol. 4, no. 1, pp. 3–28, 2003.View at: Google Scholar
M. Vitti and J. V. Basmajian, “Electromyographic investigation of procerus and frontalis muscles,” Electromyography and clinical neurophysiology, vol. 16, no. 2-3, pp. 227–236, 1976.View at: Google Scholar
T. Chen, P. W. T. Yuen, K. Hong et al., “Assessment of tissue blood perfusion in-vitro using hyperspectral and thermal imaging techniques,” in 2011 5th International Conference on Bioinformatics and Biomedical Engineering (iCBBE), Wuhan, China, May 2011.View at: Google Scholar
A. Tseng, R. Bansal, J. Liu et al., “Using the circumplex model of affect to study valence and arousal ratings of emotional faces by children and adults with autism spectrum disorders,” Journal of Autism and Developmental Disorders, vol. 44, no. 6, pp. 1332–1346, 2013.View at: Publisher Site | Google Scholar
A. Lindell, “Chapter 9-lateralization of the expression of facial emotion in humans,” in Progress in Brain Research, G. S. Forrester, W. D. Hopkins, K. Hudry, and A. Lindell, Eds., pp. 249–270, Elsevier, New York City, NY, USA, 2018.View at: Google Scholar
X. M. Zeng, Q. Wu, S. W. Zhang, Z. Y. Liu, Q. Zhou, and M. S. Zhang, “A false trail to follow: differential effects of the facial feedback signals from the upper and lower face on the recognition of micro-expressions,” Frontiers in Psychology, vol. 9, p. 2015, 2018.View at: Publisher Site | Google Scholar