Abstract
We describe analyses of the features of emotions (neutral, joy, sadness, and anger) expressed by Finger Braille interpreters and subsequently examine the effectiveness of emotional expression and emotional communication between people unskilled in Finger Braille. The goal is to develop a Finger Braille system to teach emotional expression and a system to recognize emotion. The results indicate the following features of emotional expression by interpreters. The durations of the code of joy were significantly shorter than the durations of the other emotions, the durations of the code of sadness were significantly longer, and the finger loads of anger were significantly larger. The features of emotional expression by unskilled subjects were very similar to those of the interpreters, and the coincidence ratio of emotional communication was 75.1%. Therefore, it was confirmed that people unskilled in Finger Braille can express and communicate emotions using this communication medium.
1. Introduction
The Deafblind Association of Japan estimates that there are nearly 22,000 deafblind people in Japan (2008). Deafblind people use many different communication media, depending on the age of onset of deafness and blindness and the available resources. “Yubi-Tenji” (Finger Braille) is one of the tactual communication media utilized by deafblind individuals (see Figure 1). In two-handed Finger Braille, the index finger, middle finger, and ring finger of both hands function like the keys of a Braille typewriter. A sender dots Braille code on the fingers of a receiver as if typing on a Braille typewriter. The receiver is assumed to be able to recognize the Braille code. In one-handed Finger Braille, the sender first dots the left column of Braille code on the distal interphalangeal (DIP) joints of the three fingers of the receiver and then dots the right column of Braille code on the proximal interphalangeal (PIP) joints. Deafblind people who are skilled in Finger Braille can understand the speech conversation and express various emotions because of the prosody (intonation) of Finger Braille [1]. A rule of Finger Braille is that the sender keeps touching the fingers of the receiver even when not dotting, because receivers feel uneasy in the absence of touching or tactile cues. Because there is such a small number of non-disabled people who are skilled in Finger Braille, deafblind people communicate only through an interpreter.
Various Braille input devices have recently been developed [2, 3], but they require deafblind people to wear gloves or type on a keyboard to input the Finger Braille, or to use actuators to output and convert the speech of non-disabled people to Finger Braille. With these devices, deafblind people are burdened with wearing sensors and actuators, and they must master a new communication system with such support devices.
The objective of this study is the development of a Finger Braille support device which employs the skin-contact communication of deafblind people, because skin contact is the only form of nonverbal communication for deafblind people. The concept of the Finger Braille support device is shown in Figure 2. The advantages of this device are as follows: both deafblind people and non-disabled people unskilled in Finger Braille can communicate using conventional Finger Braille, and deafblind people are not encumbered by a support device because the non-disabled people operate the support device and wear all of the sensors.
Our support device consists of a Finger Braille teaching system and a Finger Braille recognition system. The teaching system recognizes the speech of a non-disabled person and displays the associated dot pattern of Finger Braille. The non-disabled person can then dot Finger Braille on the fingers of the deafblind person by observing the displayed dot pattern [4, 5]. The recognition system recognizes the dotting of Finger Braille by the deafblind person and synthesizes this tactile communication into speech for the non-disabled person [6]. The intent of the support device is to assist not only verbal communication but also nonverbal (emotional) communication. To assist in emotional communication, we have been developing an emotion teaching system and an emotion recognition system [7]. The emotion teaching system teaches non-disabled people to express emotions by skin contact. The emotion recognition system recognizes the emotions dotted by the deafblind and presents the information as synthesized speech for the non-disabled.
In the present study, emotional expression by Finger Braille interpreters (Experiment I) and emotional communication by people who are unskilled in Finger Braille (Experiment II) were examined. This paper presents analyses of the features of emotional expression by Finger Braille interpreters and the effectiveness of emotional expression and emotional communication between unskilled people.
2. Emotional Expression by Finger Braille Interpreters (Experiment I)
To analyze the features of emotional expression using Finger Braille, emotional expression by interpreters of Finger Braille (Experiment I) was examined [8, 9].
2.1. Methods
The subjects were two Finger Braille interpreters (Subjects A and B) with 20 and 11 years of experience, respectively. Both subjects were right-handed. Both subjects gave informed consent after hearing a description of the study. In the experiment, the subject became a sender and a tester became a receiver. The same tester, who was one of the authors, was used throughout the experiment.
Two dialogues were used. “Rain has fallen” or Ame futte kita (7 characters, 3 clauses, “t” is a double consonant, “me” and “te” are characters at the end of a clause) and “It is the time” or Jikan desuyo (8 characters, 1 clause, “Ji” is a combination of voiced sound and “si”, “de” is a combination of voiced sound and “te”). These dialogues did not include words about emotion.
Plutchik proposed a multidimensional model of emotions and defined joy, anticipation, anger, disgust, sadness, surprise, fear, and acceptance as the basic human emotions [10]. Bridges observed emotional development in early infancy and found that distress (sadness), delight (joy) and anger developed from excitement in early infancy [11].
We decided to investigate the expressed emotions of neutral, joy, sadness, and anger, which are basic emotions developed in early infancy of human beings. These emotions were common to those in several previous studies of emotional expression and communication [12–20].
In the previous studies, actors or announcers were selected as the subjects, because of their experiences expressing emotion [16, 18]. In the present study, the subjects were neither actresses nor announcers; so we set six scenes (two dialogues with joy, sadness, and anger) for the subjects to express the emotions easily. The scenes are as follows.
Scenes of “Rain has fallen”
Joy: In the morning of a sport activity, you do not want to participate. If it is raining, the activity would be cancelled. Now rain is falling.Sadness: On a holiday morning, the weather is fine. You are changing your clothes to go outside. Now rain is falling.Anger: The probability of rain is 0% today. You have gone out without an umbrella. Now rain is falling.Scenes of “It is the time”
Joy: A TV program that you want to watch is going to start. You call your family.Sadness: You have come to an airport to send off your friend who is going abroad to study. Now it is time to depart.Anger: It is the closing time of your shop, but a customer will not go home.In this experiment, we adopted two-handed Finger Braille (the Perkins Brailler type, which is the most popular type of Finger Braille) and one-handed (right hand) Finger Braille.
In the two-handed Finger Braille experiment, the variables compose a 4 × 5 within-subjects design of expressed emotion (neutral, joy, sadness, anger) and position of characters (inner clause, end of clause, end of sentence, voiced sound, double consonant). In the one-handed Finger Braille experiment, the variables compose a 4 × 6 within-subjects design of expressed emotion (neutral, joy, sadness, anger) and position of characters (inner character, inner clause, end of clause, end of sentence, voiced sound, double consonant). The dependent variables are the duration of code, duration of pause, and finger load. The inner character is the dotting of DIP joints if the character includes the dotting of both DIP and PIP joints.
In our previous work [7, 8], we designed each emotion factor and each character factor and conducted two analyses of variance (ANOVA) for both dialogues (e.g., the 4 × 7 within-subjects design in “Rain” of the two-handed Finger Braille experiment, and the 4 × 8 within-subjects design in “Time” of the two-handed Finger Braille experiment). It was difficult to generalize the features of emotional expression and prosody using the character factor of the specific dialogues. Thus, in the present study, we redesigned this experiment.
The experimental flow is shown in Figure 3. First, the two-handed Finger Braille experiment was conducted followed by the one-handed Finger Braille (right-hand) experiment, with each experiment including one practice session and five experimental sessions. The subject rested after every 2-3 sessions.
In one experimental session, the subjects dotted Finger Braille on the fingers of the tester eight times in the order of “Rain” with neutral, “Time” with neutral, “Rain” with joy, “Time” with joy, “Rain” with sadness, “Time” with sadness, “Rain” with anger, and “Time” with anger. The dialogue and emotion associated with the dot was displayed on the LCD of a notebook PC that was placed in front of the subject. The subject did not hear and transfer the emotional speech of the dialogues but expressed the impressions of emotions using Finger Braille. The subject did not alter the dialogue in any way and only expressed emotions by changing the dotting speed and applied pressure. To eliminate the influence of the previous dotted dialogue with emotion, the subject dotted at intervals (1-2 minutes) and confirmed the next scene of dialogue with emotion.
As is customary in communication using Finger Braille, the subject and tester sat side by side during the two-handed Finger Braille experiment and face to face during the one-handed Finger Braille experiment. The hands of the subject and tester were in constant contact during the two-handed Finger Braille experiment.
The tester put his fingers on a pressure sensor sheet (Tactile Sensor System, Nitta) which measured the change of pressure as a result of dotting. The sampling frequency was 137 Hz, the measurement range was 15–150 kPa, the sensibility was 527 Pa, the size of the sensor sheet was 84 × 84 mm, and the sensor matrix was 44 × 44. The sensor sheet was segmented into three blocks (width: 28 mm) and the tester put his index, middle, and ring fingers into the blocks.
2.2. Results
2.2.1. Calculation of Duration and Finger Load
The load of each finger was determined by the change in pressure applied to the three blocks, and the start, end, and maximum load of dotting were measured. The duration and finger load for each dotting were also calculated. Sample data from the one-handed Finger Braille experiment are shown in Figure 4.
The duration of the code was defined as the time from the start of dotting to the end of dotting. The duration of the pause was defined as the duration from the completion of dotting to the start of the next dotting sequence. The finger load was defined as the difference between the maximum load and the mean load between the start and end of dotting. If multiple fingers were dotted at once, the finger load was defined as the mean of the finger loads for the dotted fingers.
2.2.2. Results of Two-Handed Finger Braille Experiment
(a) Duration of Code
The mean of the duration of the code as a function of the emotions and position of characters is shown in Figure 5.
A 4 × 5 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,580) = 119.3, P < .001) and the position of characters (F(4,580) = 121.0, P < .001). There was a significant interaction term of the emotion and position of characters (F(12,580) = 15.73, P < .001).
A test of the simple main effect of emotion revealed a significant effect of emotion in all the positions of characters (P < .001). Scheffe tests on the emotion factor in each position of the characters revealed the following: the duration of code of sadness was significantly longer than the other durations in the inner clause (P < .001); the duration of code of joy was significantly shorter than the other durations in the inner clause (P < .001); the duration of code of anger was significantly shorter than the duration of code of neutral in the inner clause (P < .001); the duration of code of sadness was significantly longer than the other durations in the end of clause (P < .041); the duration of code of sadness was significantly longer than the other durations in the end of the sentence (P < .001); the duration of code of neutral was significantly longer than the durations of code of joy and anger in the end of the sentence (P < .001); the duration of code of sadness was significantly longer than the other durations in the voiced sound (P < .050); the duration of code of joy was significantly shorter than the other durations in the voiced sound (P < .001); the duration of code of joy was significantly shorter than the other durations in the double consonant (P < .011).
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (P < .001). Scheffe tests on the factor of the position of characters in each emotion revealed the following: the durations of code of the end of the sentence were significantly longer than the other durations in neutral (P < .001); the durations of code of the voiced sound and double consonant were significantly shorter than the other durations in neutral (P < .011); the duration of code of the end of the clause and end of the sentence was significantly longer than the other durations in joy (P < .020); the duration of code of the end of the sentence was significantly longer than the other durations in sadness (P < .001); the duration of code of the end of the clause was significantly longer than the durations of code of the inner clause, voiced sound, and double consonant in sadness (P < .003); the duration of code of the double consonant was significantly shorter than the other durations in anger (P < .001).
(b) Duration of Pause
The mean of the duration of the pause as a function of the emotions and position of characters is shown in Figure 6. The duration of pause before the double consonant was shorter than the other durations of pause. So, we defined the duration of pause before the double consonant as the duration of pause of the double consonant and the duration of pause after the double consonant as the duration of pause of the inner clause. There was no duration of pause at the end of sentences.
A 4 × 4 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,504) = 5.615, P < .001) and the position of characters (F(3,504) = 51.837, P < .001). Scheffe tests on the emotion factor revealed that the durations of pause of sadness and neutral were significantly longer than the durations of pause of anger and joy (P < .003). Scheffe tests on the factor of the position of characters revealed that the duration of pause of the double consonant was significantly shorter than the other durations (P < .001), and the duration of pause of the end of the clause was significantly longer than the other durations (P < .001).
(c) Finger Load
The mean of the finger load as a function of the emotions and position of characters is shown in Figure 7.
A 4 × 5 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,580) = 344.6, P < .001) and the position of characters (F(4,580) = 31.49, P < .001). There was a significant interaction term of the emotion and position of characters (F(12,580) = 16.52, P < .001).
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (P < .001). Scheffe tests on the emotion factor in each position of characters revealed the following: the finger load of anger was significantly larger than the other finger loads in all positions of characters (P < .001); the finger load of joy was significantly larger than the finger loads of neutral and sadness in the inner clause (P < .036); the finger load of joy was significantly larger than the finger load of sadness in the end of the clause and the voiced sound (P < .017).
A test of the simple main effect of position of characters revealed a significant effect of position of characters in all emotions (P < .001). Scheffe tests on the factor of the position of characters in each emotion revealed the following: the finger load of the voiced sound was significantly larger than the finger loads of the end of clauses, inner clause and end of sentence in neutral (P < .001); the finger load of the voiced sound was significantly larger than the other finger loads in joy (P < .002); the finger load of the voiced sound was significantly larger than the other finger loads in sadness (P < .006); the finger loads of the end of the sentence and voiced sound were significantly larger than the other finger loads in anger (P < .007).
2.2.3. Results of One-Handed Finger Braille Experiment
(a) Duration of Code
The mean of the duration of the code as a function of the emotions and position of characters is shown in Figure 8.
A 4 × 6 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,1013) = 180.3, P < .001) and the position of characters (F(5,1013) = 75.36, P < .001). There was a significant interaction term of the emotion and position of characters (F(15,1013) = 14.06, P < .001).
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (P < .001). Scheffe tests on the emotion factor in each position of characters revealed the following: the duration of code of sadness was significantly longer than the other durations in the inner character (P < .001); the duration of code of joy was significantly shorter than the other durations in the inner character (P < .001); the duration of code of anger was significantly shorter than the duration of code of neutral in the inner character (P < .008); the duration of code of sadness was significantly longer than the other durations in the inner clause (P < .001); the duration of code of joy was significantly shorter than the other durations in the inner clause (P < .001); the duration of code of sadness was significantly longer than the other durations in the end of clause (P < .003); the duration of code of joy was significantly shorter than the duration of code of neutral in the end of clause (P < .048); the duration of code of sadness was significantly longer than the other durations in the end of sentence (P < .001); the duration of code of joy was significantly shorter than the duration of code of neutral in the end of sentence (P < .003); the duration of code of sadness was significantly longer than the other durations in the voiced sound (P < .015); the duration of code of joy was significantly shorter than the duration of anger in the voiced sound (P < .024); the durations of code of sadness and neutral were significantly longer than the durations of code of joy and anger in the double consonant (P < .019).
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (P < .001). Scheffe tests on the factor of the position of characters in each emotion revealed that the duration of code of the end of sentence was significantly longer than the other durations in all emotions (P < .030), and the duration of code of the end of clause was significantly longer than the duration of the double consonant in sadness (P < .024).
(b) Duration of Pause
The mean of the duration of the pause as a function of the emotions and position of characters is shown in Figure 9. We defined the duration of pause before the double consonant as the duration of pause of the double consonant and the duration of pause after the double consonant as the duration of pause of the inner clause. These definitions are the same as those for the two-handed Finger Braille experiment. There was no duration of pause at end of the sentences.
A 4 × 5 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,935) = 4.009, P < .008) and the position of characters (F(4,935) = 4.101, P < .003). Scheffe tests on the emotion factor revealed that the duration of pause of sadness was significantly longer than the other durations (P < .036). Scheffe tests on the factor of the position of characters revealed that the duration of pause of the end of the clause was significantly longer than the duration of pause of the inner character (P < .005).
(c) Finger Load
The mean of the finger load as a function of the emotions and position of characters is shown in Figure 10.
A 4 × 6 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,1013) = 261.4, P < .001) and the position of characters (F(5,1013) = 23.11, P < .001). There was a significant interaction term of the emotion and position of characters (F(15,1013) = 5.496, P < .001).
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (P < .001). Scheffe tests on the emotion factor in each position of characters revealed the following: the finger load of anger was significantly larger than the other finger loads in all positions of characters (P < .001), and the finger load of joy was significantly larger than the finger loads of neutral and sadness in the inner character, inner clause, and end of clause (P < .011); the finger load of joy was significantly larger than the finger load of sadness in the double consonant (P < .030).
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (P < .034). Scheffe tests on the factor of the position of characters in each emotion revealed the following: the finger loads of the double consonant, end of sentence, and inner character were significantly larger than the finger loads of the end of clause and inner clause in neutral (P < .034); the finger load of the end of sentence was significantly larger than the finger loads of the end of clause, inner clause, voiced sound, and inner character in anger (P < .013).
2.2.4. Questionnaire Results (Interpreters)
After the experiment, the subjects were asked how they expressed the emotions, and their responses are presented in Table 1. In addition, the subjects answered that they sometimes hit the hand of the receiver or dotted by claw-like fingers in anger.
Subjects also responded that it was more difficult to express emotions using one-handed Finger Braille with people sitting face to face than using two-handed Finger Braille sitting side by side, because the skin contact of the hands and the distance of each person were considered important for emotional communication.
2.3. Discussion
2.3.1. Features of Emotional Expression Using Finger Braille
To analyze the contribution of the variables to emotional expression, a discriminant analysis was conducted on data from both the two-handed and one-handed Finger Braille experiments (N = 520 and 955, resp.). The discriminant variables were the duration of code, duration of pause, and finger load. Dotting of the ends of sentences was excluded, because there was no duration of pause.
Standardized coefficients of canonical discriminant functions and contribution ratios were calculated for both the two-handed and one-handed Finger Braille experiments. The results are presented in Tables 2 and 3, respectively.
According to the standardized coefficients, discriminant function 1 represented the finger load, discriminant function 2 represented the duration of code, and discriminant function 3 represented the duration of pause. According to the contribution ratios, the finger load was the variable contributing the most to emotional expression, followed by the duration of code. The duration of pause did not contribute to emotional expression.
The discriminant ratios of emotion for two-handed and one-handed Finger Braille are presented in Tables 4 and 5, respectively. The total discriminant ratios were 68.5% in two-handed Finger Braille and 71.1% in one-handed Finger Braille. The discriminant ratio of joy was the highest (two-handed Finger Braille 86.9%, one-handed Finger Braille 85.2%); the discriminant ratio of anger was second (two-handed Finger Braille 80.8%, one-handed Finger Braille 69.9%). The dottings of neutral and sadness were frequently misdiscriminated with each other.
As indicated by the results of ANOVA (2.2.2–2.2.3), the features of emotional expression were as follows: (1) the duration of the code of joy was significantly shorter than that of the other emotions; (2) the duration of the code of sadness was significantly longer than that of the other emotions; (3) the finger load of anger was significantly larger than that of the other emotions; (4) the finger load of joy was significantly larger than that of sadness and neutral; (5) the duration of the code of anger was significantly shorter than that of sadness and neutral.
Next we discuss the features of emotional expression by the other communication media. In our previous work [9], we presented a brief discussion of other media. In the present study, we go into more detail.
Laukka analyzed the emotional expression of speech [12]. The results of his analysis were as follows: anger was characterized by a high mean pitch, high voice intensity, and fast speech rate; happiness was characterized by a high mean pitch, medium-high voice intensity, and fast speech rate; sadness was characterized by a low mean pitch, low voice intensity, and slow speech rate.
Bresin and Friberg analyzed the emotional expression of music [13]. The results were as follows: anger was characterized by a very rapid tempo, loud sound level, and mostly nonlegato articulation; happiness was characterized by a fast tempo, moderate or loud sound level, and airy articulation; sadness was characterized by a slow tempo, low sound level, and legato articulation.
Dahl and Friberg analyzed the emotional expression of a musician’s body movement in performances on the marimba [14]. The results were the following: anger was characterized by large, fast, uneven, and jerky movements; happiness was characterized by large and somewhat fast movements; sadness was characterized by small, slow, even, and smooth movements.
Clynes and Panksepp proposed a “sentic form” to express the fundamental human emotions. Sentic form can be measured by the patterns of finger pressure [15]. The results were the following: anger was characterized by high pressure and short duration; joy was characterized by short duration; sadness was characterized by low pressure and long duration.
The duration of code in the present study (Experiment I) was similar to the speech rate, tempo of music, speed of movement, and duration of finger pressure. The finger load in Experiment I was similar to the voice intensity, sound level of music, largeness of movement, and strength of finger pressure. Therefore, the features of emotional expression using Finger Braille are similar to the features of the emotional expressions of speech, music, body movement, and finger pressure.
2.3.2. Features of Prosody of Finger Braille
As indicated by the results of ANOVA (2.2.2–2.2.3), the subjects dotted dialogues with prosody (intonation) in addition to emotional expression. The followings were the features of prosody: (1) the duration of the code of the end of the clause was significantly longer; (2) the duration of the pause of the end of the clause was significantly longer; (3) the duration of the code of the end of the sentence was significantly longer; (4) the finger load of the end of the sentence was significantly larger; (5) the duration of the code of the double consonant was significantly shorter; (6) the duration of the pause before the double consonant was significantly shorter; (7) the duration of the code of the voiced sound was significantly shorter; (8) the finger load of the voiced sound was significantly larger.
Dousaka et al. and Fujimori et al. analyzed the features of prosody of Finger Braille [21, 22]. A comparison of the durations of code, pause, and dotting between the four emotions of two-handed Finger Braille in our study and in previous studies is presented in Figure 11. The values of the standard deviation were not clear in the previous studies, but the durations of code and pause of Dousaka’s study were similar to the durations of code and pause of joy and anger in the present study; the duration of dotting of Fujimori’s study was similar to the durations of dotting of neutral and sadness. The range of the duration of dotting between Dousaka’s study and Fujimori’s study was also similar to the range of the duration of dotting of the four emotions in the present study. Thus, it was considered that the features of prosody of Experiment I are natural variances of the dotting of Finger Braille.
Miyagi et al. analyzed the changes of prosody in emphasized clauses [23]. The followings are the features for prosody without emphasized clauses: the characters at the end of the clauses were dotted strongly; the duration of pauses after the ends of the clauses was long; the duration of codes at the ends of the sentences was long. When clauses were emphasized, the characters at the ends of clauses were dotted longer and more strongly; the duration of pauses after the ends of clauses was longer. In the present study, the durations of code at the ends of clauses and sentences were longer for sadness; the finger loads at the end of sentences were stronger for anger (Experiment I). These results indicate that the prosody of Finger Braille must also be similarly changed to express emotions and to emphasize clauses.
3. Additional Experiment of Emotional Expression by Finger Braille Interpreter
To confirm the features of emotional expression using Finger Braille, an additional experiment of emotional expression by the Finger Braille interpreter was conducted.
3.1. Methods
The subject was Subject A in Experiment I. The additional experiment was conducted one year and eight months after Experiment I. The subject gave informed consent after hearing a description of the study.
Two new dialogues were used. “Let’s hurry up” or Hayaku shimasyou (8 characters, 2 clauses, “syo” is a combination of a diphthong and “so”, “ku” is a character at the end of a clause), and “Is it really?” or Hontou desuka (8 characters, 1 clause, “de” is a combination of a voiced sound and “te”). These dialogues did not include words about emotion.
The expressed emotions were neutral, joy, sadness, and anger, which are identical to those used in Experiment I. We set six scenes (two dialogues with joy, sadness, and anger) for the subject. The scenes were as follows.
Scenes of “Let’s hurry up”
Joy: You are planning to go out with your close friend.Sadness: Your family member has to leave home because he has a new job. Now you help pack his bag.Anger: Your work is late but your colleague has not started to work.Scenes of “Is it really?”
Joy: You have received news that you passed a difficult examination.Sadness: You have received news that your close friend is in the hospital.Anger: Suddenly, you have been wrongly accused.In this experiment, we adopted two-handed Finger Braille (the Perkins Brailler type) and one-handed (right hand) Finger Braille, as in Experiment I.
In the two-handed Finger Braille experiment, the variables compose a 4 × 4 within-subjects design: expressed emotion (neutral, joy, sadness, anger) and position of characters (inner clause, end of clause, end of sentence, voiced sound). In the one-handed Finger Braille experiment, the variables compose a 4 × 5 within-subjects design: expressed emotion (neutral, joy, sadness, anger) and position of characters (inner character, inner clause, end of clause, end of sentence, voiced sound). The diphthong was categorized as a voiced sound. The dependent variables were duration of code, duration of pause, and finger load.
The experimental flow is shown in Figure 12. First, the one-handed Finger Braille (right hand) experiment was conducted, followed by the two-handed Finger Braille. Each experiment included one practice session and five experimental sessions. The subject rested after every 2-3 sessions.
In the experimental session, the subject dotted Finger Braille on the fingers of the tester eight times in the order of “Hurry” with neutral, “Really” with neutral, “Hurry” with joy, “Really” with joy, “Hurry” with sadness, “Really” with sadness, “Hurry” with anger, and “Really” with anger.
The other procedure and conditions were identical to those used in Experiment I.
3.2. Results
We calculated the duration of code, duration of pause, and finger load for each dotting by the change of pressure, as in Experiment I (see Figure 4).
3.2.1. Results of Two-Handed Finger Braille Experiment
(a) Duration of Code
The mean of the duration of the code as a function of the emotions and position of characters is shown in Figure 13.
A 4 × 4 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,283) = 152.8, P < .001) and the position of characters (F(3,283) = 56.87, P < .001). There was a significant interaction term of the emotion and position of characters (F(9,283) = 9.918, P < .001).
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (P < .001). Scheffe tests on the emotion factor in each position of characters revealed the following: the duration of code of sadness was significantly longer than the other durations in all positions of characters (P < .001, the voiced sound P < .015); the duration of code of joy was significantly shorter than the other durations in the inner clause, end of sentence, and voiced sound (P < .048, the inner clause P < .001); the duration of code of joy was significantly shorter than the durations of code of sadness and anger in the end of clause (P < .006).
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (P < .001). Scheffe tests on the factor of the position of characters in each emotion revealed the following: the duration of code of the end of the sentence was significantly longer than the durations of code of the inner clause and voiced sound in neutral (P < .003); the duration of code of the voiced sound was significantly shorter than the durations of code of the end of the sentence and the inner clause in joy (P < .021); the duration of code of the voiced sound was significantly shorter than the other durations in sadness and anger (P < .006); the durations of code of the end of the clause and end of the sentence were significantly longer than the other durations in sadness and anger (P < .004).
(b) Duration of Pause
The mean of the duration of the pause as a function of the emotions and position of characters is shown in Figure 14.
A 4 × 3 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,247) = 5.353, P < .001) and the position of characters (F(2,247) = 15.09, P < .001). There was a significant interaction term of the emotion and position of characters (F(6,247) = 2.993, P < .008).
A test of the simple main effect of emotion revealed a significant effect of emotion in the end of the clause and voiced sound (P < .003). Scheffe tests on the emotion factor in these position of characters revealed the following: the duration of pause of anger was significantly longer than the durations of pause of neutral and joy in the end of the clause (P < .045); the duration of pause of sadness was significantly longer than the duration of pause of neutral in the end of clause (P < .038); the duration of pause of joy was significantly shorter than the durations of pause of neutral and anger in the voiced sound (P < .042).
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (P < .050). Scheffe tests on the factor of the position of characters in each emotion revealed the following: the duration of pause of the end of the clause was significantly longer than the duration of pause of the inner clause in joy and sadness (P < .046); the duration of pause of the end of the clause was significantly longer than the other durations in anger (P < .002).
(c) Finger Load
The mean of the finger load as a function of the emotions and position of characters is shown in Figure 15.
A 4 × 4 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,283) = 229.5, P < .001) and the position of characters (F(3,283) = 50.11, P < .001). There was a significant interaction term of the emotion and position of characters (F(9,283) = 6.707, P < .001).
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (P < .001). Scheffe tests on the emotion factor in each position of characters revealed the following: the finger load of anger was significantly larger than the other finger loads in all positions of characters (P < .001); the finger load of sadness was significantly smaller than the other finger loads in the inner clause, end of sentence, and voiced sound (P < .001); the finger load of joy was significantly larger than the finger load of neutral in the inner clause and end of clause (P < .028).
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (P < .013). Scheffe tests on the factor of the position of characters in each emotion revealed the following: the finger load of the voiced sound was significantly larger than the finger loads of the end of clauses and inner clause in neutral, joy, and anger (P < .033); the finger load of the end of sentence was significantly larger than the finger loads of the end of clauses and inner clause in joy and anger (P < .041).
3.2.2. Results of One-Handed Finger Braille Experiment
(a) Duration of Code
The mean of the duration of the code as a function of the emotions and position of characters is shown in Figure 16.
A 4 × 5 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,579) = 109.9, P < .001) and the position of characters (F(4,579) = 29.90, P < .001). There was a significant interaction term of the emotion and position of characters (F(12,579) = 12.27, P < .001).
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (P < .001, the end of clause P < .034). Scheffe tests on the emotion factor in each position of characters revealed the following: the duration of code of sadness was significantly longer than the other durations in the inner character, inner clause, and end of sentence (P < .001); the duration of code of joy was significantly shorter than the other durations in the inner character, inner clause, and end of sentence (P < .028); the duration of code of sadness was significantly longer than the durations of code of joy and anger in the voiced sound (P < .007); the duration of code of joy was significantly shorter than the duration of neutral in the voiced sound (P < .024).
A test of the simple main effect of the position of characters revealed a significant effect of position of characters in all emotions (P < .001, neutral P < .031). Scheffe tests on the factor of the position of characters in each emotion revealed the following: the duration of code of the end of clause was significantly longer than the other durations in joy (P < .001); the durations of code of the end of clause and end of sentence were significantly longer than the durations of code of the inner character, inner clause and voiced sound in sadness (P < .025); the durations of code of the end of clause and end of sentence were significantly longer than the duration of code of the inner character in anger (P < .021).
(b) Duration of Pause
The mean of the duration of the pause as a function of the emotions and position of characters is shown in Figure 17.
A 4 × 4 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,541) = 4.390, P < .005) and the position of characters (F(3,541) = 7.827, P < .001). Scheffe tests on the emotion factor revealed that the duration of pause of neutral was significantly longer than the other durations (P < .001). Scheffe tests on the factor of the position of characters revealed that the duration of pause of the inner character was significantly shorter than the durations of pause of the inner clause and end of clause (P < .009).
(c) Finger Load
The mean of the finger load as a function of the emotions and position of characters is shown in Figure 18.
A 4 × 5 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,580) = 106.5, P < .001) and the position of characters (F(4,580) = 36.00, P < .001). There was a significant interaction term of the emotion and position of characters (F(12,580) = 2.999, P < .001).
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (P < .001). Scheffe tests on the emotion factor in each position of characters revealed the following: the finger load of anger was significantly larger than the other finger loads in all positions of characters (P < .001); the finger load of joy was significantly larger than the finger loads of neutral and sadness in all positions of characters (P < .017); the finger load of sadness was significantly smaller than the other finger loads in all positions of characters (P < .028).
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (P < .001). Scheffe tests on the factor of the position of characters in each emotion revealed the following: the finger load of the end of sentence was significantly larger than the finger load of the inner clause in neutral, joy, and anger (P < .001); the finger load of the end of sentence was significantly larger than the other finger loads in sadness (P < .013).
3.3. Discussion
As indicated by the results of ANOVA (3.2.1–3.2.2), the features of emotional expression were as follows: (1) the duration of the code of joy was significantly shorter than that of the other emotions; (2) the duration of the code of sadness was significantly longer than that of the other emotions; (3) the finger load of anger was significantly larger than that of the other emotions; (4) the finger load of joy was significantly larger than that of sadness and neutral. These features were very similar to the results of Experiment I.
As indicated by the results of ANOVA (3.2.1–3.2.2), the followings were the features of prosody: (1) the duration of the code of the end of the clause was significantly longer; (2) the duration of the pause of the end of the clause was significantly longer; (3) the duration of the code of the end of the sentence was significantly longer; (4) the finger load of the end of the sentence was significantly larger; (5) the duration of the code of the double consonant was significantly shorter; (6) the duration of the code of the voiced sound was significantly shorter; (7) the finger load of the voiced sound was significantly larger. These features were also very similar to the results of Experiment I.
In addition, there was a tendency for the finger load of the additional experiment to be larger than the finger load of Experiment I. This tendency could be a daily variance of the dotting strength of Finger Braille.
Therefore, we can confirm that the features of emotional expression and prosody are independent of the dotted dialogues.
4. Emotional Communication by the Unskilled (Experiment II)
To analyze the effectiveness of emotional expression and emotional communication between people who are unskilled in Finger Braille, we conducted another experiment of emotional communication using Finger Braille (Experiment II) [24].
4.1. Methods
The subjects were twelve non-disabled college students (ages: 21–26 years old) who were not unskilled in Finger Braille. All subjects were right-handed. All subjects gave informed consent after hearing a description of the study.
The dialogues were “Rain has fallen (Ame futte kita )” and “It is the time (Jikan desuyo )”, which were the same dialogues used in Experiment I.
Expressed emotions were neutral, joy, sadness, and anger, which were identical to those used in Experiment I. We set six scenes (two dialogues with joy, sadness, and anger) for subjects, which were also identical to the ones used in Experiment I.
In this experiment, we adopted one-handed Finger Braille (right hand), because one-handed Finger Braille is easier than two-handed Finger Braille for unskilled individuals.
The variables compose a 4 × 6 within-subjects design: expressed emotion (neutral, joy, sadness, anger) and position of characters (inner character, inner clause, end of clause, end of sentence, voiced sound, double consonant). The dependent variables were duration of code, duration of pause, and finger load.
The experimental flow is shown in Figure 19. The subjects were divided into four groups. The experiment was conducted for four days. On day 1, two practice sessions were conducted, followed by two intragroup experiment sessions. On days 2–4, three intergroup sessions were conducted. In the experiment session, a subject became a sender and another subject became a receiver. The experiment consisted of 132 sessions for all pairs of subjects: 11 pairs as sender and 11 pairs as receiver.
The sender and receiver sat side by side during the experiment to promote the emotional expression by the sender. To simulate deafblindness, the receiver wore earplugs, headphones playing white noise, and a night shade. The sender dotted one-handed Finger Braille (right-hand) on the fingers of the receiver eight times per session in a randomized order of dialogues with emotions. To eliminate the influence of the previous dotted dialogue with emotion, the subject dotted at intervals (1-2 minutes) and confirmed the next scene of dialogue with emotion. The sender observed a sheet of dot patterns of the experimental dialogues, and the receiver responded to the dialogue and emotion after the sender dotted one time. Unlike the interpreters in Experiment I, the subjects were not taught the features of emotional expression and prosody. The subjects expressed emotions freely by changing the strength and speed of dotting.
The receiver put his fingers on the pressure sensor sheet (Grip Force Sensor System, Nitta), which measured the change of pressure as a result of dotting. The sampling frequency was 150 Hz, the measurement range was 20–200 kPa, and the sensibility was 781 Pa. The sensor sheet was divided into 20 blocks, and the size of each block was 16 × 16 mm. The sensor matrix of the block was 4 × 4. The receiver put his index, middle, and ring fingers on the three blocks.
4.2. Results
4.2.1. Results of Emotional Expression
We calculated the duration of code, duration of pause, and finger load for each dotting by the change of pressure, as in Experiment I (see Figure 4).
(a) Duration of Code
The mean of the duration of the code as a function of the emotions and position of characters is shown in Figure 20.
A 4 × 6 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,13689) = 675.6, P < .001) and the position of characters (F(5,13689) = 39.13, P < .001). There was a significant interaction term of the emotion and position of characters (F(15,13689) = 5.153, P < .001).
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (P < .001). Scheffe tests on the emotion factor in each position of characters revealed the following: the duration of code of joy was significantly shorter than the other durations in all positions of characters (P < .001); the duration of code of anger was significantly longer than the other durations in the inner character, inner clause, end of sentence, voiced sound, and double consonant (P < .001, double consonant P < .016); the duration of code of anger was significantly longer than the duration of code of neutral in the end of clause (P < .006).
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (P < .001). Scheffe tests on the factor of the position of characters in each emotion revealed the following: the durations of code of the inner clause and end of clause were significantly longer than the durations of the voiced sound and double consonant in neutral and sadness (P < .012); the duration of code of the end of the sentence was significantly shorter than the durations of code of the inner clause, inner character, voiced sound, and end of clause in joy (P < .017); the duration of code of the voiced sound was significantly shorter than the durations of code of the inner clause, end of clause, and end of sentence in anger (P < .016).
(b) Duration of Pause
The mean of the duration of the pause as a function of the emotions and position of characters is shown in Figure 21. We defined the duration of pause before the double consonant as the duration of pause of the double consonant and the duration of pause after the double consonant as the duration of pause of the inner clause, as in the Experiment I. There was no duration of pause at the ends of sentences.
A 4 × 6 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,12607) = 43.926, P < .001) and the position of characters (F(5,12607) = 320.3, P < .001). Scheffe tests on the emotion factor revealed that the duration of pause of sadness was significantly longer than the other durations (P < .001), and the duration of pause of anger was significantly shorter than the other durations (P < .001). Scheffe tests on the factor of the position of characters revealed the following: the duration of pause of the end of the clause was significantly longer than the other durations (P < .001); the duration of pause of the inner character was significantly shorter than the other durations (P < .001); the duration of pause of the inner clause was significantly longer than the durations of pause of the voiced sound, and double consonant (P < .001).
(c) Finger Load
The mean of the finger load as a function of the emotions and the position of characters is shown in Figure 22.
A 4 × 6 within-subjects analysis of variances (ANOVA) revealed two significant main effects: emotion (F(3,13689) = 728.9, P < .001) and the position of characters (F(5,13689) = 378.2, P < .001). There was a significant interaction term of the emotion and position of characters (F(15,13689) = 25.29, P < .001).
A test of the simple main effect of emotion revealed a significant effect of emotion in all positions of characters (P < .001). Scheffe tests on the emotion factor in each position of characters revealed that the finger load of anger was significantly larger than the other finger loads in all positions of characters (P < .001), and the finger load of sadness was significantly smaller than the other finger loads in all positions of characters (P < .002).
A test of the simple main effect of the position of characters revealed a significant effect of the position of characters in all emotions (P < .034). Scheffe tests on the factor of the position of characters in each emotion revealed that the finger loads of the inner character and double consonant were significantly larger than the finger loads of the end of clause, end of sentence, inner clause and voiced sound in all emotions (P < .001).
4.2.2. Results of Emotional Communication
In this experiment, the receiver answered the recognized dialogue and emotion dotted by the sender after each dotting. As a result, the accuracy of recognition of the dialogues was a total of 96.5%. The relationship between expressed emotions by the senders, recognized emotions by the receivers, and their coincidence ratios is presented in Table 6. The coincidence ratio of anger was the highest (84.5%), and the coincidence ratio of sadness was second (78.4%). If the coincidence ratio was high, the emotions expressed by the sender were communicated well to the receiver. If the coincidence ratio was low, the emotions expressed by the sender were communicated poorly to the receiver. The coincidence ratios as a function of the emotions and subject (sender) are shown in Figure 23. Subject 2 was the best communicative sender (coincidence ratio 90.9%); Subject 7 was the worst communicative sender (coincidence ratio 55.7%).
The change of coincidence ratios as a function of the emotions and order of dotted dialogue in a session are shown in Figure 24. In dotted dialogues 1 and 2, the coincidence ratios of the four emotions were almost 60%. In dotted dialogues 3–6, the coincidence ratios of anger and sadness were 80%–90%. The coincidence ratio of joy was almost 70%, and the coincidence ratio of neutral was almost 60%. In dotted dialogues 7 and 8, the coincidence ratios of the four emotions were 80% or higher.
4.2.3. Questionnaire Results (Unskilled Subjects)
After the experiment, the subjects were asked how they expressed these emotions, and their responses are listed in Table 7. These responses were similar to the ones given by the interpreters in Experiment I.
Six subjects answered that they dotted Finger Braille while sometimes glancing at the sheet of dot patterns. The other six subjects answered that they dotted Finger Braille while continually observing the sheet of dot patterns. As a degree of emotional expression, one subject answered “well expressed”; six subjects answered “almost expressed”; five subjects answered “somewhat expressed.”
All subjects recognized the dotted dialogues by the first two characters or the last two characters. Some subjects also answered that it was hard to recognize the expressed emotions if the first emotion in the session was neutral, and two or three dialogues in the session were necessary to recognize the neutral level of the sender.
4.3. Discussion
4.3.1. Comparison of Emotional Expression between Interpreters and the Unskilled
There were similar features of the emotional expression between the interpreters in Experiment I and the unskilled subjects in Experiment II: the duration of joy was significantly shorter than that of the other emotions; the finger load of anger was significantly larger than that of the other emotions. As different features, the duration of the code of anger by the interpreters was slightly shorter, but the duration of the code of anger by the unskilled subjects was significantly longer. In a previous study, we found that the unskilled subjects faced difficulties in dotting strongly in a short time [25]. Therefore, the duration of code of strong dotting by the unskilled subjects had a tendency to be long.
Since the unskilled subjects were not taught the prosody of Finger Braille, they could not express its metrical structure. The unskilled subjects dotted Finger Braille by glancing at or continually observing the sheet of dot patterns, and then many of them paused at the end of a clause and checked the patterns.
According to the questionnaire results, the impressions of emotional expression by the unskilled subjects were similar to those by the interpreters.
4.3.2. Discriminant Analysis of Emotional Expression
To evaluate emotional expression by the subjects in Experiment II, twelve discriminant analyses on the data of each subject were conducted (N = 12,627 total). Discriminant variables were the duration of the code, duration of the pause, and the finger load. The dotting of the end of the sentence was excluded, because there was no duration of pause.
The total discriminant ratios of emotion are listed in Table 8. The total discriminant ratio was 60.6%. The discriminant ratio of joy was the highest (80.9%), and the discriminant ratio of anger was second (63.6%). The dottings of neutral and sadness were frequently misdiscriminated between each other. The discriminant ratio as a function of the emotions and subject is presented in Figure 25. If the discriminant ratio is high, the subject expressed emotions well; if the discriminant ratio is low, the subject expressed emotions poorly. Subject 9 was the best expressed subject (discriminant ratio 74.9%); Subject 7 was the worst expressed subject (discriminant ratio 47.0%).
The represented variables for discriminant functions, contribution ratios, and significance probability are listed in Table 9. For Subjects 2, 4, 8, 9, and 11, discriminant function 1 represented the duration of the code, discriminant function 2 represented the finger load, and discriminant function 3 represented the duration of the pause (group “CLP”). For Subjects 1, 3, 5, 7, 10, and 12, discriminant function 1 represented the finger load, discriminant function 2 represented the duration of the code, and discriminant function 3 represented the duration of the pause (group “LCP”). For Subject 6, discriminant function 1 represented the duration of the code, discriminant function 2 represented the duration of the pause, and discriminant function 3 represented the finger load (group “CPL”). For Subjects 2, 4, 9, and 12, discriminant functions 1, 2, and 3 contributed to emotional expression. For Subjects 1, 3, 5, 6, 7, 8, 10, and 11, discriminant functions 1 and 2 contributed to emotional expression. Subjects of the group CLP and LCP could control both the duration of the code and the finger load, and subjects of the group CLP could control the duration of the code well.
The mean of the discriminant ratio of group CLP was 66.5% (S.D. = 6.0), and the mean of the discriminant ratio of group LCP and group CPL was 56.4% (S.D. = 7.3). A t-test revealed a significant difference between group CLP and groups LCP and CPL ((10) = 2.54, P < .030). To discuss the level of learning of emotional expression using Finger Braille, we calculated the discriminant ratio of experimental sessions 1–4 and 5–11. As for the experimental sessions 1–4, the mean of the discriminant ratio of group CLP was 59.0% (S.D. = 11.6), and the mean of the discriminant ratio of group LCP and group CPL was 49.3% (S.D. = 10.3). A t-test revealed no significant difference between group CLP and groups LCP and CPL ((10) = 1.55, P < .153). As for the experimental sessions 5–11, the mean of the discriminant ratio of group CLP was 70.7% (S.D. = 4.9), and the mean of the discriminant ratio of group LCP and group CPL was 60.4% (S.D. = 7.3). A t-test revealed a significant difference between group CLP and groups LCP and CPL ((10) = 2.73, P < .021). These results indicate the following: the discriminant ratio of group CLP was significantly higher in experimental sessions 5–11; controlling the dotting strength (finger load) was easier for the unskilled people; in addition, controlling the dotting speed (duration of code) was a concern about expressing emotions well using Finger Braille. Group CLP had learned the emotional expression well during experimental sessions 1–4. Although we had not checked the dexterity of subjects concerning dotting Finger Braille (e.g., playing piano or typing on a keyboard) in the present study, we surmise that group CLP was more dexterous than groups LCP and CPL.
The relationship between the discriminant ratio and coincidence ratio of each subject (sender) is shown in Figure 26. As a result of regression analysis (N = 12), the correlation coefficient R was 0.862 (P < .001). Therefore, if the sender could express emotions well, the expressed emotions could be recognized accurately by the receiver.
4.3.3. Features of Emotional Communication
As shown in Figure 24, the receivers could recognize anger and sadness in dotted dialogue 3 and neutral and joy in dotted dialogue 7. These results indicate that the receivers first recognized the strength of dotting, and after the receivers better comprehended the level of neutral by the senders, they could recognize the duration of dotting.
Now, we discuss the features of emotional communication by other communication media.
Shirasawa et al., Iida et al., and Kinjou et al. analyzed emotional communication by speech [16–18].
In Shirasawa’s experiment [16], one speaker, who had experience as an actor and announcer, recorded eight phrases with neutral, joy, sadness, anger, surprise, and disgust. Then, 55 subjects heard the phrases and identified the emotions. The coincidence ratio was 51.5%. The coincidence ratio of sadness was the highest (64%).
In Iida’s experiment [17], two speakers recorded four phrases with neutral, joy, sadness, anger, surprise and disgust. Then, seven subjects heard the phrases and identified the emotions. The coincidence ratio was 50.0%. The coincidence ratio of sadness was the highest (91.1%).
In Kinjou’s experiment [18], six speakers, of whom three had experience as actors, recorded 24 words with neutral, joy, sadness, and anger. Then, 55 subjects heard the words and identified the emotions. The coincidence ratio was 77.1%. The coincidence ratio of neutral was the highest (83.3%).
Yoshikawa and Morisaki analyzed emotional communication by facial expression [19]. One subject expressed emotions by facial expression and another subject identified the emotions. Two situations with joy, sadness, anger, surprise, disgust, fear, and annoyance were prepared. The subjects were nine acquainted pairs and nine unacquainted pairs. The coincidence ratio was 61.6% for the acquainted pairs. The coincidence ratio of joy was the highest (97%) for the acquainted pairs. The coincidence ratio was 58.8% for the unacquainted pairs. The coincidence ratio of joy was the highest (93%) for the unacquainted pairs.
Ohgushi and Hattori analyzed emotional communication by vocal performance [20]. Three female singers sang the vowel sound “a” with joy, sadness, anger, fear, and neutral emotion. Fifty-one subjects heard the vocal performance and identified the emotions. The coincidence ratio was 52.8%. The highest coincidence ratio was anger (66%).
The results of the present study showed that the coincidence ratios of Experiment II were equal to or more than the coincidence ratios of these previous studies. Thus, it was considered that the unskilled people can express and communicate emotions using Finger Braille, which is a new tactile communication medium.
However, some subjects expressed emotions poorly, and it was difficult for the receivers to recognize the dotted emotions. Thus, some assistance in emotional communication was needed for the non-disabled people. One of the methods to teach emotional expression to unskilled people is to teach the impression of emotional expression by applying the similarities of the impression of emotional expression between the interpreters and the unskilled people (e.g., “rhythmically” for joy, “weakly and slowly” for sadness, “strongly and quickly” for anger). Another method of teaching emotional expression is to introduce a dot pattern with some symbols directly representing dotting speed and strength (e.g., a dot pattern with a long or short array for dotting speed, and a dot pattern with a colored background for dotting strength). Some example patterns for teaching emotional expression to people unskilled in Finger Braille are shown in Figure 27. We are planning to develop and implement these methods of teaching emotional expression in future studies.
5. Conclusions
In the present study, emotional expression by interpreters using Finger Braille (Experiment I) was first examined, and the features of emotional expression using Finger Braille were analyzed. The resulting features were as follows: (1) the duration of the code of joy was significantly shorter than that of the other emotions (neutral, anger, sadness); (2) the duration of the code of sadness was significantly longer than that of the other emotions; (3) the finger load of anger was significantly larger than that of the other emotions; (4) the finger load of joy was significantly larger than that of sadness and neutral; (5) the duration of the code of anger was significantly shorter than that of sadness and neutral. As shown by the results of the additional experiment, these features of emotional expression are independent of the dotted dialogue.
Next, emotional communication by people unskilled in Finger Braille (Experiment II) was examined, and the effectiveness of emotional expression and emotional communication between these individuals was analyzed. The results indicate that the features and the impression of emotional expression by the unskilled people were very similar to those by the interpreters, and the coincidence ratio of emotional communication was 75.1%. Therefore, it was confirmed that unskilled people can express and communicate emotions using Finger Braille.
The followings are plans for future studies: (1) expansion of the emotions expressed and the number of subjects used in Finger Braille experiments, (2) development of methods to teach non-disabled people emotional expression using Finger Braille, (3) analysis of the relationship between emotional expression using Finger Braille and speech, and (4) development of an emotion recognition algorithm.
Acknowledgments
The authors greatly thank Ms. Satoko Mishina and Ms. Megumi Fukuma (interpreters of Finger Braille) for their support. This study was supported by the Japan Society for the Promotion of Science under a Grant-in-Aid for Scientific Research (no. 21500522) and the Ministry of Education, Culture, Sports, Science and Technology of Japan under a Grant-in-Aid for Scientific Research (no. 16700430). This study was partly supported by Kanagawa Academy of Science and Technology (KAST) under a research grant.