Abstract
Steady improvements in technologies that measure human emotional response offer new possibilities for making computer games more immersive. This paper reviews the history of designs a particular branch of affective technologies that acquire electrodermal response readings from human subjects. Electrodermal response meters have gone through continual improvements to better measure these nervous responses, but still fall short of the capabilities of today's technology. Electrodermal response traditionally have been labor intensive. Protocols and transcription of subject responses were recorded on separate documents, forcing constant shifts of attention between scripts, electrodermal measuring devices and of observations and subject responses. These problems can be resolved by collecting more information and integrating it in a computer interface that is, by adding relevant sensors in addition to the basic electrodermal resistance reading to untangle (1) body resistance; (2) skin resistance; (3) grip movements; other (4) factors affecting the neural processing for regulation of the body. A device that solves these problems is presented and discussed. It is argued that the electrodermal response datastreams can be enriched through the use of added sensors and a digital acquisition and processing of information, which should further experimentation and use of the technology.
1. Gaming and Affective Information Streams
The evolution in the past two decades of low-cost microcontroller unit-(MCU-) based-analog-to digital conversion (ADC) technology has made possible the real-time integration of the gamers emotional and physical state into the gameplay. Despite the possibilities, use of these technologies has so far been limited, often with a game objective of controlling stress rather than making the game more interesting.
The measurement of emotional response through observations of physiological changes, such as sweating and involuntary muscle contractions, has been employed for a variety of purposes, from the spiritual to the forensic, since prehistory. Whether or not these measurements were actually a measure of emotion, or anything else, is difficult to say, but is probably in any precise sense, incorrect. From a phenomenological point of view, any physiological measure you can make, even with a high accuracy and reliability, is only a representation of the phenomenon you try to study. But, it is not the phenomenon itself, we should always be cautious when linking high brain functions with physiological signs. I will use the term “emotion,” as that thing that is being measured by EDR for simplicity throughout the text. But the reader should be aware that this usage is undoubtedly incorrect, or at least needs exploration before it can become a fully developed concept. The emotion we consider in this research is the “emotion” that is measured by the devices discussed here.
There are numerous historical examples of inventions to read “emotion”, most often in a forensic context. In ancient China suspects were told to hold a handful of rice in their mouths during a prosecutor's argument. The suspect was considered guilty if, by the end of that argument, the rice stayed dry, because salivation was believed to cease in times of anxiety. Firewalking has been a spiritual mainstay of Eastern Orthodox Christians in parts of Greece and Bulgaria, of !Kung Bushmen, of Japanese Taoists and Buddhists, and of young girls in Bali in a ceremony called Sanghyang dedari. In medieval Russia, Yaroslav the Wise, Grand Prince of Kiev, legalized the use of trial by ordeal in disputes between the individuals, saying that the plaintiff can require that the defendant proves his innocence through iron test whereas if the claim is less than half a grivna, then a water test could be used. The accused would hold a red-hot iron in his hands or remove a ring from a pot of boiling water after which the judges would wrap and seal the hands. Innocence would be shown by a complete lack of injury three days later. These tests depended on the subjects secreting copious amounts of sweat and saliva as trial by ordeal and firewalking protected the palms and soles with a Leidenfrost effect.
Similarly, involuntary muscle movements were also seen as a sign of guilt and were behind West African traditions that required anyone suspected of a crime to be made to hand a bird's egg to another person. Anyone who broke the egg would be considered guilty, based on their nervousness. Similarly, in ancient India, a suspect was given neutral words and critical words closely related to the crime he had allegedly committed. The person was to blurt out the first word that came to his mind and hit the gong at the same time. It was observed that whenever a critical word was given, the suspect would hit the gong more forcefully. Recently, interest has risen in the reading of involuntary facial micromovements to detect lying.
One potential information source for affective data acquisition has particular attractions for gaming—electrodermal response. Electrodermal response has, over the past 150 years, been at one time or another called galvanic skin response and other terms. Today it competes with several other technologies—functional magnetic resonance imaging (fMRI), positron emission tomography (PET), and electroencephalography (EEG) being the main technologies for affective data acquisition. Table 1 delineates the main differences in these approaches and their data streams.
In comparison to the alternatives, electrodermal response is:(1)noninvasive, where alternatives may require harnesses or piercing of skin which are potentially unhygienic, limit mobility, and thus may limit emotional response;(2)inexpensive and cost effective;(3)relatively sensitive when properly implemented and can supply real-time data streams, but which are not as data rich as alternatives.
The use of the electrical measurement to detect emotion dates to the time of its invention by Italian physiologist Luigi Galvani in his paper on animal electricity in 1791. The seminal study of psychosomatic bioelectric effects appeared in the 1870s in the work of Féré [1] in his study of basal skin resistance and stimulated skin resistance in connection with psychological state. He suggested applying his observations to the budding fields of criminology and hypnotherapy among other uses. Russian physiologist Tarchanoff [2] independently studied a similar correlation between electricity and emotion about the same time. This line of research was approached in a systematic fashion by Wilhelm [3] work as the basis for the use of electrodermal response instruments by Jung [4] which made these methods popular with practicing psychotherapists. Jung‘s 1906 book Studies in Word Analysis described clinical techniques for electrodermal measurement while words are read to the subject from a prepared list. If a word on this list was emotionally charged, there was a change in body resistance that caused a reading. These protocols evolved in the 1930s (most notably by the Chicago police department) as police began using polygraphs in their forensic work—poly because they integrated mechanisms for measuring skin resistance, blood pressure, respiration and so forth. Electrodermal measurement devices were used for a variety of types of research in the 1960s through the late 1970s, with a decline in use as more sophisticated techniques (such as EEG and MRI) replaced it in many areas of psychological research.
Linking EDA to emotion needs to be approached with caution. EDR is a witness of the autonomic nervous system activation through its orthosympathetic component. It is an indicator of the activation of some human system or systems and can respond to both pleasant as well unpleasant stimuli. EDR gives information about the intensity of a stimulus but not about its valence. Nonetheless, the technology merits study simply because electrodermal measurement devices are cheap, and possibly the least invasive of available techniques to measure emotional response. These are important in many applications, but the low entry costs and efficient user interfaces required of computer gaming make them especially demanding.
Electrodermal measurement devices are currently used in (1) biofeedback and protocols designed to help control epileptic seizures [5, 6] relaxation and analysis within various spiritual groups [7, 8], as a controller for video games and other computer interfaces [9–11] and in forensic interviews. Forensic applications through computerized polygraph testing are the largest part of the market for electrodermal measurement. There are several polygraph companies—Lafayette Instruments with 85% of the global market, Stoelting Company, Polygraph Technology Asia from Singapore and TH Polygraphs from TsingHua Tongfang in China—as well as Limestone Technologies which sells software for interpreting results. Though financial data is sparse, the market appears to be growing at about 10% annually, and faster outside the USA. Polygraphs sell for several thousand dollars and typically include electrodermal activity channels as two of between 5–8 affective data channels. The US Department of Commerce requires an export license for polygraph systems for any destination other than Australia, Japan, New Zealand, or NATO Countries excluding the Czech Republic, Hungary, and Poland. There are around two dozen polygraph schools around the USA.
This paper:(1)takes a critical look at what is actually being measured by these devices; (2)assesses how the user interface design of its various incarnations has helped or hindered these measurements; (3)reviews how electrodermal measurement device designs have evolved and assessed their design strengths and weaknesses; (4)designs an electrodermal response mechanism and protocols suitable for affective information streams in gaming.
The early electrodermal meters were not simple to use, and this constrained research. They were fundamentally Wheatstone bridge null detectors (see Figure 1) that ran a small current through the subject's palms or soles of his feet ( in Figure 1) while varying a resistor ( in Figure 1) until a moving arm galvanometer read zero. They lacked amplification (the electron tube had not yet been commercialized) and were crude and finicky.
In a basic Wheatstone bridge (Figure 1) Kirchoff’s Laws balance the bridge where current between 1 and 2 is zero. Detecting zero current can be done to extremely high accuracy. At the “null” point the ratio of resistances is . Detection of the “null” point is done in electrodermal meters by varying what was called the “tone arm” resistor ( in Figure 1, the name was derived from musical metaphors and had little to do with science) until the current is zero. Often the electrodermal meters have preset implying that , that is, that the resistance across the tone arm potentiometer will “read” the resistance across the hand held electrodes at the null setting. Using this “null” configuration was useful in the early days, when temperature and humidity might make component values drift—when they drifted in the same direction, rough accuracy would be maintained. The idiosyncrasies of early e-meters set one of the operating design parameters for later devices. Rather than directly reading out , the operator continually adjusts the tone arm until the meter reads zero. With the commercialization of vacuum tubes in the beginning of the 20th century, electrical engineers began experimenting with amplifying the Wheatstone bridge-galvanometer “null detector” setup that existed.
Though the Depression and WWII made materials scarce and diverted amateur electrical engineering expertise away from such projects, a parallel stream of engineering in forensic lie detection and polygraphs continued, with Chicago police becoming first movers in adopting the technology for forensics in the 1930s. The development of the polygraph integrated electrodermal measurement devices with respiration, blood pressure, and perhaps temperature and other physiological responses to stress. The layering of psychophysiological measurement devices reflected the limitations in any particular physiological measurement, but also reflected a frustration with the delicacy and inaccuracy of many of the early instruments, which were just figuring out how to use vacuum tube amplification to their advantage.
After WWII, war surplus electronics became widely available, amateur interest in electrodermal response grew, and many electrodermal response devices were marketed. To gain an idea of the level and popularity of this technology at this point in time, the reader can visit an intriguing collection of pictures and descriptions of approximately two dozen electrodermal measurement devices from the late 1940s and 1950s at the Polygraph Museum website at http://www.lie2me.net/thepolygraphmuseum/id24.html.(1)Electrodermal measurement technologies have grown considerably more reliable and affordable with the development of digital electronics; they are now popular with hobbyists, and many designs have been posted to the Internet. Lego NXT blocks along with Lego-LabView programs have been used to build a low cost PC-based electrodermal measurement device [12]. Low-cost analog-to-digital converter boards are now widely available and can be used to provide the electrodermal measurement device resistance measurement in digital form, with all of the meters and dials being transferred to the computer.
Detecting emotional information begins with passive sensors which capture data about the user's physical state or behavior without interpreting the input. The data gathered is analogous to the cues humans use to perceive emotions in others. For example, a video camera might capture facial expressions, body posture, and gestures, while a microphone might capture speech. Other sensors detect emotional cues by directly measuring physiological data, such as skin temperature and electrodermal resistance [13]. Recognizing emotional information requires the extraction of meaningful patterns from the gathered data.
In e-learning applications, affective computing can be used to adjust the presentation style of a computerized tutor when a learner is bored, interested, frustrated, or pleased [14]. Potential applications are widespread: in theory, robotic systems capable of processing affective information can be more flexible in uncertain or complex environments; digital pets and appliances can provide a higher degree of autonomy and responsiveness; social monitoring in machines such as cars can monitor the emotion of all occupants and engage in additional safety measures; in human computer interaction, affective mirrors allow the user to see how he or she performs; emotion monitoring agents send a warning before one sends an angry email; or even music players select tracks based on mood. Affective computing applications have been suggested for communicative technologies for use by people with autism [15]. Further afield, affective computing can help devices exhibit either what might be perceived as innate emotional capabilities or convincingly simulated emotions, allowing conversational agents to enrich and facilitate interactivity between human and machine [16]. While human emotions are often associated with surges in hormones and other neuropeptides, emotions in machines could be associated with abstract states associated with progress (or lack of progress) in autonomous learning systems. Indeed, [17] states that emotion is “not especially different from the processes that we call thinking” though that may merely reflect Minsky’s personal views. Damasio [8, 18] extensively describes the brain research memory processing and implications for our decision making. There are, at least, two streams of brain processing of sensory data that affect the brain’s long-term imprints—a declarative and an emotional stream. The two interact in the complex ways, and ultimately the implementations of affective streams added to declarative streams of information in gaming would benefit greatly from some of the ideas distilled in these books.
In forensics, polygraphic systems—ones that incorporate EDR as well as heart rate respiration and other bodily measurements—are preferred. Where measurements are used in tasks that have legal, employment, or other severe consequences, more information is better as any improvement in accuracy can be justified given the high cost of type I and II errors. Gaming interfaces are subject to different demands. Game economics require a low-cost (but not necessarily reliable) user interface for most games; inaccuracy of responses can be incorporated into the game script in engaging ways, for example, as gambles, as personal characteristics, or as acts of God.
The evolution in the past two decades of low-cost microcontroller unit-(MCU-) based analog-to-digital conversion (ADC) technology has made possible the low-cost real-time integration of the affective emotional and physical state into the gameplay.
Despite the possibilities, use of these technologies has so far been very limited, most famously within a certain genre of computer gaming that rely on biofeedback. Commercial integration of biofeedback was explored in a game called Journey to Wild Divine released in 2001, ostensibly to promote stress management. Graphics and gameplay were similar to the 1991 game Myst—basically a montage of stills which controlled production cost and affordability. In comparison, modern games emphasize a point of view and evolving story. Wild Divine’s gameplay can be likened to the silent films of Sergei Eisenstein, which influenced the work of later artists, but whose montage form was overtaken by the Hollywood style that emphasized point of view. Other biofeedback games have followed with similar objectives. ThoughtStream sells a similar biofeedback system; HeartMath, sells emWave, a finger-GSR/ear-pulse sensor that emits LED display and sounds; !StressEraser and Resperate also sell similar devices designed to lower blood pressure.
2. What Is Electrodermal Response Measuring?
Féré [1] and Wundt’s [3] research recognized that at least three physiological processes were influencing readings—(1) resistance through the body; (2) skin resistance having to do with the conductivity at the skin-metal interface; (3) micromovements in muscles, in particular of the hands holding the electrodes. Since these pioneering studies, many researchers have attempted to explain the phenomenon (e.g., Lacey and Siegel [19]; McCleary [20]; Bitterman and Holtzman [21]; Darrow [22]; Montagu and Coles [23]; Shahani et al. [24]; Boucsein [25]). Such interest was understandable. Since the adoption of the polygraph for forensic evidence collection by the Chicago Police Department in the 1930s, the need to better understand the exact nature of the evidence being presented in court (and how to defend its validity) was critical. The proliferation of commercial so-called “lie detection” technology in the late 1940s and early 1950s increased interest as well as availability of equipment for these studies.
The internal resistance of the human body ranges from approximately 300 ohms to 1,000 Ohms. Within the body, the tissues with the greatest resistance are bone and fat—nerves and muscle have the least resistance. Men have lower resistance than women because of the greater proportion of fat in women’s body mass. Additionally, body resistance varies with the thickness of limbs and torso—larger cross-sections conduct more current for the same type of tissue and thus have lower resistance. Since men are on average larger than women, this also lowers their resistance. Resistance also depends on length of the path between electrodes—when the current goes in the left hand and out the right foot, resistance will be much higher than if it goes in and out of adjacent fingers.(1)Electrodermal response associated with sweat gland activity is well established. Convincing evidence arises from experiments in which a direct correlation is seen between EDR and stimulated sweat gland activity. Furthermore, when sweat gland activity is abolished, then there is an absence of EDR signals [26]. The phenomenon of electrodermal resistance arises in the following way. The stratum corneum—the dry dead skin of the epidermis (from the Latin for horned layer)—is much less conductive than the underlying dermis and other internal tissues [27]. It is composed of large, flat, polyhedral, plate-like envelopes filled with keratin. These dead cells lack nuclei and are largely made of keratin, a protein that helps keep the skin hydrated by preventing water evaporation. Keratin is a form of protonic semiconductor (as opposed to an electronic semiconductor) meaning that the water content of keratin is not liquid water, but at best a hydrogen-bonded network of water molecules. It is effectively an insulator [28].
The stratum corneum can absorb water, aiding in hydration and explaining why humans and other animals experience wrinkling of the skin on the fingers and toes (“pruning”) when immersed in water for prolonged periods. In addition, this layer is responsible for the “spring back” or stretchy properties of skin. A weak glutenous protein bond pulls the skin back to its natural shape.
The thickness of the stratum corneum varies according to the amount of protection and/or grip required by a region of the body. For example, the hands are typically used to grasp objects, requiring the palms to be covered with a thick stratum corneum. In a similar manner, the sole of the foot is prone to injury, and so it is protected with a thick stratum corneum layer. In general, the stratum corneum contains 15 to 20 layers of dead cells, with a total thickness between 10 and 40 μm.
The stratum corneum is also porous and perforated with ducts from the eccrine sweat glands whose purpose is to dampen the epidermis in response to stress and environmental factors, particularly heat and humidity. The skin of an adult contains around 3 million eccrine sweat glands. Palms and soles have the highest numbers; estimates are that the palm of the hand has about 420 glands per square centimeter.
Eccrine sweat glands (Figure 2) are around 3.5 millimeters in length, with a pore size of 0.05 to 0.07 millimeters (for comparison, the diameter of a human hair is around 0.08 to 0.1 millimeters in diameter). Each pore sets on top of a palmer fingerprint ridge. Its physical purpose is to provide a liquid cooling system for the exterior of the human body. The palmer surfaces, palms and finger, and the plantar surfaces, soles of the feet and the toes, have an average of 420 pores per square centimeter of ridge friction skin surface. This compares to approximately 62 pores per square centimeter of the balance of the body's skin surface.
In fingerprinting, the eccrine pores help differentiate the ridges used in comparison, to incipient ridges (interpapillary lines, interstitial rudimentary/nascent ridges) that occur in the furrow between the papillary lines on the volar epidermis, which are smaller, lower and mostly shorter than normal papillary lines. There are no sweat glands present.
The coiled gland (Figure 2) is located in the deep dermis or at the border of the dermis and subcutaneous fat. It is composed of one distinct layer of clear and dark cells. The clear cells secrete glycogen, water, and electrolytes—which are responsible for the resistance changes measured by the electrodermal response mechanisms—and the dark cells secrete sialomucin. These secreatory cells are surrounded by contractile myoepithelial cells enclosed within a hyaline basement membrane with peripheral collagen fibers. The contractile myoepithelial cells are made of alpha smooth muscle actin and can contract and quickly expel the secretions of the sweat glands. Direct neural stimulation of the contractile myoepithelial cells can change the skin conductivity within hundredths of a second and can be responsible for the very quick responses seen in the electrodermal response mechanisms. Slower cycles in resistance may be due to the general excitation resulting from the diffusion of hormones, but are a result of the same contractions. Because the eccrine sweat glands are small and distances are short, they can change the contact resistance very quickly in response to neural impulses or less quickly to diffusion of hormones and neurotransmitters. This change in pore size is called the sympathetic skin response [29] and is influenced by the body’s homeostasis mechanisms. The sympathetic system is also very sensitive to external stimuli and emotional states.
The following details the surface characteristics of skin that are essential to understanding the response of EDR electronics that read dry electrodes (as is the case in the implementations of EDR in the survey instruments used in this research). The stratum corneum—the dry dead skin of the epidermis (from the Latin for horned layer)—is much less conductive than the underlying dermis and other internal tissues [27]. It is composed of large, flat, polyhedral, plate-like envelopes filled with keratin. These dead cells lack nuclei and are largely made of keratin, a protein that helps keep the skin hydrated by preventing water evaporation. Keratin is a form of protonic semiconductor (as opposed to an electronic semiconductor) meaning that the water content of keratin is not liquid water, but at best a hydrogen-bonded network of water molecules. It is effectively an insulator [28]. The stratum corneum can absorb water, aiding in hydration and explaining why humans and other animals experience wrinkling of the skin on the fingers and toes (“pruning”) when immersed in water for prolonged periods. In addition, this layer is responsible for the “spring back” or stretchy properties of skin. A weak glutenous protein bond pulls the skin back to its natural shape. The thickness of the stratum corneum varies according to the amount of protection and/or grip required by a region of the body. For example, the hands are typically used to grasp objects, requiring the palms to be covered with a thick stratum corneum. In a similar manner, the sole of the foot is prone to injury, and so it is protected with a thick stratum corneum layer. In general, the stratum corneum contains 15 to 20 layers of dead cells, with a total thickness between 10 and 40 μm. The stratum corneum is also porous and perforated with eccrine sweat glands whose purpose is to dampen the epidermis in response to stress and environmental factors, particularly heat and humidity. The skin of an adult contains around 3 million eccrine sweat glands. Palms and soles have the highest numbers; estimates are that the palm of the hand has about 420 glands per square centimeter.
Eccrine sweat glands are around 3.5 millimeters in length, with a pore size of 0.05 to 0.07 millimeters (for comparison, the diameter of a human hair is around 0.08 to 0.1 millimeters in diameter). Each pore sets on top of a palmer fingerprint ridge. Its physical purpose is to provide a liquid cooling system for the exterior of the human body. The palmer surfaces, palms and finger, and the plantar surfaces, soles of the feet and the toes, have an average of 420 pores per square centimeter of ridge friction skin surface. This compares to approximately 62 pores per square centimeter of the balance of the body's skin surface.
In fingerprinting, the eccrine pores help differentiate the ridges used in comparison, to incipient ridges that occur in the furrow between the papillary lines on the volar epidermis, which are smaller, lower, and mostly shorter than normal papillary lines. There are no sweat glands present.
The coiled gland is located in the deep dermis or at the border of the dermis and subcutaneous fat. It is composed of one distinct layer of clear and dark cells. The clear cells secrete glycogen, water, and electrolytes—which are responsible for the resistance changes measured by the electrodermal response meter—and the dark cells secrete sialomucin. These secretory cells are surrounded by contractile myoepithelial cells enclosed within a hyaline basement membrane with peripheral collagen fibers. The contractile myoepithelial cells are made of alpha smooth muscle actin and can contract and quickly expel the secretions of the sweat glands. Direct neural stimulation of the contractile myoepithelial cells can change the skin conductivity within hundredths of a second and can be responsible for the very quick responses seen in the electrodermal response meter. Slower cycles in resistance may be due to the general excitation resulting from the diffusion of hormones, but are a result of the same contractions. Because the eccrine sweat glands are small and distances are short, they can change the contact resistance very quickly in response to neural impulses or less quickly to diffusion of hormones and neurotransmitters. This change in pore size is called the sympathetic skin response [29] and is strongly influenced by the body’s thermoregulation and homeostasis mechanisms.
Human eccrine sweat is produced in the coiled gland (Figure 5) then travels up to the surface through the eccrine duct. It is composed chiefly of water with various salts and organic compounds in solution. It contains minute amounts of fatty materials, urea, and other wastes. The concentration of sodium varies from 20 to 38 g/kg (~35–65 mmol/L or about 2.0% to 3.8% concentration; almost exactly the same concentration of salt in seawater) and is lower in people accustomed to a hot environment. Sweats conductivity is linearly dependent on ion concentration.
When totally dry, the cells of the stratum corneum, the skin’s outer layer, have a typical resistance of over 1 megaohm; the normal hydration of these cells pulls the resistance found in living humans to between 50,000 ohms and 100,000 ohms. The skin’s resistance is much lower if it is wet, burned, or blistered, but “dry” skin encountered in electrodermal response applications typically has a resistance in excess of 50,000 ohms. In addition to acting like a resistor, the stratum corneum acts like a capacitor if placed in contact with a piece of metal—the underlying tissue acts as one plate of a capacitor and the metal surface as the other plate, with the dry stratum corneum as the “dielectric.” This effect is used in touch sensitive electronics.
The resistance of the epidermal layer will change dramatically if it is soaked with water-containing electrolytes—particularly if there are clear paths through the dead, dry cells of the epidermis all the way through to the more conductive underlying layers. It is just such a conductive pathway that is provided by the eccrine sweat glands.
The conductivity of a solution, for our purposes, is the inverse of its resistance . Since the shape of the conductor affects conductivity values, standardized measurements are expressed in specific conductivity units (Siemens/cm) to compensate for variations in electrode dimensions. Specific conductivity , where is the length of the column of liquid between the electrode and is the area of the electrodes. Sweat is not a great conductor (see Table 3) but has much higher conductivity than fat or other tissue, or than that of the stratum corneum of the palm. Thus any salt water sweat pathway through the eccrine ducts will essentially short the contacts on the outside of the epidermis to the conductive tissue in the body.
In the case of cylindrical handheld electrodes—the type commonly used in early EDR measurement devices—the stratum corneum may be seen as a large flat insulating layer (dielectric with resistance above 1 megaohm when dry, but usually 50,000 to 100,000 ohms due to residual moisture retained in the keratin) with an electrode on each side (the 300 to 1000 ohm resistance of the inner tissues being comparatively conductive).
The electrical resistance of an electrical element measures its opposition to the passage of an electric current; in resistive direct current circuits, the inverse of the resistance measure is electrical conductance, measuring how easily electricity flows along a certain path. The SI unit of electrical resistance is the ohm (Ω), while electrical conductance is measured in siemens (S) or the older term, mho (ohm spelled backwards). An object of uniform cross-section has a resistance proportional to its resistivity and length and inversely proportional to its cross-sectional area. EDR measurements assure a consistent measurement of resistance, or its inverse conductance, by running a regulated (i.e., voltage stabilized at all current levels) current through the material. All materials show some resistance, except for superconductors, which have a resistance of zero.
Any layer of sweat on the palms will be flattened by a handheld electrode. The eccrine gland can be thought of as a tube of ionized saltwater extending through the eccrine duct to the more conductive tissues underneath. The eccrine ducts are about 3.5 mm long by 0.05 mm wide at maximum, for a cell constant of about . Conductivity for a single sweat gland will be specific conductivity of salt water at the sweat concentration of 35 g/kg at a body temperature of 37°C (this is from Table 2) divided by this cell constant, for a measured conductivity of which corresponds to a measured minimum resistance of ohms across a single eccrine duct. Since parallel conductances (like those of adjacent sweat ducts) can simply be added, we can sum over the total number of conducting sweat glands to get the total conductivity (Figure 3).
Resistance drops dramatically as soon as a few eccrine sweat ducts open and fill with sweat; that resistance can just as quickly raise as the contractile myoepithelial cells, under the direction of the autonomic nervous system and the anterior hypothalamic nucleus, contract, and cut off the circuit. As there are around 420 eccrine sweat glands per square centimeter on the palms and a total of 100 cm2 on each palm, there are about 84,000 eccrine sweat ducts on the two hands together. Resistance between cylindrical metal electrodes held in each hand can, as a result of stress, quickly shift from the inherent >50,000 ohm resistance of the dry epidermis to near zero.
Another way of visualizing the change in resistance is to assume that all of the pores are equally dilated and that this dilation is somehow related to the level of excitation or stress. Assume that a particular stress level results in the diameter of the pore being some percentage of its 0.05 mm full dilation (call this proportion); then . Greater stress will contract the myoepithelial cells at the base of the eccrine duct and secrete the saltwater to the surface of the palm ridge. In this scenario, the 84,000 eccrine sweat ducts on the two hands together are all about 3.5 mm long by and for a measured conductivity of which corresponds to a measured resistance of ohms across a single eccrine duct.
In practice, there will be a great number of uncertainties. The diameter of the ducts will vary, and the thin layer of sweat that gets pushed around on top of the palm can be sucked up through capillary action or extended through the ridges of the palm. Additionally, changes in grip strength can squeeze sweat up through the ducts and increase conductivity. Skeletal muscles in the hand are stimulated through a complex pathway that involves several centers in the brain. At the behavioral and the autonomic nervous system levels, the motor cortex is involved, but not the unique structure from which motor commands are originated to make the hands muscles contracting themselves (information from the red nucleus is delivered to hand muscles). Additionally, the hypothalamus is one of several cerebral structures controlling EDA and EDR. This physiological index is also controlled by several parts of the cerebral cortex, for example, insular, cingular and prefrontal cortices.
Greater grip pressure can squeeze ducts open or closed and can spread sweat over the palms and electrode contacts changing the conductivity. It is easy to see, with a combination of skeletal muscles squeezing the sweat around the palms and the contractile myoepithelial cells (a type of small muscle in the eccrine gland that pushes up the sweat through the duct) expelling the secretions of the sweat glands, how conductivity at the skin’s surface can change very rapidly to a complex array of stimuli—in tenths and even hundredths of a second.
Eccrine glands have three functions which are only loosely affected by emotional responses.(i)Thermoregulation. Sweat cools the surface of the skin and reduces body temperature. This cooling is the primary function of sensible perspiration, and the degree of secreatory activity is regulated by neural and hormonal mechanisms. When all of the eccrine sweat glands are working at maximum, the rate of perspiration may exceed a gallon per hour, and dangerous fluid and electrolyte losses can occur. For this reason athletes in endurance sports must pause frequently to drink fluids. (ii)Excretion. Eccrine sweat gland secretion can also provide a significant excretory route for water and electrolytes, as well as for a number of prescription and nonprescription drugs.(iii)Protection. Eccrine sweat gland secretion provides protection from environmental hazards by diluting harmful chemicals and discouraging growth of microorganisms.
There is a marked difference between the two systems of sweat glands. Those which are concentrated on the palms and the soles (also in the axillary and pelvic regions) do not have a thermoregulatory function. The sweat released is often called “activation sweat”, by opposition with thermoregulatory sweat. From a phylogenetic point of view, releasing sweat from the palms and the soles results from high activation (resulting from alert or imminent danger) fight or flight is then facilitated when skin is hydrated thus helping in grasping. In addition, activation sweat has a different chemical composition from thermoregulatory sweat—activation sweat contains much less fatty materials, urea, and other wastes by comparison with thermoregulatory sweat. Thus emotional processes and thermoregulatory processes proceed more or less independently. The temporal inertia of releasing thermoregulatory sweat is greater than that originated from activation sweat (EDR responses are recorded within a time-delay of less than 2 seconds). When you perceive an external stimulus (someone calling your name for example while you did not expect that), EDR is recorded as early as the information is integrated by the central nervous system. This is entirely independent from thermoregulation.
Stress, anxiety, and socially unpleasant situations can cause the hands to sweat because of the secretion of sympathetic-like materials/hormones (mediators) from different organs in the body. Emotional factors such as a sudden shock or a feeling of impending danger can effectively trick the body into reacting as if there is an external rise in temperature that would threaten the temperature of the body. This phenomenon can occur during periods of intense fear or begin to function in the aftermath of a crisis situation. This response is controlled by sympathetic cholinergic nerves which are controlled by a centre in the hypothalamus. The hypothalamus senses core temperature directly and also has input from temperature receptors in the skin and modifies the sweat output, along with other thermoregulatory processes.
Hand sweating beyond that that is needed for temperature regulation is linked to high levels of activity or stress in the sympathetic nervous system. In cases of palmar hyperhidrosis (hand sweating) the responsible sympathetic ganglia are located within the upper part of the chest cavity. In cases of plantar hyperhidrosis (foot sweating) the responsible sympathetic ganglia are located within a section of the lumbar sympathetic chain.
Stimulation of eccrine sweat production is mediated predominantly through postganglionic C fiber production of acetylcholine. Emotional stressors tend to induce sweating that is confined mainly to the palms and soles. All eccrine units can be utilized to respond to the body's changing thermoregulatory needs.
There are two major measures of the electrodermal response. The first, involving the measurement of resistance or conductance between two electrodes placed in the palmar region, was originally suggested by Féré [30]. It is possible also to detect voltages between these electrodes; these potential waveforms appear to be similar to the passive resistance changes, though its interpretation is less well understood. This measurement was pioneered by Tarchanoff [2]. The first type of measurement is referred to as exosomatic, since the current on which the measurement is based is introduced from the outside. The second type, which is less commonly used, is called endosomatic, since the source of voltage is internal, which is less controllable, potentially leading to less accurate measurement (Figure 4).
(a)
(b)
Figure 5 shows signals characteristic of SCR and SPR waveforms. Those identified as having slow recovery, shown in Figure 5(a), have a duration of around 40 seconds, with phasic amplitudes of around 2 microseconds for conductance and 10–20 mV for potential. Rapid-recovery SCRs and SPRs are shown in Figure 5(b). (1)Present-day EDR measurement practice places a regulated voltage (constant current and constant voltage at all loads) of around volt for -type electrodes and around 2.5 volts for dry contact electrodes. Neural responses are typically classified into tonic, or regular spiking, responses, and phasic or bursting responses—Geddes and Baker [31] suggest 0–5 Hz for tonic measurements, with 0.03–5 Hz being adequate for phasic measurements. Electrodermal activity (EDA) measurement is associated with the (tonic) background level (L); electrodermal response (EDR) is associated with the time-varying (phasic) response (R) type (Figure 6). Recommendations for electrodermal measurements were drawn up by a committee selected by the editor of the journal Psychophysiology and published by that journal [32, 33].
3. Integrating EDR into Game User Interfaces
Applications of EDR in gaming are still relatively undeveloped, but could be significantly advanced, in the author's opinion, by borrowing techniques from nearly a century of research in polygraph technology. The following represents a complete protocol for integrating EDR data streams with those acquired elsewhere in the gaming interface. Polygraph protocols, for example, those adapted from the Department of Defense Polygraph Institute, Law Enforcement Pre-Employment Test (January 2002, pp.3-5) can give guidance in using EDR affective datastreams in gaming interfaces. EDR information in pilot tests has been accurately and cheaply acquired through the circuitry shown in Figure 7, which uses a Wheatstone bridge circuit to measure the resistance of hand electrodes and digitizes the signal in an inexpensive ADC.
Summarizations of the affective EDR information stream over a sequence of measurements at short-time intervals (≈1/50 second in the equipment used in this research). The response is not expected to be regular and periodic; thus many of the standard tools of time series analysis are unsuitable for summarization of that affective information stream. This is a common problem with most affective computing, as the emotional readings vary, and must be taken at a finer, unsynchronized time resolution that differs by perhaps several orders of magnitude from the frequency with which other system measurements are taken.
Costs of EDR datastreams can be kept very low, as is demonstrated in the injector circuit shown in Figure 7 using a standard LM 324 OpAmp along with inexpensive components which can be purchased for less than $10 at a retail electronics outlet. The amplified voltage from the Wheatstone bridge in the circuit is injected to an analog-to-digital converter costing $25 to $50. This feeds a time series of digital values of skin resistance to a computer, to be recorded in a database and synchronized with other computations necessary for the survey research.
Summarizations of the EDR information stream over a sequence of measurements at short-time intervals ( second in the equipment developed in this research). There are typically a very large number of intervals of length between user interface responses in gameplay, and responses are unlikely to be regular and periodic; thus many of the standard tools of time series analysis are unsuitable for summarization of affective information streams. Emotional readings may vary and must be taken at a finer, unsynchronized time resolution that differs by perhaps several orders of magnitude from the frequency with which other user interface data is acquired.
Optimal statistical summaries of affective data streams will undoubtedly evolve and improve over time. This research suggests as a starting point a summary mechanism from optimal control theory [34, P.48]—the proportional-integral-derivative (PID) control method. Whereas PID controllers are designed to quickly, smoothly, and accurately make corrections to a closed-loop feedback system, its use here will be to summarize a large number of the brain’s emotional adjustments (if we assume that is what is being acquired in the EDR datastream) in a series of short intervals , that is, the time between responding to the game interface.
This way of summarizing the time-varying EDR is attractive because it gives us empirical insight into the brain’s control inputs—the sum of all of the mental and emotional processes described in the last section with respect to EDR—that change EDR resistance over some time period . Empirical findings can be continually refined and analyzed. The three statistics describe the brain’s activity as follows (where error is the difference between the current resistance, and where the brain wants resistance to move).(1)The proportional term (also called gain) measures change that is proportional to the current error value; pure proportional control retains a steady-state error (droop or drift) that is a function of the proportional gain and the process gain. The integral and derivative terms are invoked in loop tuning to remove the effect of droop. (2)The integral term (also called reset) is proportional to both the magnitude of the error and the duration of the error.(3)The derivative term measures the rate of change of the process error.
The implementation here ignores the gain terms from the PID algorithm (i.e., sets them to 1). Define the points in time of recording the answer times for two questions and , . This allows affective data stream summaries for measurements taken in this research every to be synchronized with the survey responses. Three statistics summarize an affective data stream of measurements over the time interval .(1)The proportional statistic appears in the same data column as the survey response to and is the average over the time interval : .(2)The integral statistic is the sum of changes: .(3)The derivative statistic is the net change over the interval: .(4)Elapsed time: .
The equipment developed to take such readings can be constructed from off the shelf components for around $60, making it very affordable (and potentially scalable). This section looks at ways that affective data acquisition can be affordably extended and enriched, taking advantage of low-cost alternatives from new technologies to access an expanded set of affective datastreams. It takes cues from successful development in polygraph technology and the availability of low-cost analog-to-digital conversion chips such as SiLabs popular C8051F530 selling for around $10 (similar to the Lego approach taken in MIT’s Affective Computing Research Project).
4. Discussion
Electrodermal response measurement in its current incarnation suffers from six issues that tend to confound the extraction of meaningful affective information streams from the readings:(1)electrodermal response mechanisms readings confound measurements of:(a)skeletal muscle micromovement response to emotions and stress, (b)response of the eccrine sweat glands to emotions and stress,(c)response in internal tissue conductivity to emotions and stress,(2)electrodermal response values can be read at only one point in time. There do not exist objective, time-stamped, and complete records of resistance values, synchronized with other measures (e.g., temperature, humidity, grip strength, etc.) and synchronized with subject verbal and written responses;(3)time series analysis of readings is limited, and recording of dynamics and time series of readings is idiosyncratic and spotty, dependent on the protocols. Consequently, statistical and signal processing methods are unavailable for time series analysis;(4)the traditional null meter user interface in many devices requires excessive attention from the experimenter. This is an artifact of the crude circuitry early 20th century circuits and can easily be dispensed with;(5)subject responses are recorded separately from electrodermal response, forcing experimenter shifts of attention between electrodermal response readings and transcriptions of responses.
These problems can be resolved by collecting digital information, adding relevant sensors in addition to the basic resistance reading, and integrating these on a computer platform. This allows multivariate statistical analysis that can untangle the impact of (1) body resistance; (2) skin resistance; (3) muscle movements; other (4) factors affecting the neural processing for thermal regulation of the body. Reference [35] conducted related research in gaming and interactive computing interfaces using EDR, in their case, simply added an accelerometer to detect physical activity. This move towards a polygraph’ approach can be economically enabled in the 24-bit analog-to-digital converters on SiLabs’ C8051F530 (which has a C-language programmable MCU for data summarization and also provides the bias voltage) with setups recording the effect of internal body resistance (measured through transdermal electrodes), the skin-electrode interface, and involuntary muscle movements (measured with a strain gauge). Since the main changes in resistance are due to thermoregulatory signals from the hypothalamus, it is also reasonable to add sensors for ambient temperature and humidity, along with the light level, all of which may affect this response. Multiple streams of affective data can then be processed and sent as a sequence of measurements to a computer as an array whose time stamps can also be synchronized with audio and video channels streamed in through inexpensive microphones and video cameras. Flexible user interfaces could follow MIT Affective Computing Lab’s Lego approach, using National Instrument’s Labview software; otherwise this can easily be programmed in any of the Visual Studio languages, allowing a simple building block approach to integration, readouts and signal processing.
The research in this paper has demonstrated the potential for using affective streams to enrich gameplay. It would be naïve to assume that such an innovation would not meet with some resistance. In particular, it opens up new privacy concerns. Any new technology is a double-edged sword; with added information richness come added risk. Many of the same objections that have dogged polygraphs throughout their history are likely to be introduced to the dialog on affective datastreams. Polygraphs were perceived by early psychologists as a powerful tool for counseling patients, giving the analyst insights to focus and speed treatment. Pioneering applications by the Chicago Police Department in the 1930s to criminal psychology and forensics brought an end to innocence and altruism. Polygraph testing took on a sinister, adversarial bent. Unlike polygraphs, affective datastreams used in gaming are most likely to be assistive rather than forensic and adversarial. So it is likely, in order to reap the benefits of access to affective data, that some of the controls and user interfaces that sites such as Facebook have adopted with their existing databases and datastreams may be suitable for assuaging consumer concerns over affective data capture. Fundamentally, these will amount to quick, informed, and explicit consent, coupled with feedback on the information captured.
An arena of gaming into which affective computing has already made forays is massively multiplayer gaming. Though conceived as entertainment, gaming and the entertainment industry trends have a long history of setting protocols for my sublime business, government, and security applications—especially where teams and social networks are concerned. Much of the theory and fine tuning of gameplay (paralleling screen editing) seeks to control and modulate the enthusiasm, emotions, and stress of the player—to feedback into the gameplay an interpreted transform of affective information acquired. Too much stress can make a game confusing and frustrating; too little leaves players bored. When biofeedback loops are introduced to gaming, the game can automatically adjust its script and difficulty to the player; it can adapt for experience, innate ability, and day-to-day variations in the player’s attitude and demeanor.
Electrodermal response provides a noninvasive and inexpensive way to accurately measure a variety of emotional and stress responses from a subject. It has gone through considerable development over the 20th century and is now being rivaled by technologies, such as fMRI, that can read directly from the neural system. Yet the electrodermal response mechanism is still competitive from standpoints of cost, complexity, portability, and user interface. The current paper has analyzed, in depth, the most widely used inexpensive technology for monitoring stress and emotional state—the electrodermal response. It has argued that, with a good understanding of the mechanism behind electrodermal response, it is possible to accurately monitor a number of internal emotional activities, and with the right set of sensors, disentangle the differing sources of change and stress over time. Since the accurate capture modulation of emotional state is a promising component in search of richer more immersive gaming, it is argued that such measurements can significantly add to the quality and the attractiveness of gaming innovations in software.