Wearable Technology for Detecting Significant Moments in Individuals with Dementia

Lai Kwan, Chelsey; Mahdid, Yacine; Motta Ochoa, Rossio; Lee, Keven; Park, Melissa; Blain-Moraes, Stefanie

doi:https://doi.org/10.1155/2019/6515813

BioMed Research International

On this page

Abstract Introduction Materials and Methods Results Discussion Conclusions Data Availability Conflicts of Interest Acknowledgments Supplementary Materials References Copyright Related Articles

Special Issue

Assistive Technology Innovations in Neurological Conditions

View this Special Issue

Research Article | Open Access

Volume 2019 | Article ID 6515813 | https://doi.org/10.1155/2019/6515813

Wearable Technology for Detecting Significant Moments in Individuals with Dementia

Chelsey Lai Kwan,¹Yacine Mahdid,¹Rossio Motta Ochoa,¹Keven Lee,²Melissa Park,²and Stefanie Blain-Moraes^1,2

Guest Editor: Johnathan Collett

Received01 May 2019

Revised05 Aug 2019

Accepted19 Aug 2019

Published25 Sept 2019

Abstract

The detection of significant moments can support the care of individuals with dementia by making visible what is most meaningful to them and maintaining a sense of interpersonal connection. We present a novel intelligent assistive technology (IAT) for the detection of significant moments based on patterns of physiological signal changes in individuals with dementia and their caregivers. The parameters of the IAT are tailored to each individual’s idiosyncratic physiological response patterns through an iterative process of incorporating subjective feedback on videos extracted from candidate significant moments identified through the IAT algorithm. The IAT was tested on three dyads (individual with dementia and their primary caregiver) during an eight-week movement program. Upon completion of the program, the IAT identified distinct, personal characteristics of physiological responsiveness in each participant. Tailored algorithms could detect moments of significance experienced by either member of the dyad with an agreement with subjective reports of 70%. These moments were constituted by both physical and emotional significances (e.g., experiences of pain or anxiety) and interpersonal significance (e.g., moments of heighted connection). We provide a freely available MATLAB toolbox with the IAT software in hopes that the assistive technology community can benefit from and contribute to these tools for understanding the subjective experiences of individuals with dementia.

1. Introduction

The prevalence of dementia is expected to dramatically increase in the coming decades: in Canada, it is anticipated that the number of individuals over the age of 65 living with dementia will double within the next 20 years [1]. Dementia is a set of symptoms caused by disorders and diseases affecting the brain, including Alzheimer’s disease and vascular dementia. Individuals with dementia typically experience memory loss, difficulties with problem-solving, language and orientation, and altered mood and behavior. These symptoms diminish the individual’s ability to perform activities of daily living and disrupt their relationships, often reducing their ability to participate in society and requiring significant support in their daily lives. At present, hospitals in Canada already provide full-time care to approximately 51,000 individuals with dementia [1], and almost half a million more are cared for by family and informal caregivers. The indirect annual cost of caring for those with dementia is conservatively estimated at $1.2 billion; this estimate is projected to double by 2031 [1].

In response to the escalating demands of caring for persons with dementia, intelligent assistive technologies (IATs) for this population have proliferated. IATs have been developed for nearly every aspect of daily living and range from distributed systems (e.g., smart homes and integrated sensor systems), personal robots (e.g., socially/physically assistive), mobility and rehabilitation aids (e.g., powered wheelchairs and electronic canes), handheld/multimedia devices (e.g., smart phones, personal digital assistants (PDAs), and tablets), software applications (mobile or web-based apps), voice-prompting systems [2], wearable devices (e.g., smartwatches and e-textiles), and human-machine interfaces (HMIs) [3]. Of particular note is the growth in wearable technologies for this population, which are primarily used to monitor the well-being and safety of individuals with dementia. Wearables have been employed to track sleep and toileting patterns [4], to aid memory retrieval [5], and to aid way finding and navigation [6–9].

While the majority of IATs for individuals with dementia have focused on maintaining and improving their functional and cognitive abilities, there has been minimal focus on tracking the subjective experience of individuals with dementia. Understanding the subjective emotional and mental experience of an individual with dementia has important implications for the quality of life of these individuals and their caregivers (familial and professional). Insight into the significant events for individuals with dementia can make visible what is meaningful to them [10] and can, thus, empower caregivers to understand and ameliorate care decisions based on what matters to them [11], as well as intervene before a minor annoyance or frustration can build into an outburst or an aggressive behavior. Moreover, the communication of meaning and important moments is critical to maintaining a sense of connection and thus to preserving relationships between individuals with dementia and their caregivers [12]. In this paper, we describe a novel, wearable IAT that can track and detect moments of significance for individuals with dementia. As dementia is associated with progressively declining cognitive and motor abilities, this IAT focuses upon detecting significant moments from their physiological signals.

The feasibility of detecting significant mental and emotional reactions from changes in physiological signals has long been established [13]. Physiological signals have been used to provide insight into an individual’s significant reactions in fields such as polygraphy [14] and biofeedback [15] for decades. Additionally, physiological signals such as electrodermal activity (EDA), heart rate, and skin temperature are amenable to being recorded from noninvasive, wearable sensors. The IAT presented in this paper records physiological signals from a device worn on the hand and uses a custom software to detect individual-specific moments of significance from changes in their physiological patterns. We demonstrate that this IAT is able to accurately detect moments of physiological significance and moments experienced as significant interpersonal interactions in individuals with dementia and their caregivers. Furthermore, we make the algorithms and software freely available through a Matlab toolbox, which can be effectively used to detect moments of significance from patterns of physiological signals recorded from wearable sensors.

2. Materials and Methods

2.1. Description of Assistive Technology

2.1.1. Hardware

Autonomic nervous system (ANS) signal data were collected from a wearable device called the Triple Point Sensor (TPS) (Thought Technology Ltd. ©). The TPS was designed to be worn on the fingertip by securing it with a loose elastic or Velcro strap. Three physiological signals were recorded at a sampling frequency of 15 Hz: (1) electrodermal activity (EDA); (2) skin temperature; and (3) heart rate (HR). Signals were transmitted via Bluetooth to a paired Android phone, which stored the data in a custom database.

2.1.2. Software

Custom software, called Events Finder, a freely available MATLAB toolbox (https://github.com/BIAPT/Events-Finder), was designed to detect salient moments from individual-specific physiological signatures. Broadly, the software detected characteristics from the ANS signals that are known a priori to be associated with changes in emotional state. Analysis of the EDA signal focused on the detection of electrodermal reactions (EDRs)—transient increases of 0.05 µs or more within 10 seconds, which have been associated with heightened emotional arousal [16, 17]. Skin temperature analysis focused on detecting changes between vasoconstrictor and vasodilatory responses [18], and the heart rate was monitored for unusual patterns of acceleration and deceleration [19]. The software consists of three components: (a) preprocessing; (b) signal quality assessment; (c) event detection, which are detailed in the sections below.

(1) Preprocessing. Each of the three ANS signals was preprocessed to remove nonphysiological artifacts. Filters were chosen to be compatible with real-time processing, to ensure that the final software had translational potential for real-world use. First, missing datapoints or datapoints that resulted from hardware error were identified by flagging samples with (1) a value of 0 and (2) whose value was greater or less than three standard deviations from the average of the preceding second of data. These datapoints were replaced by interpolation using a 1-D median filter (EDA filter order n = 75; skin temperature filter order n = 1). Second, all signals were sent to a moving average filter with a non-overlapping, 0.5-second window; the resulting smoothed data had a sampling frequency of 2 Hz. Third, the signals were sent to a modality-specific filter that enhanced particular features associated with a salient reaction. To minimize jitter and lag in the EDA signal, a one Euro filter was applied with parameters mincutoff = 50 and beta = 4 [20]. An exponential decay filter was applied to the skin temperature signal to remove the high-frequency noise inherent in the signal (smoothing parameter ). Finally, the cubic smoothing spline function from MATLAB was applied to the heart rate data (smoothing parameter ). The effects of these modality-specific filters on their respective physiological signals are illustrated in Figure 1.

(2) Signal Quality Assessment. After preprocessing, a signal quality index (SQI) algorithm was applied on a sliding window of 0.5 seconds across all three physiological signals. The index ranged from SQI_x = 1, when data from physiological signal x was physiologically valid, to SQI_x = 0, when data from physiological signal x was wholly contaminated by noise resulting from the actions of the user, such as shifting the sensor’s position on the hand, scratching around the sensor, and applying pressure to the sensor. EDA signals were assigned a lower SQI_EDA when the rate of change exceeded physiological possibility, when it registered no signal for 25 seconds or longer, or when the absolute value exceeded the bounds of a normal physiological range. Skin temperature signals were assigned a lower SQI_temp when it registered no signal for 25 seconds or longer, or when values were less than 15 degrees Celsius (e.g., below normal physiological range). The specific parameters used to generate SQI_x are reported in Table 1. Upon completion of a data recording session, the mean (µ) and standard deviation (σ) of the SQI_x for each individual physiological signal were calculated. Subsequently, a binary time series (SQI_all) of non-overlapping 0.5 second steps was created, where SQI (t)_all = 0 if SQI (t)_EDA < μ (SQI_EDA) − σ (SQI_EDA) or SQI (t)_temp < μ (SQI_temp) − σ (SQI_temp), and SQI (t)_all = 1 otherwise. Any datapoint where SQI (t)_all = 0 was not considered in the subsequent event detection algorithm.

(c) Physiological Event Detection. The objective of this subcomponent was to detect events in the cleaned, high-quality data that corresponded to a salient moment for the user. The dominant physiological modality that manifests emotional processing varies from individual to individual [21, 22], and the characteristics of the changes within this dominant modality also vary according to multiple factors, including sex, age, and time of day [23]. Thus, in broad terms, event detection required (1) tuning the parameters for detecting specific features within each signal and (2) varying the relative weight of contribution of each of the three physiological signals.

Signal-specific feature detection consisted of a set of adjustable parameters for each physiological modality. Within the EDA signal, electrodermal reactions (EDRs) are associated with an orientation response and heightened emotional arousal [16, 17]. Canonically, an EDR consists of an increase greater or equal to 0.05 μS within a 10-second interval [16]. Tuning the event detector to capture EDRs for each individual required varying (1) t_EDR, defined as the time interval within which to search for an EDR and (2) A_EDR, defined as the minimum amplitude (μs) increase across t_EDR that was considered to be an EDR. Heart rate has natural patterns of acceleration and deceleration, driven by factors such as respiratory sinus arrhythmia [19, 24]. Salient events were detected at time points where heart rate acceleration and deceleration varied outside of these baseline patterns. Acceleration and deceleration patterns were tracked using the find_peaks algorithm in MATLAB. Tuning the event detector to capture significant changes from these patterns required varying minPeakProminence, the threshold for which one peak must deviate with respect to height and location from the other peaks in the heart rate time series to be flagged as an event. Fingertip temperature changes associated with salient responses are driven by transient changes in the cutaneous microcirculation [18, 25]. Vasoconstriction and vasodilatory responses manifest as changes between the rate of cooling and heating in the fingertip. The event detector for fingertip temperature was tuned by varying (1) t_temp, defined as the time interval within which to consider trends in temperature and (2) T_max − T_min, defined as the difference between the maximum and minimum temperature (°C) within the interval t_temp.

Applying the above rules for detecting features in each physiological modality yielded three time series, with a step size of 0.5 seconds, and values corresponding to the strength of each feature extracted from the physiological data: EDR (t), , and HR_var (t). Subsequently, the three physiological modalities were assigned a weighting factor to scale the relative contribution of each signal to the overall score S, indicating the magnitude of change in the weighted physiological features.where . All three weighting factors (a, b, and c) where initialized to 1 and adjusted in an iterative fashion as described in Section 2.3.

2.2. Study Design

2.2.1. Participants

Participants were recruited from a nonprofit organization that provided a variety of services to the community for individuals with dementia and their caregivers. The assistive technology was calibrated and tested on three dyads, each dyad consisting of an individual with dementia and his/her primary caregiver. Pseudonyms and the relationship between members of the dyad are listed in Table 2. Written consent was obtained from all caregivers for themselves and for the individual with dementia—who also provided written assent—after a careful discussion of risks and benefits. This study was approved by the Institutional Review Board of McGill University (A06-B25-17B).

2.2.2. Data Collection

Participants engaged in an 8-week movement-based program and ran over the course of 15 weeks. The program was held at the nonprofit organization—a familiar environment that was chosen to support the engagement and self-expression of the individuals with dementia [26]. Prior to the beginning of each session, each member of the dyad was outfitted with the assistive technology. Participants were given the choice of securing the sensor onto their fingertip or onto the palm of their hand based on comfort and ease of movement. Sensors were secured with a Velcro band. Collection of ANS data from the sensors was initiated from a paired Android smartphone prior to the movement session by a research assistant. Sessions were also video and audio recorded by using a video camera mounted near the ceiling; audiovisual data were timestamped and synchronized with the ANS data from each participant.

Following each 45-minute movement session, ANS data from each participant were run through the Events Finder software described above. Events with the highest S score were selected, and the video recording was spliced 10 seconds prior to and 10 seconds following each event, resulting in 10-second–20-second clips of physiologically triggered salient moments for each dyad.

After selected sessions, these 20-second video clips were presented to the respective dyad on a laptop computer. Dyads were interviewed to assess whether or not the salient moments detected by the algorithm corresponded to a subjective recollection of a significant experience. Significant experiences were defined as events [10] or heightened moments that stood out and/or were memorable (e.g., see also [27]). A research assistant guided their reflection about the moments captured in the video using the following questions:(i)What was happening at this point during the session? Was this in relation to your partner?(ii)Did this moment stand out to you from the ordinary flow of the session? If yes/no, why?(iii)Are there moments we did not go over where you felt an especially strong sense of connection or disconnection to your partner?

Participant responses were audio recorded and used to adjust the parameters of the event detection algorithm.

2.3. Personalization of the Assistive Technology

Prior to the first movement session, all parameters of the Event Finder algorithm were set to default values. All physiological modalities were equally weighted (i.e., a = b = c = 1). Default parameters for feature detection within each physiological signal were empirically derived from preliminary data: electrodermal reactions were initiated as increased in 0.05 μs across 10 seconds (i.e., t_EDR = 10; A_EDR = 0.05); heart rate acceleration and deceleration were flagged at minPeakProminence = 10 bpm; skin temperature trends were considered to change if there was a change of 0.01°C across 10 seconds (i.e., T_max − T_min = 0.01; t_temp = 10).

After each of the first four sessions, a research assistant reviewed the physiological data of each participant to individually tailor each of the parameters to match the particular idiosyncratic patterns of his/her physiological responsiveness. First, the audiovisual recording was observed in full for any moments that stood out from the ordinary flow of the session. Time points associated with these moments were flagged on the synchronized physiological signal recording. For each of these time points, a 10-second epoch surrounding each event was segmented, and EDR (t), , and HR_var (t) were calculated. The parameter value (Table 1) associated with each feature was calculated and used to replace the default parameters. Subsequently, the Event Finder algorithm was re-run, and the accuracy of detection of significant moments was calculated. The parameter set that produced the highest accuracy was retained for the next movement session.

At the end of some of the last four sessions, the algorithm parameters were iteratively adjusted to align with participants’ subjective report of the experience associated with detected physiological responses. Participants were presented with 20-second video clips containing salient moments detected by the personalized Event Finder algorithm. Their responses of whether or not the video detected a significant event were used to iterate on the parameters to decrease the overall score S of the nonsignificant moments. If participants reported significant moments that were not detected by the algorithm, parameters were adjusted to increase the overall score S of these moments. The algorithm with the adjusted parameters was used to detect significant moments in the participant’s next movement session.

2.4. Data Analysis

For all sessions where participants provided feedback about the salience of the moments identified by Event Finder, two research assistants independently reviewed the video and interview data to assess the performance of the assistive technology. True positives (TPs) consisted of events identified by the algorithm that corresponded to a salient moment for the user. These moments were recognized by the participant’s ability to vividly recall their subjective experience in the video clip associated with the event. False positives (FPs) consisted of events identified by the algorithm that did not correspond to a salient moment for the user. FPs occurred when (1) the event was flagged as a result of a signal artifact (e.g., the video showed the user adjusting the sensor) or (2) the participant could not recall what happened at the moment depicted in the video, or described only what he/she was observing in the video. All true-positive events were further coded to assess whether they corresponded to a moment of physiological significance (TP_phys) or a moment of interpersonal significance (TP_pers). TP_phys included vivid recollections of subjective feelings of discomfort, effort, pain, surprise, relaxation, or stress in relation to the event. TP_pers were defined as events where participants described positive or negative feelings induced by interactions with another individual. These events were typically accompanied by an underlying explanation or story.

3. Results

Each dyad participated in 2-3 sessions where they provided oral feedback that was used to calibrate Event Finder and to assess the performance of the algorithm. The total number of events per session presented to the dyads for feedback varied according to Table 3, with fewer events in the earlier sessions to help dyads become accustomed to the interview process. The session ID reflected the state of the internal parameters of the algorithm (e.g., default or customized).

3.1. Persons with Dementia Have Distinct, Personal Characteristics of Physiological Responsiveness

While the Events Finder software was initialized with the same default parameters across all participants, individual-specific patterns were identified by the end of the fourth session. The tailored parameters used in Events Finder at the end of the final session are presented in Table 4 for all participants with dementia. Mary’s salient reactions were characterized by EDRs, which morphologically increased sharply in amplitude (i.e., 0.24 μs in 10 seconds) (Figure 2(a)). Elisa’s salient reactions were manifest in heart rate accelerations and decelerations (i.e., peak prominence >25 bpm) (Figure 2(b)). Irene’s reactions were primarily reflected in changes in the rate of change of her peripheral skin temperature (Figure 2(c)).

(a)

(b)

(c)

Figure 2

True positive events detected for participants with dementia in their final movement session. Preprocessed and quality-checked signals for electrodermal activity (blue), heart rate (green), and skin temperature (yellow) are presented for Mary (a), Elisa (b), and Irene (c). Red boxes indicate the 30-second “event” that was detected by the final tailored algorithm using parameters presented in Table 4. For each detected event, represents the physiological modality dominating each change. Each participant presents varying patterns of physiological responsiveness, and their event-detection algorithm is dominated by different physiological modalities, illustrating the need for personalizing the software for each individual.

Examples of the physiological responses that triggered the detection of an event for each person with dementia are presented in Figure 2 and for each caregiver in Figure 3.

(a)

(b)

(c)

Figure 3

True-positive events detected for caregivers in their final movement session. Preprocessed and quality-checked electrodermal activity (blue), heart rate (green), and skin temperature (yellow) signals are presented for Liam (a), Giselle (b), and Sophie (c). Liam’s significant events are triggered by electrodermal reactions; Sophie’s by changes in vasodilatory and vasoconstriction responses in skin temperature. Giselle’s significant events are triggered by a combination of both skin temperature and heart rate responses. The specific parameters for each individual’s algorithm are presented in Supplementary Data, Table 1. The unique patterns of responsiveness illustrate the need to tailor the event detection algorithm for caregivers.

3.2. Assistive Technology Can Identify Salient Events from Physiological Reactions

The performance of the assistive technology in identifying salient events was assessed through classifying each flagged event as either TP or FP, based upon video observation and participant feedback. Two research assistants independently classified each event; the interrater reliability was 96.6%.

Across all participants, 70% of significant moments detected by the best tailored Event Finder algorithm were true positives. The performance of the algorithm improved upon individual tailoring for both persons with dementia (Figure 4(a)) and their caregivers (Figure 4(b)).

(a)

(b)

3.3. Events Identified through Physiological Reactions Include Interpersonal Moments

Within the TP events identified by Event Finder, TP_pers ranged from 40 to 89%. While the software predominant caught moments of TP_phys such as anxiety or pain over performing a specific movement, a significant portion was related to moments of interpersonal interaction. The physiological characteristics of three such events are presented in Figure 5, and the context of these events are described below. In contrast to Figure 4, where data were presented over the course of a long session (e.g., more than 10 minutes), Figure 5 illustrates a signal event detected in 30 seconds of physiological data.

(a)

(b)

(c)

Figure 5

Examples of physiological signals associated with experiences of interpersonal significance. Red boxes highlight the detected event for (a) a moment of connection experience by an individual with dementia with her spouse; (b) a moment of recognition between an individual with dementia and a staff member; (c) a moment of connection experienced by a caregiver with her mother. The physiological modality dominating the event-detection. However, the patterns of physiological changes triggering events differ within (a) and (b) and between (b) and (c) participants; all are associated with subjective experiences of interpersonal connection.

The first event was triggered by a sharp decrease in temperature, identified by the algorithm as a negative 0.01°C change within 10 seconds (T_max = 35.8367°C and T_min = 35.8267; t_temp = 10) measured (Figure 5(a)). Prior to this moment, Mary was disengaged from the movement activity. Liam described the change in their interpersonal interaction that accompanied this event:

“I pretend to be pulled away from Mary. Then she gave me a light headbutt on my back. And that’s when I touched her forehead with mine (forehead) back. She was playing!”

The second event was from dyad 1, triggered by Mary’s EDR and shift from increasing to decreasing fingertip temperature (Figure 5(b)). Prior to this event, Mary had made eye contact with a staff member, Meredith, from the nonprofit organization, who she had known for several years but had recently forgotten. Liam describes the moment accompanying the physiological change:

“Oh, it was you! Meredith! You got a response. It was Meredith’s engagement. Do you remember dear [to Mary] when Meredith was looking at you? You can see you and Meredith in the picture over there. And… when you saw her, you wanted to move with her.”

While Mary could not self-report about her experience of the moment, the significance of this event was described by several other participants. Meredith reported experiencing a significant connection with Mary during this moment:

“She looked at me and her face lightened up… something changed in her eyes and she smiled at me. I do not know how to say it… it’s something you feel… I think it was recognition! Recognition! She recognized me… she remembered who I was.”

Sophie (caregiver in dyad 3) also reported that she noticed the significance of this moment. She began to cry as she explained what she witnessed: “I was so happy to see [Mary] happy, so that made me happy too. Because she was really smiling and she was really happy, and I thought, “Oh good for her!” Meredith got [Mary] up dancing… and [Mary] was enjoying it.”

The third event was from dyad 2, triggered by Giselle’s increase of 0.17°C in skin temperature over 20 seconds (T_max = 33.96°C and T_min = 33.79; t_temp = 20) (Figure 5(c)). Giselle described the excitement that triggered this event: “I was watching [Elisa] move and realizing she’s doing it more than usual. Sometimes she only gets into it later. I didn’t realize she was still moving. I think she moved today more than she ever did!”

These examples demonstrate that the assistive technology is not only able to capture moments of physiological relevance but also able to capture moments of emotional and interpersonal significance.

4. Discussion

Intelligent assistive technologies (IATs) have emerged as promising tools to meet the escalating demands of caring for persons with dementia. While many wearable IATs have been developed for the purposes of tracking activity and maintaining functional and cognitive abilities, few technologies have focused on accessing the subjective experiences of individuals with dementia. In this study, we present a novel IAT that detects moments of significance for individuals with dementia through patterns of their physiological signals. This IAT is tailored to the individual’s idiosyncratic response profile and is trained using the domain expertise of the caregiver with respect to the emotional state of the individual with dementia. We demonstrate that this wearable technology is able to accurately track significant moments associated with both physical and emotional events. Such a technology has the potential to accompany frameworks such as deep assessment [11] to improve care for people with advance dementia and to support the quality of life and care decisions of professional and more distal carers. Physiological markers of “significance” or meaning can support the understanding of what matters to persons with dementia, which is now core to health policy [28]. To accompany this article, we developed a freely available open-source toolbox for detecting significant events from physiological signals (http://www.github.com/BIAPT/Events-Finder). Our toolbox enables nonexperts to flag segments of a video recording with a high probability of physical or emotional salience, facilitating discussion and feedback of the event from the individual with dementia and their caregiver. The open-source nature of our toolbox allows researchers to customize individual functions for their needs and to incorporate individual sections into their respective analysis protocols.

Every individual has a distinct pattern of physiological responsiveness to salient stimuli. Thus, the performance of the IAT is dependent on the process of tailoring the parameters of the algorithm to an individual’s idiosyncratic physiological characteristics. Tailoring the algorithm resulted in an increase in true-positive events and a decrease in false-positive events for both individuals with dementia and their caregivers (Figures 2(a) and 2(b)). The initial steps of this tailoring are time-consuming, as they involve reviewing video recordings synchronized with the physiological signals for visual moments of significance. However, this customizes the Event Finder algorithm so that it is sufficiently tuned to an individual’s particular patterns of responsiveness to capture events with a high probability of salience (Figures 3 and 4). These events include both moments of physical significance and experiences of interpersonal significance (Figure 5). The latter steps of tailoring the algorithm parameters involved incorporating feedback from both the individual with dementia and their caregiver about the identified events. This combination of subjective self-report and third-person observation enabled us to incorporate the perspectives of individuals with dementia into the tailoring process and has been shown to be an effective strategy in the previous research with this population [26]. As symptoms of dementia include difficulties with abstract reasoning, such as remembering events and reflecting on their meaning [29], supporting the feedback process with video recordings and caregiver reflections was key to the latter stages of tailoring the algorithm parameters. Feedback from the participants was also key to identifying the valence of the event, which was less evident from the physiological data alone [30]. It was evident in the feedback process that caregivers were extremely attuned to the nonverbal behavior of the individual with dementia, which was an evocative means of self-communication and interpersonal communication [31]. Integrating their expertise into the behavior of the algorithm was a critical process in achieving high-performance accuracy in the detection of significant events. Furthermore, this integration may also facilitate the eventual acceptance of the IAT for everyday use, as low levels of user involvement in technology design is a codeterminant of low IAT adoption rates by individuals with dementia [3].

The IAT presented in this paper is one of a small handful IATs that use physiological signals to gather information about an individual with dementia. Smartwatches have been developed for this target population to use a combination of ANS and accelerometry signals to track physical activity [4, 32] as well as to detect changes in mental and emotional states. The technologies developed to-date use EDA alone [33] or in combination with heart rate, temperature and accelerometry signals to recognize stress and agitation in individuals with dementia [34, 35]. The IAT presented in this paper advances the capabilities of those developed to-date in two significant ways. First, existing IATs focus on tracking states of negative valence (e.g., stress and agitation) in individuals with dementia; we have demonstrated that our IAT can also detect moments of positive valence, including meaningful interpersonal connection, in individuals with dementia and their caregivers. Second, existing technologies focus on changes in overall state, such as activity and mood, and ignore transient, short-term changes. In contrast, our IAT is specifically tailored to detect “moments” of significance, which are transient by nature, in an effort to make visible what is meaningful to individuals with dementia [10]. By synchronizing the moments detected by physiological changes with 30-second video clips, our IAT enables individuals with dementia and their caregiver to focus on the transient events that are meaningful during an activity or an interaction, as opposed to their general baseline state. As such, our IAT advances the capabilities of existing technologies by highlighting events that signify what matters to an individual with dementia [11].

While the number of IATs designed for individuals with dementia double approximately every five years, less than 2% are intended to assist with the social and relational challenges associated with this condition [3]. Technologies that address interpersonal interactions are significantly more popular in other conditions, such as autism spectrum disorder [36, 37]. Moreover, IATs designed to detect emotions from physiological signals have been developed for decades in the field of affective computing [13, 38]. These technologies use advanced features calculated from ANS signals to train feature-based machine learning algorithms to detect discrete states of emotion [39, 40] or deploy an end-to-end deep learning approach for the dimensional recognition of emotions [41]. Drawing from the techniques and knowledge of these established fields will significantly accelerate the development of IATs that support the social and interrelation fabric of individuals with dementia.

This study has several limitations. First, the algorithm parameters were only tailored across the last four movement sessions. As some participants were absent during one or more sessions, this only allowed for two to three meaningful iterations of the algorithm parameters per participant. While each iteration improved Event Finder performance, we believe that the algorithm’s performance would continue to improve with more sessions for tailoring the parameters and that the results reported herein do not represent the full capabilities of the Event Finder software. Second, the process we present for customizing the algorithm runs the risk of overfitting the parameter set to an individual’s quotidian physiological characteristics. As the progression of dementia has been associated with changes in autonomic functions [42–44], it is possible that the algorithm would need to be retrained periodically on an individual with dementia as their physiological characteristics changed over time. Third, while signals from the autonomic nervous system are being recorded, it is important to not exclusively link the detected events with physiological responses. For example, clenching of the fists reflexively during significant moments has been associated with increases in electrodermal activity [16] and would have triggered an event detection by the Event Finder algorithm. Fourth, each physiological signal was processed independently, though it is known that interactions between the physiological modalities can discriminate between different mental and emotional states [45–47]. A multivariate approach to detecting significant changes in physiological state may result in better algorithm performance [48]. Finally, a limited number of features were extracted from each physiological modality; future improvements to the algorithm could involve the addition of features such as heart rate variability, which have shown strong correlation with an individual’s emotional state [39].

5. Conclusions

We present a novel intelligent assistive technology (IAT) that detects salient physical and emotional events in individuals with dementia through patterns of their physiological signals. Our IAT consists of software that displays video recordings of events that are synchronized with potential moments of significance detected from physiological changes. Gathering feedback from individuals with dementia and their caregiver on their subjective experience of these moments enabled us to tailor algorithm parameters and improve algorithm performance. The algorithm was able to accurately detect moments of both emotional significance and moments of significant interpersonal connection in individuals with dementia. We provide our software as an open-source toolbox for the detection of significant physiological events from individually-tailored parameters in hopes that the assistive technology community will be able to benefit from and contribute to these tools.

Data Availability

The physiological data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

This research was supported by the Canadian Institute of Health Research (CIHR) Operating Grant (150704) and by a Natural Science and Engineering Research Council (NSERC) Undergraduate Student Research Award. The authors would like to thank Alzheimer’s Group Inc., for their support of this study, and the participants for their openness and valuable feedback.

Supplementary Materials

Supplemental Table 1: creating customized algorithms for caregivers from ANS signals. (Supplementary Materials)

References

Alzheimer Society of Canada, “Report summary prevalence and monetary costs of dementia in Canada (2016): a report by the Alzheimer Society of Canada.,” Health Promotion and Chronic Disease Prevention in Canada: Research, Policy and Practice, vol. 36, no. 10, pp. 231-232, 2016.
View at: Google Scholar
M. C. Carrillo, E. Dishman, and T. Plowman, “Everyday technologies for Alzheimer’s disease care: research findings, directions, and challenges,” Alzheimer’s & Dementia, vol. 5, no. 6, pp. 479–488, 2009.
View at: Publisher Site | Google Scholar
M. Ienca, J. Fabrice, B. Elger et al., “Intelligent assistive technology for Alzheimer’s disease and other dementias: a systematic review,” Journal of Alzheimer’s Disease, vol. 56, no. 4, pp. 1301–1340, 2017.
View at: Publisher Site | Google Scholar
C. Boletsis, S. McCallum, and B. F. Landmark, “The use of smartwatches for health monitoring in home-based dementia care,” in Human Aspects of IT for the Aged Population. Design for Everyday Life, vol. 9194, pp. 15–26, Springer, Berlin, Germany, 2015.
View at: Google Scholar
G. Wilson, D. Jones, P. Schofield, and D. J. Martin, “Experiences of using a wearable camera to record activity, participation and health-related behaviours: qualitative reflections of using the Sensecam,” Digital Health, vol. 2, 2016.
View at: Publisher Site | Google Scholar
A. Firouzian, Z. Asghar, J. Tervonen, P. Pulli, and G. Yamamoto, “Conceptual design and implementation of indicator-based smart glasses: a navigational device for remote assistance of senior citizens suffering from memory loss,” in Proceedings of the 2015 9th International Symposium on Medical Information and Communication Technology (ISMICT), pp. 153–156, Kamakura, Japan, March 2015.
View at: Publisher Site | Google Scholar
L. T. D’Angelo, J. Neuhaeuser, Y. Zhao, and T. C. Lueth, “SIMPLE-use—Sensor set for wearable movement and interaction research,” IEEE Sensors Journal, vol. 14, no. 4, pp. 1207–1215, 2014.
View at: Publisher Site | Google Scholar
Y.-J. Chang, C.-N. Chen, L.-D. Chou, and T.-Y. Wang, “A novel indoor wayfinding system based on passive RFID for individuals with cognitive impairments,” in Proceedings of the 2008 Second International Conference on Pervasive Computing Technologies for Healthcare, pp. 108–111, Tampere, Finland, January 2008.
View at: Publisher Site | Google Scholar
L. E. M. Grierson, J. Zelek, I. Lam, S. E. Black, and H. Carnahan, “Application of a tactile way-finding device to facilitate navigation in persons with dementia,” Assistive Technology, vol. 23, pp. 108–115, 2011.
View at: Publisher Site | Google Scholar
M. Jackson, Existential Anthropology, Berghahn Books, New York, NY, USA, 2005.
G. Lyons, M. Arthur-Kelly, A. Eidels, and A. Mavratzakis, “Deep assessment: a novel framework for improving the care of people with very advanced Alzheimer’s disease,” BioMed Research International, vol. 2015, Article ID 749451, 8 pages, 2015.
View at: Publisher Site | Google Scholar
M. Duffy, Ed.in Handbook of Counseling and Psychotherapy with Older Adults, Wiley, New York, NY, USA, 1st edition, 1999.
R. W. Picard, E. Vyzas, and J. Healey, “Toward machine emotional intelligence: analysis of affective physiological state,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 10, pp. 1175–1191, 2001.
View at: Publisher Site | Google Scholar
H. W. Gödert, H.-G. Rill, and G. Vossel, “Psychophysiological differentiation of deception: the effects of electrodermal lability and mode of responding on skin conductance and heart rate,” International Journal of Psychophysiology, vol. 40, no. 1, pp. 61–75, 2001.
View at: Publisher Site | Google Scholar
N. Birbaumer and H. Flor, “Applied psychophysiology and learned physiological regulation,” Applied Psychophysiology and Biofeedback, vol. 24, no. 1, pp. 35–37, 1999.
View at: Google Scholar
W. Boucsein, Electrodermal Activity, Plenum Press, New York, NY, USA, 1992.
P. H. Venables and M. J. Christie, “Electrodermal activity,” in Techniques in Psychophysiology, I. Martin and P. H. Venables, Eds., pp. 2–67, John Wiley, New York, NY, USA, 1980.
View at: Google Scholar
A.-L. Krogstad, M. Elam, T. Karlsson, and B. G. Wallin, “Arteriovenous anastomoses and the thermoregulatory shift between cutaneous vasoconstrictor and vasodilator reflexes,” Journal of the Autonomic Nervous System, vol. 53, no. 2-3, pp. 215–222, 1995.
View at: Publisher Site | Google Scholar
D. L. Eckberg, “Point: counterpoint: respiratory sinus arrhythmia is due to a central mechanism vs. respiratory sinus arrhythmia is due to the baroreflex mechanism,” Journal of Applied Physiology, vol. 106, no. 5, pp. 1740–1742, 2009.
View at: Publisher Site | Google Scholar
G. Casiez, N. Roussel, and D. Vogel, “1  filter: a simple speed-based low-pass filter for noisy input in interactive systems,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2527–2530, ACM, Austin, TX, USA, May 2012.
View at: Publisher Site | Google Scholar
S. Blain, S. Kingsnorth, L. Stephens, and P. McKeever, “Determining the effects of therapeutic clowning on nurses in a children’s rehabilitation hospital,” Arts and Health, vol. 4, no. 1, pp. 26–38, 2012.
View at: Publisher Site | Google Scholar
J. G. Arena, S. J. Goldberg, D. L. Saul, and S. H. Hobbs, “Temporal stability of psychophysiological response profiles: analysis of individual response stereotypy and stimulus response specificity,” Behavior Therapy, vol. 20, no. 4, pp. 609–618, 1989.
View at: Publisher Site | Google Scholar
P. H. Venables and D. A. Mitchell, “The effects of age, sex and time of testing on skin conductance activity,” Biological Psychology, vol. 43, no. 2, pp. 87–101, 1996.
View at: Publisher Site | Google Scholar
J. M. Karemaker, “Counterpoint: respiratory sinus arrhythmia is due to the baroreflex mechanism,” Journal of Applied Physiology, vol. 106, no. 5, pp. 1742-1743, 2009.
View at: Publisher Site | Google Scholar
M. Elam and B. G. Wallin, “Skin blood flow responses to mental stress in man depend on body temperature,” Acta Physiologica Scandinavica, vol. 129, no. 3, pp. 489–497, 1987.
View at: Publisher Site | Google Scholar
G. Hubbard, M. G. Downs, and S. Tester, “Including older people with dementia in research: challenges and strategies,” Aging and Mental Health, vol. 7, no. 5, pp. 351–362, 2003.
View at: Publisher Site | Google Scholar
C. Mattingly and M. Lawlor, “The fragility of healing,” American Anthropologist, vol. 29, no. 1, pp. 30–57, 2001.
View at: Publisher Site | Google Scholar
M. J. Barry and S. Edgman-Levitan, “Shared decision making—The pinnacle of patient-centered care,” New England Journal of Medicine, vol. 366, no. 9, pp. 780-781, 2012.
View at: Publisher Site | Google Scholar
L. Nygård, “How can we get access to the experiences of people with dementia?: suggestions and reflections,” Scandinavian Journal of Occupational Therapy, vol. 13, no. 2, pp. 101–112, 2006.
View at: Publisher Site | Google Scholar
J. R. Bergstrom, S. Duda, D. Hawkins, and M. McGill, “Physiological response measurements,” in Eye Tracking in User Experience Design, J. Romano Bergstrom and A. J. Schall, Eds., pp. 81–108, Morgan Kaufmann Publishers, Burlington, MA, USA, 2014.
View at: Publisher Site | Google Scholar
G. Hubbard, A. Cook, S. Tester, and M. Downs, “Beyond words: older people with dementia using and interpreting nonverbal behaviour,” Journal of Aging Studies, vol. 16, no. 2, pp. 155–167, 2002.
View at: Publisher Site | Google Scholar
V. Ahanathapillai, J. D. Amor, and C. J. James, “Assistive technology to monitor activity, health and wellbeing in old age: the wrist wearable unit in the USEFIL project,” Technology and Disability, vol. 27, no. 1-2, pp. 17–29, 2015.
View at: Publisher Site | Google Scholar
C. Melander, J. Martinsson, and S. Gustafsson, “Measuring electrodermal activity to improve the identification of agitation in individuals with dementia,” Dementia and Geriatric Cognitive Disorders Extra, vol. 7, no. 3, pp. 430–439, 2017.
View at: Publisher Site | Google Scholar
B. Kikhia, T. Stavropoulos, S. Andreadis et al., “Utilizing a wristband sensor to measure the stress level for people with dementia,” Sensors, vol. 16, no. 12, p. 1989, 2016.
View at: Publisher Site | Google Scholar
G. E. Sakr, I. H. Elhajj, and H. A.-S. Huijer, “Support vector machines to define and detect agitation transition,” IEEE Transactions on Affective Computing, vol. 1, no. 2, pp. 98–108, 2010.
View at: Publisher Site | Google Scholar
A. L. Wainer and B. R. Ingersoll, “The use of innovative computer technology for teaching social communication to individuals with autism spectrum disorders,” Research in Autism Spectrum Disorders, vol. 5, no. 1, pp. 96–107, 2011.
View at: Publisher Site | Google Scholar
F. D. DiGennaro Reed, S. R. Hyman, and J. M. Hirst, “Applications of technology to teach social skills to children with autism,” Research in Autism Spectrum Disorders, vol. 5, no. 3, pp. 1003–1010, 2011.
View at: Publisher Site | Google Scholar
M. Egger, M. Ley, and S. Hanke, “Emotion recognition from physiological signal analysis: a review,” Electronic Notes in Theoretical Computer Science, vol. 343, pp. 35–55, 2019.
View at: Publisher Site | Google Scholar
K. H. Kim, S. W. Bang, and S. R. Kim, “Emotion recognition system using short-term monitoring of physiological signals,” Medical and Biological Engineering and Computing, vol. 42, no. 3, pp. 419–427, 2004.
View at: Publisher Site | Google Scholar
C. Maaoui and A. Pruski, “Emotion recognition through physiological signals for human-machine communication,” in Cutting Edge Robotics 2010, V. Kordic, Ed., InTech, London, UK, 2010.
View at: Publisher Site | Google Scholar
G. Keren, T. Kirschstein, E. Marchi, F. Ringeval, and B. Schuller, “End-to-end learning for dimensional emotion recognition from physiological signals,” in Proceedings of the 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 985–990, IEEE, Hong Kong, China, July 2017.
View at: Publisher Site | Google Scholar
O. Collins, S. Dillon, C. Finucane, B. Lawlor, and R. A. Kenny, “Parasympathetic autonomic dysfunction is common in mild cognitive impairment,” Neurobiology of Aging, vol. 33, no. 10, pp. 2324–2333, 2012.
View at: Publisher Site | Google Scholar
A. S. S. Meel-van den Abeelen, J. Lagro, E. D. Gommer, J. P. H. Reulen, and J. A. H. R. Claassen, “Baroreflex function is reduced in Alzheimer’s disease: a candidate biomarker?” Neurobiology of Aging, vol. 34, no. 4, pp. 1170–1176, 2013.
View at: Publisher Site | Google Scholar
R. E. De Meersman and P. K. Stein, “Vagal modulation and aging,” Biological Psychology, vol. 74, no. 2, pp. 165–173, 2007.
View at: Publisher Site | Google Scholar
S. Blain, S. D. Power, E. Sejdic, A. Mihailidis, and T. Chau, “A cardiorespiratory classifier of voluntary and involuntary electrodermal activity,” BioMedical Engineering OnLine, vol. 9, no. 1, p. 11, 2010.
View at: Publisher Site | Google Scholar
J. Cacioppo, G. Berntson, J. Larsen, K. Poehlmann, and T. Ito, “The psychophysiology of emotion,” in The Handbook of Emotion, M. Lewis and Haviland-Jones, Eds., vol. 2, pp. 173–191, The Guilford Press, New York, NY, USA, 2000.
View at: Google Scholar
S. D. Kreibig, F. H. Wilhelm, W. T. Roth, and J. J. Gross, “Cardiovascular, electrodermal, and respiratory response patterns to fear- and sadness-inducing films,” Psychophysiology, vol. 44, no. 5, pp. 787–806, 2007.
View at: Publisher Site | Google Scholar
F. Cugnata, R. M. Martoni, M. Ferrario, C. D. Serio, and C. Brombin, “Modeling physiological responses induced by an emotion recognition task using latent class mixed models,” PLoS One, vol. 13, no. 11, Article ID e0207123, 2018.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2019 Chelsey Lai Kwan et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

3519

Downloads

2169

Citations