Copyright © 2008 Nuri Firat Ince et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

This paper describes a flexible, cost-effective, wireless in-home activity monitoring system for assisting patients with cognitive impairments due to traumatic brain injury (TBI). The system locates the subject with fixed home sensors and classifies early morning bathroom activities of daily living with a wearable wireless accelerometer. The system extracts time- and frequency-domain features from the accelerometer data and classifies these features with a hybrid classifier that combines Gaussian mixture models and a finite state machine. In particular, the paper establishes that despite similarities between early morning bathroom activities of daily living, it is possible to detect and classify these activities with high accuracy. It also discusses system training and provides data to show that with proper feature selection, accurate detection and classification are possible for any subject with no subject specific training.

1. Introduction

Traumatic brain injury (TBI) is one of the leading causes of death and permanent disability in the United States (US). According to the Center for Disease Control (CDC), the number of TBI patients in the US is 5.3 million [1]. About 2% of the US population has a long-term TBI and needs assistance to perform activities of daily living (ADL). This number is expected to rise with the increase in the elderly population. Males are twice as likely to sustain TBI compared to females. Furthermore, recent military actions in Iraq have led to a marked increase in TBI amongst active duty soldiers in the 18–25 age group. For example, one of a Defense and Veterans Brain Injury Center's report indicates that 62% of patients screened between July and November of 2003 were identified as suffering from brain injury [2]. Direct medical costs and indirect costs such as lost productivity of TBI totaled an estimated $60 billion in the US in 2000 [3]. The system that we describe here can decrease this cost while still allowing TBI patients to lead independent and productive lives.

Traumatic brain injury is caused by a sudden impact or a penetrating injury to the head. In general, the frontal part of the brain is damaged in TBI cases. The frontal lobe is known to control higher cognitive functions. Therefore, TBI patients have difficulties with attention/concentration, planning, memory, execution, and completion of activities.

Today, care for TBI patients is provided by health professionals. Initial treatment is given at hospitals. In late recovery stages, patients are moved from the hospital and assistance is extended into the home. Wellness monitoring of the patients becomes very important at this point. Unfortunately, with the shortage in care givers and rise in the number of TBI cases, it is becoming increasingly difficult to provide the required level of human monitoring and assistance that TBI patients require.

As indicated previously, an impact to the frontal lobe of the brain causes TBI patients to have difficulties in planning, organizing, and completing activities. To assist TBI patients in planning their daily lives, several reminder/scheduler-oriented systems have been developed. In general, these systems are based on hand-held devices that deliver messages to the patient in an “open-loop” manner. For example, the planning and execution assistant and trainer (PEAT) [4] provides automatic assistance for task planning. It uses an integrated task planning and execution algorithm that is a spin-off from NASA's robotics research. Indeed, NASA's autonomous spacecraft and rovers on Mars require the same flexibility as people to accomplish goals in uncertain and changing situations. PEAT is an application of this technology on handheld computers for the purpose of cognitive rehabilitation. PEAT and similar calendar-type systems operate on a basic alarm clock strategy that does not account for the dynamic nature of a person's daily schedule and needs. In the recovery stage, TBI subjects typically remember the daily activities that they are supposed to perform. Such subjects can find repeated alarm-clock-type reminders unnecessary and annoying. Despite its complexity and flexibility in scheduling, PEAT requires feedback from the user that could instead be provided by appropriate sensors. Within the architecture of PEAT, the monitoring of an execution of a delivered message or reminder can only be obtained by user feedback based on continuous interaction with the hand-held computer. This requires that the hand-held PC always be carried by the individual.

Fortunately, researchers and system developers are beginning to focus on monitoring activities with in-home sensor networks to complement such reminder systems. In order to overcome the limitations of PEAT, a research group from the universities of Michigan and Pittsburgh has introduced a new type of planning system called Autominder [5] for cognitively impaired people. Autominder is a reminder and scheduling system involving a robot (Pearl) which has several onboard sensors to track the activity of the patients and to deliver visual and auditory messages [6] to them. However, the sensor strategy used in the system has several limitations. First, the robot is assumed to accurately observe the actions and location of the patient. This requires the robot to be able to move to each location with the patient. This may not be practical in real life situations and may be perceived by patients as intrusive. Indeed, our discussions with TBI experts indicate that most patients dislike systems that produce video or intelligible audio recording of their activities and are perceived as intruding on the patient's privacy. A robot is also very conspicuous, adding to the stigma that TBI patients may feel. Second, the dynamic information which can be obtained from wearable wireless sensors as previously described is missing. Our experience indicates that such information is critical for accurate classification of ADLs. Finally, as with the sensor systems described above, the efficacy of such reminder/planner systems has not been studied.

The literature provides evidence that to be useful and effective, a reminder or scheduler system must accurately classify and monitor the person's activities. The two main contributions of this paper are establishing that it is possible to detect and classify activities of daily living, despite their similarities, with a cost effective system and that the system requires little or no subject dependent training. We focus on the problems of detecting, classifying, and monitoring early morning bathroom activities such as face washing, tooth brushing, and face shaving to provide evidence to an intelligent reminder/planner algorithm. The system uses fixed sensors to locate the subject at home and track daily activities at a coarse level. Data from a wearable accelerometer is then used to detect and classify the precise early morning bathroom activity of daily living performed by the subject. The proposed system uses IEEE 802.11 and IEEE 802.15.4 standard compliant wireless sensor kits. The IEEE 802.15.4 compliant wearable sensors in particular provide low power and low data rate connectivity. They are used to monitor the execution of different activities at a detailed level. The wireless in-home fixed sensors are IEEE 802.11 compatible. In more complex systems designed to identify a larger set of activities of daily living, these fixed sensors can also be used to activate the proper wearable sensors that are best suited for recognizing activities of daily living performed in a given environment. The system uses Gaussian mixture models and a sequential classifier based on finite state machine to classify the wireless sensor data. A block diagram of the proposed architecture is shown in Figure 1.

Figure 1: The schematic diagram of the proposed system.

The paper is organized as follows. In Section 2, we describe our sensor network to collect data and discuss in detail the architecture of the system. In Section 3, we explain the experimental data and our classification strategy. Finally, in Section 4, we give classification results obtained from 7 subjects and discuss future directions.

2. Integration of Wireless Sensor Networks for Activity Monitoring

As mentioned earlier, the data acquisition system developed at the University of Minnesota integrates two sensor systems. The first sensor system is a collection of fixed wireless sensors. The second system relies on wearable sensors that provide data to complement the data collected by the first system. A schematic diagram of the system is given in Figure 2.

Figure 2: The data acquisition platform which combines static home and wearable wireless sensors.

Note that other designs are also possible and may offer some advantages over the system that we constructed. For example, a system that relies exclusively on wearable sensors would be easier and cheaper to deploy. Such a system would substitute accurate localization based on wireless transmissions for the inputs obtained from the fixed wireless system that we are using. In most of the systems that we have investigated, accurate localization from wireless signal measurements requires using more than one base station and in some cases extensive signal strength surveys across a home, negating the savings achieved by not installing the fixed sensors.

2.1. Static In-Home Wireless Sensors

Many technologies have been developed for in-home activity monitoring. Most of these technologies use static home sensors which are activated by the user [7, 8]. These include thermistors positioned under the bed to measure body motion, infrared sensors to detect the presence of the subject in a specific location, magnetic sensors attached to appliances to detect their use, and so forth. The use of such sensors gives strong clues about the individual's location and activities being performed. However, the wiring between the sensors and data center is a major issue for such a system. In our study, we elected to use eNeighbor (eN), a wireless remote in-home activity monitoring system which was recently developed by RedWing Technologies and is currently marketed under the name Healthsense (http://www.healthsense.com). The eN wireless sensor network is based on the IEEE 802.11 standard. It has an Atmel Mega 128 microprocessor and includes server technology applications for externally alerting and reporting monitoring information. An IEEE 802.15.4 network standard-based version of eN will also be available soon. This system comes with several sensors such as motion, bed, chair, and door sensors that enable it to track a broad range of daily activities at a coarse level as shown in Figure 3. Each sensor communicates with the base station only in the case of an event. Therefore, the sensors have long battery life and can be used at home without maintenance for long periods of time. Each event received by the base station is exported in real time through the USB port to an external device for backup. We have developed a USB port driver to capture the messages transmitted from the base station and save these messages on a PC with a time stamp to synchronize with the other sensors in the remaining system.

Figure 3: (a) The static home sensors; from left to right: door sensor, base station, and motion sensor. (b) A typical in-home setting of static home sensors. (c) In-home sensor data transmitted from the base station to the PC while a subject is moving from the bedroom to the bathroom. (d) Wearable wireless sensor kit attached to the right wrist.

2.2. Wearable Wireless Sensors

The eN gives binary information that provides clues about the activities carried out by the individual. There are many activities where interactions with these sensors do not occur. In addition, some activities may trigger the same sensors. For instance, the subject may enter the bathroom for a washing or brushing activity. During these two activities, the same subset of sensors is activated which makes it difficult to distinguish between wash and brush activities by examining the binary sensor data of the eN.

To get detailed information about the activity of the patient, we use wearable sensors attached to the wrist and installed on a wireless networked embedded system (see Figure 3(d)). In particular, we selected the MICAz wireless nodes developed by Crossbow Technology Inc. (http://www.xbow.com) for wearable data collection. Data transmission and reception on the MICAz is handled by a Chipcon CC2420 radio chip, which is IEEE 802.15.4 compliant. It has a 250 Kbps radio throughput rate. The onboard expansion slot enables the designer to interface several sensors to the microprocessor. The microprocessor runs TinyOS 1.1.7, a small open source operating system for the embedded sensor networks. The microprocessor is programmed with the NesC programming language to collect and transmit the sensor readings to the PC. NesC is a new programming environment for networked embedded systems. It significantly simplifies the efforts for application development under TinyOS (http://www.tinyos.net).

In our system, we used the MTS310 multisensor board to record movement and environmental parameters. The MTS310 has onboard light sensors, temperature sensors, a 2-axis accelerometer, a 2-axis magnetometer, and a microphone. These sensors are connected to the multichannel 10-bit ADC of the mote kit.

In this paper, we will restrict ourselves to the presentation and analysis of accelerometer data. The onboard sensor is an Analog Devices ADXL202JE dual-axis accelerometer.

The use of accelerometers for activity recognition is not new. Initial applications of accelerometers have concentrated on the recognition of sitting, standing, and walking behavior [9]. The system of [9] used two biaxial accelerometers attached to waist and leg to estimate body position and lower-limb gestures. The accelerometer sensors are wired to a PDA for data collection. The wiring is a critical issue which limits the user activity in real life situations. In another system that consists of five biaxial accelerometers attached to several locations on the body has been used for activity recognition [10]. In order to remove the wirings between the sensors and data center, the system used hoarder boards. The data was locally stored with time stamps on these boards and post processed offline for synchronization and classification. By using decision tree classifiers, the system was able to recognize 20 everyday activities with an overall accuracy rate of 84%. The studies of [9, 10] showed that the flexible data collection is a critical step to give the subject the freedom to do his/her daily activities.

In order to transfer accelerometer data to the PC we used an MIB510 serial getaway. The MICAz mote communicates with the MIB510 gateway using a wireless IEEE 802.15.4 link. The gateway transmits the received sensor readings to the PC through an RS-232 port. In the current system, the data communication rate is limited to 56 Kbps on the RS-232 side. This data rate was high enough to transmit data from the sensors since the sensors outputs are sampled at the rate of 50 samples/s. The reader can find detailed information about the data acquisition system in [11].

On the PC side, we developed another serial port driver to capture the packets received from the MIB510 gateway. We saved the sensor readings in an ASCII file with time stamps similar to those used by the eN system for further processing. We developed software to capture the serial messages transmitted by the eN system and the MIB510 using ActiveX components built on top the MS Windows application programming interface (WINAPI). This could have also been done using the Matlab (MathWorks Inc, Natick, Mass, USA) serial line programming interface in order to bypass detailed WINAPI.

3. Detection of Activities of Daily Living

Let us now describe the data that we collected to design and test the system, explain the classification procedure we constructed, and discuss system training.

As mentioned earlier, the system that we developed relies on a two-phase approach for detecting, classifying, and monitoring ADLs. In phase I, we localize the subject within a specific room of a home and perhaps on a specific piece of furniture using the fixed wireless sensors, for example, eN in our case. This allows us to constrain the list of most likely activities that the subject may be executing. In phase II, we rely on the wearable accelerometer sensor to detect, classify, and monitor the progress of ADLs. In this phase, we rely only on accelerometer data.

3.1. Early Morning ADL Data

ADLs can be classified into 3 different categories: basic, instrumental, and enhanced ADL. According to [12], basic ADL deals with personal hygiene and nutrition such as washing, toileting, and eating. The authors state that all people living independently should be able to execute these basic ADLs. Instrumental activities can be managing a medication intake, maintaining a household, and so forth, while enhanced ADLs involve activities outside one's residence and social interactions. We have selected several basic early morning ADLs for initial investigation.

Our initial studies and system design are based on healthy subjects since data collection from TBI patients is difficult and most TBI patients do not have any upper limb disability preventing them from carrying out their early morning ADLs. We will continue to design, refine, and test the system with data collected from healthy subjects. Once we achieve an acceptable performance level, we will test our system on TBI patients and refine it further.

3.1.1. Data Collection

In this paper, we focus in particular on the classification of three ADLs. These are face washing, tooth brushing, and face shaving. The data was recorded from seven healthy subjects with the system described above. A single mote kit is attached to the wrist to record hand movements. After a small training period, the wireless sensor system and user friendly data acquisition software installed on a notebook PC were given to the subjects to record the ADL data in their home setting. For privacy reasons, no audio or video data were recorded. In order to provide the ground truth for recorded wearable and static home sensor data, we conducted a single trial based recording paradigm. The subjects freely executed one of the three early morning activities listed above and the data were labeled manually after each recording. The number of available trials for each activity is given in Table 1. Sample signals corresponding to these activities are shown in Figure 4. In addition to the 3 distinct activities, subjects were also asked to record data related to activities that have no specific purpose or do not correspond to the three early morning activities listed above. Examples of such activities include changing a towel, arranging items on the sink. All such activities are categorized as other-activity (OAct).

Table 1: Available trials.

Figure 4: Typical recordings obtained from 2-channel accelerometer sensor (Ax and Ay) attached to the right wrist; (a) tooth brushing, (b) face washing, and (c) face shaving.

During the data collection process, subjects reported that tooth brushing and face shaving were generally preceded and followed by a face wash activity. Although we attempted to record a single activity, many tooth brushing and face shaving recordings included a short duration of face washing. Therefore, in our final decision evaluation, we ignored washing outputs when they are observed just before and after tooth brushing and face shaving activities.

3.2. Classification of Early Morning ADL Data

3.2.1. Feature Extraction

There are several possibilities for generating activity state models and ADL classification methods. In this study, we use a hierarchical classification system as indicated in Figure 5 because of its simplicity and performance. The system combines Gaussian mixture models (GMM) and a sequential classifier. We use GMMs to model the activities such as tooth brushing, face washing, and face shaving. GMMs are widely used in continuous classification of EMG signals for prosthetic control and speaker identification problems due to their robustness and low computational complexity [13, 14]. The main motivation of using a GMM is that it provides a generative model of each task. The mixtures in the model are believed to represent the sub activities executed by the subject when engaged with a specific task. Furthermore, the number of mixtures can account for variability across subjects as well.

Figure 5: (a) The schematic diagram of the proposed classification system which is based on GMM followed by a sequential classifier. (b) The dyadic frequency bands used to extract frequency-domain features.

We extracted time-domain (TD) and frequency-domain (FD) features from the accelerometer data which were input to the GMM. The 2-axis accelerometer sensor provides two types of outputs for each channel. The DC component of the accelerometer sensor is related to the tilt information and the AC component is related to the acceleration signals. The time-domain features are extracted from raw data. We believe that it reflects the hand position. Frequency-domain features are extracted from the AC component measurement. Therefore, we combine both feature sets in the final classification. The time-domain features consist of the mean, root mean square, and the number of zero crossings in a 64 sample time segment. After applying a first-order high-pass Butterworth filter, we calculate the frequency-domain features for the AC component of the acceleration signal. We extend the feature set with energies in different frequency bands. The Fourier transforms of the accelerometer data along the two axes are calculated from each 64 sample time segments along with the time-domain features. The time segments are shifted with 50% overlap across the signal. In each segment, we calculate the energy in dyadic frequency bands as indicated in Figure 5(b). Frequency-domain features are then converted to log scale and combined with time-domain features related to the same time segment. This resulting feature vector has a dimension of 16 in each time segment [15].

3.2.2. GMM Classifier and Preliminary Decision

A GMM probability density function (pdf) is defined as a weighted combination of Gaussians:(1)Here, is the model, is the feature vector, is the -dimensional Gaussian pdf:(2)with mean vector and covariance matrix . Parameter is the weight of each component and satisfies(3)

A new observed feature vector can be assigned to one of the four classes () after evaluating the posterior probability of each GMM. Specifically, the label assigned to an observed vector is calculated as(4)

Model order selection plays a big role in determining the performance in GMM based systems. While a low number of mixtures can poorly represent the geometry of the activity in a -dimensional space, a high number of mixtures generally over fit the data. We have found that by varying the number of mixtures from 1 to 6 we are able to find the optimal value for classification.

3.2.3. Postprocessing and Final Decision

The evaluation in (4) gives a class label for each time point. This corresponds to the continuous classification of the streaming data from the sensors. However, we noticed that the arm movements during each task contain many sub-segments where the activity is not locally related to the task being executed. In addition, as we emphasized before, a single task can be executed by visiting many subtasks that also involve the 3 activities we focus on. For example, a face shaving task may start with face washing, then applying cream to the face, shaving with the razor, and at the end again washing the face. Therefore, the GMM outputs give many local outputs that cause a high false positive recognition rate. In our previous work, we utilized a fixed window majority voter (MV) procedure to remove local errors [15]. The majority voter used 16 points (10s) windows to decide whether the observation sequence is related to any of the tasks of interest. Although several time points were used for voting, we noticed that the classifier performed poorly during state transitions. We also noticed that the execution times of the three tasks that we are studying were quite different. A fixed window size does not provide enough flexibility to deal with these differences.

To improve performance, we used a sequential classifier that acts as a finite state machine (FSM) as described below. Instead of calculating the posterior probabilities for each feature vector on the GMM outputs, first we evaluate the output probabilities over an 8-point time window with a naive Bayesian classifier to smooth the GMM outputs. Specifically, we compute(5)We calculate the posterior probabilities of each naive Bayesian classifier and then convert them to discrete symbols that are processed by a sequential classifier. We remove observations which have low posterior probability at the input stage of the sequential classifier. Specifically, we use(6)(7)where is the naive Bayesian classifier posterior probability of , , , and nodes, is the prior probability of each task and is the input labels to the sequential classifier. Equation (7) removes outputs with low probabilities that occur at the beginning and end of each task, since these correspond to time intervals where uncertainty is high. This stage also converts the continuous input sequence to discrete labels such as . These discrete inputs from the wash, brush, shave, and OAct nodes are processed by the sequential classifier as indicated in Figure 7. The sequential classifier essentially tracks the states by counting the labels in the input sequence and deciding whether the resulting sequence is related to one of the 4 tasks that we study. If not, it provides a NoAct output. Note that, rather than using a fixed window size majority voter, the sequential classifier provides a state tracking capability and flexibility in the task specific selection of the processing window size. Since we do not know where the real activity starts and ends, the sequential classifier provides great flexibility and accounts for the temporal variability in the data.

In a similar study, a hidden Markov model- (HMM-) based approach has been used for activity modeling [16]. The authors have used a fixed size time window HMM and shifted the window along the signal to get classification outputs. In our study, the sequential classifier works without any window size limitation on the observed sequence. The window sizes for a particular activity are adjusted to subject differences. In our experimental studies, we observed that in most cases the washing activity takes much less time than the tooth brushing and face shaving activities. Furthermore, many segments of activities may involve similar movement of the arm. For instance, if a subject engages in the face shaving task, we often obtain brush labels in the beginning of the task due to common movement patterns between applying shaving cream to the face and tooth brushing. Both activities include circular hand movements which induces oscillatory components in the accelerometer sensor. A fixed size HMM can miss this when it is run in the beginning of a task. In the transition regions between states, the HMM may then provide several local errors. On the other hand, the sequential detector implements a sequential test. It waits until enough evidence has been collected before making a final decision. When an input is observed, it waits until the system classifies the next state which will give further information about what task is/was being executed. For example, if a tooth brushing input is observed, the system waits to see if the next state is putting cream/shaving, in which case it would classify the entire activity as face shaving rather than tooth brushing.

4. Results

In order to evaluate the performance of the extracted time-domain and frequency-domain features and their combination in classification, we conducted several “leave one subject out” (LOSO) experiments. In particular, we collected data from 7 subjects and used the data of one of them for testing and the remaining subjects' data for training the system. This procedure was repeated for all 7 subjects to obtain classification performance and was averaged to obtain overall classification accuracy. The classification results obtained with the LOSO method provide information about the subject generalization capability of the proposed system. Table 2 provides classification results for time-domain features, frequency-domain features, and their combination.

Table 2: Classification accuracies of different feature sets (%).

Table 3: Classification accuracies (%) obtained from combination with sequential classifier post processing. The NoMix stands for the number of mixtures in GMM.

Table 4: Classification accuracies (%) obtained from combination with majority voter post processing. The NoMix stands for the number of mixtures in GMM.

Table 5: The confusion matrix for combination and sequential classifier postprocessing for

Table 6: The confusion matrix for combination and majority voter postprocessing for

The combination of time-domain and frequency-domain features yields better classification performance than using time-domain or frequency-domain features alone. This suggests that the acceleration and the arm's tilt data carry significant information for activity recognition. In addition, the classification performance of the technique sequential classifier was better than the majority voter approach. The classification results for different number of mixtures are given in Tables 3 and 4 for the sequential classifier and majority voter approaches. We noticed that the best classification accuracy is obtained with 2 mixtures for sequential classifier and majority voter approaches. Increasing the number of mixtures for both approaches decreased the classification accuracy. A higher number of mixtures may result in over learning in the GMM stage. We believe that a low number of mixtures provide smoothness and enhance the correctness of the classifier. The confusion matrices related to the best mixture indexes for the sequential-classifier- and majority-voter-based approaches are given in Tables 5 and 6, respectively.

As mentioned earlier, in our experimental studies we noticed that there is a significant overlap in the feature space between the activities of tooth brushing, putting soap, and applying shaving cream to the face. All of these segments include circular hand movements that cause sinusoidal waveforms in the accelerometer. As can be seen from the confusion matrices, the face washing and face shaving activities are mostly classified as tooth brushing in these regions. In particular, putting soap or applying shaving cream is locally recognized as a tooth brushing activity. A representative trial is shown in Figure 6. The sequential classifier eliminated many of these false positives by using different time window thresholds. For the brushing activity, a higher brush count (BC) is used for final decision.

Figure 6: (a) The Bayesian posterior probabilities of the classifiers during a washing task. (b) The input votes () entering the sequential detector. Note that the putting soap section is locally classified as tooth brushing. (c) The Bayesian posterior probabilities related to brush activity and the input votes entering the sequential detector (d). Note that tooth brushing task is followed by a washing activity due to giving rinse. They are ignored in final evaluation ().

Figure 7

It should be noted that in our final evaluation of the classification performance, face washing outputs preceding and following brush/shave activities are ignored. Most of the time, subjects washed their faces prior to shaving or rinsed after brushing.

Note that the local OAct decisions are not evaluated as false positives. Such decisions are ignored because it is possible that the subjects can interrupt the main task for a short while. In addition, it takes several seconds for the subjects to start with the main task. For instance, when subjects grab the brush or the shaver, the classifier mostly produced an OAct or NoAct output. Therefore, OAct and NoAct outputs are merged in the final evaluation and are not evaluated as a false positive if they are locally present. As indicated previously, the main purpose of including OAct trials into the dataset is to account for activities where the subjects are not really performing the ADLs that we studied here.

In order to assess the efficiency of the GMM, we replaced it with a linear discriminant classifier (LDC) that models the feature vectors corresponding to each activity as Gaussian vectors with identical covariances and activity dependent means. In this way, we could evaluate the recognition accuracy of a discriminative approach working in the lower level of the system. In particular, we used a pair-wise classification strategy by constructing several linear discriminant classifiers. Each LDC discriminates a single task from another. In particular, every feature vector is processed by the pair-wise LDC bank. Then, each time point was stamped with a discrete label by evaluating the LDC bank outputs. As in the GMM case, the discrete sequence was then fed to a sequential classifier for final decision. The classification results obtained with the LDC are compared with the GMM approach using one or two mixtures, denoted as GMM-1 and GMM-2, respectively, in Table 7. Interestingly, the linear discriminant classifier provided very high recognition accuracy for the face shaving activity and outperformed the results obtained with GMMs. However, we noticed, while recognizing the tooth brushing and face washing activities, that the results obtained with the LDC are worse than the GMM-2-based results. Furthermore, the OAct trials are misclassified as face shaving activity. The results that we obtained thus indicate that the LDC-based approach is biased towards the shaving activity. The confusion matrix of LDC-based classification system is given in Table 8.

Table 7: Classification accuracies (%) of different classifiers.

Table 8: The confusion matrix for LDC-based classification system.

5. Limitations and Future Work

During the experimental studies we noticed that some subjects changed their active hand during task execution. For instance, one of our subjects switched his hand during brushing trials. This behavior eliminated the accelerometer observations and the system went to OAct state.

When the instrument used to perform the activities that we studied is electric, the measured patterns change. Electric tooth brushes and shavers need to be treated in a different manner. Currently, the authors are exploring the use of acoustic recording in the recognition of these activities when an electric shaver and brush is utilized. Another possibility is to use tiny modules which include an accelerometer and a radio attached to the electric shaver or tooth brush. When the electric shaver or brush is turned on, accelerometer data are transmitted to the system.

We also noticed that face washing of different subjects exhibited two distinct motion patterns. In particular, we observed that one group of subjects were applying soap, drawing water, and rinsing the face. The other group of subjects washed their face by simply splashing water onto their face. Although, few different patterns were observed within each group, in general, any washing activity involved one of the two patterns mentioned above. We noticed that when the training data were biased to one group, then the classification accuracy corresponding to face washing was much lower compared to when the training data was balanced. This shows that unless similar patterns are present in the training set, the classifier will not be able to correctly classify activities. One solution to overcome this problem is to refine the classifier with a small number of trials from the user or the subject himself. This allows the system to adapt to the unique patterns [17].

Wearable wireless sensors are one of the main components of this system. The continuous monitoring task involves continuous packet exchanges between the computational center and the wearable sensors. It is well known that the power consumption of wireless embedded systems increases while communicating. A straightforward online data transfer can decrease the battery life dramatically. In such a case the wearable system will need frequent maintenance. Therefore, an intelligent and adaptive data collection and communication strategy is necessary. In-home static sensors can be used to decide when and how to collect wearable sensor data. Furthermore, after a certain period we expect to capture the lifestyle of the person so that the system can then infer from this information to create adaptive data collection strategies.

6. Conclusion

In this paper, we described the infrastructure of an in-home activity monitoring system based on wearable and fixed wireless sensors. The system is intended to assist people with cognitive impairments due to TBI. In particular, we focused on the problems of detecting early morning bathroom activities of daily living at home. The proposed system uses IEEE 802.11 and IEEE 802.15.4 standard compliant wireless sensor kits. Finally, the data collected from both sensor networks are processed by intelligent algorithms. We showed experimental results from 7 subjects engaged in face washing, face shaving, and tooth brushing activities. Our preliminary results are quite promising. The integration of the activity detection algorithms with the reminder and planner modules may allow TBI patients to freely continue their individual life in the future.

References

1. National Center for Injury Control and Prevention, 2006, http://www.cdc.gov/ncipc/tbi/TBI.htm.
2. Defense and Veterans Brain Injury Center (DVBIC), http://www.dvbic.org/cms.php?p=Blast_injury.
3. E. A. Finkelstein, P. S. Corso, and T. R. Miller, The Incidence and Economic Burden of Injuries in the United States, Oxford University Press, New York, NY, USA, 2006.
4. R. Levinson, “The planning and execution assistant and trainer (PEAT),” Journal of Head Trauma Rehabilitation, vol. 12, no. 2, pp. 85–91, 1997.
5. M. E. Pollack, “Planning technology for intelligent cognitive orthotics,” in Proceedings of the 6th International Conference on Automated Planning and Scheduling, pp. 322–331, Menlo Park, Calif, USA, April 2002.
6. N. Roy, G. Baltus, D. Fox, et al., “Towards personal service robots for the elderly,” in Proceedings of the Workshop on Interactive Robots and Entertainment (WIRE '00), Pittsburgh, Pa, USA, April 2000.
7. T. Tamura, T. Togawa, M. Ogawa, and M. Yoda, “Fully automated health monitoring system in the home,” Medical Engineering and Physics, vol. 20, no. 8, pp. 573–579, 1998.
8. M. Ogawa and T. Togawa, “Monitoring daily activities and behaviors at home by using brief sensors,” in Proceedings of the 1st Annual International Conference On Microtechnologies in Medicine & Biology, pp. 611–614, Lyon, France, October 2000.
9. S.-W. Lee and K. Mase, “Activity and location recognition using wearable sensors,” IEEE Pervasive Computing, vol. 1, no. 3, pp. 24–32, 2002.
10. L. Bao and S. S. Intille, “Activity recognition from user-annotated acceleration data,” in Proceedings of the 2nd International Conference on Pervasive Computing and Communications (PERVASIVE '04), A. Ferscha and F. Mattern, Eds., vol. 3001 of Lecture Notes in Computer Science, pp. 1–17, Springer, Vienna, Austria, April 2004.
11. N. F. Ince, C.-H. Min, and A. H. Tewfik, “Integration of wearable wireless sensors and non-intrusive wireless in-home monitoring system to collect and label the data from activities of daily living,” in Proceedings of the 3rd International Summer School and Symposium on Medical Devices and Biosensors (ISSS-MDBS '06), MIT, Cambridge, Mass, USA, September 2006.
12. E. D. Mynatt, I. Essa, and W. Rogers, “Increasing the opportunities for aging in place,” in Proceedings of the ACM Conference on Universal Usability (CUU '00), pp. 65–71, Arlington, Va, USA, November 2000.
13. Y. Huang, K. B. Englehart, B. Hudgins, and A. D. C. Chan, “A Gaussian mixture model based classification scheme for myoelectric control of powered upper limb prostheses,” IEEE Transactions on Biomedical Engineering, vol. 52, no. 11, pp. 1801–1811, 2005.
14. A. D. Reynolds and C. R. Rose, “Robust text-independent speaker identification using Gaussian mixture speaker models,” IEEE Transactions on Speech and Audio Processing, vol. 3, no. 1, pp. 72–83, 1995.
15. N. F. Ince, C.-H. Min, and A. H. Tewfik, “In-home assistive system for traumatic brain injury patients,” in Peoceedings of the 32nd IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '07), vol. 2, pp. 565–568, Honolulu, Hawaii, USA, April 2007.
16. J. Lester, T. Choudhury, N. Kern, G. Borriello, and B. Hannaford, “A hybrid discrimitive/generative approach for modeling human activities,” in Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI '05), pp. 766–772, Edinburgh, Scotland, July-August 2005.
17. C.-H. Min, N. F. Ince, and A. H. Tewfik, “Generalization capability of a wearable early morning activity detection system,” in Proceedings of the 15th European Signal Processing Conference (EUSIPCO '07), Poznan, Poland, September 2007.