Abstract
With the prevalence of cognitive diseases, the health industry is facing newer challenges since cognitive health deteriorates gradually over time, and clear signs and symptoms appear when it is too late. Smart homes and the IoT (Internet of Things) have given hope to the health industry to monitor and manage the elderly and the less-abled in the comfort of their homes. Smart homes have been most influential in detecting and managing cognitive diseases like dementia. They can give a comprehensive view of the ADL (Activities of Daily Living) of dementia patients. ADLs are categorized as activities of daily life and complex interwoven activities. First signs of cognitive decline appear when a cognitively impaired individual tries to perform complex activities involving planning, analyzing, calculating, and decision making. Therefore, we analyze individuals’ performance while performing complex activities as opposed to Simple ADL. Artificial Intelligence has been one of health-care’s most promising techniques for prediction and diagnosis. When applied to ADL data, machine learning and deep learning algorithms can conveniently and accurately analyze activity patterns and predict the first signs of cognitive decline. Our proposed work uses machine and deep learning classifiers to classify dementia and healthy individuals by analyzing complex interwoven activity data. We use the subset of the CASAS (Centre of Advanced Studies in Adaptive Systems) dataset for eight complex activities performed by 179 individuals in a smart home setting. decision tree, Naive Bayes, support vector, multilayer perceptron classifiers, and deep neural networks have been used for classification. Their results and performances are compared to determine the best classifier. It is observed that deep neural networks and multilayer perceptron show the best results for classifying dementia vs. healthy individuals when evaluating their complex interwoven activities.
1. Introduction
There has been a rapid increase in mental disorders and the people suffering from them in the last few years. Over 1 billion people suffer from one mental disease, addiction, dementia, or schizophrenia [1] [2]. WHO (World Health Organization) has concluded that the investment in mental health has not matched the awareness scale of mental health problems. In 2010 reduced productivity and poor health owing to poor mental health resulted in a $2.5 trillion loss worldwide. This figure is expected to rise to $6 trillion by 2030 at the current rate [2, 3]. Early intervention can reduce the healthcare system’s burden globally and eventually reduce the associated mortality rate [4, 5]. At present cognitive health is analyzed in the clinic using a cognitive function test like MMSE (Mini-Mental State Examination) and MoCA (Montreal Cognitive Assessment) and other neurological exams like CDR (Clinical Dementia Rating) and assessment of ADL. The information regarding a patient’s ADLs is gathered via a questionnaire filled by the patient himself or his/her guardian, hence making the entire assessment process subjective [6, 7]. This can result in an inaccurate assessment. It has been observed that very subtle signs and symptoms first appear in the daily activities of individuals suffering from cognitive decline, which clinicians can easily miss in a physical exam. For the said reason, HAR (Human Activity Recognition) is emerging as an effective method to monitor an individual’s movements and activities and has gained special focus in the field of research to improve healthcare systems. [8, 9]. Smart homes with multiple sensors are a promising tool for HAR to gather data regarding ADL. Smart homes have multiple networks of sensors that can gather an overview of residents’ activity patterns in terms of their health, security, safety, independent activities, and their social lives [10, 11] as can be seen in Figure 1.
The ADLs gathered via sensors in a smart home are divided into simple and complex activities. Simple activities are the activities performed to manage an individual’s basic needs, like grooming, dressing, toileting, and eating. The complex activities include activities that enable an individual to live independently in the community. This would be planning a bus route, managing medication, finances, etc. Complex activities are interrelated activities that require a degree of decision-making and calculations. While simple activities are recorded based on a single sensor event, recording complex activities is not as easy and requires input from multiple sensors [10, 12].
ML(Machine Learning) and DL(Deep Learning) tools are commonly used to perform an in-depth analysis of all activities [13]. Several researchers have analyzed ADLs, compared outcomes, and identified patterns to differentiate cognitively impaired from healthy individuals. The proposed work aims to compare and contrast the efficacy of different ML and DL algorithms to analyze a publicly available dataset CASAS for complex activities obtained by different sensor readings in a smart home setting. The dataset includes a combination of both simple and complex activities.
The biggest challenge for a patient suffering from cognitive impairment is leading an independent lifestyle. Living an independent life requires not just the ability to perform daily life functional activities but also the individual to perform several complex daily activities that are dependent on other activities but require a degree of calculation and decision making [14]. Complex activities, also known as IADL (Instrumental Activities of Daily Living), are a deciding factor in diagnosing a cognitive disease. Presently, the gold standard for assessment is an in-clinic exam that can be subjective. Our primary motivation is to analyze the complex activities in the daily lives of individuals to detect the earliest signs of cognitive decline. Hence we aim to accurately and timely predict the presence of cognitive impairment or dementia by analyzing the complex activities done in daily life. We use different ML and DL techniques to find the most accurate prediction model and then compare and contrast each model’s results to determine the efficacy and determine which model is best for classification. This paper aims to contribute to research in the following aspects: (i)Proposes an approach to classify dementia individuals by analyzing complex activities performed in a smart home setting to detect the earliest signs of dementia using machine learning and deep learning(ii)Present a comparison between machine learning and deep learning algorithms to evaluate the best model and provide a baseline study(iii)Deep learning algorithm enhances dementia individuals’ detection rate compared to a machine learning algorithm and overperforms the baseline paper detection rate
The rest of the paper is structured as follows: Section 2 is a literature review of the previous work in the early detection of dementia via examining ADL data. Section 3 gives a detailed overview of the proposed approach. Experimental analysis and results are presented in Section 4, where all the different techniques employed for classification are discussed. Section 5 provides the discussion on experimental analysis. The results of all the classifiers are compared and contrasted. Finally, Section 5 concludes the paper by sharing the best model and performance.
2. Literature Review
Smart homes have been the solution of choice for the aging population and those with disabilities. Researchers have gathered ADL to analyze, diagnose, and predict cognitive health problems.
In one such research [4], the author has used the DL technique to detect the early symptoms of MCI (Mild Cognitive Impairment). Using the time-series prediction technique, he has further handled the issue of missing sensor signals, which often arise in real-time data gathering due to sensor failure. Furthermore, the author proposes an autoencoder-based technique to reduce the dimension of the data so that deviation in human behavior can be detected using an RNN-based approach. A persisting abnormal behavior indicates a problem and alerts of MCI. In [15], a robot-enabled activity support system has been proposed and is evaluated in a smart home testbed. The robot is useful in monitoring the activities of the residents and can assist in daily activities where needed.
The possibility of detecting changes in psychological, cognitive, and behavioral symptoms of Alzheimer’s disease by using unobtrusively collected smart home behavior data and machine learning techniques was evaluated in [16]. The authors have analyzed the publicly available CASAS dataset and have tried to handle imbalanced data using the Weka tool. Four models, Support Vector Regression, Linear Regression, Support Vector Regression with a Radial Basis Function kernel, and k-nearest neighbors algorithms, were used to predict mobility, cognitive, and mood-related symptoms from gathered in-home behavior data.
In another research paper [17], the author uses unobtrusively collected smart home behavioral data to diagnose functional health decline. Activity data from the CASAS dataset was obtained from 38 smart homes, and the functional health assessment of participants was conducted using the IADL-C questionnaire. This data was then analyzed using different ML algorithms. [8] uses a multisensor approach to recognize complex activities using a CNN (Convolutional Neural Network) and an LSTM (Long Short Term Memory) model and compare the performance of both models. In [18], the authors use a machine learning model to assess the quality of activity in smart homes and classify the activities as simple and complex compared to a neurologist’s assessment. The author also uses a machine learning approach to assess the accuracy of predicting cognitive health conditions like dementia, MCI, and Alzheimer’s disease.
3. Proposed Approach
The proposed work aims to predict healthy vs. dementia patients based upon the analysis of ADL data about complex activities as shown in Figure 2. The responses of individuals towards complex activities will help classify individuals and enable early detection of the onset of dementia. The proposed approach is divided into five steps: data selection, preprocessing, features extraction, and ML and DL classifiers to classify healthy individuals and those with dementia.
3.1. Data Selection
Choosing the most suitable dataset and determining the right instrument for data collection is of utmost importance in an experimental setup. Therefore, after a careful selection process, we selected s subset of the CASAS dataset. The dataset has been made publicly available by “The Centre for advanced studies in adaptive systems,” a department at Washington State University in the School of Electrical Engineering and Computer Science. It aims to research the use of smart home technology to test real data. The dataset comprises a mix of simple and complex activities that were used to perform our analysis.
Our dataset contains 179 individuals, 145 of whom are healthy, 32 suffering from MCI, and two being diagnosed with dementia. The dataset contains 24 daily life activities, where the first eight activities are simple tasks, and last eight activities are complex tasks. Tasks from 9-16 are unlabeled and not classified as simple or complex hence they have been excluded from our current research. Each participant was evaluated on the eight complex activities that were part of daily life activities like selecting a magazine from the coffee table to read it during a commute on the bus, heating a heating pad for 3 minutes using the microwave to take along, before leaving for the bus take medicine for motion sickness, calculating the bus route and estimating the time to leave for the bus and the total journey from the map, and then the individual was expected to calculate the bus fair and gather the correct change needed. The individual was also assessed on the complex activity of finding a recipe from the recipe book for spaghetti sauce and gathering all the ingredients for making it. The picnic basket activity involved making a basket by gathering all necessary things from the cupboard and putting them in the basket. Lastly, the individual was expected to exit towards the door with the picnic basket. These were the eight complex activities that the individuals were evaluated.
Multiple sensors in a smart home setting were utilized to gather the activity data. A combination of motion sensors, door sensors, burner sensor, temperature sensors, etc. has been used to evaluate a complex task.
3.2. Data Preprocessing
The ability to extract information from data is directly linked to data quality. The quality of data depends on how clean and meaningful the data is. In order to make our data suitable for evaluation, we preprocessed the data.
3.2.1. Checking for Missing Values
All attributes were checked for missing values. Missing values, i.e., sensor values with no readings, were replaced with zeros.
3.2.2. Categorizing into Numeric Values
All nonnumeric attributes were assigned categorical values. The attribute diagnosis had two values, healthy and dementia. ‘1’ was assigned to healthy while ‘0’ was assigned to dementia.
3.2.3. Scaling All Quantities
Values of all sensors were then normalized within a range using a min-max scaler. This scales the sensor readings between 0 and 1 without changing the distribution’s shape and retaining the data’s original properties. ML and DL results improve considerably if the data is scaled.
3.3. Feature Extraction
In any given dataset, certain features are irrelevant to specific research or include details that do not contribute anything significant to the research process. In order to eliminate unnecessary processing, extracting the relevant features from the dataset is essential. The redundant sensor readings were eliminated by performing the Pearson correlation. The threshold was identified at 90%. All attributes greater than 90% were eliminated because they were highly correlated. Almost 58 sensors were highly correlated. Hence, they were eliminated. The classification was applied to the readings from the remaining sensors.
3.4. Classification Models
Artificial Intelligence techniques can be accurately used in healthcare for the prediction and diagnosis of disease in an objective manner. ML and DL are the most commonly used AI techniques for disease prediction. We applied four ML models on to our dataset comprising complex activity data. They were DTC (decision tree classifier), NB (Naive Bayes), SVC (support vector classifier), and MLP (Multilayer perceptron) classifier. We also used a deep neural network comprising four dense layers to classify our data.
3.5. Evaluation Metric
For each of the models implemented, several evaluation metric tools were used to evaluate the performance of each model and to compare which model yields the best output. We compare and contrast the following:
3.5.1. Cross-Validation Scores of the Training Data
Since K-fold cross-validation (CV) is an effective measure in model selection, we performed a 10-fold CV on our training dataset for each model [19]. Then we computed the mean accuracy after the validations process for all ten iterations and the means’ standard deviation to ensure the data’s homogeneity.
3.5.2. The Mean Accuracy and the Standard Deviation of Accuracy of the Training Data
When performing the CV, it is important to calculate the average of all ten results to get an overview of the model performance and also to include a measure of the variance of all ten outcomes in order to rule out any unusual outcomes in the form of outliers [20].
3.5.3. The Time Each Model Took to Train
How many seconds it took for the ML model to calculate the accuracy? Training time is usually more than testing since training data is a bigger proportion of the dataset.
3.5.4. The Accuracy of the Testing Data
The models are then applied to the testing data, and their accuracy is computed.
3.5.5. The Time Each Model Took on the Testing Data
Testing the model of test data extracted from the dataset.
3.5.6. Precision
It is defined as the
Precision is useful in determining how accurately the model predicted the real positive outcomes out of all the positive outcomes predicted.
3.5.7. Recall
It is defined by the formula:
Recall helps identify the accurate positive predictions in proportion to the actual positive values in the dataset.
3.5.8. F1-Score
It is defined by the formula:
Since Precision and Recall are not accurate in determining the true picture of the model performance, F1-score is used to determine the combined effect of Precision and Recall by calculating their harmonic mean.
3.5.9. Support
This is simply the number of instances/records fed to a model for training or testing.
3.5.10. Confusion Matrix
Gives a summary of the no. of instances: True Positive, False Positive, False Negative, and True Negative.
3.5.11. Area under the Curve (AUC)
It is a graphical representation of how well a model can distinguish between two classes. The higher the area, the better the performance of the model.
4. Experimental Analysis and Results
The proposed approach aims to diagnose dementia using complex activity data from a publicly available dataset. We train different classifiers with the given data and analyze the outcome to obtain the model that gives the most accurate results. The experimental analysis was performed over the famous web-based IDE notebook—Google Colab, and implementation was done using Python 3.6. The first task before data preprocessing was to split the data into train and test samples. We used a 78 : 22 ratio for train and test, respectively. Of the 616 records available, 480 were used for training and 136 for testing.
4.1. Decision Tree Classifier
Decision tree algorithms are supervised ML classifiers. They operate by splitting the dataset into categories based on a criterion. The data is iteratively split till a homogeneous subset containing records with the same class labels is obtained. Gini Index, Information gain, and Gain ratio are commonly used indices to perform the split [21]. Our DTC uses the default “Gini Index” to perform the classification.
Table 1 shows that the mean accuracy obtained from the ten iterations of 10-fold cross-validation yielded 97% accuracy with a minimal standard deviation of 0.0208, indicating that all iterations gave somewhat similar results. The training time was 0.215 seconds, whereas the prediction was very fast and took only 0.03 seconds.
Table 2 is an overview of the performance of our decision tree classifier model. It can be observed that the precision-recall and F1-score all give a score of 97%. Out of the total participants in the test data, 59 dementia patients and 77 healthy individuals are in our test dataset. Figure 3 is a confusion matrix of the model. It gives a summary of predicted vs. actual outcomes. Our model accurately predicted 57 out of 59 dementia patients and 75 out of 77 healthy individuals via the decision tree classifier. There were only four misclassifications. Figure 4 is the ROC curve for our decision tree classifier. The model effectively differentiated between dementia and healthy individuals with 97% accuracy. It can be seen in the graph that the True Positive rate rapidly increased to 0.97 while the False Positive rate was fairly low, and after reaching 0.97, the graph becomes flat, and the rate becomes almost constant.
4.2. Naive Bayes Classifier
The NB classifier is a probabilistic classifier based on the Bayes theorem. By using conditional and prior probabilities, we can ascertain the probability of a class. These complex calculations are important to determine class probability since prior, and conditional probabilities can be easily obtained from the given dataset. NB Classifier fundamentally operates on the probabilistic principle as defined by where is the posterior probability, is the likelihood, is the class prior probability, and is the prediction probability.
Table 3 shows that the mean accuracy obtained after a 10-fold CV for the NB classifier is 0.831 with a standard deviation of 0.053. The training and prediction time for the NB classifier is much less than the decision tree classifier because the NB classifier is a relatively simple and easy classifier, which is an advantage in processing time but is a significant drawback owing to its Naive nature. The overall prediction accuracy achieved was 0.85.
Table 4 summarizes important findings from the model implementation. The precision value for dementia is 0.74, recall is one, and the F1-score is 0.85%, whereas the precision for healthy individuals is 1.00, recall 0.73, and the F1-score is 0.84. The overall accuracy of test data was 0.85.
Figure 5 is the confusion matrix of the actual vs. predicted outcomes. The model was able to classify all dementia patients correctly but misclassified 21 healthy individuals, thus decreasing the accuracy of the model. Only 56 healthy individuals could be predicted from the test data set. The ROC curve in Figure 6 shows an area under the curve equal to 0.99.
4.3. Support Vector Classifier
SVC is a supervised ML algorithm that can be used for classification or regression challenges. The proposed work uses SVC to classify dementia and healthy individuals by analyzing data of complex daily activities in a smart home environment. Each data point is mapped onto an n-dimensional plane, where each class is then separated using a hyperplane [22]. The objective is to find a hyperplane with maximum distance from the points closest to the line, also known as the support vectors.
Table 5 shows that the mean accuracy obtained after applying 10-fold CV is 0.967, and the standard deviation was nominal, i.e., 0.025. Table 6 shows that the precision, recall, and F1-score range from 0.97-0.99 for dementia and healthy values, indicating that our algorithm gives quite accurate results. Training and prediction time for SVC are less than for DTC and NB.
Figure 7 is a confusion matrix for the SVC. The matrix shows that the algorithm can successfully classify 58 instances of dementia and 75 instances of healthy individuals with only three misclassifications. The ROC curve in Figure 8 shows an area under the curve equal to 1.00, which shows the excellent performance of the algorithm.
4.4. Multilayer Perceptron Classifier
An MLP neural network is a neural network where each neuron imitates the way a human brain works and learns results using mathematical operations. The input layer consists of neurons that receive the data; after processing at each neuron, the data is passed to one or more hidden layer that performs mathematical operations and passes it to the output layer that predicts output [23]. Backpropagation is used whereby the neural network learns through the errors that occur. The error is computed between predicted and actual output, and adjustments are made, allowing the model to learn. Table 7 shows the performance of our MLP classifier for predicting dementia vs. healthy patients from complex activity data. The mean accuracy obtained from a 10-fold CV was 0.971, and the standard deviation was only 0.0298 between the ten iterations. The MLP classifier achieved about 99% prediction accuracy by training the neural network in 11.95 seconds and obtaining the predictions using the test data in 0.011 seconds. Although the training time for MLP is quite a lot because of the complex computations involved, it has been observed that they can achieve high accuracies. Table 8 is the classification report of the MLP classifier and indicates that precision, recall, and F1-score for both classes are between 0.99 and 1, which is close to 100%.
Figure 9 is a confusion matrix for our MLP classifier, and as can be seen, our algorithm successfully classified all the healthy individuals and only misclassified dementia patients. The ROC curve in Figure 10 also indicates a 100% accuracy with the area under the curve equal to 1.
4.5. Deep Neural Networks
A DNN (Deep Neural Network) is a type of Neural Network that has multiple hidden layers that are densely connected. The capability of a DNN to extract features from raw sensor data and give a meaningful output using complex mathematical operations makes DNN the state-of-the-art Artificial Intelligence technique. DNNs have been successfully used in healthcare, where they have exceeded human accuracy by far. Using complex activity data, the proposed work employed DNNs to predict healthy vs. people living with dementia. A DNN model was constructed using 4 dense layers. A ReLu activation function was used in the first three layers, and a sigmoid function was used in the fourth layer that yields the output predicting whether the individual is a healthy or dementia patient. Twenty epochs were used to train our model with a batch size of 16. The training accuracy quickly jumped from 0.57 after the first epoch to 0.78 after the second. The accuracy gradually increased from 0.78 to 0.986 in 20 epochs. Similarly, the prediction accuracy jumped from 0.75 after the first epoch to 0.85 after the second epoch and rose to 0.96 after 20 epochs. As shown in Figure 11, the accuracy for train and test rapidly increased during the first three epochs and then became steady after three epochs. Figure 12 is the confusion matrix of our classification using the DNN model. Only one dementia patient was misclassified; the rest of the healthy individuals and the dementia patients have been correctly classified, indicating excellent accuracy.
Figure 13 illustrates the loss/error of train and test data behavior per epoch. It is observed that the training error was very high after the first epoch, i.e., 0.64, and the testing error was 0.62. For training and test data, the loss steadily decreased per epoch. The decrease slowed after ten epochs when the curve became less steep and flatter. The final loss after 20 epochs for training data was 0.061, and for test data, it was reduced to 0.13. The ROC curve also indicates an AUC of 1, indicating an excellent classification of both classes in Figure 14.
5. Discussion
We tested five different ML classifiers on the publicly available dataset comprising data of 179 individuals, 145 of which were healthy individuals, 32 were suffering from MCI, and two were diagnosed with dementia. Data collected from sensors in a smart home setting were gathered against eight complex tasks. The resulting data were then classified using the DTC, the NB classifier, the SVC, the MLP classifier, and a DNN model. The results obtained are summarized in Table 9.
The DNN and MLP classifier yielded the best accuracy and area under the ROC curve. The NB and SVC returned less accuracy in predicting dementia vs. healthy individuals. While the MLP classifier gives accurate results, it takes the longest training time; therefore, it is not the most efficient in processing and takes longer due to its complex algorithm. DNNs have also yielded excellent accuracy but require a longer processing time than DTC, NB, and SVC. For SVC, although it takes the least processing time, it yields poor accuracy. So it can be concluded that in terms of Accuracy and ROC, MLP and DNN are the best classifiers for predicting dementia using complex activities data, given that training time is not of much relevance.
6. Conclusion
The proposed work attempts to compare and contrast the performance of the different classifiers for the prediction of dementia using ML techniques on complex activity data. We observed that most classifiers successfully classified dementia and healthy from the given data with a slight variation in accuracy. Deep neural networks and multilayer perceptron performed very well in classifying our classes. We conclude that AI techniques are very effective in the early diagnosis and prediction of dementia. Using smart homes, we can conveniently diagnose dementia patients by observing their behavior in their complex daily activities. Multilayer perceptron and deep neural networks have been deemed the best classifiers for this classification task since they can achieve accuracies as high as 99%. One of the earliest signs of dementia appears when individual attempts to perform complex daily activities that involve cognitive brain functions like planning, analyzing, and calculating. Our proposed work can detect these behavioral changes very early in individuals, thus helping medical professionals detect dementia earlier and accurately. The sooner the diagnosis is made, the easier it is to manage the disease.
Data Availability
The [Complex Interwoven Activities] data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that they have no conflicts of interest.