Abstract
The rate of world population aging is increasing. This situation directly affects all countries socially and economically, increasing their compromise and effort to improve the living conditions of this sector of society. In environments with large influxes of elderly people, such as nursing homes, the use of technology has shown promise in improving their quality of life. The use of smart devices allows people to automate everyday tasks and learn from them to predict future actions. Additionally, smartphones capture a wealth of information that allows to adapt to nearby actuators according to people’s preferences and even detects anomalies in their behaviour. Current works are proposing new frameworks to detect these behaviours and act accordingly. However, these works are not focused on managing multidevice environments where sensor and smartphone data are considered to automate environments with elderly people or to learn from them. Also, most of these works require a permanent Internet connection, so the full benefit of smart devices is not completely achieved. In this work, we present an architecture that takes the data from sensors and smartphones in order to adapt the behaviour of the actuators of the environment. In addition, it uses this data to learn from the environment to predict actions or to extrapolate the actions that should be executed according to similar behaviours. The architecture is implemented through a use case based on a nursing home located in a rural area. Thanks to this work, the quality of life of the elderly is improved in a simple, affordable, and transparent way for them.
1. Introduction
The world is experiencing population aging, a trend that is both pronounced and historically unprecedented. Over the past six decades, countries had experienced only a slight increase in the share of people aged 65 years and older, from 8% to 10%. However, in the next four decades, this group is expected to rise to 22% of the total population, a jump from 800 million to 2 billion people [1].
Due to low population density and the migration of young people to richer regions, the elderly often live alone or in nursing homes where the elderly live there temporarily or continuously. These nursing homes are modernizing their facilities to provide better services and care to their residents. These improvements affect the quality of life of the elderly and the costs of nursing homes [2]. During the last few years, they are making use of technologies [3–5], to make life easier for their residents and help healthcare professionals to monitor their health and daily activities.
Over the past few years, both for aging populations and for the population at large, significant research efforts have been devoted to various aspects of health monitoring and active aging activities. One of the main objectives of the scientific community is to monitor and accurately identify the activity patterns of the elderly. Different studies have shown that the activity patterns of the elderly are a valid parameter to predict their quality of life [6]. There are approaches in this area to propose new algorithms, techniques, or systems to improve activity monitoring such as [7], focused on collecting different types of home surveillance technologies for monitoring behaviour of older people, or [8], where an architecture is developed that exploits the benefits of the Internet of Things (IoT) to capture location and other data to detect patterns of behaviour in older people in a nonintrusive way, or [7], where an IoT detection infrastructure for the city is presented that through REST and Linked Open Data application programming interfaces (APIs) collect and present data related to older people.
In this line, there are different reviews in the literature in which the different alternatives regarding the integration of Ambient Assisted Living (AAL) technology in residences are studied [9, 10]. Reviews in which network projects, middleware, sensors, communications, or actuators are proposed, although most of them with a very particular focus and for specific scenarios, mainly focused on ubiquitous sensor systems and telehealth devices.
An alternative to these residences is home care, automating and monitoring the daily activities of the elderly [11]. There are also reviews of studies and projects aimed at the elderly for the home as an alternative to nursing homes [12, 13]. But this alternative is not sufficient in rural and dispersed population environments due to infrastructure costs and the difficulty of arriving at these homes in a minimum time in case of emergency.
However, most of the current solutions provided by the research community and companies offer too closed solutions or address a very specific problem [14, 15]. Moreover, most of these solutions do not allow the integration of devices from different manufacturers or using different communication protocols [16]. Furthermore, collaboration between these types of devices remains a challenge, considering that the heterogeneity of devices and manufacturers makes this collaboration difficult. In most deployment environments, resources are limited as in rural environments or nursing homes. In this sense, the economic cost of implementation and deployment (mainly the cost of devices) of technological systems that improve monitoring is a major barrier. Moreover, due to the poor Internet connections that exist today in certain rural areas, it is necessary that the monitoring system does not depend on a permanent Internet connection to ensure its proper functioning. Therefore, it is necessary to integrate all these technologies under a common system that helps in the care and monitoring of the elderly, saving costs and giving the possibility to any nursing home to create their own smart ecosystem.
This paper presents an architecture that facilitates the integration and collaboration of IoT devices from different manufacturers. Also, this architecture makes use of machine learning (ML) techniques to improve the automation and detection of elderly people’s activities in nursing homes located in rural and sparsely populated regions. In particular, this architecture can be applied to any nursing home that can conduct the monitoring of its patients through the use of the latest technologies in an affordable way and independent of the type of technology used. In addition, the architecture has been validated and promising results have been obtained in detecting people’s behaviours.
The rest of the paper is structured as follows. In Motivations, the main motivations and some related literature for this paper are presented; then, in Architecture Proposal Based on IoT and ML, our proposal for rural environments is detailed, while in Use Case Description a use case is shown for a better understanding of the proposal. Next, in Results and Discussion, the obtained results and some discussions are described, and finally, in Conclusions and Future Works, the conclusions of this work are exposed.
2. Motivations
The high rate of population aging means that there are more and more elderly people in nursing homes, and as a consequence, more nursing homes are created to accommodate them. One of the main activities of these homes is to monitor the activities of the people who live there to detect behavioural routines or possible abnormalities. Current technology allows this monitoring to be carried out in a simple way through intelligent devices such as sensors and actuators [17]. These sensors and actuators are usually integrated through a central node or controller that manages the communication and the information exchange. However, in environments with limited resources such as less populated rural areas, the implementation and deployment of these IoT systems can be difficult to achieve.
The monitoring of elderly people’s activities is possible thanks to the latest technologies we have today. Among these technologies, we can highlight the latest generation of sensors that can detect movement when people are walking, if a door is open or closed, if a tap leaks, if there is smoke in a room, or trigger an alarm in an emergency situation, for example. The use of sensors of this type allows nursing homes to monitor the daily activities of their inhabitants in order to detect behavioural routines or possible anomalies. There is a multitude of sensors of all types that are offered to the market by different suppliers. The choice of some sensors or others will depend on the needs that the residence has and the available budget and it will also depend on the rest of the devices to integrate into the ecosystem. Creating an intelligent ecosystem can be problematic when acquiring sensors or devices. This is due to the lack of compatibility that currently exists between devices from different manufacturers or those using different communication protocols [18]. Because of this, in environments with limited resources, alternative solutions are needed to address this problem of integration and connectivity, to provide nursing homes located in rural areas the ability to create their smart ecosystem in a simple and affordable way. In order to monitor the daily activities of elderly in nursing homes, different technology projects have been launched with the aim of establishing a comprehensive care plan in the community through the adoption of smart health and care in the area of geriatric care in nursing homes [19]. The main idea of the new smart nursing homes is the incorporation of technological devices for the monitoring of determined vital signs or specific actions of the elderly, to tackle specific aspects of health and care in these residences. On the one hand, there are solutions for personal care and integrity, such as bed-exit alarm systems [4] based on multiple sensing such as infrared, ultrasonic, and triaxial accelerometers on the route that patients pass by most often; facial recognition systems for emotion detection with smartphones to improve the quality of life of the elderly [20–22]; a smartwatch-based communication system for nursing homes [23], which improve communication between residents and caregivers, thus reducing staff response time and improving residents’ safety; or an experimental smart diaper [24] as an indicator of saturation for diaper change in people with dementia living in nursing homes. On the other hand, there are rehabilitation assistance projects, such as a new electro-informatics assistive medical system [25] used for the communication with neuromotor disabled patients, which allows bidirectional communication through using an interface with a software application by using different types of sensors including switch-type sensors or eye tracking devices; a system for monitoring and rehabilitation services for elderly [26], based on mobile and wearable technologies ready to be used in residential long-term care facilities to reduce the risk of depression and social isolation; a telecare and telerehabilitation system using computer vision techniques [27]; or a software application for tablet device [3] to support social connections and reducing responsive behaviours of people with dementia while in a care setting, such as nursing homes. The use of touch screen tablets, such as an iPad, may offer the possibility of helping people in rehabilitation with dementia to remain in a care setting.
Also, other initiatives facilitate the work of caregivers and medical staff, such as devices that assist in the early intervention of diseases such as diabetes [5], dementia [28], or other diseases [29]. The authors of this paper are working on different lines to improve the life and health of the elderly; among the many works we can highlight are an extensible environment for monitoring and detecting symptoms of depression [30], different systems for food and beverage monitoring [31–33], or a voice assistant to remind the pharmacological treatment [34]. Although in this work we will only focus on the self-management architecture based on IoT and machine learning for rural areas, these approaches are interesting to be aware of the common diseases and how the caregivers can be supported.
Elderly monitoring provides valuable information to those responsible for nursing homes. Thanks to it, daily routines or possible anomalies in the behaviour of the elderly can be detected. These routines can be related to the time they get up, when they have breakfast, at what time and where they go for a walk, and even with whom they relate to most. Also, these routines can help detect abnormal behaviours such as a person spending too much time in their room, not relating to others, or performing actions outside of their routine on a regular basis [35]. All of this information is valuable to nursing homes and can be used to take action. The processing of this information must be carried out in an intelligent device through machine learning technologies. Today, there are a large number of devices that can perform this type of computing tasks, from small microcontrollers to servers specifically dedicated for such purposes. Most of the existing proposals are focused on capturing the information obtained from the cloud and processing it there [36]. In this sense, smartphones are the smart devices that are evolving the fastest and achieving more computing capacity. Thanks to this, the processing of the information that determines people’s daily routines can be carried out on their own smartphone. In addition, smartphones have different types of sensors that can further enrich the information of the elderly and make their monitoring more effective. However, this processing does not allow us to obtain behaviours and anomalies from a group of people, which is essential in social environments such as nursing homes. Besides, this process should be performed in a central device that integrates the different sensors and actuators of the residence. In addition, in this paper, the authors propose to make use of these techniques in a balanced way between the central controller and the smartphones of the elderly that will allow to make better use of the available resources and avoid possible overloads.
All these proposals are a great step forward, but each of them involves the development and maintenance of a system independently. This situation complicates the development of smart ecosystems for nursing homes, as well as raising costs. For all these reasons, the main efforts should be directed towards the integration of all these proposals into a single system [37].
3. Architecture Proposal Based on IoT and ML
The proposal presented in this work is aimed at taking advantage of the benefits that the IoT offers in combination with machine learning algorithms to monitor people in order to detect pattern behaviours and to predict future actions in nursing homes with no or intermittent Internet connection. This is achieved by developing an architecture that integrates different sensors and actuators in nursing homes to improve the monitoring of elderly people and processing the gathered data by these devices. Furthermore, the correct behaviour of this architecture does not depend on the Internet connectivity as it is designed to work in rural areas with limited or no Internet connections.
Although the proposal is designed for environments with a limited Internet connection, it can be implemented in places where the connection is stable such as nursing homes located in smart cities, where there is a growing awareness of intelligent devices and they are being incorporated into people’s daily lives. The proposed architecture consists of three main parts: inputs, a controller, and outputs (Figure 1). The following sections detail these three aspects of architecture.
3.1. Collecting Data: Inputs
The inputs represent those devices that can provide information about the environment, such as (1) sensors of temperature, humidity, movement or luminosity; also (2) people’s smartphones that can be used to track their locations within the nursing home or frequent interactions with other devices and to provide contextual information about the person employing the different sensors they possess (accelerometer and gyroscope), such as whether they are moving, walking, and standing or any other detection that can be deduced from the smartphone’s internal sensors; and (3) web services that provide additional information about the seniors’ residence, such as virtual sensors to monitor weather conditions, to know the TV guide or upcoming events of interest.
In this sense, smartphones can process each person’s personal information to detect patterns of behaviour through machine learning techniques. This information is related to location, interaction with other devices, data sensed by their sensors, or time invested in a room. The information is obtained through the different sensors that the smartphones have, but in addition, it can be combined with the data offered by the physical sensors placed in the residence to obtain a richer and more accurate prediction. Nowadays, mobile devices are increasingly powerful and capable. This allows increasingly expensive operations to be carried out, computationally speaking, and also at runtime, without the time delay of these processes being increased [38]. That is why the computation of the preferences of the elderly could be done directly on their mobile (ML1) devices and sent to the controller once processed, with the aim of offloading processes to the controller. In this line, many libraries offer this type of computing. TensorFlow (https://www.tensorflow.org) offers the TFLite library for running predictions on thin clients (such as Android) on models previously defined and trained with TensorFlow. Also, from the latest versions of Android (API 27), Google is working on the inclusion of an API for Neural Networks (https://developer.android.com/ndk/guides/neuralnetworks), which allows for definition, training, and prediction on the same device. Regarding this task, there are previous works [39] developed by the authors of this paper based on the use of neural networks to automate actions with IoT devices, where details of how to identify a person’s behaviour and actions can be found. In that work, the detected actions depend on information from the context, such as the time, the day of the week, or the type of IoT device with which the user interact with. In addition, sensors’ values and other inputs mentioned in Figure 1 are added to those input variables. Therefore, the sensors’ values are considered together with the previously mentioned context variables, to determine which conditions cause an elderly to interact in one way or another with an actuator. For example, the ambient temperature captured by a temperature sensor can be decisive in how the elderly person acts with an air conditioner. The used neural network is a Multilayer Perceptron Neural Network (Figure 2), where there is an input layer of the size of the context variables that is used as an input, two intermediate layers of 45 neurons each, and a bias neuron in each layer, to reduce the bias of data, and an output layer of the number of actions that can be performed with the actuators. In this way, each neuron in the input layer has an associated context variable, taking its value as input to make predictions. In addition, each neuron in the output layer provides an associated action of a specific device, indicating the probability of performing that action with that device, based on the inputs specified in the input layer. The number of intermediate layers, as well as their number of neurons, is determined by adjusting the loss function and evaluating the percentage of under- and overfitting during training. This machine learning model provides the ability to discover people’s pattern behaviour during the monitoring with the aim to predict actions over the actuators, but it does not provide the ability to detect behaviours that change over time (actions that stop being executed in favour of new actions). To detect these changes in elderly behaviour, it is necessary to make the neural network aware of time, to enable it to analyze not only the context variables that condition behaviour with an actuator but also the change in behaviour with the actuators (for the same or other context variable values) over time. In other words, it analyzes a time series with the changes in the way the actuators are used. For this, recurrent neural networks (RNNs) are used, which allow the neurons to be provided with memory. Specifically, Long Short-Term Memories (LSTMs) are usually used, a type of recurrent neural network that solves the short-term memory problem of classic RNNs, so long data sequences can be analyzed. An alternative to all this is to use the same neuronal network as now, creating a mechanism, like a system of weights, to evaluate positively the records of newer interactions against older ones. However, this solution is less appropriate and may give worse results than analyzing the time series of interactions with an actuator, using an LSTM. At this point, both the information processed by the smartphones and that produced by the different physical sensors are sent as output from ML1 device and as inputs to the controller, for identifying actions and social behaviours from a set of people and smart devices.
3.2. Processing Data: Controller
The controller, which can be from a simple Raspberry Pi to a dedicated server, processes the information coming from the different sensors and smartphones (inputs) to interpret it. The controller consists of three main components. The first component, integration, is responsible for ensuring proper integration among the different devices (sensors and actuators). To do this, one of the most popular assistants for the automation of smart environments is used: Home Assistant (HASS) (https://www.home-assistant.io). HASS is a system designed for the configuration and automation of many devices from different brands. It is developed in Python, which is a free and open-source software with a large community behind 216 and large support of brands (https://www.home-assistant.io/integrations) such as Amazon Alexa, Google Assistant, Ikea, Philips, Sonos, or Xiaomi. HASS allows the integration of devices (sensors, actuators, mobile phones, etc.) through a great number of different protocols like Bluetooth, BLE, WiFi, ZigBee, or MQTT, to be controlled or to make automation in a concrete environment, like a smart home, an office, or as in the case of this work, a nursing home. Also, this system allows triggering specific actions when an event occurs, such as turning on a lamp when motion is detected or sending a notification when an alarm is triggered. HASS also allows nearby devices to be detected through different mechanisms, such as those connected to the same network (NMAP), or within Bluetooth or BLE range. This is particularly interesting for finding out who is nearby, where they are by tracking devices such as smartphones, smartbands or smartwatches. The wide range of possibilities that HASS offers is an interesting option to perform the automation of the nursing home, allowing the responsible of the nursing home to purchase a wide range of smart devices. Once the devices are integrated, the interoperability component is responsible for ensuring the collaboration among the devices. To achieve this collaboration among smart devices, we must link elder’s needs to the services that the devices offer. We define as needs the preferences that people have and that need to be covered in order to perform a certain task, such as for example, selecting a certain luminosity level. These needs can be manually specified by the person or be detected by another device, for example, selecting a specific luminosity level when a person is entering in a room. These needs will be covered by the services that the smart devices possess within the nursing home. To this end, ontologies are used to define the characteristics of the sensors, actuators, and smartphones, with the aim of creating logical relationships between them and facilitating their collaboration.
There are numerous ontologies that currently exist for healthcare in smart environments and that could be reused for the proposed architecture. In this sense, the authors of this paper developed in [40] a study of the most relevant ontologies mainly related with healthcare. Some of the ontologies included in the study are HealthIoT, HOTMES, or FIESTA-IoT. These ontologies can represent medical devices, provide information to perform personalized services to certain patients, represent smart environments dependent on certain conditions or parameters, or monitor different people’s activities. However, none of them gathers all the information that is necessary to represent the correct communication and collaboration between inputs and outputs. A complete description of these ontologies is detailed in [40] as well as their main properties’ pros and cons. The development of ontologies is usually done to solve a specific problem. It is for this reason that although the ontologies considered have classes and types of data that could solve our problem, they do not achieve it completely and do not allow us to represent the required information in the way we intend to do it. This information is related to the identification of inputs and outputs, associated services such as turning on a light or triggering an alarm, parameters that indicate the operation of the services, such as the intensity of the lighting, specific preferences of people, and values of those preferences to suit the devices. For this reason, we propose an ontology to improve the interoperability among smart devices in nursing homes. The proposed ontology is defined as Ontology for Smart Monitoring (Ont4SM) (Figure 3). The aim of this ontology is to represent the information of smart devices belonging to different application domains and manufacturers as well as the information about people to achieve a semantic relationship.
Given the fact that both people and devices have services and needs, we use a single class (Entity) to represent people and the different sensors and actuators that we treat them equally. In addition, this class allows us to represent the entity’s personal information so that it can be correctly identified and for the moment is not necessary to make a separation. Besides, this ontology is able to represent the services (Service) the entities have, as well as the needs (Need) that will be solved by the available services. We must bear in mind that the impact of services on needs will not always be the same and will depend on different parameters. For this reason, a class (Parameter) has been introduced that receives the necessary parameters to make the invocation and to adapt the service according to the characteristics of the need that it is going to solve. The following example shows how the ontology works and how the information represented in JSON format is treated. Let us suppose that a person, Bob, from the nursing home is in the living room. Through the information collected by his smartphone and contextual information, his needs (Need) are detected and interpreted by the controller and the nearby devices can adapt their services (Service) to this person as well as additional information to identify him in the nursing home (Figure 4). In this case, Bob would like a medium illumination (6/10). To solve this, the intelligent light bulb in the living room, a SmartBulb, receives the data from the person through the controller to adapt the lighting level to the detected need. In this case, the service that allows changing the lighting is invoked and receives the “illumination” parameter that acts directly on the detected need when it is solved (Figure 5). Also, the SmartBulb has an associated need to optimize energy consumption. If there is a nearby entity that can perform this optimization, for example, lifting a blind for natural light entering so that the bulb can be turned off temporarily, the process would be repeated so that this need would be resolved by the service of the nearby smart blind.
The processing of the ontology goes through several phases from the entities which are detected in the nursing home, until the information is processed to improve the elderly monitoring. This is a brief description of the ontology, so all the details about it and its processing is available in [40]. The last component of the controller, ML2Anomalies, is responsible for processing the information collected to detect possible anomalies in people’s behaviour. In addition, this component allows us to discover social behaviour that can be identified within the nursing home, such us the frequency with the people interact between them, what are the most interesting topics they talk about or if they develop certain task together, such as play a game or go for a walk. This can be done by using machine learning techniques. Thanks to these algorithms it is possible to process the information gathered from the inputs and to make a decision that allows the adaptation of the actuators or to predict future actions in the environment. To do this, unsupervised algorithms are used for anomaly detections. Anomaly detection, in data mining, is the process of detecting, for a large dataset, observations from this dataset that differ significantly from the values taken by most other observations. Therefore, it is used in tasks such as fraud detection [41]. Due to the unsupervised algorithms work with unlabeled data, the previous data that was used as input (sensors) and output (actuators) is used as input for the anomaly detection algorithm. This allows certain areas to be identified where the elderly is interacting normally with the actuators, according to the input data from the sensors, for example. When unusually operating with the actuators according to the inputs, this interaction produces data that is outside any of the zones where most of the data is, assuming that the elderly are performing an action that differs from their usual behaviour. Also, it is important to indicate that, although the information of which elderly performs a certain interaction with the sensors/actuators is saved and it is considered as part of the model entries, the information will always be processed globally for the whole residence, so the models generated will be at the residence level (one model to automate actions and another to detect anomalies) and not models for each elderly. In this way, it will be possible to extrapolate the models from a residence to a similar residence, as will be seen in Results and Discussion.
3.3. Taking Actions: Outputs
Finally, the outputs represent the actuators that are in charge of triggering different actions depending on the information that has been previously processed. These actions will vary depending on the type of actuator, such as setting a certain illumination, regulating the temperature of a room, tuning to a specific channel on the television, and turning on/off a device connected to the electrical current. Additionally, push notifications, or any other type of notifications, could be sent to caregivers in case of emergency. One of the main advantages that this architecture has is that it does not require an Internet connection. Once the architecture is implemented, the sensors and actuators are connected through different communication protocols to the controller, which is in charge of performing all the information processing locally. The integration of these three parts of the architecture, inputs, a controller, and outputs, contributes to improving the elderly monitoring in a transparent way for them mainly in environments with limited or no Internet connection such as rural areas or small villages, isolated from the big cities.
4. Use Case Description
The proposed use case is based on a nursing home located in a rural area with limited resources. In this residence, it has been proposed to monitor the daily activity of the elderly living in the second floor to check what activities they do or when and even to detect strange behaviours. To do this, the nursing home’s caregivers want to use the latest technologies that allow data to be collected and processed in the most efficient and easy way. Due to limited resources, such as Internet connectivity or a minimal budget, it required a solution that they can perform simply and affordably.
For the proposed use case, it has been decided to use three different communication protocols (WiFi, ZigBee, and Bluetooth) to take advantage of the benefits of each of them and to cover a wider range of different devices’ specification. In addition, the use of different technologies serves as a complement to evaluate the integration, collaboration, and interoperability among different sensors and actuators. The selected sensors and actuators are shown in Table 1, as well as the different smartphones that the elderly living in the residence have.
In Figure 6, the floor plan of the second floor of the nursing home on which the current case study is conducted is shown. In this floor, there are four rooms for people, a living room, and the central corridor automated with sensors (green) and actuators (red). Each room is provided with a sensor to control when the windows are opened and closed, as well as a bulb to change the luminosity or color, and a socket to turn on/off any electronic device connected to it, for example, a TV or a radio. Besides, sensors to control the opening of windows and smoke detection, temperature, and movement sensors have also been installed in each living room. In addition, the living room has sockets that allow elderly to activate or deactivate any device connected to it, a smart button that allows elderly to interact with the light bulbs or other smart devices, a bulb that can change its intensity or color apart from being turned on/off, and a thermostat that allows to regulate the temperature of the room. Different devices have also been installed in the corridor to monitor the activities of the elderly. In the corridor, there are a movement sensor, a smoke detector, and a general alarm that allows for notification of possible incidents. On this second floor, four elderly people (yellow) are currently living, who move freely between the rooms and the living room to perform personal, social, or leisure tasks. Each person has its personal smartphone that collects information about their position, who is near them and even the most used applications, to recognize behaviour patterns in their daily routines. These smartphones are constantly gathering data from the different sensors they have with the aim to detect behaviour pattern about their owners by using the machine learning techniques previously explained. Also, in this second floor, a controller is located, a Raspberry Pi (blue), that receives and sends information from sensors, actuators, and elderly smartphones via WiFi, Bluetooth, and ZigBee, depending on the type of device. All these devices have been integrated through HASS to perform the appropriate automations as well as to adapt their behaviour to the environment. The information collected in this environment is processed through unsupervised algorithms previously detailed in the controller to detect anomalies in elderly behaviours.
Thanks to the structure of the proposed architecture, as well as the technologies it uses for the integration of devices and information processing, the data collected in this use case are potentially usable for the detection of behaviour patterns and anomalies in residences with similar characteristics (number of people, arrangement of sensors and actuators, size of the building, location, etc.).
5. Results and Discussion
The need for mechanisms to facilitate the monitoring of older people is increasingly real. The increase in the average age of the population poses new challenges to societies and healthcare systems. Nonetheless, the emergence of the area of IoT research, wireless communications, and mobile computing is raising hopes for automated assisted environments. These environments combine the advances of sensor networks with those of run-time monitoring systems, to create intelligent nursing homes capable of monitoring the elderly. However, although various AAL systems have been proposed in the last years, the goal of realizing an effective support system for elderly is still far from reach. That is why, in this paper, we have presented a project aiming to reengineer a set of everyday life objects, equipping the nursing home with different types of sensors and actuators, thus monitoring the condition of older people in their nursing homes and providing security while preserving the autonomy and independence of the persons, in an environment with limited resources such as rural areas. This solution is based on an architecture capable of managing different types of devices, such as sensors, actuators, and smartphones with an algorithm to detect pattern and anomalies in elderly behaviour.
The developed algorithm to detect pattern behaviours has shown promising results. The following results are an extract of the main work developed by the authors of this work [39] where the neural network developed and evaluated in different scenarios for the detection of behavioural patterns in different scenarios of people is presented.
In this case, these results are related to the (ML1: Routines) component of the proposed architecture. The results shown correspond to the behaviour of an elder in the nursing home who performs different activities from Monday to Friday, on Saturday he/she goes for a walk, and on Sunday he/she goes to his/her children’s house. The validation lasted three weeks. The two first weeks were used for training the model with information about the different actions that the users performed with the present devices for this scenario. And the last week was used for the testing phase in order to get prediction and evaluate them. Some predictions were also made during the two weeks of training, to check how much data were needed for the system to start learning new behaviours and how the predictions improved with new records. However, this depended a lot on how frequent the behaviour is. First, regarding the responsiveness, it is interesting to indicate that on average 11.05 seconds are required to refit the neural network with the inputs of each user every day and 1.35 milliseconds on average are required to make a prediction. As can be seen, the responsiveness is very good for almost any IoT application. Other similar proposals [42] are getting prediction times of 896.1 milliseconds (when the number of devices is small) and 1.21 seconds (when the number of devices is higher). Training times are not included in these proposals. Second, regarding the accuracy, the results obtained for the scenario can be seen in Table 2. In this table, the number of predictions, the correct and the incorrect ones, is detailed.
By analyzing the joint results for that scenario, we have to notice that 33 predictions were not correct. According to this, we detected that 29 were produced by trying to predict actions that do not have a specific behaviour pattern, since there were records of different actions for the same device in the same context and for the same user. For example, imagine a person who sets a different television channel every day when arriving at the nursing home. If the system tries to predict which channel the person will choose, the probability of getting a correct prediction is low. For the other 4 incorrect predictions, the model was not able to learn the action. Therefore, it can be said that only 4 out of the 245 predictions were not correct.
In addition to analyzing the results offered by the model in terms of accuracy, it is important to analyze other functions measuring the quality of the developed model. For this purpose, the loss function has been used. The loss function for the model discussed in this section is shown in Figure 7. It can be seen how the function tends to the value zero (target value in this function), which allows us to know if the value of the learning rate or the value of the batch size is appropriate. To do this, the evolution of the loss function through the epochs is compared with the ideal evolution of the loss function [43]. Thanks to this analysis, it can be determined that the learning rate is good, although it could still be reduced a little. The noise that can be observed in the loss could be reduced by decreasing the batch size a little but is not significant.
In addition, we also evaluated the probability of success for every prediction in order to identify a threshold that allows us to know when a predicted action should be performed or not. In Table 3 can be appreciated the average, the maximum, and the minimum value and standard deviation for the probability of the correct and failed actions.
Finally, to determine the threshold, a univariate distribution has been created with the value of the probability of success of the correct (Figure 8(a)) and failed (Figure 8(b)) predictions. Thanks to these data and reducing the number of false positives even if the accuracy is reduced, it can be determined that the best threshold is over 0.4 and 0.5. Besides, in Table 4, it can be observed that a threshold of 0.44 is better, because a lower threshold includes more false positives.
(a)
(b)
With the obtained results, it can be stated that the algorithm has been able to correctly predict 86% of the interactions, failing only in 1% of the predictions or when the performed actions do not follow a pattern.
Regarding the machine learning algorithm development, another way to implement them to detect pattern behaviours and anomalies can be by developing a single model using the neural network for both functions. Each time an elder performs an action manually, he/she employs the neural network to check the probability of execution of that action, according to the model trained with previous records and the context (inputs). By assigning a minimum probability threshold, if the action being executed by the elder does not exceed the threshold in the prediction model, it is considered an anomaly. This is because the elder is performing an action that has very little probability of being performed according to his/her usual behaviour. Also, it is necessary to verify that there are actions that the elder could perform at that moment that exceed the threshold. If not, the reason why the performed action did not exceed the probability threshold could be that the model is not sufficiently trained for that situation and not because of an anomaly. However, this method is more susceptible to errors, since there are already dedicated algorithms to do this. It could be useful in case we wanted to reduce to the maximum the amount of computation and complexity of the controller, using a machine learning model for everything. This may be a way to reduce the energy footprint on the controller.
Although the selected use case is based on a nursing home with specific characteristics, the proposed architecture can be implemented in another nursing home. This is especially beneficial. This work is part of a project on gerontechnology, in which one of the tasks is to define an architecture that can be used in all residences in the regions of Extremadura (Spain) and Alentejo (Portugal). This architecture and implementation allows to reduce the costs of the residences and be more efficient in caring for their residents. The benefits of the proposed architecture should be considered when implementing this as well as the requirements of the destination nursing home. These benefits include the wide variety of sensors and actuators that can be integrated, regardless of the manufacturer and technology used. More and more devices are compatible with systems like HASS, which allow users to create their own intelligent environment without having to rely on third-party applications or hardware to operate and communicate with each other. This allows the application to be used in rural environments, even if it has no Internet connection or is limited, because all information processing is done locally at the controller. To implement the architecture in any other scenario, it would be necessary to study the available resources and the available budget. Furthermore, it would be necessary to study the requirements, mainly the type of sensors or actuators to be used, depending on the type of monitoring and the performance to be achieved. For example, monitoring the movement of people or if the lights remain on for a long time. In addition, it must be decided what type of controller will be suitable for managing the environment, from a microcontroller to a dedicated server with more power and resources. This decision will vary depending on the size of the environment and the number of devices being managed. Once the sensors, actuators, and the controller have been identified, they must be integrated into the HASS installed in the controller so that the information from the sensors, actuators, and smartphones can be processed by the machine learning algorithm to detect behaviour patterns or predict certain actions. If the new scenario in which the architecture is to be implemented has the same number and type of sensors and actuators as any other previously known scenario, the machine learning models can be exported from the known scenario to the new one.
Otherwise, if some of the inputs or outputs were different, it would be necessary to modify the inputs and outputs of the models, having to generate new models and train them from scratch. From this moment, it is possible to start with data collection and environment automation to perform the necessary tasks.
The system developed promises to be of great help to older people, especially in environments with reduced or no Internet connection such as rural areas. As mentioned above, the proposed architecture can also be applied in residences located in settings with good Internet connections. These environments can even enhance the characteristics of the proposal, allowing tasks to be performed remotely, such as controller management, using cloud services for more complex data processing or monitoring the status of people in the residence from other locations. In the case of remote management, HASS makes it easy to do so through the Nabu Casa (https://www.nabucasa.com) service, which allows access to the control panel from anywhere through the Internet browser. In this way, the HASS can be managed and the information of the sensors installed in the nursing home can be evaluated. For data processing in remote servers, the great versatility that the Raspberry Pi offers would allow one to easily cloudify these services, using Amazon Web Services (AWS) (https://aws.amazon.com) or Microsoft Azure (https://azure.microsoft.com), to send the data and to be able to recover the results once processed. This feature could be added in the future to expand the possibilities of the architecture and support new features. Although the potential benefits of the proposed architecture have already been shown, further work is needed to overcome current limitations. Among these limitations is the amount of data required for the machine learning model to be able to act correctly. In this sense, machine learning models are more accurate when they have a large amount of data, so when used in small environments with few interactions with actuators, data collection can take a long time until the model begins to generate more reliable predictions. For the studied scenario, Figure 9 shows a line chart with the evolution of the accuracy of predictions with respect to the amount of data that has been collected. This chart shows the evolution of the accuracy in the 3 weeks that the validation lasted. However, only data for 15 days are shown, because the user did not use the system every day of these 3 weeks. When there were no interactions, the model was not retrained. These days are not represented in the chart in order to improve its readability. In this chart, it can be seen how, as the size of the datasets grows, the gap of the accuracy between the training and testing datasets (the overfitting) is reduced. It is important to mention that the training is done with data from real users. Then, the accuracy evolution depends on how they use the system. If a user always uses the system with a strongly determined behaviour, the amount of data needed to obtain good predictions will be low. If not, more data are needed and the algorithm will not be able to offer high accuracy predictions in all cases. During the training period, the model is trained with the actions performed in the different devices. Therefore, if the user has a random behaviour with any of them, the accuracy goes down, even if the probability of success of the model for the rest of the devices is high. To finish analyzing this figure, it is important to note that the accuracy values for the last measurements are not identical to those in Table 2. This is because the accuracy shown in Table 2 is obtained based on the number of actions that is automated by the user with the last trained model and the number of these actions that the user accepts or corrects immediately. The accuracy shown in this graph is the one obtained at the moment of training the model and testing it. Figure 10 shows, for the same period, the time needed for training (using CPU) and the space required to store the dataset. The training time is divided into two phases that are part of the training phase: data cleansing and fit network. In this way, the amount of training time dedicated to each of these tasks is shown. From the chart, it is possible to see how adding more data to the dataset does not considerably affect the time needed to train the model. This is because the amount of data that is collected in this timeframe is small, as can be seen in the size dataset (on the order of kilobytes). The number of inputs that the neural network model has for this scenario is small. If the number of inputs in the model grows, the increment of the space occupied by the dataset with respect to the timeframe will be greater.
Another current limitation is that the information collected from smartphones is limited to space and time variables. To obtain other types of information, it would be necessary to install a mobile application that provides more detailed information.
In addition to these limitations, a procedure to evaluate the architecture has been performed using the ATAM (Architecture Tradeoff Analysis Method) methodology [44]. ATAM allows the evaluation of software architectures through nine steps grouped into four phases (presentation, research and analysis, testing, and reporting) to mitigate risks related to quality, performance, availability, security, and modifiability. One of the benefits of ATAM is that it can be used during different parts of the architecture’s life cycle, either at the beginning when the architecture is being designed, throughout its life cycle, when the architecture is defined but barely developed, and even for a fully defined architecture. In this sense, the evaluation of the architecture proposed in this paper is made in a preliminary way to know the most important risks that it may contain. (i)In terms of the quality of the architecture, the proposal advocates offering nursing homes the latest technologies for the domotization of smart environments. Among these technologies, the use of machine learning algorithms (TensorFlow) for data processing, or Home Assistant for the integration of intelligent devices, stand out. Also, these technologies have wide support from the scientific community, which allows the quick identification and solution of problems and a more than acceptable capacity of expansion. The risks associated with the quality of the architecture reside in the degradation of the physical devices over time and with the conditions of the environment, where interference with other devices or the physical layout of the walls of the residence could affect the architecture behaviour(ii)The performance of the architecture depends on the type of devices selected for monitoring the nursing home. The architecture is designed to accept any type of device and taking into account that this implies a cost for the nursing home’s managers. Moreover, since the controller is the main device, it can be more or less powerful depending on which ones are chosen. In the example proposed, we have chosen to use a low consumption microcomputer (Raspberry Pi) whose characteristics are sufficient for a small-medium size environment. However, the capacity of this device can be reduced depending on the number of devices that are integrated and the size of the nursing home. This is why this device can be replaced by higher performance elements such as a dedicated server to carry out all the required processing(iii)In an intelligent environment, the availability is a key property. However, this feature often depends on the network infrastructure. In this case, the nursing home does not have to offer an Internet connection, but it must have at least one gateway (a router) enabling a local network with intelligent devices. Furthermore, although only one controller has been proposed to be installed, this could be replicated in another area of the nursing home to make the architecture fault-tolerant in case errors are triggered by a controller. For environments where there is an Internet connection and where the characteristics of the architecture can be extended, again there must be a backup system that allows the output to the Internet in case of an error(iv)Also, the security supposes a fundamental aspect in software architecture. On the one hand, the architecture contemplates that the devices can be connected through secure P2P protocols against the controller, using WiFi, Bluetooth, or ZigBee technologies. This means that once configured, the devices only communicate with the controller to send and receive data. On the other hand, the privacy of people’s data is a crucial aspect. This is since sensitive personal data is handled. Therefore, the authors of this paper are firmly committed to data privacy and have been working for some time to guarantee this privacy. In this sense, the authors of this paper are working on a framework that allows people to decide what information to share and with whom [45]. This allows residents to decide what information will be shared from their mobile devices with the rest of the devices and the controller. Therefore, data privacy is guaranteed and only the data that people want is shared. Another important point is that taking into account that no Internet connection is required and that the processing is done locally, the data does not travel to remote servers, thus reducing the chances of eavesdropping by third parties(v)Finally, the characteristics of the architecture allow it to be easily adapted or modified, improving the system scalability. Both the integration of new devices and the configuration of existing ones are done intuitively from the controller by using the GUI provided by Home Assistant. However, it must be taken into account that the evolution of smart devices advances at great speed and that the communication protocols are constantly updated. This means that the controller must have its libraries updated to ensure the greatest possible compatibility with new devices. Also, Home Assistant is such a versatile software that any type of script can be incorporated to modify the behaviour of the controller or to add new functionalities easily. As for data processing, the machine learning algorithms used to allow them to be retrained with new data to detect new patterns or anomalies in patients easily. The previous paragraphs largely summarize the most important risks detected through ATAM. Although this is a preliminary report, the authors of this work plan to continue with this methodology during the architecture’s life cycle to detect new risks that may arise to offer a solution with the highest possible quality
6. Conclusions and Future Works
This paper deals with the problem of automating environments to make everyday tasks easier for the elderly, through the use of smart IoT devices and machine learning techniques. To this end, an architecture has been proposed that, through a controller, allows data to be collected from multiple types of sensors and smartphones, to modify the behaviour of the available actuators under people’s preferences. To favour multidevice environments and give users the freedom to use different types of sensors and actuators, the architecture is capable of working with WiFi, ZigBee, and Bluetooth (among others) communication protocols. Also, a neural network model has been developed that, from the data collected, allows the controller to be able to learn about the habits and routines of older people, predict future behaviour, and detect anomalies. This information analysis is also valuable for informing family members or health experts of the habits or routines that older people normally follow, reporting abnormalities or notifying in emergencies.
To validate the proposed architecture, the implementation has been done in a daily environment where many elderly people live: a nursing home. Thanks to the conducted automation, it is possible to detect when one or several people enter a room and turn on the TV, to turn off the lights when they leave, or to turn on the heating at the time they normally go to the living room. These interactions are also valuable in improving the model more and more. When data from a longer period becomes available, the use of the mentioned LSTM networks will be evaluated if it is necessary to allow the system to learn to identify changes in the way elderly interacts with the actuators. As future work, the implementation for the detection of anomalies (ML: Anomalies) by means of unsupervised algorithms will be performed. Besides, the conducted machine learning model will be extrapolated to an environment with similar characteristics to evaluate its operation and to be able to adjust those parameters that are necessary.
Thanks to this proposal, better monitoring of elderly people in nursing homes is achieved through the use of all kinds of intelligent devices, thus improving their quality of life and the effectiveness of their caregivers. Also, the flexibility and scalability that the presented architecture offers allow nursing homes to implement the system with a wide range of devices and without the need for an Internet connection.
Data Availability
The data used were obtained from tests conducted with real users. They are available upon request.
Conflicts of Interest
The authors declare that there is no conflict of interest regarding the publication of this paper.
Acknowledgments
This work was supported by 4IE+ project (0499_4IE_PLUS_4_E) funded by the Interreg V-A España-Portugal (POCTEP) 2014-2020 program, by the Spanish Ministry of Science, Innovation and Universities (RTI2018-094591-B-I00 project and FPU17/02251 and FPU19/03965 grants), by the Department of Economy and Infrastructure of the Government of Extremadura (GR18112, IB18030), and by the European Regional Development Fund.