Abstract
Gathering physiological data when they are performing experiments requires a great effort from researchers. Very often, a considerable time is required to prepare the signal acquisition equipment, conduct the experiments, and properly label the data of each participant. Nevertheless this data is valuable for the analysis of personal characteristics, such as behavior, health conditions, and preferences. With the aim of assisting researchers with such tedious tasks, we have developed the DAFIESKU system. This system serves to acquire several types of physiological data. DAFIESKU facilitates the creation of new datasets with physiological data by means of mobile and wearable devices. The usability of the system was evaluated in two case studies in a two-step iterative process. Before conducting the second case study, the whole system was improved using the feedback obtained from the first case study. The results achieved show that usability was enhanced in the second version of DAFIESKU.
1. Introduction
People interact with computing systems in a way that is continuously changing: from batch communication to interactive systems and from interacting at certain times and places with desktop computers to continuous ubiquitous mobile interaction with wearables. New paradigms of human-computer interaction (HCI) have arisen, such as augmented reality, ubiquitous computing, and affective computing, and new applications have been discovered, such as eLearning, eHealth, and eGovernment [1–4].
Initially, specific equipment was used to enable these new applications, but nowadays almost any commercial equipment may serve. The use of mobile smart-phones is common, alone or in combination with other components such as smart-watches, glasses, microphones, and clothes. Most of them have integrated sensors, for example, accelerometers, gyroscopes, temperature sensors, pressure sensors, acoustic sensors, and thermography. With this equipment it is easy to record physiological data from users. These data, if adequately stored and preserved, may be used to train machine learning systems to analyze health conditions, user behavior, characteristics, and other similar topics.
There is a huge number of public online repositories storing datasets [5, 6]. Nevertheless, in some cases researchers need to create new datasets to work with, as the task of acquiring the datasets is an essential step in the research process [7].
The way data are obtained is an important issue directly relating to the validity of the data under consideration. Approaches to recording data are based either in controlled environments such as research laboratories or in uncontrolled environments such as in real life [8–10]. The first approach has the advantage of minimizing noise and recording mainly appropriate data, but its principal disadvantage is that naturalness is lost. That is, subjects may feel uncomfortable in a fairly unfamiliar place for them, probably while being observed by several people, that is, researchers. The latter approach has its advantages and disadvantages contrary to the former: subjects are in their real life contexts and therefore data is generated by more natural reactions, although environmental factors may affect the quality of the recordings. However the popularization of smart-phones and advances with wearable devices have made this option more appealing in the last decade.
In this paper, the authors review which design issues are to be taken into account and present a system named DAFIESKU aimed at acquiring physiological data in a mobile uncontrolled environment. Its main objective is to allow the design of experiments to be applied in real conditions and then to record natural data in real environments instead of recording them in laboratories. This system allows researchers to conduct experiments remotely with the collaboration of participants. The initial versions of the DAFIESKU system and the case studies employed to test them are also detailed here.
This paper is an extension of a paper presented at the UCAmI 2016 conference [11, 12]. The system proposed, aimed at remotely recording physiological data of participants without the help of research assistants, has been developed by the usual iterative development cycle proposed on User-Centered Design [13]. Reference [12] presents, schematically, the first cycle of the iterative development. This new paper extends the information given in the previous one. Moreover, this one presents the second iteration which expands the original framework, focusing on improving the guidance necessary for the user to carry on the experiments with DAFIESKU. New screens and arrangement have been added in the mobile application of DAFIESKU following the recommendations obtained in the first case study involving five participants. A new case study has been performed to test the changes and its appropriateness to meet the goals of the new approach. The results of the aforementioned case study, which involved 17 new participants, are also shown in this paper. New sections and subsections have been included and several figures and tables have been added in order to clarify explanations and to show the results achieved. Two tables outline the lessons learned during both experiments.
2. Related Work
Evaluation and testing of research questions and hypothesis are a recurring subject for human-computer interaction researchers. For this reason, it is not surprising to find an abundance of frameworks, toolkits, and platforms devoted to remote evaluation in the scientific literature [14, 15]. Most often these works are related to the web domain and the usability or accessibility of web interfaces. For instance, RemoTest [16] proposes a platform designed for the remote evaluation of usability and the study of interaction with web applications. RemoTest enables experiments to be set up and data to be collected from participants’ web browsers using plug-ins.
Over the last decade, the high popularity of smart-phones and their applications, commonly called apps, have presented an opportunity to carry out remote evaluations in mobile environments. For example, Funf [17] is an extensible framework based in Android which is able to collect and upload data from different sensors embedded in smart-phones with the aim of studying social interactions. There is a version called Funf in a Box [18] that automatically creates a phone-sensing app with the guidance of a user. Aware [19] is a framework devoted to the instrumentation of the smart-phone to understand human behavior better. The framework is oriented to context-awareness and it provides mechanisms to interact with the user while acquiring data from the sensors.
In recent years, the use of physiological signals has emerged in the computer science area due to the vast amount of wireless wearable devices that provide such signals. Hence, similar systems for the recording and storing of physiological signals can be found in the literature. Biosignal Ignitor toolkit [20] introduces a set of tools for the acquisition of different physiological signals. The toolkit provides a desktop application to visualize and record the signals and also includes its own sensor platform called BITalino [21]. Focused in the medical domain, Physiodroid [22] presents a framework for the creation of medical applications using sensors including wearable devices. This framework is oriented to developers who want to create their own application for the medical or context-aware domain rather than researchers who want to conduct experiments. These works are focused on the remote acquisition of physiological signals without taking into consideration the whole experimentation process.
There are other paradigms such as Participatory Sensing which “will task deployed mobile devices to form interactive, participatory sensor networks that enable public and professional users to gather, analyze and share local knowledge” [23] or Mobile Crowd Sensing which is “a new sensing paradigm that empowers ordinary citizens to contribute data sensed or generated from their mobile devices, aggregates and fuses the data in the cloud for crowd intelligence extraction and people-centric service delivery” [24]. These paradigms are closely related to this proposal and could be easily merged with it.
As an example of these paradigms, [25] proposes a system where crowdsourced physiological data and subjective emotions could be used for cross-validation of the emotions that people feel when visiting different urban spaces. However this mobile app is being considered for compiling subjective emotions not for collecting physiological data. This is precisely where DAFIESKU could be of help.
Another crowd-sensing platform is VITA [26], a mobile cyber-physical system for crowd-sensing applications. Similarly to DAFIESKU, VITA software architecture is divided into two parts, one for the mobile part and the other for the cloud. VITA allows human resources to be allocated and supports user participation in various mobile crowd-sensing applications through interaction with other participants. On the other hand, users with a participant role in DAFIESKU interact only with the mobile application following the tasks designed by the researcher.
The main objective of the DAFIESKU system is to provide resources and tools to researchers in order to facilitate the deployment of an experiment from its design to the analysis of the results. Experiments will be made in real or natural environments (familiar to participants) by using wearable technology and ubiquitous computing. The data obtained is expected to be more naturalistic than that obtained in controlled places, such as research laboratories (unfamiliar to participants).
Therefore, DAFIESKU can be used in research studies to acquire physiological data, including studies in medical settings in order to monitor personal health conditions and in sports settings or even when studying at home in order to measure effort and enhance efficiency. It can also be used for affective computing when detecting emotions [27].
3. Design Issues
There are different types of experiments involving physiological data in computer science research. DAFIESKU aims to collect physiological data with activity labels from the participants of the experiments. Furthermore, the system must work autonomously without the presence of research assistants when the experiments are conducted. Subsequently, the system was designed to interact directly with the experiment participants. In this section design issues on the development of the system (see Figure 1) are explained in deeper detail.
3.1. Involved Roles
When developing DAFIESKU, two main user roles have been identified. The first role is the researcher or the designer of the experiments, while the second one is the experimental participant.
The role of the researcher is to think up, design, and supervise experiments in which real users are going to be involved. They usually work in a team, but they may also work alone. The researcher comes up with research questions and the way to measure responses to them. They also decide the apparatus to be used in the experiment and estimate the duration of the experimentation and the size of the population involved. The methods to obtain data (e.g., videotaping, interviewing, recording speech), where to store them, and whether to make them available or not for the community are also decisions to be taken by the researcher. It should be underlined that DAFIESKU is aimed to help in the setting up of experiments, the selection of participants, and the storing of data. Analyzing data is beyond the scope of DAFIESKU.
On the other hand, there are the experiment participants. They have to decide whether they are willing to collaborate with researchers in order to create data that will be further analyzed. They have to explicitly give their consent and, if requested, they and their data are to be deleted from the experimentation. Experiments are made individually or in groups, depending on the design made by the researchers. Participants are also known as experimental subjects.
3.2. Experimental Design
Researchers usually take several steps in order to define experiments and the experimental setups to be carried out with final users [9]. Experimental research analyzes and explains how manipulating a known variable (“independent variable”) has a direct influence on another variable (“dependent variable”). There also are other types of variables named “control variables” which are to be kept constant across the whole study in order to ensure that the achieved results are indeed caused by the independent variable.
The main types of experiments are those called “randomized experiments” and “quasi-experiments.” In randomized experiments, participants are randomly assigned to every independent variable condition while in quasi-experiments participants are manually assigned to experimental conditions.
Randomized experiments are typically between-subjects or within-subjects. In between-subject experiments, participants are assigned to a single group which is then exposed to a single condition. If the number of conditions is high, the number of required groups increases and so in turn does the number of required participants. In within-subject experiments participants are exposed to every experimental condition and it is easier to find differences caused by the treatment. The number of required participants is lower than in between-subjects experiments, but a learning effect could appear. In order to minimize these effects, conditions are usually counterbalanced (complete counterbalance and/or Latin square).
Another type of experiment is the Factorial Design, used when more than one independent variable has to be tested. Factorial experiments can be within-subject, between-subject, and mixed (the last one, for example, when testing treatments related to age and/or gender).
In DAFIESKU, we have been focusing on experiments to get data from participants, especially physiological data. Therefore, we aim to provide researchers with an application that will help them while designing their experiments to obtain physiological data.
3.3. Making Experiments Available to Participants
When experiments are to be made in controlled environments such as research laboratories in presence of researchers, these can give all the information to participants directly. However, when researchers and participants are in different locations, a protocol of collaboration needs to be established.
In DAFIESKU, there are these uncontrolled environments which form the principal focus, for example, the participant’s habitual dwelling. In this environment, participants feel more comfortable, responding as they might do in their day to day lives, and the recorded data is more realistic, reflecting user’s natural characteristics and behavior.
The experiment, its related information, and the required equipment (e.g., sensors) have to be made available for participants. This is achieved by means of their mobile phones, and the data is sent through the same phones. Therefore, when a participant is ready to perform an experiment, first they have to open the application which contains the experiment, next follow the steps detailed in the experiment, and, finally, send the data captured using sensors.
3.4. Wireless Sensors, Data Acquisition, and Transmission
Nowadays, sensors are more comfortable due to miniaturization. Normally, these sensors are integrated in all-in-one hardware platforms, such as Biosignalsplux [28] or Shimmer [29], including batteries and wireless transmission capabilities. In short, these sensor platforms are useful for experiments performed in uncontrolled environments, for instance, at the home of the participant.
Nevertheless, these platforms still have limited storage capabilities and they use Personal Area Network (PAN) connectivity, such as Bluetooth, which can not be connected directly to the Internet.
In order to overcome these limitations, a common solution is to combine sensor platforms with mobile phones carried by users. Thanks to the mobile devices, the data obtained from the sensors are transmitted first to the storage of the mobile device and finally to the system server which may be available on the cloud.
3.5. Data Storage and Analysis
Once all the ethical and privacy issues have been taken into account [30], recorded data should be processed and filtered and, in order to make further analysis easier, standards, such as HDF5, [31] should be followed.
The physiological data must be available in the system, not only for the researcher but also for the participant. Access to their own physiological data contributes to ensuring transparency in the process of data acquisition.
Of course, if the data is to be used by different research teams, various agreements have to be made between the owners or administrators of the databases, in order to ensure that researchers do not make illicit use of these data when processing them. For instance, the anonymity of users involved in the recording of data has to be preserved even when the data are made publicly available.
3.6. Technical Details
The server side of DAFIESKU was developed in a VirtualBox virtual machine with the Ubuntu 14.04 operative system. Various technologies are used to implement the different parts of the server side of the system. The server configuration is carried out with Apache 2.4.7. The web application is implemented with HTML5, PHP 5.5.9, JavaScript (jQuery 3.0.0), and CSS3. To implement the REST API, the Slim micro framework 2.6.3 is used, which provides an easy way for the Android device to access the data stored in the database. The protocol used to carry out the various requests is HTTP. The JSON format [32] is used to facilitate the communication between the server and the Android smart-phone/tablet by means of the API.
Depending on the request, different JSON objects have to be sent to the API. For example, when a participant finishes an experiment, the Android APP will send a request to the server via the API to update the information on that specific experiment in the database. Examples of the parameters needed for that given operation are the id of the experiment and the data that needs to be updated. If all the required parameters are sent by the Android device when calling the API, the operation is successfully carried out and a specific JSON response is sent to the Android device saying that it has been completed. The JSON response from the server is always formed by an HTTP status code and a message; this could be an error message, a message saying that the operation was successful, and/or a nested JSON message containing the data requested.
MySQL 5.5.47 is used for database management. All the experiment data are stored in the database, so it can be accessed by the API. We also needed a way to store the physiological data, so we opted for the HDF5 file format. When a participant finishes an experiment, the Android device automatically creates a compressed ZIP file containing the physiological data obtained (one text file per sensor used and another file for the time marks) and uploads it to the server using a specific request method developed in the REST API. The server then runs a program developed in C that uses the HDF5 1.8.16 library, which reads the ZIP file containing all the data, and converts it to an HDF5 format file. This file is then stored in the server and its directory path is linked in the database to the experiment to which it belongs, so the researcher has the file available for download.
The application for the Android client of DAFIESKU has been developed using the Android Studio IDE [33] and it can be run on Android devices up to API 14 (Android 4.0, Ice Cream Sandwich).
Most of the classes used for the project are Java and Android native, although some third-party libraries have been used, such as Butter Knife [34] for field and method binding, and GraphView [35] for plotting in Android views. Both libraries are open source and widely known by Android developers.
As mentioned, the communication between the Android smart-phone/tablet and the physiological data acquisition device will be performed through the Bluetooth protocol. Therefore, the Android library provided by the physiological data acquisition device should be added to the application project. In this particular scenario, a BITalino library for Android has been added. If the physiological data acquisition device does not have an Android compatible library the communication with the Android device can be manually performed using the Android Bluetooth class [36].
On the other hand, the Android device and server communication is performed using the HttpURLConnection [37] Java class following the JSON [32] format. Four different types of calls are used by the Android application to communicate with the server:(i)The login call: in this call credentials introduced by the user are sent to the server. If login succeeds an API key will be received as a response. This API key will be used to verify the user in further communications.(ii)Synchronization call: this call will update the Android application with the latest information relating to tasks saved in the server; for example, it will download to the app new tasks available to the user.(iii)Sending the extracted data: this HTTP POST call will be executed automatically when a task is successfully finished. It sends the data extracted from the physiological sensors to the server.(iv)Confirming call: once the data transmission (previous call) is finished an HTTP PUT call will be performed to update the task as carried out on the server side.
In future versions, for these communication purposes, it is planned to add the Retrofit library [38]. This will make the application easier to maintain and will increase stability.
The data acquired from the physiological sensors will be stored in temporary files inside the device’s internal memory, one file per sensor. Due to the computing limitations of mobile devices, generated files will not be processed in the client. Instead, all the temporary files will be compressed in ZIP format file and sent to the server to be further processed. Once transmission with the server is confirmed the data will be removed from the Android device.
4. DAFIESKU 1.0 System
The DAFIESKU system has been designed to make use of wireless devices to obtain a less intrusive environment for the participants during data acquisition. DAFIESKU consists of a set of wearable devices with physiological sensors coordinated by a server. Wearable devices with wireless connections usually offer Bluetooth based connectivity which can be used to send physiological data to any computer. For this reason, a mobile device (e.g., smart-phone or tablet) is used as an intermediary between the server and the wearable devices (see Figure 2).
The main advantage of this approach is the possibility it affords of configuring the data acquisition in each experiment and the possibility of guiding and supporting the participants during the experiments. It should be pointed out that both researchers and participants must register on the DAFIESKU system. Usually, the participants will be registered on DAFIESKU by the researchers.
Prior to acquiring or recording data, there are several decisions to be adopted during the configuration phase. Using a web interface the researcher prepares the experiment which the participants will be able to download to their mobile devices, thus enabling the researcher to acquire their physiological data. In this section, we present the first version of DAFIESKU, while in Section 6 we present the new version that has been developed considering the results obtained in the experimentation presented in Section 5.
4.1. Creating Experiments
With DAFIESKU, researchers may establish which data will be acquired and how and for what length of time it will be acquired in order to create databases with physiological information.
The data to be gathered depends on the scope of the studies that the researchers plan to develop in the future, the sensors that are available for acquisition, the population, and so forth.
The DAFIESKU system provides the researchers with a web client, which enables them to define how the sensors (among those available) will be used, whether they will be used together or separately, and which kind of information provided by the devices will be recorded (see Figure 3).
Figure 3 shows a form for creating an experiment in the DAFIESKU 1.0 web application. The values requested are a name for the experiment, not necessarily unique (there can be more than one experiment with the same name, as they are identified by an id number), and a description, which should briefly and clearly explain its purpose to the participant. Next, the sensors that are going to be used to get the physiological data must be selected. Afterwards, the duration of the experiment (in hours, minutes, and seconds) and the name and surname of the associated participants have to be defined.
Finally, several time marks can be defined. These time marks represent extra information such as activities that a participant can perform during an experiment, temporary emotions, and so forth. For testing reasons, in this version of DAFIESKU 1.0, a maximum of three time marks can be defined.
4.2. Providing Guidance to the Participants
In the DAFIESKU system, participants play a key role in data acquisition. Among other things, they decide when to start the experiments and they provide feedback with which to label the physiological data.
In order to facilitate these activities, the researcher must also () specify the experiment description and define the tasks for the participants, () describe the instructions for attaching the sensors, and () set a list of predefined marks or activities related to the experiment to label the resultant data set. All these texts must be comprehensive and understandable for the participants in order to avoid problems during experimentation (see Figure 4).
(a)
(b)
4.3. Data Acquisition during the Experiments
The participants will be able to download the experiment to their Android mobile device (smart-phone or tablet), thanks to a communication that uses a web service based on a REST API. Once downloaded, they will have to set the sensors and connect them wirelessly to the mobile device, that is, via Bluetooth. Finally, they will have to start to record data and to label moments, following the instructions of each experiment. These last two will be mere push button or select from list tasks, respectively. Therefore, in the first case untagged data are recorded when the user wants to, whereas in the second case data are recorded with marks relating to the activity the user is carrying out (such as running, climbing slopes, studying, and being relaxed).
Figure 4(a) shows the main activity of the DAFIESKU Android application. The aim of this screen is to show all the tasks relating to a previously logged user. Tasks are classified as being pending or finished, depending on their status. In order to perform a task, it has to be selected from the pending tasks list. By clicking on a finished task name the participant can obtain the task feedback.
Once the user has selected a task from the pending list, the screen in Figure 4(b) is shown. This screen contains all the information regarding the selected task (description, sensors to wear, maximum time estimation, and activities to perform). In order to make correct recordings, the user can check whether sensors are correctly adjusted before performing activities related to the task. When participants complete the tasks before the finish time estimation, the back button will take the participant to the previous screen. Otherwise, when the estimated time is reached, the system will automatically go back to the previous screen.
5. Case Study to Evaluate the DAFIESKU 1.0 System on Non-Classroom Learning Experimentation
A case study was carried out to test the usability of the DAFIESKU 1.0 system. For this purpose a representative experiment in a non-classroom learning situation was designed for DAFIESKU in order to test both roles in the system: that of the researcher and that of the participant. As a researcher, the experimental subject introduces the information to set up the experiment. On the other hand, as an experiment participant, the same experimental subject performed fixed tasks such as reading texts or questions, and answering questions while their physiological data were acquired. The usability of both web and mobile applications was measured by using SUS questionnaires [39].
5.1. Method
5.1.1. Participants
5 volunteers (2 females) were recruited from the surrounding research laboratories of the Faculty of Informatics of the University of the Basque Country (UPV/EHU). This is the minimum number of participants that lets you find almost as many usability problems as you would find using many more test participants, as indicated in [40]. The experimental subjects ranged from the age of 30 to 57 years (); see Table 1. Informed consent was obtained from all individual experimental subjects included in the study. Users were computing graduates except User 05a, who had a degree in industrial engineering. Users 01a, 02a, and 05a had PhDs. All of them had previously participated in other research studies, but only Users 03a and 04a had been previously involved in physiological experimentation. Furthermore, Users 03a and 04a had previously prepared experiments with users, but no physiological data had been captured on those occasions.
5.1.2. Apparatus
The web service for designing experiments runs on a virtual machine with an Ubuntu 14.04 LTS operating system. This web service was used when playing the researcher role in this experiment using a laptop. The designed apps for acquiring data run on a Samsung Galaxy Tab 2.7.0 with an Android 4.2.2 operating system (RAM: 1 GB; 1.0 GHz dual core processor; 7 screen with 1024 600 pixels). The BITalino [41] sensor platform was used to collect physiological data from the electrocardiography (ECG) and electrodermal activity (EDA) sensors. The virtual machine running on the laptop and the tablet were connected wirelessly using a WiFi router. The tablet and the BITalino were connected via Bluetooth interface.
5.1.3. Procedure and Design
After consent was obtained from the experimental subjects, their demographic data were gathered. Then, a sheet with the tasks to be completed was delivered to each experimental subject.
The first task (Task 1) to be completed was to define an experiment with certain characteristics by using the web client of DAFIESKU 1.0. This was carried out in the researcher role. Task 2 was carried out in the role of a participant. The equipment needed in the experiment was prepared (see Figure 5), it was verified that everything was working correctly, and then the activities required by the experiment defined in Task 1 were carried out. This experiment consisted of reading a text printed on paper and then reading and answering several questions relating to that text, also on paper and using a pen, as if they were doing out of classroom exercises. Participants had to indicate by using the system the time in which they were reading the text, when they were reading questions, or when they were answering questions. After the experiment was finished, they had to take off the sensors. As a researcher again, on Task 3 they verified, using the web client again, that the data of the experiment was properly stored on the server.
When finishing these three tasks, experimental subjects completed two SUS questionnaires, one as researcher and the other as participant in the physiological experiment. The time required to complete each of the tasks was measured. Finally, they were interviewed by up to two DAFIESKU 1.0 development team members in order to get more feedback, in this case qualitative feedback. The interviews were around 5 minutes long and the questions referred, in particular, to their opinion, their suggestions, the usability of the system, and any issues they had had during the process, as well as what aspects might be improved for future versions. Moreover, the interviews were voice recorded.
5.2. Results and Discussion
All the experimental subjects successfully completed the required tasks. Table 2 shows the times needed to complete the three tasks. Task 2 required the most time while Task 3 was the one requiring the least time. Minutes needed for completing the three tasks range from 25 (User 05a) to 44 (User 02a).
Concerning the usability of the system (see Table 3), the SUS scores were for the researchers web client application and for the Android application. These scores are over 70 and therefore both applications may be considered as being good from the usability point of view [39]. Nevertheless, two of the participants marked the applications lower than 70 and one of them rated the Android application with a score of 42.5, indicating that there were usability concerns to be addressed.
Experimental subjects also suggested several enhancements, almost all with the aim of improving participant experience with regard to data acquisition. One of them (User 03a) suggested including instructions about how to wear the physiological sensors required for the experiment app. Some other experimental subjects mention that wires hindered the completion of the experiment during Task 2. As DAFIESKU aims to create experiments and register data, including help or instructions for the correct adjustment of sensors directly in the app was considered a good idea for future versions. However, it has to be said that coping with wires depends on the sensor technology employed and it is beyond the scope of the DAFIESKU system. Nevertheless, the researcher should consider the characteristics of the technology employed when designing experiments for preventing usability issues that may influence the results.
Other comments advocated the enhancing of the DAFIESKU 1.0 implementation. For example, with relation to the mobile application, several participants suggested increasing the distance between task names when they are shown on the screen. They also suggested maintaining the vertical orientation of the screen while they were checking whether sensors were correctly working. With relation to the server-side application, how to express the duration of the experiments and how to select sensors involved in the experiment needed to be more intuitive. And in order to minimize errors when designing experiments, a confirmation option was suggested to allow the researchers to read the details and accept the experiments after checking if they were correct.
In Table 4 lessons learned in order to make new versions of the systems are highlighted, relating to both the server-side application and the mobile application.
There were also several issues which brought to light aspects in which further evaluations of DAFIESKU 1.0 could be enhanced. For example, our case study was made with five participants all of whom had technological profiles. For more complete and significant results, DAFIESKU should be evaluated with more participants with different profiles (for example, psychologists or people with medical profiles).
6. DAFIESKU 2.0 System
In this section the main changes with respect to the first version of the DAFIESKU system are explained, taking into account the results extracted from the previous experimentation.
6.1. Creating Experiments
With DAFIESKU 2.0, some improvements have been made to the web client in order to solve certain usability issues, also following the suggestions made by the participants that took part in the previous experimentation (see Section 5).
In this 2.0 version of DAFIESKU web client, the objective is maintained, which is to facilitate the process of creating experiments, but in a more intuitive way.
Figure 6 shows a form with which an experiment can be set up in the DAFIESKU 2.0 web application. The main differences are the order of the parameters required to create an experiment. As the participants need to be registered in the system by the researchers, before specifying other parameters for the experiment, it is important to check whether the participant is registered in the system or not. For this reason, in DAFIESKU 2.0, the first step is that of selecting the participants.
Furthermore, more sensors have been added as well as the possibility of selecting a different device. As the system is generic, DAFIESKU 2.0 is prepared to work with different devices. This also facilitates the connection and use of additional devices that allow the use of different sensors for the experiments. When creating an experiment, the system will indicate to the researcher, in real time, which devices are compatible with the sensors that they have selected.
Next, the maximum duration of the experiment must be defined, in hours and minutes. This field has been also changed from the 1.0 version. In fact, the participants had issues with the time format they needed to specify, which lead to errors when creating the experiments. This field has been simplified in order to avoid this kind of error. Moreover, the field for specifying the seconds has been removed, as it makes little sense outside of testing purposes.
Finally, certain time marks can be defined which represent the different activities a participant can perform during an experiment. In DAFIESKU 2.0, it is possible to define dynamically as many time marks as needed by the researcher, unlike in version 1.0, in which only up to three time marks could be specified.
6.2. Improved Guidance for the Participants
As mentioned in previous sections, the aim of the mobile application consists in the recording of physiological data. Due to the critical nature of these data, it is advisable to give the user some feedback to make sure that they are performing the recording correctly. This is even more important if the user has no experience of mobile applications as is the case of several elderly people. For all these reasons and taking into account the case study of the previous section, in the new version of the mobile application, four different categories of assistive screens have been included to guide the user (see Figure 7):(1)Default application information: this category describes the general information about the mobile application such as how to log into the system and what a user has to do to start an experiment (see Figure 7(a) for login screen guide). These screens are accessed automatically the first time a participant opens the application and after that moment by means of a small information icon at the top of the screen (see Figure 8(a)). This button is implemented for all the screens providing in each one the appropriate information to be able to continue using the application.(2)Device functionality and sensor placement assessment: for an inexperienced user, physiological data acquisition platforms may be complicated to use for the first time. Furthermore, the placement of electrodes or the platform itself in the body requires a careful explanation to avoid errors when recording the data. A step-by-step assessment is provided for the device and all the electrodes required in the experiment (see Figure 7(b) along with a guide in Figure 7(c) for both the device and electrode placement).(3)Sensor testing screens: once the sensors are attached to the participants body, a series of tests should be performed to determine that the sensors are working as expected. The system achieves this goal showing to the participant two figures: first, the expected wavelet for a physiological signal and, then, their own signal recorded by the application in real time (see Figures 7(d) and 7(e) for electrocardiogram signal test). The system asks the participant to continue or fix the placement of the sensors if any error was found.(4)Experiment screen instructions: the system was developed to annotate various actions during the experiments using predefined markers. A brief explanation with the functionality of the annotations is provided to the participants (see Figure 7(f)).
(a) Login instructions
(b) Device and sensors information
(c) Electrode placement
(d) Sensor signal instructions
(e) Sensor test with real signal from participant
(f) Experiment screen instructions
(a) Login
(b) Experiment list
(c) Experiment running
The first category of assistance screens is accessible at any time via an icon. On the other hand, the rest of the categories are accessed sequentially when an experiment starts. This sequence must be carefully followed by the participants to prevent some common mistakes when using physiological sensors. Overall, more than 15 different screens were introduced for the 2.0 version of the mobile application.
6.3. Usability Improvements in the Mobile Application
After the first case study, we noticed that the usability of the mobile application could be enhanced. In DAFIESKU 2.0 several improvements relating to the user experience were added.
Thus, fields that do not require interaction with the user are now colored differently on the screen to those that do require interaction. Moreover, for DAFIESKU 2.0 several images were added in the fields in which the user has to interact in order to give a more graphic description, for example, an image of a key in the password field (see Figure 8(a)).
Furthermore, another significant change was added: an update button on the main screen (see Figure 8(b)). By clicking on this button the user synchronizes the status of his local tasks with those on the server. In DAFIESKU 1.0, this synchronization was only carried out at the time of setting up or finishing an experiment.
Finally, several internal improvements were implemented for the mobile application, taking into account the first experiment and the way users interacted with the mobile application. Therefore, the implementation was slightly adapted to give greater stability to issues which were noticed during the experimental phase. For example, a more efficient keyboard management was included.
7. Second Case Study: Evaluation of DAFIESKU 2.0
The newly developed version of the system includes enhancements on the server side and in the mobile application that must be evaluated again to measure if any improvement was achieved in terms of usability (i.e., personal satisfaction and efficiency). We address two different research questions in this second case study:(i)Q1: how much more usable is the second version of the web application than the previous version of the web application?(ii)Q2: how much more usable is the second version of the mobile application than the previous version of the mobile application?
The same protocol followed in the previous case study was replicated for the second case study. Therefore, the apparatus, procedure, and design subsections of the Method section remain the same as the ones described in Section 5 and they should be considered as control variables.
In order to answer research questions Q1 and Q2, the completion times for each task and the SUS scores for each role were collected. In addition to these measures, the participants were requested to fill in a Likert-scale questionnaire with values from 1 to 5 points at the end of the experiment. The questionnaire contains 8 items which are described in Table 5.
Finally, they were interviewed by a DAFIESKU 2.0 development team member in order to get more feedback. The interviews were around 10–15 minutes long and the questions made were especially regarding their opinion, suggestions, usability of the system, and issues they had during the process, as well as what aspects should be improved for future versions. The interviews were voice recorded.
7.1. Method
7.1.1. Participants
17 volunteers (7 females) were recruited from the Faculty of Informatics of the University of the Basque Country (UPV/EHU) and neighboring faculties. The experimental subjects ranged from the age of 21 to 51 years (); see Table 6. Informed consent was obtained from all the individual experimental subjects included in the study. In general, users had a degree in computing, except User 06b who had a degree in mathematics, Users 07b and 17b with degrees in electronics and User 12b, who was studying psychology. Moreover, Users 01b, 09b, 15b, and 16b were Ph.D. graduates. Users 02b and 05b were left-handed while the others were right-handed. Users 01b, 03b, 09b, and 16b recounted that they had prepared experiment(s) with users in the past, and User 09b had previously prepared at least one experiment in which physiological information was acquired. Several users had had previous experience as participants in experiments (Users 01b, 03b, 05b, 06b, 09b, 10b, 12b, 13b, 14b, 15b, 16b, and 17b), and in some cases their physiological data had been acquired in at least one previous experiment (Users 06b, 09b, 12b, 13b, 14b, and 17b).
7.2. Results and Discussion
All the experimental subjects successfully completed the required tasks. Table 7 shows the times needed to complete the three tasks presented. Task 2 was again the one needing more time while Task 3 was the one which required the least time. The number of minutes needed to complete the three tasks ranged from 17 (User 04b) to 38 (User 11b).
Comparing these results with the previous evaluation (Section 5), there have been improvements in Task 1 (4.06′ on average while the previous study achieved 5.80′ on average), Task 2 (19.65′ on average while in the previous study 23.60′ was needed on average), and in Total (26.06′ on average compared to 31.20′ on average in the previous study), while Task 3 achieved poorer results (2.35′ on average while in the previous study an average of 1.80′ was achieved).
With relation to the usability of the system (see Table 8), the SUS scores were for the researchers web client application and for the Android application. These scores may be considered quite good, nearly excellent from the usability point of view [39]. Nevertheless, there are usability concerns to be addressed after considering the individual scores.
When compared to the results shown in Section 5, both components of DAFIESKU have achieved better scores. First, the researchers web client application scored 78.5 on average in the previous study, whereas the Android application achieved 71 on average. The improvement was notable in the case of the Android application.
Moreover, the time required to complete the tasks was lower and the usability results were higher. These data suggest that DAFIESKU 2.0 is more usable, more efficient, and friendlier. Therefore we can conclude that the changes introduced have had a positive effect on the system. With respect to the first research question, the SUS score slightly increased in the web application which indicates that the changes proposed had a minor effect in the usability perception. Additionally, with respect to the second research question, the novelties introduced in the mobile application increased the usability perception by almost 9 points on average. These changes mitigated the problems of the mobile application identified in the first evaluation.
In general, scores achieved in the final Likert-style questionnaire were also high (see Table 9). Several factors were measured, from general satisfaction, to the usefulness of the assistance instructions obtaining the agreement of the participants. Only statement 4 (“sensors do not cause inconvenience when doing the experiment”) had an average value lower than 4. This was mainly due to the wires of the sensors employed in the experimentation and the need to write during Task 2. Maybe using wireless sensors would enhance this value.
We also have to mention that User 14b did not answer Question 3 (“I am able to make experiments by using the DAFIESKU system without being aided by technical staff”). As she mentioned during her interview, she did not think she was going to perform experiments with users during her career. Therefore, she preferred to leave that question unanswered as she could not imagine the need to use a system like DAFIESKU during her activities.
In general, during the interviews, participants declared the DAFIESKU system to be easy to use, both as a researcher and as an experimental subject. Bearing in mind they were using it without prior presentation of the system, it would appear they were able to carry out the requested activities in an intuitive manner. They also thought that if they were requested to use DAFIESKU again, they would easily remember its functionalities.
Although their first language was either Spanish or Basque, almost all of them tested the English version of the application for researchers. There was only one exception: User 12b tested the Spanish version of the application during Task 1. No problems were found and the aforementioned user tested the English version of DAFIESKU during Task 3.
During the interviews several comments arose that will guide us when making new versions of the DAFIESKU system. In what follows we will set out the most relevant of these.
For example, possible errors should be clearly explained to participants in the current tablet app. During the experimentation, in several cases (Users 06b, 09b, and 10b) the Bluetooth connection between the tablet and the BITalino was broken and the message “Unfortunately, Dafiesku has stopped” appeared on the tablet screen (see Figure 9). This message did not give any information about what had happened. The way to solve this problem was suggested by the person in charge of the experimentation and consisted in restarting the tablet. Afterwards, the participants had to start Task 2 again. The error only happened once to each of them and at the second attempt they were able to complete Task 2.
When using wired sensors, several participants (Users 03b, 11b, and 16b) suggested using colors to distinguish them, as they initially thought there were no differences between them. But in general, and when completing a Likert-style questionnaire, the participants thought that wireless sensors would make experimenting with physiological signals more comfortable. They also suggested including more commercial equipment in future DAFIESKU versions to get physiological information, as only BITalino and eHealth equipment were considered in the prototype.
Several participants found this system adequate for monitoring health issues (Users 03b, 10b, 11b, 12b, and 17b). In some cases, they also mentioned other applications such as sports (Users 10b and 11b).
Other enhancements suggested by several participants (Users 10b and 14b) were related to existing on/off buttons in the experiment: if at maximum one had to be active (on), activating one should deactivate the previous one. Thereby, the effort needed (measured in operations to be carried out by the users) would be lower (only one click needed instead of one click to deactivate and another one to activate the new one).
Even the Next and Back buttons were found useful to check whether participants were wearing the sensors correctly (e.g., User 14b). There was no total agreement about maintaining the current version or changing it. For example, User 05b suggested making a vertical scroll instead of Next and Back buttons. Another option was mentioned by User 13b, who suggested showing a next button instead of a back button in Figure 7(e) when a sensor was checked and no errors were found.
Other suggestions were related to including new functionalities within the DAFIESKU system. For example, Users 08b and 16b suggested including statistical information within the researcher part, maybe by also using graphics, to analyze the evolution of the participants in the experiments. User 08b also suggested including the option of taking and sending photos in order to let the participants show how they completed the tasks and allow the researchers to check the results achieved by the participants. User 16b was of the opinion that in future versions the researchers would have to design experiments by using a tablet, mobile, or similar devices, instead of personal computers. She suggested working on this idea for DAFIESKU.
In Table 10, lessons learned in order to make new versions of the systems are highlighted, relating to both the server-side application and the mobile application.
8. Conclusions and Future Work
In this paper, we propose a system aimed at remotely recording physiological data of participants without the help of research assistants. It has been developed by the usual iterative development cycle proposed on User-Centered Design [13]. Making prototypes helps solutions to be put into practice in a fast and economical way. Moreover, evaluating developed prototypes serves to acquire feedback from potential system users. This helps developers to think about the issues raised by users while using the system. Two iterations of the system (versions 1.0 and 2.0) are presented and evaluated in this paper. The system was developed to configure data acquisition, perform data collection, and finally store data in a friendly format for the researcher.
Two case studies were carried out to test the adequacy of the proposed approach and the usability of the software. The first evaluation with 5 participants obtained promising results and raised interesting issues, concerning, among others, the lack of support for guiding the participants. Afterwards, a new iteration of the software was developed to address these issues and include enhancements suggested by the participants. Thus, a second case study with 17 participants, including a student in psychology and other graduates apart from computing, was conducted to measure whether the usability of the system had increased or not.
The results of the second evaluation show an improvement in the mobile application usability in almost 9 points and a slight enhancement of 3 points in the web application on the SUS scale [39]. Therefore the changes made in the mobile side of DAFIESKU were useful to increase the usability perceived by the participants. New feedback and suggestions were collected. Furthermore, the interviews and questionnaires carried out with the participants pointed out that the guidance level required for physiological data acquisition was achieved by the system.
Thanks to the second evaluation, we have identified new features and the need for assistance screens to be added in future versions of DAFIESKU. The next evaluation step should be performed in the homes of a group of volunteers, thereby allowing a full deployment of the system in an uncontrolled environment through the use of participants’ personal mobile devices. We also plan to provide the researchers with more resources to customize their experiments, such as the possibility of editing the text and images that appear in the mobile application assistance screens.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work has been partially funded by the Ministry of Economy and Competitiveness of the Spanish Government and by the European Regional Development Fund (Project TIN2014-52665-C2-1-R) and by the Department of Education, Universities, and Research of the Basque Government under Grant IT980-16. The last three authors belong to the Basque Advanced Informatics Laboratory (BAILab), Grant UFI11/45, partially supported by the University of the Basque Country (UPV/EHU). The authors would like to acknowledge all the participants who took part in the studies.