Definition of Guideline-Based Metrics to Evaluate AAL Ecosystem’s Usability

Romeiro, Carlos; Araújo, Pedro

doi:https://doi.org/10.1155/2022/8939072

Human Behavior and Emerging Technologies

On this page

Abstract Conclusions Data Availability Disclosure Conflicts of Interest Acknowledgments References Copyright Related Articles

Research Article | Open Access

Volume 2022 | Article ID 8939072 | https://doi.org/10.1155/2022/8939072

Definition of Guideline-Based Metrics to Evaluate AAL Ecosystem’s Usability

Carlos Romeiro¹and Pedro Araújo^1,2

Academic Editor: Zheng Yan

Received23 Jun 2022

Revised01 Sept 2022

Accepted25 Oct 2022

Published14 Nov 2022

Abstract

The elderly population growth has posed several challenges which the current healthcare systems are incapable of handling. In the past few years, there has been a close collaboration between both the scientific and industry communities to provide feasible solutions capable of addressing the growing demands from people with special needs, namely, in terms of assistance and improvement of their overall life quality, which promoted to the development of the ambient assisted living (AAL). Despite the general consensus regarding its positive impact in the user’s daily life, several challenges compromise their overall adoption. As a consequence, the research undertaken so far focused over the mitigation of technical-related limitations, overshadowing user-related limitations, namely, the ecosystem’s usability. This article presents a parametrization of the literature guidelines, which provides the end-users a consistent and accurate way of using the heuristic methodology to assert the interface’s usability without relying in external entities with specialized know-how.

1. AAL Ecosystems

The shift of the world’s demographic pyramid is a phenomenon that affects both western and eastern civilizations. The effects of the higher life expectancy, the declining birth rates, and the overall improvement of healthcare services are noticeable upon a close inspection of the current worldwide population distribution tendency (see Figure 1) [1–5].

This new paradigm presents a unique set of conditions that will have a direct impact in the modus operandis of multiple sectors, with special emphasis over the health sector, where the demands for services capable of assuring the elderly’s wellbeing have grown over the years. Demands combined with human resource shortage, lack of patient-oriented approaches, and the rapidly rising costs of elderly care threaten to hinder the core service efficiency and availability, leading to their long-term collapse. As a consequence, an attempt was made to develop an ICT-based solution capable of tackling the economic and functional endeavours of the assistance services within the health sector and promoting the elderly’s quality of life and autonomy—the ambient assisted living (AAL) ecosystems [5]. Despite the significant improvements in the field, multiple challenges still need to be tackled to make their widespread adoption on the market viable. These challenges can be segmented into technical (system’s bottlenecks from an engineering standpoint) and end-user challenges (user perceived main barriers to the product’s acceptance) (see Table 1).

Regardless, an effort was performed to overcome all these drawbacks and promote the ecosystem mass adoption. In terms of usability, multiple studies have been conducted to identify the range of factors that have a direct impact in the product’s usability and several approaches have been proposed to assist in the process used to evaluate this property within the context defined [19–24]. Despite the research effort undertaken, the results produced are still not sufficient to consider this a closed issue. To address this issue, the author’s initial proposition was to empower the product manufacturers and provide them a simple and feasible way of evaluating and monitoring the product’s usability. From all the available methodologies, the one eligible to be executed in an enterprise setup, due to its inherent cost and speed of execution, was the heuristic-based. Alas, it also presents limitations that compromise its adoption, namely, its results’ accuracy and its restricted applicability.

Considering the challenges presented, this article provides a parametrization of the usability guidelines. Our aim is to (1) optimize the subjectivity level typically found in heuristic-based methodologies, (2) optimize their overall accuracy and results consistency, (3) extend its accessibility/applicability to nonusability experts, and (4) minimize the effort typically related to their automation.

A thorough search in literature was conducted in order to identify what can be learned from the best practices depicted, how they can be applied in a practical scenario, and how the inclusion of automation can be a feasible option in an enterprise context to mitigate the flaws typically detected in multiple medical applications in the field.

1.1. Usability

Usability is a multidimensional property that reflects the scope in which a product/service is expected to be used [25, 26]. This characteristic varies depending on its end-users, product/service purpose, and application context. To ensure its compliance, it is required for the design process to take into account multiple elements [25, 27–31]: (i)Efficiency: measures the speed rate at which the users perform accurately a task in the interface(ii)Effectiveness: measures the accuracy with which the actions required to complete a task in the interface are executed by the user(iii)Satisfaction: measures the interface friendliness and user’s need compliance level(iv)Learning curve: measures the speed in which the end-user is able to assimilate the knowledge needed to use the system properly(v)Rememberable: measures the difficulty behind the retention of the required knowledge for the user to handle the system properly(vi)Context compliance: measures the compliance of the product/service created in terms of its applicational context(vii)Security: measures how the use of the depicted product/service impacts the user’s data integrity

To enforce usability throughout the development lifecycle, it is mandatory to integrate within the process practices that could aid both developers and designers. For this purpose, it is imperative to stipulate a set of guidelines to be adopted, evaluation scales and methodologies for the identification, and prioritization of the usability bottlenecks.

1.2. Guidelines

The search for a set of generic rules not bound to any applicational context to assist the team during the development cycle was a thematic thoroughly explored in the literature. The first set of principles was proposed by Gould and Lewis in 1985 [32, 33]; however, their limited applicability hindered their adoption and acceptance by the scientific community [34, 35]. To address the gap between the theoretical definition and practical application, in the following years, several authors proposed their own version of usability principles, which considered their applicability in a real environment.

In 1990, Nielsen and Molich proposed 10 heuristic principles [36–39] that emphasized the following guidelines: (1) visibility of system status, (2) match between system and the real world, (3) user’s control and freedom, (4) consistency and standards, (5) error prevention, (6) recognition rather than recall, (7) flexibility and efficiency of use, (8) aesthetic and minimalist design, (9) error diagnose and recognition, and (10) user’s assistance and documentation. In 1995, Constantine proposed 11 rules based on multiple interface-related subjects (access, efficiency, progression, support, context, structure, simplicity, visibility, reusability, feedback, and tolerance) [40]. In 1996, Powalsa proposed 10 cognitive principles [33, 37, 41] focused on a holistic analysis of the usability evaluation process. In 1998, Shneiderman proposed a set of 8 golden rules [42] whose definition stressed scopes already enforced by its predecessors. In 2000, the authors Susan Weinschenk and Dean Barker proposed 20 principles, which resulted from the combination of the Jakob Nielsen principles with vendor specific guidelines (Apple and Windows) [43] focused on multiple topics (user’s control, human limitations, modal integrity, and linguistic clarity, among others). In 2003, Tognazzini proposed another set of principles covering a sum of 19 areas that range from broad subjects, such as learnability and readability, to more practical and specific ones, such as colour blindness and Fitts’s law (human interaction prediction model published by Paul Fitts in 1954) [44, 45]. This set was the first to include accessibility elements within its heuristic set and stress its importance to the product’s acceptance.

Additionally, the emergence and mass adoption of new interface devices (smart phones and smart TVs) motivated the creation of specific guidelines capable of taking into account their design restrictions. On the mobile domain, several proposals with a well-defined set of heuristics have been made by the authors Silva et al. [46] and Inostroza et al. [47]. The smart TV domain should be highlighted in the study by Martins et al. [48], which aimed at adapting the Jakob Nielsen heuristics to the specific technical constraints.

Among the previously mentioned principles, the most widely accepted, due to their extensive applicability in multiple use cases, maturity level, and adaptability, is the Jakob Nielsen and Rolf Molich heuristic principles.

1.3. Scales

The scales used in the evaluation process are aimed either at quantifying the product usability level or at providing a formal procedure to prioritize each bottleneck detected according to its criticality and severity. They can be segmented into two groups: (i)User-based: quantification process based on the information extrapolated from the user’s reactions, opinions, desires, and feelings expressed during the interaction process (e.g., ICF-usability scale [49] and Likert [50])(ii)Heuristic-based: prioritization process performed within the scope of heuristic methodologies that identify the critical flaws in need of further improvement (e.g., Jakob Nielsen scale [51] and Joe Dumas and Ginny Redish scale [52])

1.4. Methodologies

Evaluation methods used to identify usability bottlenecks and provide the development team feedback are required to mitigate them accordingly [53]. According to their execution requirements, they are divided on a high level into three typologies, which one with its unique characteristics: (i)Inspection-based: analytic methodology whose execution depends on the availability of formal evaluators or field specialists capable of performing an accurate assessment of the interaction between system and end-user. This scope includes the following techniques: (1) heuristic evaluation, (2) feature inspection, (3) consistency inspection, (4) standard inspection, (5) formal usability inspection, (6) cognitive walkthrough, and (7) pluralistic walkthrough [28, 53, 54](ii)Enquiries-based: empirical methodology based on execution of questionnaires specifically design to take into account the inner characteristics of the product/service being evaluated. The subjectivity of the collected data is a relevant asset in the identification of the user needs and in the identification of usability bottlenecks. Additionally, their execution speed and low implementation costs make them a common choice within the usability evaluation scope [54](iii)Test-based: empirical methodology in which a human evaluator observes the user’s executing specific tasks in the interface and collects enough data to extrapolate empirical evidence to optimize interaction mechanisms within the interface [54, 55]

From the multiple methodologies described, this article focuses over the heuristic methodology. For this purpose, it defined quantifiable metrics based on literature guideline parametrization. However, before applying any guideline breakdown, it is important to identify the advantages and disadvantages behind the adoption of such methodology, in order to clarify what drove that decision.

Regarding the main advantages, the heuristic methodology is known as a quick and low-cost approach capable of providing feedback to the designers since an early stage, without the need of real end-users. This approach uses literature guidelines to evaluate the interface, and this assists the designers in identifying corrective measures needed to solve usability bottlenecks detected. In terms of disadvantages, the ones most frequently highlighted are (1) the approach’s efficiency and viability depends on expert know-how, (2) inability to evaluate usability to its full extent (for example, user’s satisfaction is out of the methodology’s scope), and (3) the lack of reliability of the produced end-results, which can be tackled by including additional specialists to perform the evaluation process [56, 57]. According to the best practices, its applicability requires a group of 3 to 5 evaluators [58].

The dependency on expert’s know-how and the execution restrictions identified are challenges that the proposed parametrization intends to mitigate, ensuring the methodology accessibility to anyone that intents to use it. However, it should be noted that applying a set of well-defined metrics to manually evaluate an interface in terms of guidelines compliance level is a time-consuming task. Since parametrization is a first step to define business rules to be consumed by a yet to define tool, it is reasonable to explore the use of automation mechanisms to handle such procedure.

2. Heuristic Methodology Roadmap

This methodology has been a thematic explored in the literature by several authors whose contributions led to its typification into four categories, each one with their own unique characteristics: (i)Interaction-based: an approach focused on the use of users’ interactions to evaluate the interface’s usability. The analysis performs a comparison between the interaction samples collected and the ones considered optimal within the applied context, so as to identify usability bottlenecks. Despite being acceptable to check the interface implemented at the interaction level, its use entails multiple consequences, such as (1) the dependence on real users and an already testable interface and (2) the results reliability and the significant number of samples required to ensure the process viability. In terms of architecture, these solutions usually delegate the sample processing and analysis to a remote server, but inherent performance drawbacks have led to the adoption of a new approach in which the load is divided between server and client accordingly with the objective of improving the results’ response times [59–63](ii)Metric-based: an approach focused on the definition metrics used to quantify the interface’s compliance level with usability guidelines defined in literature. These solutions do not require the intervention of end-users, which make them a viable option to be used in the interface’s development initial stages. Despite the correlation shown between the approaches and the manual tests outcomes, the metrics’ lack of context awareness and their overall dependence on expert’s know-how compromise their adoption in multiple use cases [59–61](iii)Model-based: an approach developed to mitigate the drawbacks identified in the metric- and interaction-based approaches through the use of artificial intelligence mechanisms; this allows for the definition of the interaction model then applied in the evaluation process. Features which make it independent from end-users enhance its context awareness. Although efforts have been made to enhance the model’s creation and training algorithms and their context adaptability, current solution scalability and performance still need improvement before being implemented in a real environment [59–61, 64](iv)Hybrid-based: an approach that combines the mechanisms of each of the previously described categories to provide a holistic solution capable of identifying usability flaws and proposing measures to address them [59, 60]

Despite their differences, there is a common denominator among the categories defined—the automation mechanisms—which aim to reduce the number of tasks to be handled by a human agent. Based on such typification, a development effort has been performed by both the scientific and industry community to create solutions capable of evaluating to a certain degree in the interface’s usability.

2.1. Industry Context

In the industry context, external factors, such as the government accessibility guideline compliance policy [65, 66] combined with the elapsed time typically required to perform an interface full assessment, fostered the development of standalone metric-based solutions capable of evaluating a specific usability subset—accessibility—for both web and mobile interfaces (please see Tables 2 and 3).

2.2. Academic Context

In the academic context, an analysis performed to 99 scientific articles ranging from 1997 to 2022 provided an overview of the heuristic methodology evolution tendency and respective trends explored throughout the years [60, 62–64, 67–142] (see Figures 2 and 3).

Based on the results obtained, it is perceivable that 40% of the scientific articles published focused over interaction-based solutions. However, such tendency has been shifting from 2005 onwards to the model-based approaches, due to the improvements achieved in the machine learning and data mining algorithms.

From the several articles sampled are highlighted the following ones: (i)In 2013, Dingli and Cassar proposed a tool that automates the usability evaluation process of web interfaces using several artificial intelligence mechanisms. The results obtained demonstrate the capability of the tool in the identification of critical bottlenecks otherwise identified by a human agent [69](ii)In 2014, Yáñez Gómez et al. compiled a set of heuristic evaluation checklists and adapted them by taking into consideration the unique requirements of mobile interfaces. As a result, an assessment tool based on a mobile-oriented best practices checklist was generated. Such tool provided the means to trained and nontrained developers to assess and identify critical usability bottlenecks in mobile interfaces in an accurate and feasible way [139](iii)In 2017, Sun proposed a usability evaluation approach based on mixed intelligent optimization focused over the assessment of educational resources software [140]. In the same year, Ferre et al. proposed an extension to the Google Analytics functionality, which stored the actions executed by the user during the usability evaluation process. The solution itself was divided into three functional blocks: the first one identifies the tasks executed and events triggered in the interface, the second one maps the event/task information sampled and logs it into the client device, and the third one focuses over the identification of interaction patterns in the samples collected through the execution of a set of data mining algorithms and compares them with the defined expected interactions [73](iv)In 2018, Othman et al. published a study that compares the Jakob Nielsen principles and the smart heuristics in the detection of usability bottlenecks within a museum guide app developed for the mobile environment. Comparison is aimed at identifying which heuristic subset was suitable to evaluate/assert the interface’s usability—a generic heuristic set (Nielsen’s heuristics) or a tailored heuristic set whose metrics have been customized to consider the unique characteristics of application developed for smartphones. The output provided highlighted the importance of the application context during the selection of the heuristic set to maximize the accuracy and efficiency of the evaluation process [141](v)In 2019, Ribeiro et al. proposed an approach to perform an assessment and identification of usability bottlenecks in an automatic manner through the use of user interactions with the interface in production environment. Such approach is aimed at mitigating the costs and complexity related with the user interaction sampling process during the execution of usability tests [105]. In the same year, Virtanen used the robot framework to automate the analysis of the interface’s compliance level with the Jakob Nielsen heuristics. The study compared the usability bottlenecks identified with the ones detected by the traditional manual approach to assert the approach effectiveness and accuracy [143](vi)In 2020, Bures et al. proposed a crawler specialized in the generation of interaction models within the context of smart TV applications. The generated models provide the end-user a mechanism to quantify the feasibility and effort related with the execution of each action within the interface and evaluate it in terms of usability [100](vii)In 2021, Ripalda et al. proposed a tool that correlates the adopted usability metrics defined in the literature with the feedback retrieved automatically from LINKERT questionnaires designed by the development team. Such tool provides designers, developers, and usability specialists the mechanisms required to evaluate the interfaces, to identify the side effects of design changes in the interaction process and to perform an assessment with recommendations meant to optimize the overall results obtained [137](viii)In 2022, Muhanna et al. proposed a set of heuristics to tackle the lack of explicit usability evaluation methods capable of assessing the usability within a specific application context—Arabic mobile games. For this purpose, Nielsen’s heuristics were revised and adapted to meet the unique characteristics of the software being evaluated. Through such adaptation it became possible for the evaluators to detect additional critical bottlenecks within the test scope defined [142]

The approach that is explored in the article is included in the metric-based type.

2.3. Heuristic Optimization

2.3.1. Principles’ Breakdown

The guidelines selected for the parametrization process were the following: (1) Jakob Nielsen’s principles, (2) Shneiderman’s golden rules, and (3) Weinschenk and Barker’s cognitive principles. Each principle was grouped according to its scope within the interface and its direct relation with the interface main building blocks, the components, and the actions [144]. As a result, four typologies were defined: component oriented (CO): focused on the compliance of the interface’s native components with the look and feel defined during the design phase. As the name implies, its applicability requires an assessment of the components within each application’s section in terms of typology (active or passive), family (button, checkbox, and input tag, among others), and name; action oriented (AO): focused over actions provided used to navigate across the interface and manipulate the business data consumed system; section oriented (SeO); and screen oriented (ScO).

For each principle, the respective parametrization is presented in Tables 4–6.

It should be noted that within the Shneiderman and Weinschenk and Barker subsets, there were principles not described, since they have a common ground in terms of concept and definition to principles whose evaluation process had already been discussed.

In the Shneiderman subset, principles such as (1) “Strive for consistency,” (2) “Seek universal usability,” (3) “Prevent errors,” (4) “Permit easy reversal of actions,” and (5) “Reduce short-term memory load” share the same evaluation process described for their counterparts in the Jakob Nielsen and Rolf Molich set (“Consistency and standards,” “Flexibility and efficiency of use,” “Error prevention,” “User’s control and freedom,” and “Recognition rather than recall,” respectively).

In the Weinschenk and Barker subset, principles relating several topics, such as (4) “Accommodation,” (10) “Accuracy,” (16) “Consistency,” (17) “Support,” and (20) “Responsiveness,” were discarded from the evaluation process. According to Nayebi et al. [156], the use of a proper language, terminology, and adequate metaphors (daily life objects within the interface components as a visual representation of their purpose) plays a major role to ensure the interface suitability to the user’s needs and behaviours. These key factors are already considered in the second Jakob Nielsen and Rolf Molich principle (“Match between system and the real world”). In terms of “Accuracy,” the principles state that the interface should be deprived of errors or, in other words, the errors should be prevented to a certain degree. This topic is already discussed and addressed by the fifth Jakob Nielsen and Rolf Molich principle (“Error prevention”). The remaining categories share a direct connection with their Jakob Nielsen and Rolf Molich’s counterparts (“Consistency and standards,” “Help and documentation,” and “Visibility of system status”).

2.3.2. Real Environment Applicability

The parametrization provided a checklist to assert the interface’s usability compliance level. To foster its development, it becomes imperative to identify current bottlenecks that may hinder its adoption by using it in a real use case. For this purpose, two e-health applications were evaluated (an academic prototype and an enterprise solution).

2.3.3. Doctor Helper (Prototype)

The Doctor Helper is an academic prototype created as an attempt to replicate the functionalities identified in typical e-health applications, namely, (1) account creation/authentication, (2) sensor samples’ registration, (3) presentation of the sensor sample history in multiple visual formats, and (4) creation of data reports and inclusion of a notification mechanism to aid the user in his/her daily tasks or to report any abnormal event in the system.

The performed evaluation considered the 106 actions and 356 components available in the 15 screens of the entire interface (see Figure 4). The end-results are presented in Figures 5–7.

Figure 4

Distribution of objects and actions among the several interface screens. Notes: 1: user authentication screen; 2: account creation screen; 3: password recover screen; 4: timeline screen; 5: sensor overview screen; 6: sensor details screen (table format); 7: sensor details screen (chart format); 8: sensor edition screen; 9: sensor troubleshoot screen; 10: sensor measurements screen; 11: look and feel settings screen; 12: user profile screen; 13: equipment settings screen; 14: notifications settings screen; 15: data management settings screen.

The evaluation results are allowed to identify a total amount of 1781 usability smells. In terms of “Visibility of system status,” the main bottlenecks in the Jakob Nielsen subset are the lack of confirmation/conclusion dialogs and the lack of a task completion rate. Regarding the “Error prevention” principle, the most prominent issues were related to the lack of an autocomplete mechanism in the components which receive an input from the user, the lack of a mechanism that automatically saves the user’s work, the lack of error messages providing clear indications of the type of unconformities detected in the user’s input, and the lack of mechanisms capable of disabling the action-related controls when the view requirements are not met. Finally, for the “Help and documentation” principle, its evaluation was mainly related to the lack of an option in the interface to access the platform official documentation.

In terms of “Support internal locus of control,” the main bottleneck in the Shneiderman subset was related to the lack of confirmation dialogues to assert the user’s intentions before executing a certain task within the interface.

In terms of “Human limitation,” it should be emphasized that, in the Weinschenk and Barker subset, the components lack the capability to store their previous state, preventing the user from being aware of his/her previous interactions without appealing to his/her memory. Regarding “Interpretation” principles, the main bottleneck detected was related to the lack of a mechanism in the system capable of predicting the user’s intentions or user’s input when the interaction process is taking place.

2.3.4. SmartAL (Enterprise Application)

SmartAL (https://http://www.alticelabs.com/site/smartal/) is a solution developed by Altice to monitor in real time chronically ill and elderly patients. This platform is intended to perform a thorough follow-up of the patients by providing mechanisms to monitor their vital signs, perform video calls with their physician, manually register vital signs samples, and schedule video call appointments among other functionalities.

During the analysis were taken into consideration 523 actions, 1918 components, and 103 screens of the entire interface (see Figure 8). Note that there is a significant discrepancy between the number of objects and actions identified between the academic prototype and a commercial application in a real environment, due to the number of functionalities and user roles supported (patient and caregiver). The interface technical depth combined with extensive number of elements to be tackled and the project time constraints implied the execution of compromises. As a consequence, the analysis scope was restricted to the most mature and emphasized subset in the literature—the Jakob Nielsen and Rolf Molich subset. The end-results are presented in Figure 9.

The results obtained highlighted 2488 usability smells. Among the evaluated principles, the ones that were not able to achieve the minimum acceptable score (70%) were related to “Visibility of system status,” “User’s control and freedom,” “Error prevention,” “Flexibility and efficiency of use,” “Help users recognize, diagnose, and recover from errors,” and “Help and documentation.”

The lack of progress bar indicators for time-consuming operations, the lack of dialogues to signal the user when a certain action is concluded, the lack of meaningful tooltips with information regarding the object or action intent, the existence of hints with system specific terminology, the lack of dialogues to assert the user’s intentions when an action is being executed, the lack of advices within the error message to mitigate the abnormal event, the lack of mechanisms capable to cancelling an action execution at any given time, the presentation of error messages with technical terminology, the lack of a link in the interface section to redirect the user to the official documentation, the lack of a tooltip in the interface controls describing the keyboard shortcut which could be used to trigger the component inherent action (see Figure 10), and the lack of a mechanism to assure the user’s work is not compromised by any abnormal event that may be triggered within the interface are some of the most prominent flaws identified in the interface which led to the obtained results.

All the results from the evaluation process were obtained through a manual analysis of the interface, a time-consuming and error prone approach. To maximize its scalability, it is required to implement automatisms capable of assisting the end-user with the evaluation tasks.

3. Implications

The AAL ecosystem’s rapid growth in the past few years led to the development of multiple use case-oriented solutions and studies addressing different subjects and challenges identified in such ecosystems. Depending on the scope of the studies and solutions devised, different approaches have been used to explore and tackle usability.

There are studies that focus over the usability assessment of existing solutions to identify critical bottlenecks and suggest optimizations accordingly. In this scope are included but not limited to the following application contexts: wearable camera system for dementia patients in 2014 by Matthews et al. [162], diabetes-monitoring applications in 2015 by Isaković et al. [163], IPTV application running in commercial environment that integrates multiple solutions to support home care by Ribeiro et al. [164], emergency alert devices in 2020 by Lersilp et al. [165], and smart bands in 2021 by Correia et al. [166], among others.

Studies focused over the usability processes adopted and their role within the development cycle. In 2017, Sili et al. presented a user-centred design process for an indoor and outdoor navigation solution suitable for the target group defined [167]. In 2021, Bastardo et al. went a step further by questioning the methodological quality of the user-centred usability evaluation of AAL ecosystems. The heterogeneity of the methods adopted in this scope imposes the definition of guidelines and instruments to assess the quality of the evaluation procedures adopted. Through an accurate assessment, it becomes possible to identify bottlenecks optimizing the process in an overall manner [168].

Other studies emphasize how the lack of attention given to usability has hindered the acceptance and adoption of AAL ecosystems’ solutions. In 2013, Agha et al. proposed a study which aimed to provide awareness to the readers of the complexities and importance of addressing human factors and usability aspects of telehealth. According to the authors, the rapid growth of mHealth technologies imposes a shift in terms of priorities during the development cycle. Feedback provided by both patients and caregivers must be considered to maximize the solution’s acceptability and adaptability [169]. In the same year, Queirós et al. performed a review of the AAL ecosystems to identify how end-users are involved during the development cycle. The end-results proved a certain lack of involvement of the end-users in the process, especially in the assessment of usability and accessibility bottlenecks [170].

Note that it is perceptible that usability is gaining momentum in AAL ecosystems. However, the way how it is tackled tends to be centred over the solution’s specific needs. Within the article’s context, usability is explored in an independent manner. The parametrization of the literature guidelines is aimed at creating the cornerstones required for the definition of heuristic metrics suitable for any solution within the AAL ecosystems’ scope. In opposition to the typical approaches adopted so far, the objective was to avoid coupling the metrics defined to any specific solution. The unique characteristics and the nature of the tasks executed within such ecosystems impose the definition of tailored made metrics. An analysis of the literature showed that such customization has not been explore thoroughly due to the heterogeneity of the solutions devised within this scope, so with this article, it is intended to perform the first step towards achieving that goal.

The findings present a practical application of the parametrization adopted and highlight how its use allows to assert the level of compliance with the guidelines covered in the study. Despite providing the foundations of the metrics to be used within the evaluation process, their scope still needs to be further refined, in order to make them suitable with the ecosystems that will be evaluated. Therefore, in a second iteration, it is expected to test the parameters in other e-health applications and collect user’s feedback regarding the usability bottlenecks identified to identify which metrics have contributed to the identification of critical issues and in which way they can be further optimized to take into account the user standpoint when handling the application.

4. Conclusions

The research performed compiled the knowledge available in the literature which is culminated in the proposed parametrization. Parametrization provided an objective manner to address the usability principles frequently applied in a heuristic evaluation process. Its main differentiating factor is related to how the metrics were defined. The definition of each principle was analysed thoroughly to identify practices which are applied with the objective of ensuring, within a certain degree, its compliance. By isolating the typically used approaches, it was possible to define metrics that can be asserted in a binary manner. Its applicability allowed the maximization of the result’s accuracy and consistency, making the heuristic methodology more accessible to users without expert usability know-how.

However, there are inherent limitations within the current process which hinder its long-term scalability: (i)Manual analysis: the interfaces’ compliance level with the metrics defined was checked by a human agent. The multiple scopes that are expected to be analysed (components, actions, and sections) in combination to the respective interface technical debt tend to significantly increase the time needed to perform a thorough analysis of the interface. Therefore, it is imperative to automate the current approach. The use of automation in the heuristic methodology brings multiple benefits such as evaluation cost reduction and test coverage maximization [59, 93](ii)Context adaptation: the current approach provides the groundwork in terms of the parametrization of the metrics applied. However, the parameters defined are too generic and must be further refined and adapted to the environment where they will be used. Therefore, user feedback through the evaluation of multiple AAL ecosystems is required to refine the parametrization process and improve their accuracy in the detection of critical usability bottlenecks. Such refinement is expected to be performed through a comparison between the usability bottlenecks identified by the current approach and by end-users

Data Availability

All the data collected within the study scope can be requested to the authors through their email ([email protected] and [email protected]).

Disclosure

A preprint has previously been published [171].

Conflicts of Interest

The authors declare that there is no conflict of interest regarding the publication of this paper.

Acknowledgments

This work was funded by FCT/MCTES through national funds and when applicable cofunded by EU funds under the project UIDB/50008/2020.

References

Eurostat, “Increase in the share of the population aged 65 years or over between 2008 and 2018,” 2019, August 10, 2021, https://ec.europa.eu/eurostat/statistics-explained/index.php/Population_structure_and_ageing.
View at: Google Scholar
United Nations, “World population prospects 2019,” p. 141, 2019, http://www.ncbi.nlm.nih.gov/pubmed/12283219.
View at: Google Scholar
U. K. Age, “Later life in the United Kingdom 2019,” May 2019, https://www.ageuk.org.uk/our-impact/policy-research/policy-positions/.
View at: Google Scholar
The World Bank, Population ages 65 and above (% of total population),, 2017, September 2019, https://data.worldbank.org/indicator/SP.POP.65UP.TO.ZS.
K. Curran, Recent Advances in Ambient Intelligence and Context-Aware Computing, 1^a Edição, IGI Global, 2014.
D. Z. A. V. Gundlapalli and M.-C. Jaulent, MEDINFO 2017: Precision Healthcare Through Informatics: Proceedings of the 16th World Congress on Medical and Health Informatics, IOS Press, 2018, https://books.google.pt/books?hl=pt-PT&lr=&id=xsFVDwAAQBAJ&oi=fnd&pg=PA166&dq=ambient+assisted+living+challenges+security&ots=c-bLrnWAOU&sig=BGFUVQYc-45DEYWjEHDALABInq0&redir_esc=y#v=onepage&q=ambient&f=false.
C. van den Broek, G. Cavallo, and F. Wehrmann, AALIANCE Ambient Assisted Living Roadmap, IOS Press, 2010.
S. T. M. Peek, E. J. M. Wouters, J. van Hoof, K. G. Luijkx, H. R. Boeije, and H. J. M. Vrijhoef, “Factors influencing acceptance of technology for aging in place: a systematic review,” International Journal of Medical Informatics, vol. 83, no. 4, pp. 235–248, 2014.
View at: Publisher Site | Google Scholar
T. Greenhalgh, J. Wherton, P. Sugarhood, S. Hinder, R. Procter, and R. Stones, “What matters to older people with assisted living needs? A phenomenological analysis of the use and non-use of telehealth and telecare,” Social Science & Medicine, vol. 93, pp. 86–94, 2013.
View at: Publisher Site | Google Scholar
P. A. S. Duarte, F. M. Barreto, P. A. C. Aguilar, J. Boudy, R. M. C. Andrade, and W. Viana, “AAL platforms challenges in IoT era: a tertiary study,” in 2018 13th System of Systems Engineering Conference, SoSE, pp. 106–113, Paris, France, 2018.
View at: Publisher Site | Google Scholar
A. Mkpa, J. Chin, and A. Winckles, “Holistic blockchain approach to foster trust, privacy and security in IoT based ambient assisted living environment,” in 2019 15th International Conference on Intelligent Environments (IE), Rabat, Morocco, 2019.
View at: Publisher Site | Google Scholar
A. Ismail and A. Shehab, Security in smart cities: models, applications, and challenges, Springer International Publishing, 2019.
View at: Publisher Site
V. Vimarlund and S. Wass, “Big data, smart homes and ambient assisted living,” Yearbook of Medical Informatics, vol. 9, no. 1, pp. 143–149, 2014.
View at: Publisher Site | Google Scholar
S. T. M. Peek, K. G. Luijkx, M. D. Rijnaard et al., “Older adults' reasons for using technology while aging in place,” Gerontology, vol. 62, no. 2, pp. 226–237, 2016.
View at: Publisher Site | Google Scholar
J. L. Helbostad, B. Vereijken, C. Becker et al., “Mobile health applications to promote active and healthy ageing,” Sensors, vol. 17, no. 3, pp. 1–13, 2017.
View at: Publisher Site | Google Scholar
R. Hoque and G. Sorwar, “Understanding factors influencing the adoption of mHealth by the elderly: an extension of the UTAUT model,” International Journal of Medical Informatics, vol. 101, pp. 75–84, 2017.
View at: Publisher Site | Google Scholar
C. Jaschinski and S. b. Allouch, “An extended view on benefits and barriers of ambient assisted living solutions,” International Journal of Advanced Life Sciences, vol. 7, pp. 40–48, 2015, http://www.iariajournals.org/life_sciences/lifsci_v7_n12_2015_paged.pdf#page=60.
View at: Google Scholar
L. T. Vassli and B. A. Farshchian, “Acceptance of health-related ICT among elderly people living in the community: a systematic review of qualitative evidence,” International Journal of Human Computer Interaction, vol. 34, no. 2, pp. 99–116, 2018.
View at: Publisher Site | Google Scholar
S. Macis, D. Loi, A. Ulgheri et al., “Design and usability assessment of a multi-device SOA-based telecare framework for the elderly,” IEEE Journal of Biomedical and Health Informatics, vol. 24, no. 1, pp. 268–279, 2019.
View at: Publisher Site | Google Scholar
A. I. Martins and M. Cerqueira, “Ambient assisted living – a multi-method data collection approach to evaluate the usability of AAL solutions,” Usability, Accessibility and Ambient Assisted Living. Human–Computer Interaction Series, Springer, Cham.
View at: Publisher Site | Google Scholar
J. D. Hallewell Haslwanter, K. Neureiter, and M. Garschall, “User-centered design in AAL: usage, knowledge of and perceived suitability of methods,” Universal Access in the Information Society, vol. 19, pp. 57–67, 2020.
View at: Publisher Site | Google Scholar
J. D. Hallewell Haslwanter, G. Fitzpatrick, and K. Miesenberger, “Key factors in the engineering process for systems for aging in place contributing to low usability and success,” Journal of Enabling Technologies, vol. 12, no. 4, pp. 186–196, 2018.
View at: Publisher Site | Google Scholar
T. Holthe, L. Halvorsrud, D. Karterud, K. A. Hoel, and A. Lund, “Usability and acceptability of technology for community-dwelling older adults with mild cognitive impairment and dementia: a systematic literature review,” Clinical Interventions in Aging, vol. 13, pp. 863–886, 2018.
View at: Publisher Site | Google Scholar
A. Teixeira, A. Queirós, and N. P. Rocha, Laboratório Vivo de Usabilidade: Living Usability Lab, ARC Publishing, 2013.
J. Pitkänen, Mobile Application Usability Research Case Study of a Video Recording and Annotation Application, Metropolia Ammattikorkeakoulu, 2016.
ISO 9241-11, “Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability,” 1998, April 2016, http://www.iso.org/iso/catalogue_detail.htm?csnumber=16883.
View at: Google Scholar
D. Quiñones, C. Rusu, and V. Rusu, “A methodology to develop usability/user experience heuristics,” Computer Standards & Interfaces, vol. 59, pp. 109–129, 2018.
View at: Publisher Site | Google Scholar
S. Saeed, I. S. Bajwa, and Z. Mahmood, Human factors in software development and design, IGI Global, 2015.
View at: Publisher Site
H. S. Naeini and S. Mostowfi, “Using QUIS as a measurement tool for user satisfaction evaluation (case study: vending machine),” International Journal of Information Science, vol. 5, no. 1, pp. 14–23, 2015.
View at: Publisher Site | Google Scholar
H. Sharp, J. Preece, and Y. Rogers, Interaction Design: Beyond Human-Computer Interaction, Wiley, 5 edition edition, 2019.
S. Lauesen and H. Younessi, Six Styles for Usability Requirements, Refsq, 1988.
J. D. Gould and C. Lewis, “Designing for usability: key principles and what designers think,” Communications of the ACM, vol. 28, no. 3, pp. 300–311, 1985.
View at: Publisher Site | Google Scholar
I. Sciences, Using Tablet Devices to Control Complex Home Appliances Aleksi Lintuvuori, Universidade de Tampere, 2015.
L. A. Suchman, Plans and Situated Actions: The Problem of Human-Machine Communication, Cambridge University Press, Cambridge, 1987.
T. Winograd and F. Flores, Understanding Computers and Cognition: A New Foundation for Design, 61997, Addison-Wesley Professional, 1986.
R. Molich and J. Nielsen, “Improving a human-computer dialogue,” Communications of the ACM, vol. 33, pp. 338–348, 1990.
View at: Publisher Site | Google Scholar
J. Nielsen, “Enhancing the explanatory power of usability heuristics,” in CHI ‘94 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 152–158, Boston Massachusetts USA, 1994.
View at: Publisher Site | Google Scholar
A. P. G. D. Carrare, C. C. Hernandez, C. Kochi, I. F. Silveira, and C. A. Longui, “Usability heuristics for clinical case-based learning assessment systems applied to medical education,” IEEE Latin America Transactions, vol. 13, no. 3, pp. 892–898, 2015.
View at: Publisher Site | Google Scholar
L. Salgado, “Investigating performance characteristics of novice evaluators in heuristic usability evaluations,” Federal University of Lavras (UFLA), 2014.
View at: Google Scholar
L. Constantine, “What do users want? Engineering usability into software,” Windows Tech Journal, vol. 4, no. 12, pp. 30–39, 1995.
View at: Google Scholar
J. G. Powalsa, “Cognitive engineering principles for enhancing human-computer performance,” International Journal of Human Computer Interaction, vol. 8, no. 2, pp. 189–211, 1996.
View at: Publisher Site | Google Scholar
B. Shneiderman, “The Eight Golden Rules of Interface Design,” 2010, September 2019, https://www.cs.umd.edu/users/ben/goldenrules.html.
View at: Google Scholar
E. Oskarsson, Institutionen för Datavetenskap Post-Deployment Usability Opportunities: Gaining User Insight from UX-Related Support Cases Final Thesis Post-Deployment Usability Opportunities: Gaining User Insight from UX-Related Support Cases [M.S. thesis], Linköpings universitet, 2016.
M. MacKenzie, “Fitts’ Law as a Research and DesignTool in Human-Computer Interaction,” Human-Computer-Interaction, Lawrence Erlbaum Associates, vol. 7, pp. 91–139, 1992.
View at: Google Scholar
B. Tognazzini, “First principles of interaction design (revised & expanded),” 2014, April, 2017 http://asktog.com/atc/principles-of-interaction-design/#readability.
View at: Google Scholar
P. A. Silva, K. Holden, and P. Jordan, “Towards a list of heuristics to evaluate smartphone apps targeted at older adults: a study with apps that aim at promoting health and well-being,” in 2015 48th Hawaii International Conference on System Sciences, pp. 3237–3246, Kauai, HI, USA, 2015.
View at: Publisher Site | Google Scholar
R. Inostroza, C. Rusu, S. Roncagliolo, V. Rusu, and C. A. Collazos, “Developing SMASH: a set of smartphone's usability heuristics,” Computer Standards & Interfaces, vol. 43, pp. 40–52, 2016.
View at: Publisher Site | Google Scholar
V. F. Martins, A. G. D. Correa, and M. de Paiva Guimaraes, “Usability inspection of a smart TV,” in 2014 9th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–7, Barcelona, Spain, 2014.
View at: Publisher Site | Google Scholar
A. I. Martins, A. F. Rosa, A. Queirós, A. Silva, and N. P. Rocha, “Definition and validation of the ICF - usability scale,” Procedia Computer Science, vol. 67, pp. 132–139, 2015.
View at: Publisher Site | Google Scholar
J. F. Proença, “Famous people participation in social marketing programs: a research focusing on public health,” Enhancing Knowledge Development in Marketing, vol. 21, p. 516, 2010.
View at: Google Scholar
J. Nielsen, “Severity ratings for usability problems,” 1995, May, 2016 https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/.
View at: Google Scholar
J. S. Dumas and J. Redish, A Practical Guide to Usability Testing, Intellect Books, 1999.
A. I. Martins, A. F. Rosa, A. Queirós, A. Silva, and N. P. Rocha, “European Portuguese validation of the system usability scale (SUS),” Procedia Computer Science, pp. 293–300, 2015.
View at: Publisher Site | Google Scholar
A. I. Martins, A. Queirós, N. P. Rocha, and B. S. Santos, “Avaliação de usabilidade: Uma revisão sistemática da literatura,” Revista Ibérica de Sistemas e Tecnologias de Informação, vol. 2013, no. 11, 2013.
View at: Publisher Site | Google Scholar
B. Martin, B. Hanington, and B. M. Hanington, “Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions,” Choice Reviews Online, vol. 49, no. 10, pp. 49–5403, 2012.
View at: Publisher Site | Google Scholar
R. Molich and J. Nielsen, “Heuristic evaluations and expert reviews,” May, 2016 http://www.usability.gov/how-to-and-tools/methods/heuristic-evaluation.html.
View at: Google Scholar
Federal Aviation Administration, “Heuristic evaluation,” May, 2016 http://www.hf.faa.gov/workbenchtools/default.aspx?rPage=Tooldetails&subCatId=13&toolID=78.
View at: Google Scholar
I. Poltronieri, A. Zorzo, M. Bernardino, B. Medeiros, and M. Campos, “Heuristic evaluation checklist for domain-specific languages,” in Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, pp. 978–989, Vienna, Austria, 2021.
View at: Publisher Site | Google Scholar
M. Bakaev, T. Mamysheva, and M. Gaedke, “Current trends in automating usability evaluation of websites: can you manage what you can’t measure?” in 2016 11th International Forum on Strategic Technology (IFOST), pp. 510–514, Novosibirsk, Russia, 2016.
View at: Publisher Site | Google Scholar
M. Bakaev, V. Khvorostov, and T. Laricheva, “Assessing subjective quality of web interaction with neural network as context of use model,” Communications in Computer and Information Science, Springer, pp. 185–195, 2017.
View at: Publisher Site | Google Scholar
I. Type and B. Chapter, “Evaluation of software usability,” International Encyclopedia of Ergonomics and Human Factors - 3 Volume Set, CRC Press, pp. 1978–1981, 2nd edition, 2006.
View at: Publisher Site | Google Scholar
M. I. L. Lunarejo, P. de Alcântara dos Santos Neto, G. Avelino, and R. de Sousa Britto, “Automatic detection of usability smells in web applications running in mobile devices,” in XVI Brazilian Symposium on Information Systems, Bernardo do Campo, Brazil, 2020.
View at: Publisher Site | Google Scholar
F. Paternò, A. G. Schiavone, and A. Conte, “Customizable automatic detection of bad usability smells in mobile accessed web applications,” in Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, Vienna, Austria, 2017.
View at: Publisher Site | Google Scholar
K. Todi, G. Bailly, L. Leiva, and A. Oulasvirta, “Adapting user interfaces with model-based reinforcement learning,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–13, Yokohama, Japan, 2021.
View at: Publisher Site | Google Scholar
E. Commission, “A digital agenda for Europe,” Communication, vol. 5, p. 42, 2010.
View at: Google Scholar
M. Pădure and S. Independentei, “Exploring the differences between five accessibility evaluation tools Costin Pribeanu Academy of Romanian Scientists,” in RoCHI - International Conference on Human-Computer Interaction, pp. 87–90, University POLITEHNICA of Bucharest, 2019.
View at: Google Scholar
N. Harrati, I. Bouchrika, A. Tari, and A. Ladjailia, “Automating the evaluation of usability remotely for web applications via a model-based approach,” in 2015 First International Conference on New Technologies of Information and Communication (NTIC),, pp. 1–6, Mila, Algeria, 2015.
View at: Publisher Site | Google Scholar
A. M. El-Halees, “Software usability evaluation using opinion mining,” Journal of Software, vol. 9, no. 2, pp. 343–349, 2014.
View at: Publisher Site | Google Scholar
A. Dingli and S. Cassar, “An intelligent framework for website usability,” Advances in Human-Computer Interaction, vol. 2014, Article ID 479286, 13 pages, 2014.
View at: Publisher Site | Google Scholar
S. Feuerstack et al., “Automated usability evaluation during model-based interactive system development,” Lecture Notes in Computer Science, Springer, vol. 5247, pp. 134–141, 2008.
View at: Publisher Site | Google Scholar
J. Vanderdonckt, A. Beirekdar, and M. Noirhomme-Fraiture, “Automated evaluation of web usability and accessibility by guideline review,” Web Engineering, Springer, vol. 3140, pp. 17–30, 2004.
View at: Publisher Site | Google Scholar
A. Beirekdar, J. Vanderdonckt, and M. Noirhomme-Fraiture, “A framework and a language for usability automatic evaluation of web sites by static analysis of HTML source code,” Computer-Aided Design of User Interfaces III, Springer, Netherlands, pp. 337–348, 2002.
View at: Publisher Site | Google Scholar
X. Ferre, E. Villalba, H. Julio, and H. Zhu, “Extending mobile app analytics for usability test logging,” Human-Computer Interaction – INTERACT 2017, Springer, pp. 114–131, 2017.
View at: Publisher Site | Google Scholar
M. Quade, “Automation in model-based usability evaluation of adaptive user interfaces by simulating user interaction,” Tech. Rep., Technische Universitaet Berlin, Germany, 2015.
View at: Publisher Site | Google Scholar
A. Rukshan and A. Baravalle, “Automated usability testing: analysing asia web sites,” pp. 1–13, 2012, https://arxiv.org/abs/1212.1849.
View at: Google Scholar
J. Mifsud and A. Dingli, “Useful: a framework to mainstream web site usability through automated evaluation,” International Journal of Human Computer Interaction, vol. 2, no. 1, 2011.
View at: Google Scholar
K. Muhi, G. Szőke, L. J. Fülöp, R. Ferenc, and Á. Berger, A semi-automatic usability evaluation framework BT-computational science and its applications–ICCSA 2013, Springer, pp. 529–542, 2013.
S. Barra, R. Francese, and M. Risi, “Automating mockup-based usability testing on the mobile device,” Green, Pervasive, and Cloud Computing, Springer, pp. 128–143, 2019.
View at: Publisher Site | Google Scholar
N. Mathur, S. A. Karre, and Y. Raghu Reddy, “Usability evaluation framework for mobile apps using code analysis,” in Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018, pp. 187–192, Christchurch, New Zealand, 2018.
View at: Publisher Site | Google Scholar
J. Grigera, A. Garrido, J. M. Rivero, and G. Rossi, “Automatic detection of usability smells in web applications,” International Journal of Human Computer Studies, vol. 97, pp. 129–148, 2017.
View at: Publisher Site | Google Scholar
J. Grigera, A. Garrido, and G. Rossi, “Kobold: web usability as a service,” in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 990–995, University of Illinois at Urbana-Champaign, Illinois, USA, 2017.
View at: Publisher Site | Google Scholar
G. E. Chanchí Golondrino, D. Pérez Oliveros, and W. Y. Campo Muñoz, “Automation of usability inspections for websites BT-human-computer interaction,” Iberoamerican Workshop on Human-Computer Interaction, Springer, pp. 124–137, 2019.
View at: Google Scholar
N. L. Liyanage and K. Vidanage, “Site-ability: a website usability measurement tool,” in 2016 Sixteenth International Conference on Advances in ICT for Emerging Regions (ICTer), pp. 257–265, Negombo, Sri Lanka, 2017.
View at: Publisher Site | Google Scholar
E. Shawgi and N. A. Noureldien, “Usability measurement model (UMM): a new model for measuring websites usability,” International Journal of Information Science, vol. 5, no. 1, pp. 5–13, 2015.
View at: Publisher Site | Google Scholar
L. B. Ammar and A. Mahfoudhi, “Early usability evaluation in model driven framework,” in Proceedings of the 15th International Conference on Enterprise Information, pp. 23–30, Angers, France, 2013.
View at: Publisher Site | Google Scholar
R. Atterer, “Model-based automatic usability validation: a tool concept for improving web-based UIs,” in Proceedings of the 5th Nordic conference on Human-computer interaction building bridges - NordiCHI '08, pp. 13–22, Lund, Sweden, 2008.
View at: Publisher Site | Google Scholar
A. Deicke, “Masterarbeit automatic usability evaluation of task models,” vol. 1612, 2013, https://pdfs.semanticscholar.org/091a/89972de496ec838f4b1644808771ff9762dc.pdf.
View at: Google Scholar
J. S. Sottet, G. Calvary, J. Coutaz, and J. M. Favre, “A model-driven engineering approach for the usability of plastic user interfaces,” Engineering Interactive Systems, Springer, vol. 4940, pp. 140–157, 2008.
View at: Publisher Site | Google Scholar
B. Shivade and M. Sharma, “Usability analyer tool: a usability evaluation tool for android based mobile application,” International Journal of Emerging Trends & Technology in Computer Science (IJETTCS), vol. 3, no. 3, pp. 247–252, 2014.
View at: Google Scholar
K. M. Malan, J. H. P. Eloff, and J. A. de Bruin, “Semi-automated usability analysis through eye tracking,” South African Computer Journal, vol. 30, pp. 66–84, 2018.
View at: Google Scholar
A. A. Solomon, O. O. Ishaq, B. A. Muhammad, and A. M. Adamu, “Integrated usability evaluation framework for university websites,” i-manager’s Journal on Information Technology, vol. 8, no. 1, p. 40, 2019.
View at: Publisher Site | Google Scholar
J. Xu, X. Ding, K. Huang, and G. Chen, “A pilot study of an inspection framework for automated usability guideline reviews of mobile health applications,” in Proceedings of the Wireless Health 2014 on National Institutes of Health, Bethesda, MD, USA, 2014.
View at: Publisher Site | Google Scholar
Z. Aliyu, L. A. Rahim, and E. E. Mustapha, “A combine usability framework for IMCAT evaluation,” in 2014 International Conference on Computer and Information Sciences (ICCOINS),, pp. 1–5, Kuala Lumpur, Malaysia, 2014.
View at: Publisher Site | Google Scholar
A. Fernández, “A usability inspection method for model-driven web development processes,” p. 371, 2012, http://riunet.upv.es/bitstream/handle/10251/17845/tesisUPV3981.pdf?
View at: Google Scholar
A. Edmonds, “Uzilla: a new tool for web usability testing,” Behavior Research Methods, Instruments, & Computers, vol. 35, no. 2, pp. 194–201, 2003.
View at: Publisher Site | Google Scholar
K. W. Ma, S. Park, and S. Park, “Automatic detection of usability issues on mobile applications,” KIPS Transactions on Software and Data Engineering, vol. 5, no. 7, pp. 319–326, 2016.
View at: Publisher Site | Google Scholar
F. Trojahn, Improving the usability of UI Prototypes by incorporating an automated evaluation in the design process, ResearchGate, 2020.
View at: Publisher Site
І. Гученко, “Mathematical Model for Software Usability Automated Evaluation and Assurance,” Proceedings of the National Aviation University, vol. 49, no. 4, pp. 83–90, 2011.
View at: Publisher Site | Google Scholar
E. Ballarini, L. Ciabattoni, D. Domenichelli, M. Domenichelli, G. Foresi, and A. Monteriù, “An Automated Procedure to Evaluate Usability of eHealth Platforms In The Wild,” in 2019 Zooming Innovation in Consumer Technologies Conference (ZINC), pp. 130-131, Novi Sad, Serbia, 2019.
View at: Publisher Site | Google Scholar
M. Bures, M. Macik, B. Ahmed, V. Rechtberger, and P. Slavik, “Testing the usability and accessibility of smart TV applications using an automated model-based approach,” IEEE Transactions on Consumer Electronics, vol. 66, no. 2, pp. 134–143, 2020.
View at: Publisher Site | Google Scholar
M. J. M. da Ponte and A. M. da Silveira, “A methodology for evaluation the usability of software for industrial automation using artificial neural networks: case study—Eletrobrás,” in 2008 International Conference on Computational Intelligence for Modelling Control & Automation, pp. 430–435, Vienna, Austria, 2008.
View at: Publisher Site | Google Scholar
J. F. Filho, W. Prata, and J. Oliveira, “Affective-ready, contextual and automated usability test for mobile software,” in Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, pp. 638–644, Florence, Italy, 2016.
View at: Publisher Site | Google Scholar
J. F. Filho, T. Valle, and W. Prata, “Automated usability tests for mobile devices through live emotions logging,” in Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, pp. 636–643, Copenhagen, Denmark, 2015.
View at: Publisher Site | Google Scholar
J. Harty, “Finding usability bugs with automated tests,” Communications of the ACM, vol. 54, no. 2, pp. 44–49, 2011.
View at: Publisher Site | Google Scholar
R. F. Ribeiro, M. de Meneses Campanhã Souza, P. A. M. de Oliveira, and P. de Alcântara dos Santos Neto, “Usability problems discovery based on the automatic detection of usability smells,” in Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, pp. 2328–2335, University of Cyprus, Cyprus, 2019.
View at: Publisher Site | Google Scholar
M. Schulz, “Demographic user characteristic sampling for model-based usability evaluation,” in Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, pp. 171–174, Helsinki Finland, 2014.
View at: Publisher Site | Google Scholar
H. Okada and R. Fujiok, “Automated methods for webpage usability & accessibility evaluations,” Advances in Human Computer Interaction, vol. 1, 2008.
View at: Publisher Site | Google Scholar
D. Bader and D. Pagano, “Towards automated detection of mobile usability issues,” in Lecture Notes in Informatics (LNI), Proceedings-Series of the Gesellschaft fur Informatik (GI), pp. 341–354, Germany, 2013.
View at: Google Scholar
Y. Tao, “Automated data collection for usability evaluation in early stages of application development,” in 7th International Conference on Applied Computer & Applied Computational Science (ACACOS ‘08), pp. 135–140, Heraklion, Greece, 2008.
View at: Google Scholar
T. N. Kiprotich, S. Kimani, and W. Cheruiyot, “Remote evaluation of an interface usability using asynchronous auto-logging methodology,” vol. 4, no. 11, pp. 133–136, 2015.
View at: Google Scholar
J. Vanderdonckt and A. Beirekdar, “Automated web evaluation by guideline review,” Journal of Web Engineering, vol. 4, no. 2, pp. 102–117, 2005.
View at: Google Scholar
A. Fourney, R. Mann, and M. Terry, “Characterizing the usability of interactive applications through query log analysis,” Conference on Human Factors in Computing Systems - Proceedings, pp. 1817–1826, 2011.
View at: Publisher Site | Google Scholar
S. Kaur, “An automated tool for web site evaluation,” International Journal of Computer Science and Information Technologies, vol. 3, p. 4, 2012, https://pdfs.semanticscholar.org/7235/8615740da86038e531f8ce1c6c12f024def3.pdf.
View at: Google Scholar
I. Pretel and A. B. Lago, Mobile-human interaction monitoring system BT - mobile lightweight wireless systems, ResearchGate, pp. 198–205, 2012.
L. M. Tobar, P. M. L. Andrés, and E. L. Lapena, “WebA: a tool for the assistance in design and evaluation of websites,” Journal of Universal Computer Science, vol. 14, no. 9, pp. 1496–1512, 2008.
View at: Google Scholar
K. Engelbrecht, M. Kruppa, S. Möller, and M. Quade, “MeMo workbench for semi-automated usability testing Deutsche telekom laboratories, Berlin University of Technology, Germany DAI-Labor, Berlin University of Technology, Germany,” Artificial Intelligence, pp. 1662–1665, 2008.
View at: Google Scholar
F. Lettner and C. Holzmann, “Automated and unsupervised user interaction logging as basis for usability evaluation of mobile applications,” in Proceedings of the 10th International Conference on Advances in Mobile Computing & Multimedia, pp. 118–127, Bali, Indonesia, 2012.
View at: Publisher Site | Google Scholar
S. Baker, F. Au, G. Dobbie, and I. Warren, “Automated usability testing using hui analyzer,” in 19th Australian Conference on Software Engineering (aswec 2008), Perth, WA, Australia, 2008.
View at: Publisher Site | Google Scholar
P. Harms, “Automated usability evaluation of virtual reality applications,” ACM Transactions on Computer-Human Interaction, vol. 26, no. 3, 2019.
View at: Publisher Site | Google Scholar
P. Harms, Automated Field Usability Evaluation Using Generated Task Trees, Institute of Computer Science, University of Göttingen, Gottingen, Germany, 2016, http://hdl.handle.net/11858/00-1735-0000-0028-8684-1.
W. Kluth, K.-H. Krempels, and C. Samsel, “Automated usability testing for mobile applications,” vol. 2, 2014.
View at: Publisher Site | Google Scholar
P. Fabo and R. Ďurikovič, “Automated usability measurement of arbitrary desktop application with eyetracking,” in 2012 16th International Conference on Information Visualisation, Montpellier, France, 2012.
View at: Publisher Site | Google Scholar
F. Lettner and C. Holzmann, “Usability Evaluation Framework: Automated interface analysis for android applications,” in Proceedings of the 13th International Conference on Computer Aided Systems Theory-Volume Part II, pp. 560–567, Las Palmas de Gran Canaria Spain, 2011.
View at: Publisher Site | Google Scholar
L. L. Hurtado, O. D. Castrillón, and G. Olivar, “Una metodología automatizada para la evaluación de usabilidad de interfaces de supervisión industrial,” Informacion Tecnologica, vol. 24, no. 4, pp. 21-22, 2013.
View at: Publisher Site | Google Scholar
M. Abulkhair, A multilingual automated web usability evaluation agent [M.S. thesis], University of Sheffield, 2004.
M. Kiura, M. Ohira, and K.-I. Matsumoto, “Human-Computer Interaction,” New Trends. HCI 2009, Springer, Berlin, Heidelberg, vol. 5610.
View at: Google Scholar
K. Sagar and A. Saha, “Enhancing usability inspection through data-mining techniques: an automated approach for detecting usability problem patterns of academic websites,” Intelligent Human Computer Interaction. IHCI 2016, Springer, Cham, vol. 10127, pp. 229–247, 2017.
View at: Google Scholar
E. M. Pivetta, D. S. Saito, C. da Silva Flor, V. R. Ulbricht, and T. Vanzin, “Automated accessibility evaluation software for authenticated environments,” Universal Access in Human-Computer Interaction. Design for All and Accessibility Practice. UAHCI 2014, Springer, Cham, vol. 8516, pp. 77–88, 2014.
View at: Google Scholar
L. S. Zettlemoyer, R. St Amant, and M. S. Dulberg, “IBOTS: agent control through the user interface,” in International Conference on Intelligent User Interfaces, Proceedings IUI, pp. 31–37, Helsinki, Finland, 1999.
View at: Google Scholar
B. Helfrich and J. A. Landay, QUIP: Quantitative User Interface Profiling, Academia.edu, 1999.
G. Al-Qaimari and D. McRostie, “KALDI: a CAUsE tool for supporting testing and analysis of user interaction,” People and Computers XV—Interaction without Frontiers, Springer, London, pp. 153–169, 2001.
View at: Google Scholar
J. Scholtz, S. Laskowski, and L. Downey, Developing Usability Tools and Techniques for Designing and Testing Web Sites, ResearchGate, 2000.
F. Paternò and G. Ballardin, “RemUSINE: a bridge between empirical and model-based evaluation when evaluators and users are distant,” Interacting with Computers, vol. 13, no. 2, pp. 229–251, 2000.
View at: Publisher Site | Google Scholar
J. W. Thatchew, Evaluation and Repair: LIFT from UsableNet, 2011, January 2021, https://www.jimthatcher.com/lifteval.htm.
L. Galko and M. Bačíková, “Experiments with automated evaluation of domain usability,” in 2016 9th International Conference on Human System Interactions (HSI), Portsmouth, UK, 2016.
View at: Publisher Site | Google Scholar
E. Chang and T. Dillon, “Automated Usability Testing,” Human-Computer Interaction INTERACT ’97, Springer, Boston, MA, 1997.
View at: Publisher Site | Google Scholar
D. Ripalda, C. Guevara, and A. Garrido, “Gestalt prototyping framework - evaluation tool,” Intelligent Human Systems Integration 2021. IHSI 2021, Springer, Cham, vol. 1322, pp. 747–752, 2021.
View at: Publisher Site | Google Scholar
N. Bessghaier, M. Soui, C. Kolski, and M. Chouchane, “On the detection of structural aesthetic defects of android mobile user interfaces with a metrics-based tool,” ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 11, no. 1, pp. 1–27, 2021.
View at: Publisher Site | Google Scholar
R. Yáñez Gómez, D. Cascado Caballero, and J.-L. Sevillano, “Heuristic evaluation on mobile interfaces: a new checklist,” The Scientific World Journal, vol. 2014, Article ID 434326, 19 pages, 2014.
View at: Publisher Site | Google Scholar
J. Sun, “Usability evaluation approach of educational resources software using mixed intelligent optimization,” Mathematical Problems in Engineering, vol. 2017, Article ID 2926904, 12 pages, 2017.
View at: Publisher Site | Google Scholar
M. K. Othman, M. N. S. Sulaiman, and S. Aman, “Heuristic evaluation: comparing generic and specific usability heuristics for identification of usability problems in a living museum mobile guide app,” Advances in Human-Computer Interaction, vol. 2018, Article ID 1518682, 13 pages, 2018.
View at: Publisher Site | Google Scholar
M. Muhanna, A. Masoud, and A. Qusef, “Usability heuristics for evaluating Arabic mobile games,” International Journal of Computer Games Technology, vol. 2022, Article ID 5641486, 7 pages, 2022.
View at: Publisher Site | Google Scholar
M. Virtanen, Applying Test Automation, Turku University of Applied Sciences, 2019.
W. O. Galitz, The Essential Guide to User Interface Design: An Introduction to GUI Design, Wiley, 2002.
A. Harley, “Visibility of system status (usability heuristic #1),” 2018, October 2019, https://www.nngroup.com/articles/visibility-system-status/.
View at: Google Scholar
A. Kaley, “Match between the system and the real world: the 2nd usability heuristic explained,” 2018, October 2019, https://www.nngroup.com/articles/match-system-real-world/.
View at: Google Scholar
J. Nielsen, “Do interface standards stifle design creativity?” 1999, October 2019, https://www.nngroup.com/articles/do-interface-standards-stifle-design-creativity/.
View at: Google Scholar
P. Laubheimer, “Preventing user errors: avoiding unconscious slips,” 2015, October 2019, https://www.nngroup.com/articles/slips/.
View at: Google Scholar
R. Budiu, “Memory recognition and recall in user interfaces,” 2014, October 2019, https://www.nngroup.com/articles/recognition-and-recall/.
View at: Google Scholar
K. Moran, “The characteristics of minimalism in web design,” 2015, October 2019, https://www.nngroup.com/articles/characteristics-minimalism/.
View at: Google Scholar
J. Nielsen, “Error message guidelines,” 2001, October 2019, https://www.nngroup.com/articles/error-message-guidelines/.
View at: Google Scholar
C. Romeiro and P. Araújo, “Guidelines’ parametrization to assess AAL ecosystems’ usability,” ICEIS (2), SCITEPRESS, pp. 309–316, 2022.
View at: Google Scholar
F. K. Mazumder and U. K. Das, “Usability guidelines for usable user interface,” Int J Res Eng Technol, vol. 3, no. 9, pp. 2319–2322, 2014.
View at: Google Scholar
E. P. Rozanski and A. R. Haake, “Human computer interaction,” Prentice Hall, 3rd edition, 2017.
View at: Publisher Site | Google Scholar
B. Shneiderman, C. Plaisant, M. Cohen, and S. Jacobs, Designing the User Interface: Strategies for Effective Human-Computer Interaction, Pearson Education, 2017, https://books.google.pt/books?id=Fvz0DQAAQBAJ.
F. Nayebi, J. M. Desharnais, and A. Abran, “An expert-based framework for evaluating iOS application usability,” in 2013 Joint Conference of the 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement, pp. 147–155, Ankara, Turkey, 2013.
View at: Publisher Site | Google Scholar
K. Whitenton, “Minimize cognitive load to maximize usability,” 2013, October, 2019 https://www.nngroup.com/articles/minimize-cognitive-load/.
View at: Google Scholar
J. Nielsen, “Website response times,” 2010, October 2019 https://www.nngroup.com/articles/website-response-times/.
View at: Google Scholar
G. Rempel, “Defining standards for web page performance in business applications,” in Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, pp. 245–252, Austin, Texas, USA, 2015.
View at: Publisher Site | Google Scholar
T. Kvasnicová, I. Kremeňová, and J. Fabuš, “The use of heuristic method to assess the usability of university website,” Congress Proceedings EUNIS, 2015, http://www.eunis.org/download/2015/papers/EUNIS2015_submission_65.pdf.
View at: Google Scholar
C. Romeiro and P. Araújo, “Guidelines’ parametrization to assess AAL ecosystems’ usability,” in Proceedings of the 24th International Conference on Enterprise Information Systems, pp. 309–316, SciTePress, 2022.
View at: Publisher Site | Google Scholar
J. T. Matthews, J. H. Lingler, G. B. Campbell et al., “Usability of a wearable camera system for dementia family caregivers,” Journal of Healthcare Engineering, vol. 6, 2015.
View at: Publisher Site | Google Scholar
M. Isaković, U. Sedlar, M. Volk, and J. Bešter, “Usability pitfalls of diabetes mhealth apps for the elderly,” Journal Diabetes Research, vol. 2016, article 1604609, 9 pages, 2016.
View at: Publisher Site | Google Scholar
V. S. Ribeiro, A. I. Martins, A. Queirós, A. G. Silva, and N. P. Rocha, “Usability evaluation of a health care application based on IPTV,” Procedia Computer Science, vol. 64, pp. 635–642, 2015.
View at: Publisher Site | Google Scholar
S. Lersilp, S. Putthinoi, P. Lerttrakarnnon, and P. Silsupadol, “Development and usability testing of an emergency alert device for elderly people and people with disabilities,” The Scientific World Journal, vol. 2020, Article ID 5102849, 7 pages, 2020.
View at: Publisher Site | Google Scholar
L. Correia, D. Fuentes, J. Ribeiro et al., “Usability of smartbands by the elderly population in the context of ambient assisted living applications,” Electronics, vol. 10, no. 14, p. 1617, 2021.
View at: Publisher Site | Google Scholar
M. Sili, M. Gira, and C. Mayer, “Usability matters,” Human-Computer Interaction User Interface Design, Development and Multimodality, Springer, pp. 384–394, 2017.
View at: Google Scholar
R. Bastardo, A. I. Martins, J. Pavão, A. G. Silva, and N. P. Rocha, “Methodological quality of user-centered usability evaluation of ambient assisted living solutions: a systematic literature review,” International Journal of Environmental Research and Public Health, vol. 18, no. 21, article 11507, 2021.
View at: Publisher Site | Google Scholar
Z. Agha, C. R. Weir, and Y. Chen, “Usability of telehealth technologies,” International Journal of Telemedicine and Applications, vol. 2013, Article ID 834514, 2 pages, 2013.
View at: Publisher Site | Google Scholar
A. Queirós, A. Silva, J. Alvarelhão, N. P. Rocha, and A. Teixeira, “Usability, accessibility and ambient-assisted living: a systematic literature review,” Universal Access in the Information Society, vol. 14, no. 1, pp. 57–66, 2015.
View at: Publisher Site | Google Scholar
C. Romeiro and P. Araújo, “Usability heuristic evaluation in AAL ecosystems,” KnE Engineering, vol. 5, no. 6, pp. 324–340, 2020.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2022 Carlos Romeiro and Pedro Araújo. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

226

Downloads

367

Citations