Abstract
Given the tremendous potential and influence of artificial intelligence (AI) and algorithmic decision-making (DM), these systems have found wide-ranging applications across diverse fields, including education, business, healthcare industries, government, and justice sectors. While AI and DM offer significant benefits, they also carry the risk of unfavourable outcomes for users and society. As a result, ensuring the safety, reliability, and trustworthiness of these systems becomes crucial. This article aims to provide a comprehensive review of the synergy between AI and DM, focussing on the importance of trustworthiness. The review addresses the following four key questions, guiding readers towards a deeper understanding of this topic: (i) why do we need trustworthy AI? (ii) what are the requirements for trustworthy AI? In line with this second question, the key requirements that establish the trustworthiness of these systems have been explained, including explainability, accountability, robustness, fairness, acceptance of AI, privacy, accuracy, reproducibility, and human agency, and oversight. (iii) how can we have trustworthy data? and (iv) what are the priorities in terms of trustworthy requirements for challenging applications? Regarding this last question, six different applications have been discussed, including trustworthy AI in education, environmental science, 5G-based IoT networks, robotics for architecture, engineering and construction, financial technology, and healthcare. The review emphasises the need to address trustworthiness in AI systems before their deployment in order to achieve the AI goal for good. An example is provided that demonstrates how trustworthy AI can be employed to eliminate bias in human resources management systems. The insights and recommendations presented in this paper will serve as a valuable guide for AI researchers seeking to achieve trustworthiness in their applications.
1. Introduction
Our daily lives have been profoundly transformed by artificial intelligence (AI) and algorithmic decision-making (DM). In this computing society, AI and algorithmic DM influence most of our daily tasks, either directly or indirectly. Nowadays, with the immense available data, advanced algorithms, and high computing power, these systems have become increasingly complex and efficient. Understanding the logic behind these systems can be rather challenging, which explains the critical need to assess them, especially in terms of trustworthiness and reliability. AI systems can be fairly biased due to their constraints, biases, and ethical issues [1–3].
More importantly, data with biases and errors often result in unfair AI systems. For example, in the case of the CalGang database, a highly biased crime dataset with errors was used to train the AI system to predict violent gang-related crimes, resulting in a flawed system. Another example involves the use of the recidivism algorithm to predict the probability of reoffending, resulting in biased DM against dark-skinned people in the US courts [4]. Using a specific algorithm, Amazon was exposed to make recruitment decisions that put candidates at risk of discrimination and gender equity [5]. In addition to data training, continuous data flow is also necessary for these AI systems. This implies the importance of data privacy and governance against malicious activities. One notable example involves the Equifax data breach, which put millions of users at risk of their personal data being illegally exposed [6]. Furthermore, it is not a straightforward process to comprehend the logic of algorithmic DM, given the complexity of the advanced algorithms behind AI systems. As a result, the systems are not fully accepted or trusted. Despite the application and advantages of AI systems for the health industry, their trustworthiness is not widely accepted, further highlighting the issues of accountability in shared decision-making [7, 8].
Addressing all of these concerns, various methods and guidelines have been proposed to ensure the trustworthiness, reliability, and security of AI systems.
The trustworthiness of AI has recently gained the global attention of governments, major organisations, and scientific communities. For example, the International Organisation for Standardisation (ISO), which focusses on technical, industrial, and commercial standardisation, explored the attributes of accountability, controllability, fairness, and transparency of AI and recommended various methods to ensure the trustworthiness of AI [9–11].
Additionally, the European Union (EU) proposed ethical guidelines for trustworthy AI to govern and facilitate the development and operations of these systems [12] and the enforcement of the General Data Protection Regulation (GDPR) that promotes individual rights to explain AI decisions [13]. There are also frameworks that can measure and increase user trust in AI systems (proposed by the National Institute of Standards and Technology, NIST) [14] or promote accountability and responsibility for AI use (proposed by the U.S. Government Accountability Office, GAO) [15]. Furthermore, there is a specific programme known as explainable artificial intelligence (XAI), which was launched by the Defence Advanced Research Project Agency (DARPA) [16, 17]. The programme focusses on making AI systems explainable and trustworthy. Considering the efforts made by these major organisations to ensure the trustworthiness of AI systems, the importance of trustworthiness in ensuring the success and security of AI for users and society is evident [18].
According to Gartner, about 30% of AI-based digital products would need a trustworthy AI framework by 2025 [19]. Moreover, 86% of the users reported their intention to place their trust and loyalty in companies that adopt ethical AI principles. These examples reaffirm the critical need to develop trustworthy AI. Consequently, there are different methods for different phases of the AI lifecycle. Certain methods are proposed to establish a solid foundation of trusting AI requirements and expectations during the design phase. In contrast, other methods are recommended for different phases of data collection, security, and preprocessing to ensure data variation, security, and fairness. There are also the modelling phase, which involves specific methods that ensure the explainability and interpretability of AI, and the implementation and oversight phases, which involve auditing and testing methods that ensure accountability and reliability of AI. According to the EU [12], human involvement is imperative in the development of trustworthy AI. The concept of collaborative intelligence is also recommended, which refers to the collaboration of humans and machines in DM [20]. In total, these methods share a common goal: developing trustworthy AI. AI systems must behave as intended with no unfavourable implications for users and society. Recently, artificial intelligence has been widely used in several applications, including those that are in touch with our lives (see Figure 1) [22]. Therefore, it is critical to make AI trustworthy.
Generally, there are three main factors in building trustworthy AI (see Figure 2): (i) trust in data to ensure that the data are free of bias, accurate, high quality, and privacy-preserving; (ii) trust in the AI model to address issues regarding model explainability, robustness, accuracy, and bias; and (iii) trust in the process to verify the evaluation process, compliance, and consistency. More details of these three factors are presented in the latter sections.
In light of the above, this review looks at the various aspects of trustworthy AI, including four questions and four contributions in response to each question.(1)Q1: Why do we need trustworthy AI?(2)Contribution 1: the reasons for the need for trustworthy AI have been explained(3)Q2: What are the requirements for a trustworthy AI?(4)Contribution 2: the key requirements that establish the trustworthiness of AI systems have been explained, including explainability, accountability, fairness, robustness, acceptance of AI, privacy, accuracy, reproducibility, human agency, and oversight(5)Q3: How can we have trustworthy data?(6)Contribution 3: the challenges of the key requirements for preparing trusted data for AI, including high-quality data, privacy, and free of bias, have been explained.(7)Q4: What are the priorities in terms of trustworthy requirements for some applications?(8)Contribution 4: some guidelines are summarised for the prioritisation of different applications, including trustworthy AI in education, environmental science, 5G-based IoT networks, robotics for architecture, engineering and construction, financial technology, and healthcare
2. Search Strategy
We reviewed significant research papers in the field published during 2018–2023, particularly those from 2021, 2022, and 2023. Our comprehensive search was mainly performed in Science Direct (SD), Scopus, IEEE Xplore, and Web of Science (WoS), including the popular publishers IEEE, Elsevier, MDPI, Nature, ACM, and Springer. Some papers have been chosen from ArXiv. The selection of articles was based on the trustworthiness of artificial intelligence (AI) in different areas. Keywords were selected based on the recommendations of experts in the field of AI to answer all questions. Therefore, keywords related to applying trustworthy components in AI were specified as (“Trustworthy”) AND (“Artificial Intelligence”), (“Trustworthy Artificial Intelligence”) AND (“explainability”), (“Trustworthy Artificial Intelligence”) AND (“accountability”), (“Trustworthy Artificial Intelligence”) AND (“fairness”), (“Trustworthy Artificial Intelligence”) AND (“acceptance of AI”), (“Trustworthy Artificial Intelligence”) AND (“privacy”), (“Trustworthy Artificial Intelligence”) AND (“accuracy”), (“Trustworthy Artificial Intelligence”) AND (“reproducibility”), (“Trustworthy Artificial Intelligence”) AND (“education”), (“Trustworthy Artificial Intelligence”) AND (“environmental science”), (“Trustworthy Artificial Intelligence”) AND (“5G-based IoT networks”), (“Trustworthy Artificial Intelligence”) AND (“robotics for architecture”) AND (“engineering”) AND (“construction”), (“Trustworthy Artificial Intelligence”) AND (“healthcare”), (“Trustworthy Artificial Intelligence”) AND (“financial technology”), (“Trustworthy Artificial Intelligence”) AND (“deep learning”), and Figure 3 depicts the search strategy.
3. Comprehensive Examination of Scientific Mapping
Due to increased contributions and applied research, it has become more challenging to identify crucial evidence discovered in previous studies. Keeping up with the literature has proved challenging due to the constant flux of practical and theoretical contributions. PRISMA is a technique that a number of academic specialists have proposed to reorganise previous studies’ results, summarise problems, and identify future research gaps. In contrast, this paper presents a review that expands the scope of the knowledge base, strengthens the research strategy, and synthesises the literature’s findings. However, despite their widespread use, evaluations continue to be beset by issues of reliability and objectivity. This is because such methodologies rely on the authors’ perspective to reorganise the results of previous investigations. Several works have provided techniques for conducting an R-tool and a VOSviewer-based comprehensive scientific mapping analysis that is more suitable. The objective of these approaches is to promote openness in summarising the findings of previous investigations. The bibliometric technique yields conclusive results, identifies voids in the study, and draws conclusions about the findings of the corpus of literature in a highly dependable and transparent manner. In addition, the tools presented do not require high expertise and are considered open source. Consequently, the bibliometric methodology, which is described in detail in the following, was chosen for this investigation.
3.1. Annual Scientific Production
There has been significant progress in developing trustworthy artificial intelligence in the last 10 years. The yearly scientific output shown in Figure 4 explains the productivity of previous theoretical and practical investigations on reliable artificial intelligence. A limited number of articles were published during the first period from 2012 to 2015. Nevertheless, there has been a significant increase in published articles in recent years. The number of articles published increased markedly from 2017 to 2018, with a respective increase of eight to 27 articles per year. The number of articles saw a consistent upward trend in 2019 and 2020, culminating in a notable surge to 50 articles in 2020. The aforementioned trend remained consistent throughout 2021 and 2022, with 101 and 147 articles produced, respectively. The year 2023 is in its early stages, with a limited number of articles, namely eight, having been published so far. The overall pattern suggests a rise in the number of reputable research articles on artificial intelligence that have been published.
3.2. Three-Field Plot
The three-field plot is a common visualisation approach to show data with three separate parameters. The left field relates to sources (SO), the centre field to titles (TI_TM), and the right field to keywords (ID) in the specified situation. The graphic is often used to examine the correlations between the three parameters, as shown in Figure 5. According to the study title (TI_TM) in Figure 5, the Journals of Information Fusion, Automation Construction, and Nature have the most citations from the sources (SO) on the left side. Furthermore, the Information Fusion journal holds a significant place among the several publications that explicitly focus on the issue of dependable and understandable artificial intelligence (AI). Furthermore, the relevant domain (ID) acknowledges that the most frequently used phrases, such as “artificial intelligence,” “trust,” “robotics,” “explainability,” “information fusion,” and “explainability,” are consistently found in the journals indicated in the intermediate field (TI_TM).
3.3. Word Cloud
In prior studies, word cloud facilitated the identification of prevalent and significant phrases. This study aims to present and discuss critical findings from previous research, focussing on essential keywords. The purpose is to provide a concise summary and reorganise the material gathered. Figure 6 displays keywords of different widths. The considerable size of the keywords implies a higher frequency of their occurrence in the literature. In contrast, the diminutive dimensions of terms imply a lower frequency of occurrence. Based on the frequency distribution shown in Figure 6, prominent topics within the domain of trustworthy artificial intelligence (AI) include artificial intelligence, with an important emphasis on trustworthiness. Furthermore, the graphic illustrates that surveys and reviews are fundamental topics within the field. Topics such as ethics, privacy, and security often emerge, underscoring the need to consider these aspects throughout the development and deployment of AI systems. Figure 6 illustrates the many applications of artificial intelligence (AI) in several domains, such as healthcare care, decision-making processes, and information fusion. Additionally, this encompasses artificial intelligence methodologies such as convolutional neural networks (CNNs), natural language processing, and robotic systems. In general, the word cloud generated from trustworthy AI publications demonstrates the diverse nature of this field, including a broad range of topics. These include technical aspects of AI and its ethical, legal, and social implications.
3.4. Cooccurrence
A cooccurrence network is another tool used in bibliometric research. A semantic network is utilised in prior research to identify and examine widely used phrases, as well as in future analysis. This network provides professionals, policymakers, and researchers with vital insights into the underlying conceptual framework of a specific field of expertise. Figure 7 depicts essential information about a cooccurrence network built from the names of credible articles on trustworthy artificial intelligence (AI). The network is made up of nodes that represent the individual words in the titles. The edges of the nodes represent the frequency with which these words occur in the same title. Figure 7 depicts a set of nodes, as well as their clusters and proximity centrality values, which measure the degree of interconnection between a node and the other nodes in the network. The observation demonstrates that the nodes are divided into eight unique clusters, with the phrases within each cluster displaying a thematic or conceptual relationship with reliable artificial intelligence. Terms in Cluster 1 include “machine,” “deep,” “models,” “trust,” “adversarial,” “privacy,” security,’ “detection,” and “classification,” suggesting a possible relationship with the development and implementation of reliable artificial intelligence systems. Artificial intelligence, trustworthiness, review, explainability, ethics, future, technology, opportunity, acceptability, health, industry, and responsibility are all words found in Cluster 2. These phrases suggest that the cluster is concerned with the ethical and explicable aspects of artificial intelligence. Similarly, additional clusters are associated with concepts such as “bias,” “analysis,” “fairness,” “construction,” “risk,” and “safety.” The closeness of a node in a network determines its centrality, which may be taken as a measure of its importance within the network. Words with higher proximity values have stronger links to other nodes in the network, indicating their greater importance with regard to the topic of trustworthy artificial intelligence. In general, the figure depicts the relationships between numerous concepts and phrases linked with the concept of dependable artificial intelligence, as evidenced in the titles of scientific works on this topic. The information provided may be useful in understanding the current level of research on this topic and in identifying areas that need more research.
4. Related Work
In this section, we provide a comprehensive summary of the most recent evaluations and analyses conducted on articles related to the concept of trustworthy AI. These reviews represent a systematic effort to evaluate and critique the body of literature surrounding the development and deployment of AI systems with a focus on their trustworthiness, ethics, and reliability.
Nazir et al. [23] presented a survey of explainable DL methods for efficient and robust pattern recognition, categorising them into three main areas: explainable DL methods, efficient DL through model compression and acceleration, and robustness and stability in DL. The authors of [24] examined the Trustworthy Artificial Intelligence (ALTAI) Assessment List, including its advantages and disadvantages. Additionally, they evaluated the tool’s ability to assist the industry in understanding the potential risks associated with AI systems and implementing effective strategies to mitigate them. The importance of integrating research and methodologies from fields such as environmental sustainability, social justice, and corporate governance (ESG) is underscored in addressing analogous challenges in the ethical advancement and implementation of artificial intelligence (AI). Furthermore, this research investigates the prospective efficacy of the instrument with respect to its adoption within the sector, considering several factors that could impact its level of acceptability. The authors of [25] further guide professionals in the development of transparent artificial intelligence (AI) systems in healthcare. Moreover, their contributions serve to formalise the field of explainable artificial intelligence. The authors propose that the justification for seeking explainability is crucial in identifying the particular elements that need clarification. Consequently, this factor affects the relative importance of the characteristics related to explainability, including interpretability and integrity. The author introduced a theoretical framework to enhance the decision-making process by selecting several types of explainable artificial intelligence (AI) approaches. The categories mentioned above include the distinction between explainable modelling and post hoc explanation, as well as the many types of explanations, such as model-based, attribution-based, and example-based. Furthermore, the theory posits a distinction between global and local explanations.
The evaluation conducted by [26] offers a comprehensive examination of the latest developments in constructing an AI system with trustworthiness and explainability. This approach emphasises the intrinsic lack of transparency in artificial intelligence (AI), which limits our understanding of its underlying architecture. The article also explores several aspects of trustworthy AI, including its inherent biases and tendencies that undermine its dependability. This study explores the need to incorporate trustworthy AI into several industries, including banking, healthcare, autonomous systems, and the Internet of things (IoTs). This method enables the incorporation of trust-building tactics in several areas, including, but not limited to, data security, pricing, expenditure management, dependability, assurance, and decision-making processes. Trusted artificial intelligence is used across diverse domains and exhibits various degrees of adoption. Furthermore, it emphasises the need to use transparent and retrospective explanatory models throughout creating an explainable artificial intelligence (XAI) system. Additionally, this study outlines the potential constraints and obstacles that are likely to arise with respect to the development of explainable AI. This report provides a complete analysis of regulations related to the advancement of trustworthy AI in the autonomous car manufacturing industry. This paper explores several methodologies for developing artificial intelligence (AI) systems with reliability, interpretability, explainability, and trustworthiness. The overarching objective is to guarantee the safety of autonomous vehicle systems.
Numerous criteria, including fairness, explainability, accountability, dependability, and acceptability, have been proposed to increase the credibility of artificial intelligence systems. The survey [18] uses a methodology grounded in existing research to assess various requirements. It thoroughly analyses several techniques that try to mitigate the possible risks connected with artificial intelligence and improve the levels of trust and acceptance among users and society. The paper also discusses the analysis of existing approaches used in validating and verifying these systems, along with the continuous efforts in standardisation to guarantee the dependability and trustworthiness of artificial intelligence. The evaluation of confidence in artificial intelligence and robotics is of significant importance as these technologies continue to gain prominence within the architecture, engineering, and construction (AEC) sector. The publication [27] offers a complete review of the results obtained from an extensive analysis of the academic literature published over the last two decades. The study focusses on two main domains: (1) the notion of trust in relation to artificial intelligence (AI) and AI-driven robotics and (2) the many implementations of AI and robots in the architecture, engineering, and construction (AEC) sector. Also, analysis research thoroughly examines the shared characteristics of trust and assesses their applicability to current applications within the architecture, engineering, and construction (AEC) sector. The results indicate an increasing interest among scholars and professionals in the academic and industrial sectors of the architecture, engineering, and construction (AEC) domain with regard to investigating and using artificial intelligence (AI) and robotic technologies. However, extensive research that methodically analyses essential trust elements, such as explainability, dependability, robustness, performance, and safety, is still needed, particularly within the architecture, engineering, and construction (AEC) domain. On the contrary, the authors of [28] comprehensively analyse explainable and interpretable machine learning techniques in various healthcare fields. They also analysed the ethical implications of using ML and DL in healthcare. They also looked at the security, safety, and robustness problems that make ML less reliable. Furthermore, the study aims to comprehensively analyse how the incorporation of explainable and trustworthy machine learning (ML) methods might successfully tackle the ethical predicaments outlined earlier. The authors of [29] conducted a comprehensive and rigorous examination of previous findings, establishing a foundation for potential future investigations. The researchers thoroughly analysed the issues underlying the factors and proposed solutions related to the topic. The present study used a rigorous scientific mapping analysis to reorganise and integrate previous research results to resolve apprehensions about reliability and impartiality. Furthermore, the research has provided solid empirical data to substantiate the dependability of artificial intelligence (AI) in healthcare care. Addressing essential research gaps in this topic has been facilitated by presenting eight modern critical appraisals. The ubiquity of artificial intelligence (AI) in various aspects of human life has led AI systems to play a pivotal role in the digital economy [30]. From the description, it seems that the map primarily focusses on the “trustworthy AI.”
Table 1 presents a comparison between our survey and a more recent survey in terms of the requirements for trustworthiness and applications. Each of the articles mentioned focusses on a single or some requirement of trustworthiness, leaving a gap in the complete picture. This paper aims to present all the necessary requirements for trustworthiness, allowing researchers to have a comprehensive understanding and apply it to their AI applications. The primary goal of this paper is to synthesise and unify the various research efforts represented by the individual nodes on the map into a cohesive and interconnected understanding of trustworthy AI.
5. The Need for Trustworthy AI
Every person has a basic form of AI at home, for example, smart digital assistants, such as Siri, Alexa, and Google Assistant, robotic vacuums, or facial recognition [34]. This field has experienced revolutionary advancements, especially in the domains of machine learning and deep learning (DL), which were introduced in the early 2010s.
The tremendous prospects of AI have significantly expanded our imagination of reality with intelligent agents, contributing to prosperity and well-being at all levels. Despite that, similar to other technologies, AI systems come with ethical, legal, and social concerns [35]. Recognising the challenges of AI, there have been calls for “beneficial AI” [36], “responsible AI” [37], or “ethical AI” [38] in recent years. Various terminologies describe AI, but these terms reflect the same purpose, namely, to advance AI by maximising its benefits and minimising or preventing its adverse risks.
In early 2019, an independent organisation, specifically known as the high-level expert group on Artificial Intelligence of the European Commission, published the “Ethical Guidelines for Trustworthy AI” [39], which has attracted a great deal of interest from researchers and practitioners. These guidelines have also established a solid base for the use of the term “trustworthy AI” in existing guidelines and frameworks, such as the AI principles of the White House [40]. In particular, trust serves as a pivotal foundation for the economy of society and the country and sustainable development. The full potential of AI systems can be reached through a trusted AI foundation [41].
Defining and realising trustworthy AI is challenging. As a highly interdisciplinary and dynamic field, trustworthy AI covers various disciplines, ranging from computer science, economics, and management to sociology and psychology. There are varying views and understandings of what trustworthy AI is and different priorities on the ethical and regulatory criteria of trustworthy AI. Moreover, the technical and nontechnical aspects of trustworthy AI continue to evolve over time. As the complex phenomenon of trust alone has continued to spark scholarly debates in recent years, the conceptualisation of trustworthy AI and what constitutes trust in AI have remained inconclusive in both theory and practice. Currently, AI, especially DL, is involved in many applications such as education, environmental science, 5G-based IoT networks, robotics for architecture, engineering, and construction, banking, and healthcare [27, 42–44]. It is possible that the AI applications go wrong. Therefore, it is important to pay attention to how to make AI trustworthy. For example, in Australia, Robodebt, which is the government’s automated welfare debt recovery process, led to the catastrophic failure to collect millions from welfare recipients [45]. Another example is that in July 2022, a robot broke the finger of a 7-year-old boy while playing chess, which is a red flag that reliability and safety are needed in AI [46]. There are more examples in which uncontrolled AI threatens enterprises (see Figure 8).
The beauty of AI is that if there is a bias, for example, then it can be fixed by adjusting the training data in terms of diversity, while it is hard to fix with humans. Therefore, it is critical to address these trustworthiness issues in the form of bias, hatred, and the propagation of bad ideas before transitioning from research to deployment. As a result, the goal is to have good AI systems capable of improving the quality of life.
6. Trustworthy AI Requirements
In this section, we will discuss the requirements for developing trustworthy AI applications (see Figure 9).
6.1. Explainability
When DM is more dependent on AI algorithms and systems, it is crucial that these decisions are explicable and can be trusted by different stakeholders. However, due to the inherent complexity of AI, it has become increasingly difficult to explain machine-learned decisions. Explainability should reflect the logic of the decisions made by AI systems and support system transparency and interpretability, leading to system improvement and better system governance [17, 33, 47–49].
An AI system that displays explainability facilitates the detection of errors or vulnerabilities and supports the establishment of the trustworthiness of the system [29, 50]. Users have the right to receive explanations about the results of an AI system, such as the thought process involved in the system to generate a particular result, the types of training data used by the system, and the metrics used to assess the validity and reliability of system results [51–53]. Additionally, an AI system should be able to provide appropriate explanations to users with different backgrounds, different expertise, and application requirements [54]. Users who clearly understand why an AI system produces a certain result are more likely to have a higher level of trust in the decision of the system [55]. It is important to understand that there are many forms of explanation that are driven by different purposes and to cater to different types of users, depending on their expertise and application requirements [56]. As a result, this has led to different levels of system interpretability, such as global interpretability and local interpretability [57, 58]. The methods of global interpretability focus on elucidating the entire logic and operation of an AI system. In other words, it provides an overview of the system in generating an explanation of the system’s outputs and reasoning. This level of interpretability is used primarily to predict global trends, such as climate change [59], which can be rather challenging in terms of practise due to the scale involved. Meanwhile, local interpretability methods, which focus on elucidating specific decisions made by an AI system, are more commonly applied. The level of interpretability is more instantaneous than the global interpretability and distinguished according to when and for which input data the explanations are provided to the users [60, 61]. Accordingly, there are ex ante and ex post explanations under local interpretability [62]. With the purpose of establishing trust in AI systems, ex ante explanations elucidate the features, use, and operation of the systems prior to the actual use. These explanations imply that the systems are well-designed, tested, and validated. On the other hand, ex post explanations elucidate the features and conditions that result in the decisions. These explanations are provided after the decisions are revealed, validating the previous ex ante explanations [63]. As stipulated by ISO [9], ex ante explanations and ex post explanations are critical aspects of explainability for trustworthy AI in terms of transparency and interpretability.
Most of the methods to ensure explainability cater to the needs of system designers and developers for the purposes of debugging and oversight [64]. More appropriate methods are needed to support nonexpert users in dealing with the gap between transparency and actual implementation [65]. Additionally, organisations often have security and privacy concerns, resulting in a reluctance to adopt AI systems. Therefore, appropriate methods that take into account the explainability of AI systems should be developed without compromising users’ privacy concerns and the security of these systems [66–69].
6.2. Accountability
How algorithmic DM is developed and operated must be monitored to avoid any unfavourable events or outcomes. These algorithms are simply computer programmes that are trained using data. Therefore, those involved in algorithmic DM are responsible for any adverse events or outcomes caused by the algorithms [70–72]. Wieringa [73] defined accountability as a networked account that assigns the responsibilities of stakeholders at different phases of the AI life cycle. Basically, algorithmic accountability involves evaluating algorithms based on relevant parameters and assigning responsibilities to those involved in the development of the algorithms.
The increasing use of algorithmic DM, especially in high-stake applications, has contributed to the critical need for accountability measures. The design, development, and application of these algorithmic models must be reliable and secure. System failure leads to irreversible losses and damages. For instance, the computer software of an aircraft had significant glitches, which contributed to the crash of a Boeing plane and the lives of 346 people [74]. Volkswagen’s electric car software had critical software architecture issues [75]. In another case, a face recognition system was found to be biased, putting women and dark-skinned people at a disadvantage [76]. All these failures can be prevented if only these algorithms are monitored accordingly. However, the question of who should be responsible for these failures remains: should it be the system developers, the ones who were in charge of data collection, or the system users who had the expertise to use the system? There is no conclusive answer to this question, which highlights the need to have a proper mechanism to establish accountability [77, 78].
There are different mechanisms to establish accountability of algorithmic DM, such as the incorporation of specific methods into the algorithm design process, the application of transparency methods, and the enforcement of strict laws and policies for better governance of algorithms. As an incremental process [79, 80], algorithms’ accountability needs proper governance of the AI lifecycle and collaboration between different stakeholders [81]. Despite that, it is not a simple process to identify who should be responsible for system failure or to determine who exactly causes the said error since the development of algorithms involves different stakeholders. Appropriate accountability measures must be established based on the application domain since there is no universal measure applicable to all domains. For better governance, the development of context- and application-dependent accountability measures is proposed [9]. In line with that, ISO highlighted the use of medical AI systems and AI-based recruiting systems. In the case of medical AI systems, doctors who are the system users are responsible for any harm caused by the system, since they are the domain experts. More importantly, the system should only be used to help doctors in their DM process. Meanwhile, in the case of an AI-based recruiting system, the system users are not responsible for any harm caused by the system, since they are not in a position to know why their application is unsuccessful. This explains why context-dependent accountability measures should be further explored [82].
6.3. Fairness
AI systems and algorithms rely on a tremendous amount of data and logic to execute a particular task and facilitate the DM process. Considering the substantial influence of AI systems in daily tasks and operations, it is important that these systems are not biased. A fair system does not discriminate against any individual or member of society [83]. The principle of fairness is in line with the concepts of ethics and moral values [84–88].
An unfair system in terms of its design, development, application, and monitoring is detrimental. There are several examples of unfair AI systems or algorithms. For instance, a judicial system was found to rely on an unfair risk assessment tool that discriminates against dark-skinned people [89]. In another case, a major tech firm was exposed to relying on an unfair hiring algorithm that discriminates against women [90]. Certain studies demonstrated discrimination against underprivileged individuals or of certain races or ethnicities in the DM process of child maltreatment screening through the use of an unfair predictive analytic tool [91].
There are various factors that affect the trustworthiness of AI, such as data bias, model bias, and evaluation bias. With the critical need for fairness in AI, studies have proposed various definitions of fairness, but there is no conclusive definition for the term “fairness” in AI. Some previous studies compared and discussed these definitions in detail [85]. In general, how fairness in AI is defined depends on the context, specifically how the AI is applied. There are two broad definitions of fairness in AI: individual fairness and group fairness [85].
First, individual fairness makes sure that individuals of the same group receive similar predictions [92]. The definition of fairness under individual fairness is related to fairness through awareness [92] or unawareness [93], as well as counterfactual fairness [94, 95]. On the other hand, group fairness ensures that all groups of society receive equal treatment [96]. The definition of fairness under group fairness is linked to demographic parity [95], equalised odds [97], equal opportunity [98], and conditional statistical parity [99].
Apart from the various definitions of fairness, there are numerous methods available to establish fairness in AI. However, it is not a straightforward process to identify a single definition and method to detect all forms of bias. Feuerriegel et al. [100] recommended the need for more studies to explore the definitions and perceptions of fairness, particularly in relation to AI applications: certain AI applications may be more sensitive to certain elements than other applications. It is imperative to establish frameworks and policies that clearly define fairness based on the context of applications. Furthermore, different stakeholders view fairness differently, which suggests the need to involve various stakeholders to ensure the trustworthiness of AI. It is also important to have more robust testing methods and measures to identify and eliminate various forms of system bias [101].
6.4. Robustness
According to the EU’s ethical guidelines for trustworthy AI, robustness is one of the three essential criteria for building trustworthy AI systems [102]. IEEE defines the notion of robustness as “the degree to which a system or component can function correctly in the presence of invalid input or stressful environmental conditions [103].” This suggests that robust AI systems should be resilient against variations in input data or the external environment. For instance, in [104], the authors define the robustness of a deep neural network as “the integrity of the network under varying operating conditions, and the accuracy of its outputs in the presence/absence of input or network alterations,” and propose that robustness is divided into two subproperties: security against possible attacks and reliability as resistance to environmental changes.
The majority of existing research on robustness in AI systems has focused on adversarial robustness, i.e., the ability to resist adversarial attacks [105]. Consequently, the term (adversarial robustness) has often been used interchangeably with (robustness) in the literature. The purpose of an adversarial attack is to mislead an AI system into making an incorrect prediction by injecting into the system deliberately altered or manipulated input, known as adversarial examples [106]. These are a subset of perturbed examples and, unlike common perturbations, are generated to exploit weaknesses in the decision limits of the models [105]. For instance, adding a small perturbation to an input image results in a wrong decision (see the two examples [107, 108] as shown in Figure 10). It is critical to consider the issue of adversarial attack when it comes to building an AI system [109, 110].
(a)
(b)
Since adversarial examples can be used to test the robustness of an AI system, adversarial robustness is also considered the “worst-case robustness” [111, 112] and can be addressed as an optimisation problem [113]. In [114], the authors generate adversarial instances to attack a predictive model and assess the model’s robustness by calculating the error rate of the model’s prediction. In [111], the authors develop algorithms to search for the smallest additive distortions in the input space that are sufficient to confuse a classifier. In [109], the authors propose three major categories of robust optimisation methods, which are adversarial training to train predictive models using adversarial examples, certified defenses against norm-bounded adversarial examples, and regularisation approach to reduce the effects of perturbation on model predictions.
In recent years, nonadversarial robustness, which refers to “preserving model performance under naturally induced corruptions or alterations” in model inputs, has also drawn attention [115, 116]. Typical examples of corruption conditions include varying noise, blur, weather, and rendering conditions. In [117], the authors consider two main forms of nonadversarial robustness, known as corruption robustness and perturbation robustness, and propose benchmarks that evaluate model performance on common corruptions and perturbations for computer vision. In [118], the authors propose a set of corruption categories and specify a robustness score to measure the ratio of a model’s corruption accuracy to its accuracy on clean input images. In speech recognition research, robustness to common corruptions (such as street noise, background chatter, and wind) is often more emphasized than adversarial audio since common corruptions are always present and remain unsolved [119].
In contrast to research on robust models for image and text data, robustness in AI systems that use structured data as input is underexplored. A few existing studies include detecting imbalanced and fraudulent tabular data using adversarial attacks to improve model robustness [120], adversarial training of predictive models that take time series data as input [121], and generating more event log data for predictive models using generative adversarial nets [122].
6.5. Acceptance of AI
Trustworthiness is a must when it comes to AI, given the growing reliance on AI-based DM systems in our daily tasks. System failure leads to lower acceptance and a lack of trust in AI by system users. AI systems must be carefully evaluated using a specific mechanism in order to promote users’ acceptance of AI and to gain trust in AI-based DM systems [123, 124]. Previous studies [125] identified several factors that help build trust, including system performance, type of task, type of application, human component and explainability of the system, and human involvement, which was identified as a crucial factor that can boost user confidence in AI systems and promote accountability for their decisions [125]. A few other prior studies [126] recommended having separate governance laws to promote the acceptance of AI. These recommendations share a similar purpose, to promote users’ acceptance of AI through system evaluation. There have been efforts made to develop mechanisms to improve the acceptance of AI.
Furthermore, there are different expectations from different users of AI systems. Undoubtedly, the lack of information and the understanding of AI systems result in overstated or understated assumptions of systems in terms of system performance, usability, reliability, and fairness. Unfulfilled expectations lead to lower acceptance of AI systems. Some previous studies [127] demonstrated the low acceptance of AI systems by users despite the benefits of these systems and linked the mistrust of users in the systems with the lack of empathy and morality of the systems. Therefore, it is important to have a specific tool or mechanism for potential users to evaluate AI systems based on their trustworthy requirements, ethical principles, and expectations [128, 129].
6.6. Privacy
AI-based DM systems involve a substantial amount of training data for their DM process. The amount of training data influences the accuracy and performance of these systems. However, there are implications to consider when it comes to the availability and application of data. The misuse of data by unscrupulous individuals, private companies, or governments has negative implications. For instance, a government misused the personal data of the citizens, resulting in inaccurate debt assessment [130]. In another case, the personal data of 50 million users of a social networking platform were collected and shared without their consent and misused to manipulate the US presidential elections [131]. These examples demonstrate the importance of ensuring data privacy, especially in gaining users’ trust in systems [132].
Users are more likely to trust a system that applies the necessary mechanisms to protect their personal data and identity. The increased availability of data is linked to more cases of data breaches. For example, the Equifax data breach exposed the personal data of millions of users [133], and the details of credit and debit cards of 40 million Target customers were hacked [134]. Through such data breaches, a tremendous amount of sensitive information details are exposed and exploited. Internal attacks or targeted attacks lead to data breaches, resulting in system collapse and users’ mistrust of AI. Therefore, securing the privacy of the AI systems and users establishes trustworthy systems [135]. There are different phases (e.g., data collection, preprocessing, modelling, and implementation phases) of AI systems that are subjected to different privacy risks. For example, privacy risks associated with the data collection phase may lie in the gathered data or data storage. The inability to distinguish sensitive data from nonsensitive data during the preprocessing and modelling phases compromises privacy [136]. Frequently querying the model results in one’s familiarisation with the internal functioning of these systems, which poses another form of threat that puts privacy at risk [137].
Studies have recommended various measures to enhance privacy, such as deidentification, federated learning techniques, and data agreements. These measures have their own strengths and limitations, depending on the requirements of the applications. How these measures are selected depends on the trade-offs between the performance and privacy-preserving overhead [138]. Despite the availability of these measures, AI systems are still susceptible to privacy risks. Appropriate laws are needed for AI systems to serve society and ensure system privacy, which can be challenging to accomplish both purposes at the same time. For instance, Rosenquist [139] highlighted the benefits of AI systems in rescuing exploited children. Still, the measure taken compromised the privacy of users—all images posted on the social media platform were subjected to facial recognition analysis, which is not possible for humans to perform. Considering that, the need for context-based privacy laws that clearly define the requirements and conditions that allow such measures to be taken is evident [140, 141].
6.7. Accuracy
Accuracy reflects the robustness of AI systems. A system that displays a certain level of accuracy yields reliable DM. An accurate AI system can deal with errors and problems and yields reproducible output with the same input under the example, a feature squeeze recommended various methods to ensure the input space-making items. For instance, a feature that squeezes problems reduces the complexity of the input space, making it less likely to be subjected to system problems [142]. Another proposed method involves incorporating problematic case examples into the system’s training data [143, 144].
6.8. Reproducibility
With the same input parameters and conditions, a trustworthy system produces the same decisions, demonstrating its reproducibility [145, 146]. To quantify the state of measureity of empirical AI research, there are several reproducibility metrics measuring different degrees of reproducibility, including the hypotheses and how they were documented. Then, the method was explained in different aspects, including experiment, data, method, and new findings [147–149].
6.9. Human Agency and Oversight
Most importantly, AI systems must be subjected to the control of users as these systems continue to support and assist humans in their tasks [150, 151]. The control over these systems depends on the risks and implications of poor decisions on the users and society. The element of human agency and oversight ensures the involvement of humans in the DM process according to the risks, as well as social and environmental implications [152, 153].
7. Trustworthy Data for AI
One of the prominent issues in algorithmic DM lies in its data, specifically its availability and quality [154, 155]. However, the scope and potential use of data today are different. The significant amount of data contributes to the feasibility of AI [156]. An AI system can learn better and improve accuracy when supplemented with more data [157]. Advanced integration technologies are used to deal with interoperability limitations [158]. Open government data and the Internet of things are key sources of big data for governments and other institutions, such as data on education, health, public safety, social welfare, and taxes. Big data may not reach its potential for innovation without AI, suggesting their mutually enabling relationship. It should be noted that data with errors introduce risks to the systems, resulting in economic loss, mistrust in the systems, and compromised legitimacy. Therefore, data quality is key to establish trustworthy AI [155]. This section first discusses the challenges of data, including data quality and management, data security and privacy, and data bias. Then, we discuss measures to address these data challenges for trustworthy AI.
7.1. Large High-Quality Data
Making improvements to data analytics has successfully gained much attention, but improvements in data management for data analysis to establish trustworthy AI have remained underexplored. Furthermore, it is challenging to create a solid foundation for the development of AI systems, as no appropriate curated data resources are available [157]. Big data should not be viewed as data of better quality, and any data moved away from its original contexts do not reflect its original meaning [159].
There are two levels of the system that demonstrate the importance of data management: the operational level and the preprocessing level. Data management at the operational level enhances the overall trustworthiness of AI systems and confirms the availability of metadata to determine how data can be applied (e.g., data origin, format, extraction, organisation, classification, and connection) [160]. Meanwhile, data management at the preprocessing level confirms the reliability of data acquisition and the reproducibility of the results, confirming the transparency of the overall analysis.
It is crucial to train DL models using a large amount of data to achieve generalisation and ensure trustworthiness [161, 162]. However, data scarcity can pose a challenge in the training of DL models. To overcome this challenge, several techniques have been developed, including transfer learning [161, 163–167].
7.2. Data Security and Privacy
The significant potential of AI in simultaneously combining a large amount of data from multiple sources leaves trails of identifiable data, highlighting the significant relevance of privacy [168]. It is legally mandatory for government agencies and other institutions to protect and secure all personal and sensitive data; unfortunately, there are still cases of data breaches, such as the recent data breach at the Office of Personnel Management [169]. When combined and analysed together, integrated datasets provide the potential for “the ability to combine multiple customer views [that] may provide inappropriate insights” [170, 171]. The integration of multiple datasets contributes to the significance of AI and big data. It is a complicated phenomenon involving the privacy and application of data, in which no measures can increase data privacy without compromising the application of data [170].
7.3. Data Bias
AI models are math-based algorithms that themselves are not biased. However, the data can be biased. Therefore, it is critical to consider data diversity in terms of training AI models [172].
The bias in AI systems with respect to the data was classified into six groups [173]:(1)Sample or selection bias when the database used underrepresents or overrepresents the target sample for the intended AI application(2)Measurement bias relates to the systematic value misrepresentation when the instrument or the device used promotes a particular result(3)Self-reporting bias is mainly present in the survey case, where absent, incomplete, and inconsistent responses might be provided(4)Confirmation bias refers to observer bias when the working hypothesis is made based only on the observer’s cognitive background and preferences(5)Prejudice bias represents human data bias which reflects prejudice against gender, age, race, or ideologies, leading to discriminatory predictions and recommendations(6)Algorithmic bias happens when the algorithm/model creates or amplifies the bias in an attempt to meet processing requirements
To address the challenges of data bias and make AI systems more trustworthy, we propose the following recommendations:(1)Datasets used to train AI algorithms and build AI systems should be balanced and representative of the target population/estimations.(2)Usage of different measurement devices/instruments, involving different observers, assessing the inter- and intra-observer reliability, and comparing the output, should be highly encouraged before engaging any dataset in AI pipelines. In fact, systematic measurement bias error cannot be addressed by simply collecting more data.(3)Missing, incomplete, and inconsistent data should be disregarded.(4)Whenever human observers are involved in the data curation of an AI pipeline, collection protocols and guidelines must be provided to ensure that the relevant characteristics of the intended use are coherently represented. Furthermore, it should be comprehensive to take into account the opinions of different observers without misinterpretations.(5)AI algorithm/model should be tailored to the available data. The algorithm design should reflect the intended use of the AI systems without creating or amplifying any data bias.(6)Evaluate and benchmark the AI algorithms/models constructed from the balanced datasets. The best AI model should be selected based on multiple evaluation metrics simultaneously. Accordingly, the essential metrics weights must be considered when evaluating the developed mode.
7.4. Addressing Data Challenges for Trustworthy AI
Addressing data challenges for trustworthy AI, studies have proposed various measures that governments and institutions should consider, which can generally be grouped into two forms of recommendations: data management and data literacy. First, data management measures serve as a fundamental foundation to establish trustworthy AI. The Data Management Association introduced the Data Management Body of Knowledge (DMBOK), which has become one of the most crucial data management frameworks [174]. It is an industry standard to evaluate data management and determine the appropriate strategies to manage data with the consideration of the following key dimensions of data management: (1) data governance; (2) data architecture and design; (3) database management; (4) data access (security) management; (5) data quality management; (6) master data management; (7) data warehouse and business intelligence management; (8) records management; (9) metadata management. Governance plays an important role in establishing the needed standards for all nine dimensions through specific structures and practices that exert authority and control (e.g., planning, monitoring, and enforcement) over the management of data assets. Through policies and standardised practices, institutions can systematically make data decisions and direct the functions of people and processes in data management [175]. Most importantly, data governance serves to manage data use with respect to established policies and practices toward achieving the key expected outcomes [176]. These sustainable and strategic data governance structures help minimise security issues and protect data creation, access, and application.
Second, data literacy plays an influential role in AI applications. Trustworthy DM relies on the integrity, security, and appropriateness of data in this digital era. Users, such as government employees, are responsible for the collection, analysis, and application of data. Through data literacy, users have knowledge of data management and have the ability to perform the tasks of creating, maintaining, and securing quality data to support daily operations [177]. Data cleaning, preparation, and review require meticulous documentation of judgment calls, further demonstrating the importance of data literacy [178]. These recommendations allow policymakers and decision-makers to evaluate and review the basis of AI development, promoting system transparency and accountability.
8. Trustworthy AI in Different Applications
This section presents six applications, including trustworthy AI in education, environmental science, 5G-based IoT networks, robotics for architecture, engineering and construction, financial technology, and healthcare. These applications are critical due to daily use; therefore, it is important to consider their trustworthiness in them (see Figure 11). The purpose of this section is to explain what the priorities are in terms of reliable requirements for the mentioned applications as listed in Table 2.
Building trustworthy AI systems is essential across various applications and industries. To achieve trustworthiness, it is crucial to prioritise requirements such as data privacy and security, explainability and interpretability, and adherence to regulatory standards. These requirements contribute to building trust, facilitating collaboration, ensuring ethical use, and improving outcomes in fields such as education, environmental science, 5G-based IoT networks, robotics for architecture, engineering and construction, financial technology, and healthcare. By addressing these requirements, AI researchers and practitioners can develop AI systems that are reliable, transparent, and effective, fostering widespread acceptance, and maximising the potential benefits of AI for society.
8.1. Trustworthy AI in Education
There are various dimensions of trust in AI. There are different ways to deal with trust in AI for different countries. It is complicated to explain certain algorithms or techniques in layman’s terms; therefore, a few countries like France have rejected the use of certain types of algorithms in public DM [179]. For China, it is about promoting social interaction and mutual trust. Meanwhile, the US take on expanding AI through research and development in order to improve the security, robustness, and trustworthiness of its systems closely mirrors the G20 AI Principles. Referring to the EU guidelines for trustworthy AI, there should be transparency, clear documentation, and identification of DM, and explainable technical processes and related human decisions. It is important that users of the AI system are informed of their interaction with the system and of the capabilities and limitations of the system [31].
Trustworthy AI performs what it is designed to execute in an unbiased manner. Taking the case of an AI-powered early warning system, students are subjected to profiling, and students who are at risk of dropping out are identified. An AI system that is not trustworthy may not be able to effectively identify these students or may be subject to misuse despite its accuracy in identifying these students. The system only matters when good (human) interventions are involved to support these identified cases of potential dropouts and deal with the potential risks involved. Addressing social justice, fairness, and nondiscrimination, in line with the G20 AI Principles, certain interventions may serve to improve the school completion rate. Still, they may also exclude students who are at risk of dropping out due to certain accountability issues that potentially lead to unfavourable circumstances for the school (e.g., negative school reputation or image). In other words, the application of AI is not merely about establishing its trustworthiness but also involves a trustworthy connection between AI and humans [180].
The applications of AI in education are not widespread. However, there are various cases that can benefit from AI, such as automatic DM. For instance, AI can be applied for school, college, or university admission. This promotes fairness and unbiased selection of students. AI may introduce certain unintended implications since a new system is likely to alter the beneficiaries of the most in-demand schools, which is why it is important that the AI system and its algorithms display transparency and explainability. Transparency can be achieved by expanding “openness” to algorithms, but for certain AI techniques (e.g., deep learning), it is difficult to achieve explainability.
AI innovations benefit individuals and societies through education and learning outcomes, especially in this era of digitalisation. When it comes to trustworthy AI in education, privacy and security are among the most important aspects. Since students are generally minors, data protection and security are important issues, given how AI is largely based on the huge amount of personal data. The efficiency of AI in education increases with the use of personal data, but the collection and storage of these data pose privacy risks. Taking into account the fear of “Big Brother” in society, privacy concerns are magnified when it comes to AI in education. Large volumes of data are gathered and stored and can be retrieved at any point in time, raising concerns that “old” data are used in DM. There is also the possibility that data are exploited for commercial purposes. These concerns reflect the “human-centered” nature of trustworthy AI in numerous ways. It is plausible that human-centred habits and practices are retained despite the prominence of AI: for example, the ability not to be constantly observed, not to risk having one’s private information publicly exposed, and the ability not to be judged based on past or irrelevant information (but now available) [181, 182].
After all, each country has its own strategies to deal with privacy and security issues. For instance, when it comes to the European Union, the use of personal data is strictly regulated with respect to a stringent framework presented by GDPR, which emphasises transparency, data and storage limitation, and accountability. There are specific requirements for one to use, share, and store data. Similarly, the U.S. applies a specific framework on the use of personal data within the context of education under the U.S. Family Educational and Privacy Rights Act. Meanwhile, China implements the Chinese Governance Principles for Responsible AI, which promotes users’ rights to be informed and to make their own selections freely and protects data privacy. In short, AI systems should consistently undergo necessary improvements in terms of controllability, explainability, transparency, and reliability without compromising the safety and security of these systems.
8.2. Trustworthy AI in Environmental Science
The applications of AI and machine learning (ML) in environmental science have recently gained growing popularity [183–186]. Environmental scientists employ AI/ML to make sense of raw data, e.g., satellite imagery and climate data, in order to come up with appropriate decisions for implementation. There have been more applications of AI, which have continued to improve predictions for various high-impact occurrences. However, unethically or irresponsibly developed AI may eventually cause more harm, which can be observed in a few high-profile cases beyond the context of weather and climate [187, 188].
The benefits of AI in terms of environmental sustainability are undoubtedly evident, such as allowing automated monitoring of the ecosystems to support accountability for climate justice, for example, monitoring changes in land cover for the detection of deforestation [189–191], counting endangered species populations in very high-resolution satellite data [192, 193], and tracking bird populations in radar data [194–196]. Through automated analysis of retrospective and real-time datasets, relevant stakeholders can continue to monitor environmental trends and provide timely and effective responses.
Unfortunately, AI in environmental science may result in unfavourable results for certain application areas. Prior to the discussion on the shortcomings of AI in environmental science, the concept of environmental justice must first be discussed. Environmental justice refers to the unbiased treatment and meaningful public participation, regardless of demographic background and socio-economic status, to ensure the necessary development, implementation, and enforcement of environmental laws, regulations, and policies (EPA, 2022). With respect to this definition, there are multifaceted perspectives to take into account, from instrumental and consequential perspectives to principled ethical perspectives. These are entangled in the biases and pitfalls that this paper explores. Debiasing has the potential to address both. Note we assume the reader is generally familiar with the concepts of AI and ML and does not focus on any specific AI/ML methods but instead on the applications across environmental sciences. The question is “How AI can go wrong for environmental sciences?” The operation of AI depends on the quality of training data. Therefore, nonrepresentative or biased training data result in unreliable and biased AI systems and models. In other words, this phenomenon is known as “coded bias” [188]. For instance, an AI system that relies on training data that only contains hail occurrences in highly populated areas is more likely to show hail prediction in highly populated areas. Then, the system is biased to the population density aspect on which the system is trained. Algorithms that are frequently used in regions or circumstances that are different from where the training data are sourced may eventually perpetuate the bias beyond the regions and circumstances on which the algorithms are trained.
Issues related to training data are rather common and inevitable in most cases due to the following reasons: (1) it is extremely challenging to gather perfect, nonbiased datasets; (2) data developers themselves are not aware of the data limitations or biases; (3) the interpretation of bias depends on the context or application; (4) there are no standardised tests to check for common biases in datasets. Problematic data represent another common issue in AI [197, 198]. Such data may result in a faulty model or inaccurate model when real-world data are used in the model. When it comes to the quality of environmental science data, it is important to take note of the human and environmental factors that affect the data for use in AI. In addition to training data issues, there are also internal issues to consider. For instance, model training choices influence every aspect of the AI model operation. Although the methods used to develop AI are generally “objective” according to specific mathematical equations, the following training choices can significantly influence the model outputs: (1) the selection of attributes for the model (e.g., environmental variables); (2) the source(s) of training data; (3) the selection of data preprocessing techniques (e.g., data normalisation, elimination of seasonality, and application of dimension reduction methods); (4) the selection of AI model type (e.g., clustering, random forest, or neural networks); and (5) the selection of hyperparameters (e.g., random forest in terms of the number of trees, maximal depth, or minimal leaf size). Each selection significantly influences the model outcomes, resulting in different results with critical implications. For example, the selection of spatial resolution is important to determine the model output for environmental justice—an AI model that is trained to predict urban heat at a low spatial resolution may overlook extreme values in small areas. In contrast, an AI model that is trained to predict urban heat at a higher spatial resolution can identify extreme values but potentially comes with noise.
On the one hand, faulty strategies are instead subjected to learning. AI models are trained to learn data patterns to produce good predictions. However, training data may not be representative of real-world settings. Faulty strategies can be identified by using interpretable or explainable AI (XAI) [199–202]. When it comes to interpretable AI, human experts must be able to understand the models [87]. However, explainable AI focusses on identifying the internal strategies used in more complicated models. Both interpretable and explainable AI enable human experts to evaluate the models for potentially faulty strategies prior to the application of the model [186].
8.3. Trustworthy AI in 5G-Based IoT Networks
The rapid expansion of wireless broadband and multimedia applications in relation to the Internet of things (IoTs) has required enhanced capacity and robust quality of service (QoS). The reliance of IoTs on communication and computational structures potentially results in performance issues, as different sensors and devices are required for monitoring and control features [203–207]. The incorporation of 5G wireless networks can improve capacity and QoS in cellular networks, which can be subsequently addressed by addressing these bottlenecks. As a new decentralised architecture, the 5G network offers numerous benefits. Even at different locations, the resources for communication and computation can be acquired. The development of IoT-based vehicular communication technologies produces reliable wireless connectivity, enabling the development of intelligent transportation systems [208].
Accordingly, the 3rd Generation Partnership Project (3GPP) presented a roadmap that governs extremely high bandwidth, ultralow latency, and high-density connections to support 5G-empowered vehicle-to-everything (V2X) services. With 5G-enabled vehicular networks, vehicles with different sensors can be connected with each other through mechanisms such as dedicated short-range communications (DSRC), LTE-V-Direct or device-to-device (D2D) millimetre wave, and by requesting and/or accessing resources from neighbouring vehicles, roadside units (RSU), 3GPP core network, or base stations [206, 209]. In addition to that, the prevalence of mobile and sensor-rich devices benefits IoT-based smart healthcare systems, particularly in supporting human activities, such as applications that recognise and monitor routine life-logging, healthcare, senior care, and personal fittings [206, 210, 211]. Similar to AI applications in other fields, IoT networks are also linked to security, privacy, and trust issues. Some academic and commercial institutions have made efforts to explore and develop vehicular networks to support this vision provided by 3GPP. Focussing on the issues of secure communication, studies have explored security, privacy, and trust in vehicular ad hoc networks (VANETs) [212–214]. Despite that, cybersecurity findings in the application of 5G in connected vehicles have remained scarce [213, 215–217]. Malina et al. [218] explored various IoT applications to categorise potential privacy leaks and then developed an innovative mechanism to protect privacy and maintain security within IoT services for actual applications, such as smart healthcare systems, intelligent vehicular networks, and smart cities. Using blockchain technology, Baza et al. [219] designed a privacy-preserving charging station-to-vehicle energy trading scheme, which can effectively identify Sybil attacks, protect the privacy of drivers, and significantly minimises communication and computation [220]. With the recent technological advancements in IoT networks, IoT users are able to make use of computational resources for IoT services. As modern vehicles in vehicular networks can exploit computational resources, drivers can have more relaxed, safer, and fuel-efficient rides, resulting in less traffic congestion and accidents. In such a scenario, the study of trustworthy privacy-preserving management in the computation is lacking and leads to the high potential risks to the vehicles [221, 222]. In particular, there are centralised and distributed mechanisms. For the centralised mechanism, data processing depends on the centralised servers. However, it does not fulfill the recent requirements, which is attributed to the immense number of users in large-scale networks (causes congestion at the centre). With 5G networks, systems can now fully make use of distributed nodes to address the significant computational burden in the centre, as well as the characteristics of IoT users and the unused computational capacity to process data in a distributed way. Meanwhile, as for the distributed mechanism, the trustworthiness of its management appears to be a major concern despite its benefits of improving the system’s capability and reliability to complete computational service tasks. This may be attributed to the potential case of malicious miners deceiving the running algorithms, as well as the presence of egocentric computing nodes in the distributed computation (unwilling to share data) and the low processing capacity of miners, resulting in unfavourable contributions or even the divergence of algorithms. For both centralised and decentralised data processing, training data must be preprocessed or reconstructed to secure and protect any sensitive data, but this may reduce system accuracy. Therefore, robust privacy-preserving computation is necessary to compensate for the trade-off between system accuracy and data privacy [223, 224]. These issues must be addressed using a fair system design.
Recent advancements in AI techniques must be fully exploited [225–227] in the development of reliable privacy-preserving computations to improve data processing accuracy, protect sensitive data, and avoid adversary machine learning attacks [228]. Privacy is the most essential aspect that influences trust in AI when it comes to 5G-based IoT networks [229, 230]. With the recent emergence of new technologies and applications and their close connections with IoT networks, the privacy of IoT users has gained growing attention. Undeniably, technological advancements, especially in vehicular networks and smart cities (e.g., autonomous vehicles, hyper-connected vehicles, 5G and beyond networks, and AI-assisted Big data management in surveillance), have brought major improvements to the overall quality of life. However, security functions are brought to attention. For instance, an application that involves the capability to track trucks aims to identify the location of trucks and drivers in the terminals, maximise the throughput of container loading or unloading, and plan the driving routes according to the dynamic circumstances. With that, the application comes with an AI-assisted framework to perform the following functions: (1) mobility-aware attack detection; (2) real-time controller area network bus intrusion detection; (3) trust management system; and (4) privacy preservation of drivers. In other words, these functions help identify potential malicious actors in a triage manner without compromising trust, privacy, and security. In this case, the anonymity approach that takes advantage of the generation of pseudonyms as unique identifiers for authentication may benefit the privacy-preserving mechanism. Through such unique identifiers, any personally identifiable data are not required. Focussing on improving the conventional Hoepman’s eight privacy design strategies, a previous study [218] demonstrated the successful application of such a privacy-preserving mechanism for private and secure IoT services, which included the preservation of data privacy (data collection, storage, and application), user authentication, communication, and computation. This simple yet effective design benefits numerous applications, such as identification systems, access control systems, smart safety systems, smart grids, and healthcare systems [231].
Apart from that, another approach to secure data communication involves the privacy preservation model under the data sanitisation process. In this sanitisation process, the key is optimally tuned by using many advanced AI-based algorithms, which include nature-inspired algorithms and DL techniques [205, 206, 232]. Moreover, the privacy-preserving model or the authentication mechanism can be designed according to the secure key management, which commonly involves secret key encryption or public key infrastructure with certificates [219, 220]. In the cryptography-based authentication process, the sender generates codes using the secret key, and the receivers verify the codes attached to the message using a shared key.
Accordingly, there are two authentication methods: (i) authentication with central authority and (ii) authentication without central authority. For authentication with the central authority, an authority provides a unique identifier and a seed value (generating short-term pseudonyms), and the receivers maintain a random keyset for authentication while preserving their privacy [233]. However, this authentication method comes with communication and storage delays and limited nonrepudiation property. Meanwhile, as for authentication without central authority or, in other words, public key-based authentication, public or private key pairs are provided for pseudonymous communication, and the digital signature of a certification authority serves for authentication. Secret keys and short-term pseudonyms generate a digital signature, enclosed with its certificate in the data packet. Certification authorities are responsible for managing these certificate-based signatures, which are authenticated by receivers without exposing the identity of senders [234]. Considering the issues involved in the revocation of pseudonym certificates, efficient management of the certificate lifecycle is necessary. Taking the case of a vehicle certificate revoked due to fraudulence, error in certificate issuance, or compromised certificate, the driver cannot obtain new pseudonyms from the certification authorities, and it is challenging to authenticate pseudonyms against a certificate revocation list promptly due to the significant number of messages and lists [235].
In addition to that, there is signature-based authentication, which involves the use of identity-based signature, certificateless signature, or group signature [236, 237]. For this application, the private key is generated and assigned by the private key generator, and the public key represents the node identity for signature authentication. Meanwhile, the certificateless signature does not require certificates for authentication, and only a partial private key is provided by the key generation centre. It depends on the users to generate the actual private key according to its secret value and the partial private key or to generate the public key according to its secret value and the public parameters. When it comes to the group signature, the valid group members sign the messages anonymously. Although this approach maintains the privacy of vehicles, it is time-consuming to authenticate signatures. Therefore, it is not practical for real-time applications in VANETs [238–240].
In view of the above, studies on privacy preservation have placed emphasis on communication and overlooked its application in computation [241]. This highlights the critical need to establish a comprehensive privacy-preserving framework that deals with the required data communication and computation today, such as in the case of hyper-connected vehicles. Additionally, the integration of a privacy-preserving mechanism and trust management to create a unique design of privacy preservation and trustworthy management is clearly necessary. It is particularly challenging to design an effective trust model in 5G-based IoT networks. For instance, vehicular networks face inefficient long-term operations to connect vehicles due to the high mobility of users, resulting in constant or rapidly changing locations and topology updates, as well as numerous types of privacy and security attacks. This clearly calls for a trustworthy model that can simultaneously deal with these attacks and maintain data privacy [242]. To sum up, it is critical to establish a comprehensive privacy-preserving framework that deals with the required data communication and computation in 5G-based IoT networks.
8.4. Trustworthy AI in Robotics for Architecture, Engineering, and Construction
For user acceptance, trust is a critical criterion for the interaction between technology and humans. Similar to other fields, the applications of AI in robotics for architecture, engineering, and construction have rapidly expanded, which calls for more studies on the trustworthiness of AI in these fields (see Figure 12), especially in terms of (1) privacy and security, (2) performance and robustness, and (3) reliability and safety [27, 244, 245]. The most crucial sociotechnical aspects of using AI and robotic technologies involve both privacy and security. Although privacy refers to one’s right not to be observed, how AI operates requires extensive learning and continuous improvements involving humans [246]. Despite the linkage between privacy and security in AI, in most cases, both aspects should not be used interchangeably. In particular, privacy in AI involves the acquisition, analysis, and use of personal data. In contrast, security in AI involves protecting data confidentiality, maintaining data integrity, and ensuring prompt data availability upon request [247].
A hackproof AI system prevents data breaches, poor system design and engineering, unplanned data corruption, and malicious intent to obstruct or limit user access to the system [248]. Most recent studies on ethical AI tend to focus on the following topics [249, 250]: opacity of AI systems; privacy and surveillance; machine ethics or machine morality; the influence of ethical DM of automation on employment; manipulation of behaviour; the interaction between humans and robots; DM bias; autonomous system control; artificial moral agents; and singularity. For the interaction between humans and robots, privacy and security are relevant forms of risk to trust [27, 251]. The protection of cybersecurity systems promotes privacy and, subsequently, trust in robots [250].
Accordingly, there are two forms of risk to trust from the perspectives of privacy and security: (i) privacy situational risk and security situational risk and (ii) privacy relational risk and security relational risk. First, privacy situational risk refers to the belief that the system is likely to expose personal data involving users or their surroundings. Second, relational risk to privacy refers to the belief that the system is likely to expose users or their surroundings to unauthorised observation or disturbance. Third, security situational risk refers to the belief that the system causes users to be vulnerable to any form of threat to their safety. Lastly, relational security risk refers to the belief that the system is vulnerable to misuse, resulting in some form of threat to safety [251]. Ethical AI (responsible AI) promotes trustworthiness [27, 248]. To realise ethical AI, the security of data is required. However, only a few studies explored the ethical challenges of applying AI in robotics for architecture, engineering, and construction. Certain studies highlighted the failure of common worker monitoring methodologies (for the training of machine learning models) when considering workers’ conscience, intentionality, and free will [252]. For applications involving AI and robotics, all data are gathered in monitoring construction tasks using telecommunications devices, wearable devices (e.g., VRs, smart, hard-hat cameras, and sensors), GPS, CCTV, drones, or smartphones [253–255]. Although there are different applications, ranging from worker safety [166] to equipment emission monitoring [256], data performance of workers and contextual information are indispensable for all applications.
Cloud computing is another example of a pervasive technology related to AI in the fields of architecture, engineering, and construction [257]. There are various privacy and security challenges of cloud computing in industries, which involve data security, access control, and intrusion prevention [258, 259]. Specific measures to improve data security in construction projects should be further explored [257] considering the importance of data security for such projects that are related to the military, government, and the public. These industries are often involved with confidential data, such as contract information, blueprints, images, and project personnel data. Careful attention is imperative to prevent any data breach [260].
Meanwhile, when it comes to the discussion on performance and robustness, machine competence in terms of technical performance and capacity serves as a key driver of trust in AI [261, 262]. The performance of AI is typically measured based on system accuracy. Accuracy functions as a performance benchmark to establish the trustworthiness of the system [261, 263]. A robust AI that is developed in a specific context can systematically deal with varying issues within and beyond the context system without compromising its performance [263]. From a theoretical point of view, performance and robustness may not go hand in hand; testing data issues can potentially result in an inaccurate classification at a high confidence level [264–266]. Despite that, both terms are typically considered when it comes to discussing the technical reliability of AI within the context of trustworthiness. In 2018, the European Commission prepared the Ethics Guidelines for Trustworthy AI. It defined “robustness” as “resilience, accuracy, trustworthiness of AI systems,” which represents one of the seven key general trustworthiness requirements [22, 267]. Some of the proposed methods or techniques to promote robustness with acceptable accuracy include explicit training against known cases of system issues, as well as regularisation and robust inference [268, 269]. In most cases, users place their trust in AI based on their observation of its stated accuracy or the performance of the system in practice. When users observe that the accuracy of a system is low, their trust in the system declines, regardless of its stated accuracy [270, 271]. However, users would not trust the algorithm as soon as they identify any error, regardless of how the performance accuracy is observed [272, 273]. Most users establish their trust based on perceived accuracy [274], although trust is more affected by system failures than system successes [275]. Apart from system performance and robustness, there are other similarly effective performance metrics that contribute to trust building, such as precision and recall. For instance, users are more likely to emphasis the recall of classifiers instead of their precision in order to determine the acceptability of the performance of classifiers. However, the application can make a difference in terms of weight [276]. AEC use cases are not different from other applications in that the performance of the technology tool plays a key role in trusting and ultimately adopting it.
In construction, there is a specific target metric for the project cost or schedule [277, 278]. However, there are other comparatively important aspects to consider when adopting innovations and technologies, such as quality, safety records, sustainability measures, productivity metrics, and inspection results [279, 280]. Construction projects gain substantial benefits in terms of performance optimisation and assessment when BIM, big visual data, and modelling of construction performance analytics [281, 282] are adopted in conjunction with AI systems. The performance and efficiency of future projects can be significantly improved when the construction site layout planning process is digitised in BIM and training models. The development of AI-based real-time analytics tools and cloud-based data analytics for site data can facilitate projects to achieve performance targets and the required quality benchmarks [283]. For example, an AI chatbox can be used to receive, review, and share new projects and activities. After all, given the project-based nature of the construction industry, various contractors and other stakeholders with specialty trades or expertise are simultaneously involved. Therefore, AI applications, systems, and algorithms can be significantly different within a single project or from one project to another [284]. This is where the robustness of AI in construction comes into play. A robust AI must be transferable across varying projects, sectors, industries, and even geographic locations, given the uniqueness of any given project, without compromising user trust. For example, robots were used to pour concrete, brush uneven layers, and dismantle forms for a 334-metre-wide dam construction work [27]. The applications of AI also benefit tunnel construction projects, including inspection, maintenance, and health monitoring tasks [285, 286]. Robust models with the programming of specific project risk, quality, and inspection criteria are important for the effective and efficient transfer and application of tasks in construction projects. Both technical construction and business processes must demonstrate performance and robustness to establish trustworthiness [287].
In addition to that, reliability and safety serve as other key components of trust in AI-powered systems for architecture, engineering, and construction. [288, 289]. From the performance point of view (instead of the ethical point of view), reliability and safety are associated with trust in AI [290]. A safe interaction with AI reflects reliability and trustworthiness [291, 292]. However, a high level of reliability can contribute to overtrust and complacency [293], which explains the need to calibrate trust within the human-robot interaction. In the case of overtrust, users believe that the system mitigates risks and eventually accepts too much risk [294]. Through trust calibration, potential risks can be accurately identified [295]. The process helps users to be aware of and familiarise themselves with the system features, consistency, and failure modes in order to prevent over- and undertrust [295]. In the case of undertrust, trust calibration helps users build trust gradually through experience and iterative interactions [296].
Besides that, the amount of effort by users versus the autonomy of AI affects the trustworthiness of AI from the reliability and safety points of view. As a human-robot interaction (HRI) factor, reliability is evaluated as a function of such autonomy. Beer et al. [297] proposed different classifications of robot autonomy (LORA) for HRI [297–299], starting from the level when users have full control over the process (manual) to the final level when the system has full control over the process (complete autonomy). These different classifications are reviewed according to the interactions between humans and robots or the evaluation of the sensing, planning, and acting tasks. In addition to LORA, there are classic Sheridan–Verplank levels of automation and recent self-driving car classification schemes [300, 301]. However, these taxonomies only consider circumstances when automation limits human control or, in other words, single-dimensional automation. Addressing these limitations, Shneiderman recommended the human-centred artificial intelligence (HCAI) framework to generate reliable, secured, and trustworthy designs [302]. According to HCAI, it is possible to simultaneously attain high levels of human control and automation (two-dimensional HCAI), which helps in enhancing the overall performance in terms of reliability, security, and trustworthiness [298, 302].
Numerous AI applications have been developed to address significant safety and health issues in the construction industry [303]. Furthermore, the use of AI to address safety issues in the construction industry has gained growing popularity [244, 304–306]. AI-controlled systems or robots can improve efficiency and minimise accident risks, which are highly favourable for high-altitude construction tasks, over a long period of time, or involve dangerous circumstances [307]. Common applications of AI for construction safety involve systems that make use of proximity monitoring of hazards. Taking the case of evaluating the interactions between construction workers and equipment, studies have proposed a system that can make use of the 5G wireless network to send images from trucks, cranes, and other construction machinery to a database for review [308, 309]. In addition to that, AI can be applied to predict appropriate safety measures, such as the severity of the injury, the type of injury, the affected body part, and the type of incident in a construction project [310]. For building construction, dynamic building information modelling (BIM) offers important information to the relevant stakeholders to develop more efficient planning, design, construction, and operation or maintenance. The use of BIM and AI-based software packages that incorporate machine learning algorithms can further improve the analysis of all aspects of a design, ensuring its reliability without compromising other building systems [283]. For instance, Augusto and Nugent [311] extensively explored the benefits of AI in designing smart homes [312].
Despite the benefits of applying AI, its new applications may gain lower trust, given their potential safety risks. Studies have demonstrated the need for any new technologies or systems to demonstrate appropriate reliability and safety in construction [313]. When it comes to industrial applications of AI, human safety is a priority, since such applications are generally more robust and larger, which poses safety risks [314, 315]. Most previous studies on the reliability and safety of human-robot interaction in construction used two validation approaches. The first validation approach involves operating robot prototypes in physically simulated working environments. This serves to determine any weaknesses of robots in reliability and safety for enhanced functionality. The second validation approach involves developing predictive models to explain the perceived reliability and safety of robots and their effects on humans [316, 317]. For example, the robot acceptance safety model (RASM) integrates immersive virtual environments (IVEs) to review the perceived safety of collaboration between humans and robots. It involves several participants working together with a 3D simulated robot for a specific task using a head-mounted display. The results obtained revealed enhanced perceived safety when human and robot work areas are separated. This segregation promotes the identification of the team and the gradual trust in robots. In other words, the overall perceived safety and comfort of working with robots reflect their acceptance of future collaboration with robots [318, 319].
Any form of accident can affect the collaboration between humans and robots and the efficiency of work in industrial settings [320]. Addressing that, studies have proposed the use of active vision-based safety systems as one of the best measures. Various reviews and technical analyses have been conducted on the advancements of vision-based technologies and alarm systems in terms of methods, sensor types, safety functions, and static/dynamic actions of robots [321, 322]. Applications of AI and its subfields (e.g., machine learning, computer vision, knowledge-based systems, and natural language processing) and AI-powered robotics in construction mainly focus on safety and health to improve reliability and trust. However, the reliability and safety of AI in establishing cognitive trust are equally crucial and should be further explored.
8.5. Trustworthy AI in Financial Technology
With the development of new business models based on the utilisation of big data, fintech can confuse established financial intermediaries, especially banks [323]. Machine learning is a variant of AI that makes it possible for a computer to learn without an explicit programme. What is “deep learning”? It tries to derive meaning from big data using layers of learning algorithms. Costs may be reduced by applying new technologies, financial intermediaries, and improved consumer products [324]. AI-based applications need to be regulated to drive change in the right direction in compliance with global rules and standards especially when it comes to funds and sensitive data involvement. It is well known that trust is at the core of our financial system and building global trust in financial institutions is not easy. This is a bank regulation that works. AI-related policy recommendations and guidelines challenges for banks and other market participants (payments, third-party providers, Apps, and fintech) enforce strict control and create a secure environment so that the user is validated and identified [325]. With the combination of the right level of regulation and the ethical approach to artificial intelligence, ultimately user trust will be built and greater acceptance of AI-based practices in financial solutions.
To realise trustworthy AI in finance and insurance, the operational set of processes, methods, and tools is based on three challenges that are rarely addressed today.(1)The model designer should choose the right topics of AI(2)The model demonstrates valid accountability and discrimination, considering fairness criteria in data preparation and model design(3)The model designer needs to achieve an unpleasant trade-off between accuracy and fairness or accountability
In this part, we will discuss the important concepts for addressing responsible AI in the financial sector.
8.5.1. Explainability
It determines how AI professionals and business leaders should address technology-specific limitations when explaining how AI models reach final and correct results. Despite the excellent performance of AI in a variety of areas, artificial intelligence (AI) is gaining in popularity. However, in addition to these achievements, lack of transparency, ambiguity, and the inability to explain and interpret most of the state-of-the-art technology are considered ethical issues [326]. Due to the complexity of AI, it is often difficult to explain and validate its predictions. This is sometimes referred to as the “black box” of machine learning [327]. Fintech is at the forefront of regulation covering accountability and equity issues. According to applicable regulations and guidelines, these systems need to be transparent. There are many different AI models today, ranging from linear regression to deep neural networks (DNNs). A simpler model may be easier to interpret, but its predictive power and accuracy are often poor compared to complex models [328]. These complex algorithms are very important in advanced fintech applications such as trading and cybersecurity. The models are intrinsically intractable models or approach interpretable models; these models are more complex and use reverse engineering [326, 327, 329, 330]. Implementing explainable fintech requires understanding the level of explainability according to the level of the user, for example, stakeholder, supplier, and third party. Thus, there are several points that should be considered when implementing XAI, including:(1)Explanations should be familiar to the users and should be simple in terms of audience knowledge(2)The XAI model should easily explain which features or variables influenced the model’s predictions and at what step the decision is made(3)The weaknesses and strengths of the model must be explained, as well as how it would work in the future
8.5.2. Privacy
One of the principles of responsible AI is privacy [331]. It is the ability to prevent people from obtaining information about others without their permission [332]. In recent years, the importance of data protection has received a great deal of attention. European Union (EU) and the Governor of California enact security laws and data protection policies to protect the rights to personal data. This gives more control to the consumers over their personal information. Therefore, the next generation of services while maintaining privacy is an open research topic [333]. Data protection is an important issue for companies that work with large amounts of data. When implementing AI solutions in the financial sector, the following items must be considered [334]:(1)Data confidentiality and data subject privacy protection.(2)Infrastructure security.(3)Maintain the right to the level of personal access to the information. When collecting data for banking applications powered by AI solutions, the following guidelines should be established.(a)The personal information is collected and processed legally and fair.(b)The agencies should collect the right data to avoid harming the accuracy. Also, these data must be updated periodically.(c)The system that uses the data should apply security safeguards to avoid accidents.
The authors in [330] addressed two methods for preserving privacy in financial AI systems:(i)Federated learning (collaborative learning) model: It allows multiple stakeholders who do not fully trust each other to work together to train the AI learning model on the combined dataset without data sharing. In this model, the local data are trained in local models, and the local models share the training parameters to complete the global model. This model can be centralised, decentralised, or heterogeneous.(ii)Synthetic data-based model: It is artificially manufactured information that is not collected from the real world but can represent it. Using sophisticated AI algorithms, such as generative adversarial network (GAN) and real-world data, the synthetic data are generated. The newly generated information has similar statistical characteristics to real-world data. The newly generated data can be fully synthetic, partially synthetic, or hybrid synthetic [335], depending on the cost and privacy of real data. When using a synthetic dataset by the organisation, the utility metric should be evaluated because the utility indicates the validity and analytical completeness of the data; thus, a trade-off between utility and privacy perception should be considered. Synthetic data can be considered as a safeguard for personal information, which is critical in financial applications and important for anonymisation. Also, synthetic data are easy to access and exchange between organisations (see Figure 13).
8.5.3. Ethics
This is related to how AI treats society. All organisations must address this to build trust in their systems and meet the needs of their stakeholders for reliable information and accurate results [336]. The need for ethical AI systems is critical in the financial services sector. Finance organisations increasingly use AI to make critical decisions that can negatively impact customers, e.g., those related to credit card applications and rejection of loans. Therefore, to ensure ethical fintech, it is important to consider the following concepts when developing financial AI algorithms:(1)The system should protect the privacy of the customers and their confidential data(2)The system should be transparent(3)The system should be unbiased(4)The system should be accountable
AI can improve fintech, but there are limits. Specifically, bias can pose an ethical hazard that calls into question the reliability of the result generated by the system. Bias can be explained by the explainability of data, reproducibility when testing consistent results, and verifiability. Bias is not always bad and intentional. In some cases, it can generate discriminatory and unfair outcomes and is referred to as unfair prejudice. Bias can manifest itself in statistical models in many ways:(1)Inherent in training data(2)Algorithm distortion
In logic-based AI, bias is introduced depending on the rules that are affected by the knowledge of the designer. In a data-driven statistical model, the bias is affected by the way the data are collected. In bias free model, it is generated according to the manner in which the system is used.
8.6. Trustworthy AI in Healthcare
Similar to other industries, trustworthy AI is critical for the advancements and applications of AI in healthcare. It has become increasingly crucial to establish trustworthy AI systems following the growing dependence on AI in diverse healthcare applications. The ethics, evidence, bias, and equity aspects are key elements that contribute to the establishment of trust among healthcare professionals and patients [337–339]. The use of robotics was very helpful during the COVID-19 crisis and helped reduce the need for person-to-person contact [340] (see Figure 14).
(a)
(b)
(c)
(d)
There are fundamental values that ethical AI must maintain in healthcare care [338]. AI-enabled technologies in healthcare are different from the applications of AI in other fields—these applications in healthcare influence the therapeutic relationship between healthcare practitioners and patients. The training data for AI in healthcare provide varying quality and completeness and are intended to be applied in various settings and circumstances. This poses the risk of inequities in patient outcomes. However, AI systems are highly adaptive and have the capacity to learn and evolve over time beyond human observations and control [341]. However, the stakeholders involved with various forms of expertise, professionalism, and objectives are responsible for AI design, development, deployment, and oversight [342].
Despite the challenges of adopting AI in healthcare, there are frameworks for the ethical design and deployment of AI in this field. For example, the American Medical Association (AMA) Code of Medical Ethics on ethically sound innovation in medical practice (Opinion 1.2.11) specifies the need to establish the scientific foundation for any innovation that has direct consequences on patient care under the guidance of healthcare professionals with appropriate clinical experience, with minimal risks to patients and maximum benefits of the introduced innovation [343, 344]. Opinion 1.2.11 further elaborates on the need to have meaningful oversight in the development and integration of innovation into the delivery of care. According to the same code, Opinion 11.2.1 focusses on professionalism in healthcare systems, specifically on the ethical need to continuously monitor and report the results of AI innovations in the delivery of care [345]. In other words, innovations in healthcare should not put patients at a disadvantage or risk. They must be implemented based on the availability of resources and infrastructure for high-value care. Institutional oversight should also consider the possibility of adverse effects beyond clinical settings despite how well innovations are designed. For example, clinical prediction models are developed and adopted to identify individuals at risk of medical conditions but stigmatise or discriminate against certain individuals or communities. In 2019, the high-level expert group on Artificial Intelligence of the European Commission published the Ethics Guidelines for Trustworthy AI, which described the key role of trust in the development and adoption of AI and highlighted the need for a framework to achieve trustworthy AI [346, 347]. The publication described trustworthy AI as ethical, robust, and lawful. Its design should be human-centred according to ethical principles, including respect for human autonomy, prevention of harm, fairness, and explicability. The publication further noted the difficulty of observing and predicting the risks of AI systems, especially when vulnerable communities are involved, and the need for a holistic approach that involves multiple stakeholders and sociotechnical processes. On top of that, the publication saw AI as a sociotechnical system that needs to be reviewed within the context of the society for which it is designed [348].
Another recent study conducted by the European Parliamentary Research Service on “Artificial intelligence: From ethics to policy,” conceptualised AI as a potentially beneficial and risky experiment in actual settings [348]. AI systems must be trained to be ethically responsible and to balance predicted benefits against potential risks without compromising the safety of humans. Recognising the highly promising prospects of AI, the study reported the significance of incorporating ethics into the design, development, and implementation of AI.
Apart from ethics, evidence plays an influential role in healthcare care, focussing on the validation of AI algorithms, but studies have demonstrated inconsistent terminology and approaches [349–351]. Achieving the highest scientific standards in design and development and providing clinical evidence of effectiveness and safety establish and enhance the trustworthiness of AI. There are specific frameworks for designing, conducting, and evaluating clinical research; for example, the US Food and Drug Administration (FDA) approved a drug and device development process that provides a model on which to base a standardised approach to meet this responsibility [352, 353]. An AI system designed for use in healthcare must at least demonstrate a clearly defined design protocol, address clinically relevant objectives and questions, and possess comprehensive documentation with scientifically rigorous and consistent validation in terms of safety and effectiveness. Besides that, a competent group of experts with diverse expertise and knowledge must review the AI system and prepare an unbiased report of its performance according to scientific standards.
These key requirements are subjected to continuous review and improvements as AI and technologies continue to evolve over time. Studies have proposed various methods to evaluate the quality and level of evidence required for the applications of AI in healthcare care. For example, the classification of recommendations, assessment, development, and evaluation (GRADE) evaluates the quality of evidence and the strength of suggestions from clinical practice [354]. In addition to that, there is a risk categorisation framework for software as a medical device (SaMD) developed by the International Medical Device Regulators Forum (IMDRF), which functions to categorise the impact on SaMDs based on the importance of the information provided for DM and the state of the healthcare condition [355, 356]. Such evidence and risk framework provide information on the levels of validation and evidence required for AI systems and deal with numerous ethical considerations, including sociotechnical and environmental considerations. Through the framework, IMDRF highlighted the importance of postmarket surveillance through a continuous learning process based on real-world evidence. Recognising the applications of AI in healthcare, ranging from administrative tasks to algorithms that inform diagnosis or treatment, acquiring evidence in proportion to the potential risks of AI for patients is highly crucial [357, 358].
Meanwhile, the applications of AI in healthcare care have highlighted the need to take into account the aspect of bias in the design, operation, or application of these adaptive systems in clinical settings [359–362]. Algorithms trained on electronic health records (EHRs), as most are currently, risk building into the model itself, whatever flaws exist in the record [363]. It should be noted that EHRs electronically capture information only from individuals with access to healthcare and that all data are not uniformly structured across these EHRs. Most of the data in EHRs consist of information captured “downstream” of human judgments. In other words, this comes with the risk that the model replicates human cognitive errors [363–365].
Despite all efforts to deal with potential biases in training data, this may result in unintended implications. For example, “race-corrected” algorithms divert resources from patients in the minority group instead of offering individualised and equitable care [350, 366, 367]. In addition, most of the medical literature is Western medicine. Thus, this may not apply to different races, the developing world, and medicine in under-resourced regions. Thus, it auto-race corrected.
The development of unbiased models must address the definition of “fairness” [368, 369] and determine the appropriate trade-offs between fairness and performance [369, 370]. Furthermore, fairly designed algorithms, hypothetically, may become biased over time when these algorithms are applied in different contexts (from what they are designed in) or continuously trained on data with uncorrected biases in a broader healthcare system [359]. In certain cases, the designed models may be applied uncritically or within certain settings that are discriminatory or biased in nature. In addition to that, these models can have biased selection or a tendency to promote certain outcomes that are not in favour of the interests of individual patients [359, 371].
Focussing on health equity, the vision of AMA is to promote a vibrant environment that provides ample resources, equitable and safe systems, and the opportunity to achieve good health, as well as equips healthcare professionals with the awareness, equipment, and resources to deal with any inequities in all aspects of the systems. Despite significant technological advances and innovations to improve health equity, existing models of resource allocation, evidence development, solution design, and market selection have overlooked the incorporation of an equity lens, risking automation, scaling, and exacerbation of health disparities rooted in historical and contemporary racial and social injustices. The use of training data that excludes or under-represents historically marginalised and minority groups contributes to the rise in equity. Significant differences in results related to the individual identity of the patients are not taken into account [372].
Furthermore, the design of the algorithm itself can exacerbate inequality when the related proxies or assumptions are discriminatory in nature. The nature of algorithms is generally more objective than that of humans, but these algorithms are developed by humans who are inherently biased [373, 374]. Marginalised communities are evidently under-represented in solution design and development, including those venture-backed start-ups, large technology firms, and academic medical centres. For instance, innovation teams and user testing efforts exclude any representations from the Black, Latinos, individuals with disabilities, and other populations. The Board of Trustees of AMA published a report on AI in healthcare back in 2018, identifying the hidden and unintentional existence of biases in training datasets that may be reproduced or normalised as a critical outcome for end users of AI systems impacts [347, 375]. According to a sociology professor and Princeton University professor Ruha Benjamin, Ph.D. in “Race After Technology,” the book highlighted a similar notion of how “coded inequities” may appear neutral compared to historical discrimination but actually perpetuate and deepen discrimination and elaborated on the lack of intentionality as an inadequate attempt to rationalise the perpetuation of such discriminatory biases [376].
Regarding the European Commission statement on Artificial Intelligence, Robotics, and Autonomous Systems [377], the development and evaluation of AI solutions in healthcare must involve the intentional application of an equity lens from the start specifically from system design, development, testing, problem framing, training data selection, and algorithm design to the evaluation of the algorithm in order to identify, document, and eliminate these biases at the earliest stage possible. Although AI solutions are designed and developed more intentionally, AI is viewed as a downstream lever linked to larger upstream issues of inequity in the healthcare system. The application of these solutions remains bounded within a system that allocates the needed resources and opportunities for optimal health at the expense of others. However, healthcare professionals still have the opportunity to look upstream and explore beyond the design of the algorithm, including the health and care of patients under specific circumstances [368, 378]. Lastly, the issue with several state-of-the-art models is a lack of transparency and interoperability, which is a major drawback in many applications, including healthcare [379, 380].
There have been several tools to explain what is inside the “black box” of DL models and how the models make decisions [381].
Two examples have been listed from our ongoing work:
Example 1. Class activation mapping tool to visualise where exactly the DL model focusses on making a decision on a test set [382]. The shoulder classification task is chosen as an example (see Figure 15). Figure 16 shows the Grad-CAM and score Grad-CAM for shoulder X-ray images with two DL models. The first image shows the region of interest (ROI) in the red circle. The first model (image a) showed a correct prediction but could not be trusted because of low confidence and out-of-the-ROI focus. However, the second model (image b) showed a high correct prediction with a focus on ROI. It is necessary to investigate the results before moving to deployment.
(a)
(b)
Example 2. DeepDream is a technique that can be applied to display features that have been recovered by the network after the training phase [383]. In light of the fact that U-NET was shown to be a trustworthy segment in view of its strong statistics results, the resulting DeepDream images must set the primary features that differ per group (COVID-19 and non-COVID-19) (see Figure 17).
(a)
(b)
(c)
9. Challenges, Conclusions, and Future Directions
This paper discussed the significance and key requirements of trustworthy AI, which represents a fundamental research field in the AI ecosystem. The discussion began on the need for trustworthy AI, followed by trustworthy AI requirements, trustworthy data for AI, and lastly, applications of AI in education, environmental science, 5G-based IoT networks, robotics for architecture, engineering, and construction, financial technology, and healthcare. Accordingly, trustworthy AI demonstrates the following criteria: explainability, accountability, fairness, acceptance of AI, privacy, accuracy, reproducibility, and human agency and oversight. The development and working of trustworthy AI systems require appropriate measures, mechanisms, standards, and legal frameworks. Numerous studies have identified several technical challenges in developing trustworthy AI, such as the lack of clear requirements and standards to establish trustworthiness for AI. There are ambiguous definitions of the principles and properties of AI and unresolved differences in the principles across different application domains [9, 37, 384, 385]. For instance, a model that applies the principle of explicability is more prone to attacks, since the model is more interpretable and transparent. Therefore, stricter laws and trade-offs involving these principles according to the application domains are necessary.
Another challenge that affects the development of trustworthy AI lies in the need for context-specific solutions, as there is no universal solution for all problems. For instance, system developers may provide an explanation that is difficult for users with a nontechnical background to understand. A multidisciplinary team of experts is indispensable in the development of AI systems. In summary, trustworthy AI represents an emerging research field that requires more studies to further enhance the reliability and trustworthiness of AI.
Based on the findings of this article, there are several recommendations for future research on trustworthy AI. First, to optimise the potential of AI, especially in contexts that prioritise security, establishing the safety and trustworthiness of AI algorithms and systems is crucial. The ability to provide explanations to all stakeholders, ranging from system designers, developers, domain experts, and nonexpert users to policy makers, influences the trustworthiness of AI [386]. For that, multidisciplinary studies that involve computer science, data science, economics, law, and sociology are recommended to develop AI applications. This form of research offers significant knowledge and expertise from multifaceted perspectives that can enhance the safety and trustworthiness of AI.
Second, considering the growing importance of AI in this digital age, it has become increasingly crucial to establish standards and policies to ensure appropriate applications of AI. The standardisation of AI promotes more effective and efficient transfer of technologies, interoperability, security, and reliability. In addition to establishing new standards, AI applications must comply with existing laws and regulations according to their usage [18]. Therefore, future research is recommended to explore the needs and requirements of AI standards and enforcement policies.
Despite the significant benefits of AI, certain studies [127, 387] demonstrated the failure to use and accept AI among potential users, who reported the need for a certain mechanism that can objectively establish the trustworthiness of AI. Studies have proposed several theoretical models [388, 389], trust models [18], and human involvement methods [390, 391] to promote greater acceptance of AI. However, there are limited mechanisms that can measure and test user acceptance, which should be further explored. For example, future research is recommended to measure the effectiveness of trustworthy AI requirements and the implications of these requirements on user acceptance.
Last but not least, it is clear that there are different expectations about the capabilities of the system from different stakeholders, which influence the levels of acceptance and trust [392]. An expectation management framework from the beginning of the development of the AI system is crucial for system designers and developers to design and develop a system that is more well-received according to user expectations [302]. Further studies are also suggested on how to address postdevelopment expectation management to capture the influence of various factors such as system information, user reasoning and understanding of the system, and first-hand experience of the system in its acceptance by users [393–402].
Collaboration between AI experts and other experts in the domain, which is called “AI + X,” is essential. It is important to bring them together to address any concerns and build a trustworthy AI. For the massive use of AI, it must be safe, trustworthy, and reliable. It is critical to ensure the development and deployment of trustworthy AI systems that benefit society while maintaining ethical and responsible practices.
Data Availability
No data were used to support this study.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Authors’ Contributions
The authors contributed equally to this work.
Acknowledgments
The authors would like to acknowledge the support received through the following funding schemes of the Australian Government: Australian Research Council (ARC) Industrial Transformation Training Centre (ITTC) for Joint Biomechanics under grant IC190100020. Laith Alzubaidi would like to acknowledge the support received through the QUT ECR SCHEME 2022, the Queensland University of Technology. Open access publishing facilitated by Queensland University of Technology, as part of the Wiley - Queensland University of Technology agreement via the Council of Australian University Librarians.