Abstract
In recent years, much attention has been paid to autonomous vehicles and security threats on such vehicles have become an important issue. One of these examples is a command injection issue on a gateway ECU, which was reported in 2016. In order to mitigate these threats, the secure design of connected vehicle systems, which is done at the concept phase during development, has become increasingly important in industry. From this perspective, a security guideline such as JASO TP15002 which specifies two concrete methods, CRSS (CVSS Based Risk Scoring System) and RSMA (Risk Scoring Methodology for Automotive System), was made public in 2015. The latest work on the application of TP15002 to the ITU-T X.1373 standard was published in 2017. However, the risk assessment in this publication seems limited. It is not clear from this publication how systematically the risk assessment task in TP15002 can be performed at the implementation level. Another interesting question is how different methods affect the risk scores of connected vehicle systems. In this paper, we focus on the risk assessment phase in JASO TP15002. For a systematic risk assessment, we introduce an idea of asset container and propose to extend CRSS to a novel RSS (Risk Scoring System), RSS-CVSSv3, by appropriately replacing CVSSv2 vulnerability scoring system on which CRSS is based with CVSSv3. To address the above questions, we perform a comparative study on CRSS, RSMA, and RSS-CVSSv3 for multiple use cases such as a CGW (Central Gateway) and a drone, to examine the efficiency and usefulness of our methods. For this comparative purpose, we devise an interesting approach for the refinement of RSMA to the obstacles in comparing CRSS with RSMA.
1. Introduction
1.1. Positioning of This Paper
This paper is an extended version of the paper that appeared in the proceedings of the DECSoS 2017 Workshop [1]. This extended version includes about 200% new material. The main changes and increments are as follows:(i)Revision of the abstract(ii)Revision of the introduction for better description(iii)Addition of preliminary work regarding general concepts and JASO TP 15002 based(iv)Addition of new problems of difficulty in comparing CRSS and RSMA(v)Addition of new proposals of RSS-CVSSv3 and approach for comparing CRSS and RSMA(vi)Supplementation of all necessary data for case study 1 on connected car system(vii)Addition of a new result on a case study on a UAV (Unmanned Aerial Vehicle) system(viii)Addition of discussions and limitations on our proposals(ix)Supplemental related work.
1.2. The Background and Our Contributions
In recent years, there has been a significant amount of attention devoted to autonomous vehicles. These vehicles have a variety of purposes for which many systems and devices have been developed, ranging from self-driving cars, autonomous robots to aid people in their work in hazardous cities, and UAV which are commonly known as drones.
Recently, it has been known that the security is critical in vehicle systems. Many devices, objects, or entities are highly digitalized and networked in a sense, where vehicle systems can be regarded as Internet-of-Things (IoT) nodes, which allows the users to use various services via Internet. From security point of view, the communication interfaces that allow the vehicles to connect to the Internet can be considered “Entry Points” or “Attack Surfaces” for the attackers to exploit.
Since 2010, researchers have reported attacks on real vehicles, where the attacker can manipulate the control functions in such a way that the car can be controlled in an unintended manner [2, 3]. In 2016, a command injection issue on a gateway ECU, commonly known as the CWE-77 weakness, was reported, for which in CVE 2016-9337 [4] a CVSS Score 6.8, which is computed by means of the formula specified in [5], was assigned.
Regarding the security of the in-vehicle network, in particular, CAN BUS is viewed as a problem [6]. To solve the problem, in [7], the authors studied the new in-vehicle network consisting of several subnetworks, where one subnetwork is connected to another via Central Gateway (CGW).
In addition to the recent hacking of cars, researchers have also found vulnerability in certain types of UAVs (drones), which exploited information security vulnerability in 2015. The attackers took control of driving systems by drones by exploiting sensor systems [8] or interfering with flying functionality by means of attacks via Wi-Fi communications.
For UAVs, although some vulnerabilities were pointed out even before this case [9–11], it is still important to pay attention to these vulnerabilities because UAVs will be incorporated into a larger system, namely, the surrounding environment in the future. For example, the UAV may connect and share its data with other UAVs and the surrounding environmental system [12], or its flight may be controlled in a way that amateurs do not carelessly violate the security and privacy of others [13].
In response to these security concerns, secure design of connected vehicle systems, which is performed at the concept phase during the development, has become increasingly important in industry. From this perspective, a security guideline which incorporates the risk management procedures of information security has been developed [14].
One particularly useful approach is to measure software attack surfaces, which indicate susceptibility to attack. With respect to the security analysis based on attack surface, model-based metrics have recently been developed [15], dealing with the dependability evaluations and the assurance of critical systems.
Regarding IT (Information Technology) security, the concept of risk has been investigated from various perspectives such as risk management [16–18] and risk analysis (cost assessment) [19]. IT security risk assessment is discussed with respect to asset identification, lifecycle clarification, system model definition, and conceptual frameworks for risk assessment [20–23]. There is a document describing the Common Evaluation Methodology (CEM) for Information Technology Security Evaluation, which helps evaluators learn how to apply the CC (Common Criteria) when they perform formal evaluations. ISO/IEC 15408 [24] is based on the security evaluation standard developed by the CC project.
In the automotive industry, several security guidelines and methods have been proposed such as SAE J3061 [25]. JASO TP15002 [26, 27] also deals with high-level methods, as explained in a sequel. In this guideline, unlike in the previous phases, namely, model definition and threat identification, the risk assessment phase has two concrete methods, CRSS (CVSS Based Risk Scoring System) and RSMA (Risk Scoring Methodology for Automotive System). CRSS is a risk evaluation criterion based on CVSS v2 (Common Vulnerability Scoring System), where two metrics Attack Ease and Effect Factor are assigned for each identified threat. RSMA computes the risk score by referencing the Risk Level Decision Table, which consists of Occurrence Possibility and Effect Factor.
To the best of our knowledge, there has been no prior work on these methods in a comparative way, nor on which use case (in one of) these methods work reasonably.
For the last few years, Internet-connected services used in modern vehicles such as cars and drones have received a lot of attention from the public. However, in response to the recent hacking of such services, such as the exploitation of information security vulnerability, the secure design of connected vehicle systems, which is performed at the concept phase during the development, has become increasingly important in industry. From this perspective, a security guideline, JASO TP15002, was made public in 2015. The risk assessment methodology described in TP15002 is applied to the ITU-T X.1373 standard that was published in 2017, dealing with an important security use case, namely, remote software update for vehicles. However, the risk assessment of this work seems limited. The main issues are as follows:(a)It is not clear how systematically the risk assessment task in TP15002 can be performed at the implementation level.(b)Another interesting question is how different methods affect risk scores of connected vehicle systems. One of the obstacles is the difficulty in comparing CRSS and RSMA.
In this paper, we focus on the risk assessment phase in JASO TP15002. The importance of JASO TP15002 can be explained by the fact that the significant time and effort of a large automotive committee in Japan have been spent as well as their consensus on what has to be taken into account in the product design phase of the automotive development. Another explanation on its importance can be supported by the fact that the methodology of TP15002 is applied to recently developed ITU-T standard X.1373 [28] that addresses one of the most important use cases, over-the-air software updates for the vehicle devices. As a result, TP15002 identified 17 classes of threat on the model of the mobile vehicle gateway.
For a systematic risk assessment, we introduce the idea of an asset container and propose to extend CRSS to a novel Risk Scoring System (RSS)s, RSS-CVSSv3, by appropriately replacing the CVSSv2 (Common Vulnerability Scoring System version 2) vulnerability scoring system on which CRSS is based with CVSSv3. To address the above issues in an objective manner, we perform a comparative study on CRSS, RSMA, and RSS-CVSSv3 for multiple use cases such as a CGW and a drone, to examine the efficiency and usefulness of our methods. For this comparison purpose, we devise an interesting approach of refinement of RSMA to the obstacle in comparing CRSS and RSMA.
As a result, we provide some interesting data which could serve as information for better understanding of original and proposed risk assessment methods. We expect that our research will offer an information basis on which future system designers can choose the most suitable security risk assessments for designs of connected vehicles.
The analysis of our method is efficient enough to minimize reworking the guideline. This saves time by using a format which organizes items which allows an engineer to just write a simple description without any ambiguity.
The Need from Industry. This work is motivated by a need from industry with respect to the implementation of secure system design based on a high-level security guideline, such as JASO TP15002. This need is summarized by conducting a secure system design with appropriate efforts. With a naive implementation of JASO TP15002, the following problems require more effort to keep the quality of the result:(i)There are nondetailed matters in JASO TP 15002. Thus individual designers have to refine or instantiate these matters in their own ways.(ii)Typically a very large number of threats are identified, which cannot be managed by a designer.(iii)The quality of output, such as granularity, the amount of description, or validity depends on individual designers.
The Industrial Impact. Our work contributes to the secure development of devices such as a CGW, telematics, or end ECUs that play key roles in Over-The-Air (OTA) applications. This is supported by the evidence that JASO TP15002 is used as a risk assessment methodology in the on-going ITU-T development “secure software update capability for intelligent transportation system communication devices” which was discussed in the UNECE World Forum for Harmonization of Vehicle Regulations (WP.29) forum in 2016 [29]. Consequently, we expect that our work will help in reducing efforts for secure system design, with sufficient quality.
Our Contribution. Our first contribution in this paper is for engineering. For system designers who aim to comply with JASO TP15002, we propose the notion of an asset container which leads to a systematic way for secure design, which is consistent over phases 1, 2, and 3, in JASO TP15002. Our approach is to identify gaps between the descriptions in JASO TP15002 and the actual design tasks. Our second contribution in this paper is scientific. We propose a risk assessment method RSS-CVSSv3 based on the well-known vulnerability scoring system CVSS v3 for a reference to future update of JASO TP15002. To pursue our interest in comparing risk assessment methods included in this guideline, we provide an approach to refine RSMA to make the discussion of our risk assessment methods more concrete and more comparative, we focus on particular types of vehicle systems, such as connected cars employing CGW and UAVs.
1.3. Related Work
1.3.1. Vulnerabilities in Cars
Although applying Information Technologies accelerates the evolution of automobiles, it also provides attackers with opportunities to exploit automobiles. As mentioned in [30], implementing intelligent features, especially, like lane keeping assist, may allow critical components, like the steering system, to be controlled from the outside.
In 2010, experimental attacks on actual cars were reported [2, 3]. In those reports, external devices were wired to the in-car network. Attackers could access the network directly; they analyzed traffic and injected fake messages in order to cause irregular behaviors on the car. Furthermore, if the attacker can physically access the car, the attacks are even easier to make. For example, a denial of service attack called the bus-off attack is one of such attacks. Attackers have to use sequences of signals according to the specification, when the car is not physically accessible [31]. Attackers can attack by sequences of signals not according to the specification if they access a car physically [32, 33].
The authors of [2] and [3] discussed remote attacks on actual cars in [34] and [30], respectively. In [30], vulnerabilities in the telematics of a car system and exploits to them were shown in detail. In this case, an attacker could access the telematics easily and send commands to equipment in the car via wireless communication. In 2016, an attack on an actual car was reported by finding a vulnerability assigned CVE 2016-9337 [4] of the Command Injection, for which the CVSS v3.0 Base Score, equal to the RSS-CVSSv3 score, was 6.8. The relevant module was the vehicle’s gateway ECU, which our Case Study 1 dealt with in Section 6. Due to this vulnerability, an attacker installed malicious software by sending messages to the vehicle’s CAN BUS.
With a fake Wi-Fi access point, attackers accessed the target car system, and then they exploited vulnerabilities in the browser [35]. They were then able to control the car even when it was running. The vulnerabilities allowing a remote attack via a cellular network Telematics Control Unit, when present on actual vehicles produced in a certain period, were reported. For one of these vulnerabilities assigned CVE 2018-9318 [36], the CVSS v3.0 Base Score, equal to RSS-CVSSv3 score, is very high, 9.8.
Attacks on specific functionalities of cars using close-range wireless communication have also been revealed. For instance, a car immobilizer is a system that typically requires the presence of a key fob to allow a car to run. An attack where the attacker was able to track car keys by exploiting vulnerabilities in a car immobilizer was reported in [37] in 2012. Cracking car immobilizer systems damages the functionalities and a cracked car system may run without the owner’s key or stop suddenly on a high way [38]. Security concerns in Tire Pressure Monitoring Systems (TPMSs) were reported in [39]. In this case, when messages are communicated, neither authentication nor input validation can take place.
1.3.2. Vulnerabilities in Drones
For many years it has been clear that drones categorized as Unmanned Aerial Vehicles (UAVs) have been used for military purposes of surveillance, but currently drones have become widely known to the public. Companies have given much easier access to drones than before.
In 2017, researchers pointed out that certain drones sold in the market are vulnerable [8].
Consequently, US-CERT summarized these vulnerabilities and informed the public that certain types of drones allow full file permissions to anonymous users by providing FTP access over its own local access points [40].
The vulnerability note VU#334207[40] and CVE-2017-3209[41] assigned to it can be used to achieve several goals, including that the attackers intentionally cause accidents on a drone. In 2015, in Seattle, an accident of this kind actually occurred [42].
1.3.3. Standardization Activities and Security Guidelines
For the automotive industry, several security guidelines and methods have been proposed. SAE J 3061 [25] is a high-level guideline which includes a definition of the lifecycle process, information for existing tools, and methods.
OCTAVE Allegro [43] is a process-driven threat assessment method. It includes methods for threat identification, risk assessment, and the selection of mitigation approaches. The EVITA method [44] refines abstract attacks into concrete attacks through the construction of Attack Trees and the evaluation of the risks of abstract attacks using those Attack Trees. In the EVITA methodology, risk is evaluated by severity, which includes 4 aspects (safety, finance, privacy, and operation) and attack probability based on the framework of Common Criteria (CC) in which system users can specify their security functional requirements.
The term impact in TP 15002 has the same meaning as in EVITA.
Since 2016, there has been an important standardization joint working project between ISO and SAE, namely, ISO/SAE. This standardization activity on cybersecurity engineering, expected to be published as ISO/SAE 21434, adopts the risk-oriented approach. It attempts to specify Cybersecurity Assurance Level (CAL) [45] which indicates the required level of cybersecurity process rigor, though it is now unclear whether CAL will be normative or informative. In the project, the CAL and the methodology to determine the CAL level will be provided for all the systems and devices throughout the entire automotive supply chain.
1.4. Organization
In Sections 2 and 3, we will introduce preliminary work related to this paper. In Section 4, we will identify the problems in previous work and then we propose our method in Section 5. In Sections 6 and 7, we conduct case studies on the secure designs of publicly available automotive systems. Finally, in Sections 8 and 9, we will discuss our approach and our results and explain the limitations of our work. In Section 10, we present our conclusion.
2. Preliminary 1: Concepts and Framework
2.1. Common Concepts Underlying Safety and Security
We consider that safety is relevant in this paper because the risk score systems we address such as CRSS incorporate criticality of safety as impact factor during their computations. In the automotive industry, it is increasingly important to develop methods for evaluating multiple attributes at the same time and to provide useful information on the entire system. For instance, a methodology for modeling automotive software security, privacy, usability, and reliability has been developed by Ford [46].
The perspectives of CIA (Confidentiality, Integrity, and Availability) of security are increasingly important in safety-critical systems and a security-for-safety paradigm is necessary when hazards originate from threats. A considerable amount of published work exists in this area, for example [47–49].
Automotive systems are typically known as safety-critical systems and the system development process follows the standard ISO 26262 [50], which deals with safety issues. In safety-critical systems, several methodologies of safety analysis and design have been established [51–55], but security was not considered. Various analysis and design methods of security have been proposed in Information Technology [43, 48, 56–62]. Safety-critical systems other than automotive systems have begun to consider the relationship between safety and security. In railway control systems, security affects safety and economy [63]. This relationship between safety and security is called “security informed safety” [64]. Thus, in railway control systems, security is analyzed from two aspects, unsafe train movement and no train movement. In the avionic security standard DO-326A [65], security is analyzed independently. Security-related activities refer to the results of safety related activities but not vice versa. In the FP7 research project SESAMO (SEcurity and SAfety MOdelling), security and safety are analyzed independently [66]. Security and safety requirements derived from analysis converge on a trade-off analysis. Thus, in the SESAMO, security and safety are considered at the same level and affect each other. In industrial control systems, several approaches for combining safety and security including the above concepts are also considered [67, 68].
Referring to well-established work in related disciplines [48], here we give a solid foundation for safety and security engineering by recognizing the similarities and differences in these disciplines. In [48], security and safety are defined as follows:(i)Safety is the degree to which accidental harm is prevented, reduced, and properly reacted to.(ii)Security is the degree to which malicious harm is prevented, reduced, and properly reacted to.
As SAE J3061 showed some overlaps between safety and security exist. But the differences are not entirely clear. For instance, a malicious attack that compromises the integrity of an ECU could eventually lead to accidental harm caused by an operation parameter change in terms of safety. A more generic quality factor that includes these two quality factors is dependability, which is defined as follows:(i)Dependability [69] is the degree to which various kinds of users can depend on a work product.
Hazard is a situation that increases the likelihood of one or more related accidents [70]. In ISO 26262, a hazard is defined as a potential source of harm caused by the malfunctioning behavior of the item. Note that this definition is restricted to the scope of this standard. A hazard caused by unintended failure and a threat of attacks by attackers can cause harm to assets. More specifically, hazards and threats are defined as follows:(i)Hazard is the potential source of harm to an asset due to unintended failure of the system. Note that the harm is typically restricted to humans or the environment.(ii)Threat is the potential source of harm to an asset due to attackers.
The information models of safety and security engineering are significantly similar. For this reason, safety and security requirements can be analyzed regarding a risk-oriented, asset-based approach that considers the accompanied hazards and threats from which these assets have to be protected.
In the above definitions, safety deals with hazards, while security deals with attacks. In order to establish a link between safety and security, the “security-for-safety” paradigm can be introduced in the sense that threats (e.g., intentional hacking attempts) cause hazards that are a potential source of harm.
2.2. The ISO/IEC 15408 Framework in Comparison with ISO 26262
In the automotive industry, there has been an increasing need to develop a framework in which security functions are used in reference to the information system field where ISO/IEC 15408 plays a key role. The purpose of ISO/IEC 15408 is to establish security objectives and evaluation procedure for a Target of Evaluation (TOE), which is a statement that counters identified threats and satisfies assumptions.
In the context of the evaluation of IT products, ISO/IEC 15408 [24] provides a framework and defines security concepts. This standard uses the term TOE where there are modules employing the assets to be protected. In this paper, after defining a TOE, threat identification is conducted and security objectives regarding the major threats are considered.
Referring to [71], we explain the similarities and differences between the ISO/IEC 15408 TOE and an ISO 26262 item definition. Regarding the ISO 26262 item, most required parts of the reference, the overview and description of ISO/IEC 15408 TOE, have been already described. However, ISO/IEC 15408 TOE is extended with an overview of the included security features such as the CIA (Confidentiality, Integrity, and Availability) perspectives and the functionality of the ISO 26262 item. Afterwards, hazard analysis and risk assessment are performed so that potential hazards and their operational situations are identified and an Automotive Safety Integrity Level (ASIL) is determined. Then safety goals are determined for hazardous events (the combination of a hazard and an operational situation [50]).
3. Preliminary 2: Security Evaluation in JASO TP15002
Since 2012, the Society of Automotive Engineers of Japan (JSAE), Inc., has developed a standard procedure for security system design, yielding the guideline JASO TP15002. Its purpose is to describe standard procedures that define the security functions. It considers the prevention of unauthorized operations of the functions that transmit the information to control ECUs (e.g., Body Domain ECUs) from servers and other devices.
In JASO TP15002, the secure system design process consists of five phases: TOE definition, threat analysis, risk assessment, defined security objectives, and security requirement selection. The first three phases are related to security evaluation, while the remaining phrases are about the treatment of major security risks. Hence the tasks and deliverables for the first three phases are reviewed in this subsection.
Another example of security design is that Firesmith [72] defines an asset-based risk-driven procedure for the identification and analysis of security requirements, which consists of 13 general steps.
Also ISO/IEC 27005 [73], based on the international standard ISO 31000 for risk management [74], specifies a security risk management process consisting of 5 iterative activities with 3 activities conducted in activities other than the design phase of the product. Security processes in this literature start with identification of the target or the environment. The processes extract threats and then evaluate and prioritize them. Finally the processes decide treatment of threats as requirements.
3.1. Phase 1: TOE Definition
This phase clarifies the structure of TOE as a model. This model is produced from the assets to be protected and from Data Flow Diagrams (DFD) that specify data flows between the modules in TOE. Thus, the model shows the network structures and entry points in the TOE. Next, properties to protect the CIA perspectives are assigned for each asset in the modules. It is important that the functions of an automotive system operate as correctly as expected, and therefore integrity or availability should be ensured for functions. Similarly, confidentiality or integrity should be ensured for the information exchanged among devices, and for external central servers as well. Information used in the sequel, such as lifecycles and modules of the TOE, is also specified in this phase. Finally, all information related to security evaluation is formalized and shared.
3.2. Phase 2: Threat Analysis
Threats together with the situations they occur in are listed exhaustively in this phase. First, assumptions are stated for feasible analysis. It can be assumed that events or situations that may occur logically but do not actually happen for some reasons are ignored in analysis. Next, adverse actions are investigated exhaustively. The situation of an adverse action is identified with 5W perspectives (“Who,” “When,” “Where,” “Why,” and “What,”). These perspectives take values from information identified in Phase 1 (see Table 1). For example, the factor “When” is valued in the lifecycle phases of the TOE. For each combination of entry points and assets, possible adverse actions should be considered. Moreover, organizational security policy is stated in this phase. Regulations or policies that should be treated with but are not derived from the deliverables of Phase 1 are identified.
3.3. Phase 3: Risk Assessment
This phase estimates the risk levels for all the threats that have been identified in the previous phase. The result tells us about critical threats we have to treat. Two risk evaluation criteria are mentioned in JASO TP15002, namely, CRSS (CVSS v2 [75] Based Risk Scoring System) and RSMA (Risk Scoring Methodology for Automotive Systems). They are reviewed in the following subsections.
In this risk assessment phase, major threats are analyzed more deeply. Using fault trees, a threat is decomposed into combinations of concrete events or situations step by step. Finally basic events, which are the origins of a threat, are identified. We treat threats of TOE by considering the treatment of basic events (conducted in the next phase).
3.3.1. CVSS Based Risk Scoring System
CRSS is a risk evaluation criterion based on CVSS v2 (Common Vulnerability Scoring System version 2). For each threat identified in Phase 2, two metrics, exploitability and impact, are assigned, and the risk value is computed from them. Exploitability depends on how close an attacker must be to the TOE for the attack, how far the entry point is from the asset logically, and how much an attacker must pass authentication to reach the asset. Impact depends on the severity of the threat. Finally, the risk for a threat is categorized into three levels according to the computed risk value: Level III (Critical), Level II (Warning), and Level I (Caution). An example of a result at Phases 2 and 3 in JASO TP15002 is shown in Table 1.
3.3.2. Risk Scoring Methodology for Automotive Systems
In addition to CRSS, JASE also proposed RSMA (Risk Scoring Methodology for Automotive Systems) in JASO TP15002. RSMA is a method adopting the concept defined by ISO/IEC 27005 [73]. It computes the risk score by referencing the Risk Level Decision Table, which consists of Effect Factor and Occurrence Possibility. More specifically, RSMA has the following procedures:(1)Based on the attack scenario of the threat, decide on the values of the five metrics in the Likelihood list shown in Table 2 (see Section 6.6), sum up their values, and decide on the Occurrence Possibility level with reference to Table 3.(2)Classify the asset to be attacked into three categories (Damage Category), and decide on the Effect Factor.(3)Decide on RSMA Risk Level referring to Table 4.
3.4. Inputs and Outputs at Each Phase in JASO-Based Risk Assessment
The outputs and inputs at each phase in JASO TP15002 and the inputs that we propose in our risk assessment proposals and their relations are shown in Table 6, in Table 5, and in Figure 1, respectively.
4. Observations and Problems
4.1. A Concept Model of JASO TP15002
We have reviewed the concepts in JASO TP15002. Figure 2 shows a partial result; concepts in Phases 1, 2, and 3 are identified and related transversely. All concepts are classified into three groups depicted as packages. The first package system-related information represents information about the entire vehicle system. Concepts in this package are the inputs to the TP15002 process. System outline consists of three concepts: system configuration, system functions, and information that the system handles. The second package TOE system represents information actually dealt with in the TP15002 process. Concepts in this package are determined by referring to concepts in system-related information. Component of the TOE system is a part of TOE system, including entry point as a subconcept. Information flow is a part of TOE system as well. Module function is related to asset to be protected and refers to component of the TOE system. The third package threat information represents the result of risk assessment in TP15002. Threat is determined with the help of concepts in TOE system. Attack Ease (AE) and Effect Factor (EF) are calculated by referring to threat, and they determine risk value as a result. Cause is analyzed for threat when required.
Our model makes it clear that there is in-dependency of lifecycle from TOE model. In Phase 2, the lifecycle contributes to the perspectives of “Who” and “When” and thus can be dealt with independently.
4.2. Observations and Problems with Previous Work
We made the following observations and identified the following problems with each phase.
4.2.1. Observation in JASO TP15002 Phase 1
Vehicle systems are typically complex. For instance, a whole car employs up to 70 ECUs, which are connected via various protocols such as CAN. It is possible that the in-vehicle devices are developed by different suppliers and vendors. Therefore, the way to form a TOE model depends on them. This means that the TOE could be very different and the model may not be elaborated in a suitable manner. Since a TOE model consists of just functions (and assets in them) and information flows, systems dealing with them in vehicles can be TOEs in general. For instance, it is reasonable that car manufactures consider the whole car as the TOE. However, some suppliers consider only a group of in-vehicle devices responsible for infotainment as the TOE. We observe that, even when there are two TOEs (the whole vehicle and some parts of it) for the same vehicle, it is important to the outcomes of the security design that these two TOEs are consistent with each other and any interactions have to be clear.
4.2.2. Problems in JASO TP15002 Phase 2
Phase 2 of JASO TP15002 suggests that the 5W (“Who,” “When,” “Where,” “Why,” and “What”) perspectives should be considered at the same time for each threat. However, we encounter a problem, namely, the difficulty in exhaustively identifying all the possible threat descriptions from the perspectives of “Where” and “What.”
We point out that this difficulty originates from ambiguity in the definitions of the 5W characteristics. More specifically, there is a problem with the definition of “What,” specifically, what the attacker does. The description for the perspective of “What” could be ambiguous or abstract such as ‘breaking the central gateway;’ hence, the threat identification (Phase 2) might be performed in an ad hoc manner. This could result in producing useless and unnecessary descriptions (data) of threats for Phase 3. On the other hand, important threats might not be identified as a consequence of this ad hoc search and the entire process might end up with a reworking of Phase 2. The 5W method can identify few hundreds or even thousands of threats. Phase 2 might not be performed in a systematic manner; therefore, an exact description of “What” is unclear. Another problem is that it is unclear how logically and closely this phase task is related to tasks in Phase 1 and Phase 3. More specifically, a question arises as to which metric for risk assessment is related to “Where,” “When,”..., or, “What.”
4.2.3. Problems in JASO TP15002 Phase 3
CRSS specified in JASO TP15002 is based on the CVSS v2 vulnerability scoring system that was published in 2007. However, CVSS v3 has also been published. Therefore it is necessary to consider whether CRSS is suitable for current or future vehicle systems. RSMA, another method, may be more suitable than CRSS. One of the obstacles in determining a suitable method is the difficulty in comparing CRSS and RSMA.
4.2.4. Problem in Application of JASO TP15002 to ITU-T X.1373
The latest work on JASO TP15002 can be referred to in the ITU-T X.1373 [28] standard that was published in 2017, dealing with an important security use case, namely, remote software updates for vehicles. The standard defines TOE for VMG (Vehicle Mobile Gateway) which is recognized as a key component of a secure software update.
The problem we identified with this work is that the investigation of the risk assessment seems limited. The TOE contains only one module, VMG, which supports nine functions that include mobile communication function, software get function, remote software update function, GPS reception function, Wi-Fi connection function, USB connection function, CAN communication function, CAN gateway function, and OBD connection function, each of which has one asset or a few assets. The threat analysis result shows the relatively small number of 19 threats. In addition, the risk scores are computed for only 3 threats.
This work is reasonable for the purpose of grasping the idea of how the JASO-defined 3-phase risk methodology works. However, in practice, the system is likely to be more complex. There is an important gap between this work and a thorough study. It is not clear how systematically the risk assessment task in TP15002 can be performed at the implementation level. Another interesting question is how different methods affect the distribution of risk scores for the same connected vehicle system.
4.3. A Taxonomy Observation of the JASO Design Process
Our analysis identified a problem with the JASO design process by investigating the well-known dependability taxonomy dealing with safety and security described in Section 2 based on [48]. As mentioned there, some security threats cause failure in the target system, which is a concern in safety. Indeed, in our case study in Section 6, the threat that control function in Chassis module suffers from a malware is identified. This threat can easily cause an accident in a vehicle.
JASO TP15002 describes the CIA perspectives of security but it does not show a link to safety. In addition, although JASO TP15002 focuses on security, its method allows us to pick up safety-related threats. Hence it is worth distinguishing safety-related threats in threat analysis, and letting safety evaluation refer to the result.
5. Our Proposed Methods
In this section, we propose our methods for solving the problems stated in the previous section. Our methods include an efficient and systematic way of implementing the 3-phase risk assessment methodology specified in the JASO TP15002 guideline.
5.1. Proposed Notion: An Asset Container for a Systematic Approach in Phases 2 and 3
To solve the problem stated in the previous section, we propose a new description for “What” using “At” (attack destination) and “Asset” (the targeted asset) at the threat analysis phase (Phase 2). Our objective is to improve the efficiency of the total work of Phase 2 (threat identification) and Phase 3 (risk evaluation).
Our idea behind this approach is that the Phase-2 work identifies threats in a systematic manner and this work only outputs a list of threats whose description contains information necessary and useful for the risk-score computation in Phase 3.
The perspective of “Where” represents one of the entry points in JASO TP15002, but, in our approach, the perspective of “What” describes what asset of what asset container is attacked. Next, we define the asset container as a module that contains the assets to be targeted as the “At” perspective (see Figure 3).
Therefore, it is important to distinguish these two perspectives strictly from the beginning of Phase 2. On the other hand, only two perspectives, “Where” and “What”(= “At” + “Asset”) are necessary to cover all threats, and the remaining three are not necessary up to Phase 3. Though a division of “What” into “At” and “Asset” is added, the number of perspectives for risk analysis drops from five to three, so the workability will be increased.
In our paper, by considering the interdependency we described in the previous section, we propose a 2-step threat identification as follows.
Step 1. Exhaustively identify threats, each of which describes the following three perspectives: “Where,” “At,” and “Asset.”
Step 2. For the identified threats, identify associated threats each of which adds descriptions of the following three perspectives: “Who,” “When,” and “Why.”
Step 1 refers to the TOE model and the module function list, both of which have been produced at Phase 1 in the JASO design process. Step 2 refers to the lifecycle list, which has been produced at Phase 1 in the JASO design process.
After applying the above two steps, our method outputs information shown in Table 7 which includes the data produced by Phases 2 and 3 in JASO TP 15002.
Advantage of Our Method. Our method has three advantages over a naive implementation of JASO:(i) From a working procedural perspective, we make progress in formalizing a security evaluation job. Our method can, especially, separate this job into two tasks, thus enabling an optimization of this job in the sense that our method can refactor in factors necessary for risk assessment. By introducing an asset container which covers “At” and “Asset” in addition to the“What” perspective, all intrusion routes are clearly and systematically identified and all attack targets can be grasped without exception.(ii) From a delivery perspective, our method can improve the quality of the delivery (output) of security because we can remove ambiguity from the threat descriptions. An advantage may exist that encompasses all threats and an accurate computation of risk scores for each can be realized.(iii) From an operational perspective, we propose our method as a practical technique that makes clear what the developer and the factory operator who are not necessarily security specialists should keep in mind.
5.2. Proposed Method: RSS-CVSSv3
We propose a novel risk assessment method, RSS-CVSSv3: The risk scoring system used CVSS version 3 in place of CVSS version 2 of CRSS.
5.2.1. The Idea Inspired by the CRSS Construction
We take a close look at CRSS based on CVSS metrics. We notice that this method makes a clear distinction between what the system designers can take into account during the concept phase and what has to be taken into account after the release of the product to the market. Our view is that this is why CRSS metrics only include six CVSS v2 base metrics (AV, AC, Au, C, I, and A) that are reasonably considered during the concept phase while these metrics exclude CVSS v2 metrics categorized as three temporal or five environmental metrics. We observe that this distinction comes from the important difference between what risk assessment is in the context of security design and what vulnerability analysis is. We note that the former is within the scope of CRSS and the latter is within the scope of CVSS.
As mentioned above, JASO TP15002 adopts CRSS as the quantification method for risk assessment at Phase 3. However, CVSS used for CRSS (version 2)[76] has a new version (version 3)[5]. Many security standards or guidelines have been updated to mitigate security attacks. These standards have only got better and new vulnerabilities are reported on a daily basis. It is natural for us to assume that TP 15002 could be updated to address the latest trends of vulnerability in the future.
5.2.2. RSS-CVSSv3 Score Computation
Against this background we propose a novel risk assessment method, namely, RSS-CVSSv3, based on applying the scoring systems, CVSSv3 (Common Vulnerability Scoring System version 3), which is a new system that scores vulnerability in embedded systems and standardized in ITU-T X.1521(03/2016)[5].
The idea is to include only the basic metrics of CVSSv3 shown in Table 8 because our focus is on security design rather than vulnerability analysis. Compared with CVSSv2, there are some changes of metrics in CVSSv3. To illustrate this, Table 8 also shows some differences of the base metrics between the two standards.
“S”(Scope) is a new metric of CVSSv3, which indicates whether the vulnerability spreads to other resources beyond the exploited component, and from which the formula changes a little. As Scope is unchanged normally, the formula of RSS-CVSSv3 is similar to that of CRSS.
The Formula for RSS-CVSSv3: Based on [5], the score is calculated as follows, depending on parameter .
If the Scope is Unchanged(S=U):Else if the Scope is Changed(S=C):
5.3. Proposed Approach for Comparing CRSS with RSMA
RSMA is a system that evaluates risk based on an attack scenario and the environment of the attacker, whereas CRSS is a system that evaluates risk based on the asset to be attacked. Therefore, in this paper we attempt to compare their results and to analyze the causes of the difference.
As mentioned previously, JASO TP15002 adopts CRSS as the quantitative risk assessment method at Phase 3, while RSMA is not quantitative. Therefore, we need to customize RSMA in a way that we can calculate figures as the assessed value, compare their results, and analyze the cause of the differences.
In the following formula we adopt RSMA methodology to our risk scoring system. Hereinafter we refer to this method as Q-RSMA (Quantifiable-RSMA). In the original RSMA, the Occurrence Possibility for a threat is decided by evaluating the sum of metrics at the three levels. Here we apply the values of metrics directly. Four levels for Effect Factor (None, Small, Medium, and Large) are converted in numerical forms. In the formula, the Effect Factor values (0.0, 10.0, 20.0, and 30.0) and the constant (34.0, and 100.0) are decided so that the value takes between 0 and 10 and their histogram has one peak.
This method uses the multiplication of Occurrence Possibility and Effect Factor, whereas CRSS uses the addition of Attack Ease and Effect Factor. RSS-CVSSv3 also uses the addition of Exploitability and Impact. Q-RSMA might have different values and prioritized threat lists from CRSS and RSS-CVSSv3. There is a difference in multiplication and addition. The reason why we choose different approach is that we are interested in how different the results of different risk scoring systems are for the same TOE. We expect that our work will be a good reference for the designers to choose more appropriate risk scoring systems for their TOE.
The Formula for Q-RSMA:
In Table 9 there is an Effect Factor list used for Q-RSMA. The RSMA Effect Factor is numerized in Table 9. This table is added for the quantification of the RSMA result.
6. Case Study 1: A CGW-Employed Connected-Car System
Here we present our results on a case study where the target system is a connected-car system employing CGW as a key component. We apply our method to this case study and then we show the prioritized threats. First, we define the following system specifications of our connected car system:(i)Functional safety of each ECU is maintained by itself.(ii)Safety is maintained, even if any of the communications between the ECUs is made impossible.(iii)The connected-car system is not automated.
6.1. Results of Phase 1: TOE Definition
As an interface to the outside, there is an on-board diagnostics (OBD) connector, a mobile communication module, a global positioning system/global navigation satellite system (GPS) signal reception device, a wireless fidelity (Wi-Fi), a Bluetooth connection, a CAN connection, a universal serial bus (USB) connector, and a secure digital (SD) connector. Based on the vehicle architecture shown in [77], in Figure 4, TOE is defined as the area surrounded by the dotted line and it communicates with the outside of the vehicle via a connection interface.
6.1.1. : TOE Model
(See Figure 4).
6.1.2. : Module Function List
Based on the TOE model, its submodules and assets to be protected are shown in Table 10. As for assets in the CRSS method, their importance is determined from the viewpoints of CIA for security, Confidentiality, Integrity, and Availability.
6.1.3. : Lifecycle List
After writing the details of the TOE model, we write the lifecycle of TOE, and the people involved with this lifecycle. Table 11 shows the lifecycle list.
6.2. Results of Phase 2: Threat Analysis
We present the results on out threat extraction. In this phase, we confirmed the assumptions and conditions of the TOE modeled at Phase 1 and extracted the threats.
6.2.1. : Assumption List
Before performing this evaluation, we assume that no security functions are employed in CGW because our position is that the security functions to be employed will be made clear after the implementation of the final phase of JASO TP15002. Our study makes clear how we can prioritize the CGW-related threats, by considering all of them related to the entire vehicle structure, which we take as the TOE model.
Table 12 shows the preliminary assumptions in advancing risk assessment, which also include assumptions other than CGW.
6.2.2. : Threat List
Table 13 is a part of the threat list made by applying the asset container perspectives. We carefully cover all the combinations of the perspectives of “Where,” “At,” and “Asset,” and ‘add’ or ‘remove’ threats according to other perspectives (e.g., “Who” or “What”) and the assumption list.
In this case study, 145 threats in total have been extracted, including 15 threats related to CGW.
6.3. Results of Phase 3: Risk Assessment in the CRSS Case
In Phase 3, we prioritize these threats by their risk value, which is quantified by the CRSS method.
In advance, we need to set the criteria for each metric used for CRSS, by which each condition gets a numeric value.
6.3.1. and : Attack Ease and Effect Factor for CRSS
Table 14 shows the metrics and the criteria related to Attack Ease (AE). These AE metrics indicate ease of attack from the viewpoint of the attack route. We allocate a combination of the entry point and the module to the rank of a metric.
Table 15 shows the metrics and the criteria related to Effect Factor (EF). These EF metrics indicate the importance of the asset from the viewpoint of the security CIA and the seriousness of the attack when it is successful.
6.3.2. : Prioritized Threat List
Our study identified 145 threats in total, with the top 16 shown in Table 17. 9 threats are classified as the ‘Critical’ risk (Level III). 93 threats are classified as the ‘Warning’ risk (Level II). 43 threats are classified as the ‘Caution’ risk (Level I). Table 16 shows the definition of each risk level.
This result shows that the more highly evaluated significant asset in the vicinity of the entry point which can be accessed from a remote place is more likely to be attacked.
With respect to CGW, our study makes the following points clear:(i)Regarding the attack path, the entry point via the telematics leads to a Level III threat. This was not expected before our study. Intuitively, the OBD interface might lead to this kind of threat. However, this is not the case. Some connected cars have already had updated software (and firmware) OTA by using telematics, and the number of cars updated over the air will certainly increase in the future. Attackers will be using the OTA update mechanism to reprogram ECUs to take control of the vehicle. To prevent this attack, authenticity and integrity need to be updated and implemented in CGW. Threats from telematics and attacks to CGW control functions impact safety mechanisms.(ii)From our risk analysis result with respect to CGW, there is no significant bias among the 15 related threats with respect to the interface. This implies that uniform security countermeasures are necessary, in contrast to focusing on protecting attacks via one or a few interfaces (see Table 18).
6.4. Results of Phase 3 in the RSS-CVSSv3 Case
As mentioned in Section 5.2.2, we prioritized the threats using the RSS-CVSSv3 method in this case study (see Table 20).
6.4.1. : Exploitability for RSS-CVSSv3
In Table 19, Exploitability list is used for RSS-CVSSv3. Table 19 shows the metrics and the criteria related to exploitability. These exploitability metrics indicate ease of attack from the viewpoint of the attack route similar to the AE metrics in the CRSS method. We allocate a combination of the entry point and the module to the rank of metric.
6.4.2. : Impact for RSS-CVSSv3
: Impact for RSS-CVSSv3 is similar to the EF metrics of as shown in Appendix.
6.4.3. : Prioritized Threat List in the RSS-CVSSv3 Case
Table 20 is the prioritized thread list.
6.5. Results on Comparison of Proposed RSS-CVSSv3 with JASO CRSS
Because CVSSv3 is the evolved system of CVSSv2, the result of RSS-CVSSv3 is similar to that of CRSS. But there are some changes. For example, the number of ranks for the Attack Vector becomes four and the value of the rank “Local(L)” increases (0.395 to 0.55) in RSS-CVSSv3. As a result, the risk assessment result from OBD-II has shifted to the higher level (see Table 21).
This is the result obtained by classifying OBD-II as a higher-level network by dividing it from other physical access points such as USB, because exploiting OBD-II can affect a wider range. Since research about attacks via OBD-II has increased in recent years, we think this difference is timely. Furthermore, because the threats that ranked higher are related to CGW, the countermeasures against the CGW are also urgent ( Table 20).
6.6. Results of Phase 3 in the Q-RSMA Case
As mentioned in Section 5.3, we prioritized the threats using the Q-RSMA method in this case study.
Q-RSMA uses the same Occurrence Possibility list used for RSMA; =
Table 22 is the prioritized thread list. From the perspectives of “Who,” “When,” and “Why,” assigned values are always the same. The “Who” is ‘Outsider,’ the “When” is ‘In regular use,’ and “Why” is ‘Maliciously,’ respectively, in all the 14 high risk threats. Hence, these corresponding three columns are omitted from this table.
Although the top two show the same results as in Table 17, 8 threats are newly replaced. With their tendency to change priority, “Opportunity (O)” and “Device (D)” metrics seem to emphasize the risk value at Q-RSMA. These threats are emphasized because there is enough opportunity to attack and the attack is easy to carry out by remodeling commercial devices.
In addition, depending on the features of Q-RSMA, this method uses multiplication of “Occurrence Possibility and Effect Factor,” and the magnitude of the Effect Factor tends to influence the result significantly. For example, #141 is the fourth highest value tied in Table 17 but is the 141st in the Q-RSMA result (Value=2.30), because the asset “The Authentication Function of Infotainment” is not considered important (property/corporate value = Small) and is one-third the value compared with High (Table 23).
As mentioned in Table 23, the differences are seen as the method evaluated from the attack scenario or as a quantification method by multiplication compared with CRSS.
6.7. Safety-Related Threats Included the Result of Phase 3 in Each Case
It is interesting that the threats as an output of Phase 3 include some safety-related threats.
Both the threat on the Power-Train ECU (access) and the threat on the Chassis ECU (brake and steering) can be viewed as safety-related threats. In this case study, Threat #6, #14, and #31 are the safety-related threats.
For comparison purposes, we now examine three safety-related threats shown in Table 24, for which the risk values and the prioritized rank are associated. Although there are some differences in risk value between CRSS and RSS-CVSSv3, they are simply rank fluctuations because the classification change of Access Complexity (AC) has been made to CVSSv2 on which CRSS is based. Consequently, we consider that there is no fundamental difference in risk assessment between CRSS and RSS-CVSSv3 regarding these three safety-related threats.
For each of the three methods that we used in this study, the Impact metrics reflect how important the asset is and the Exploitability metrics reflect the ease and the technical means including the route in which the attack can be mounted from a remote place at any time (by the devices easy to purchase in the Q-RSMA case). Hence, their scores belong to a higher rank in the prioritized threat list in each case. As described above, our proposed methods identified the safety-related threats in this case study.
7. Case Study 2
In this section, we describe the components of the intradrone network formed by the interconnection of the multiple modules in the drone.
There are two reasons why we add this case study:(i)To verify whether our proposed method can be applied to other systems, that is, not only limited to the automobile industry. Precisely, we are going to confirm whether the anticipated threats are reasonably quantified by our proposed method.(ii)To visualize the threats influenced by new features, which are likely to be applied to automated driving technology. The drone has an autonomous control function in addition to a remote operation function, and we attempt to investigate what threats will be regarded as more important in the future, such as attacks on logs of the sensor information and position information in this case study.
7.1. Results of Phases 1 and 2
7.1.1. : TOE Model
There is prior research on attacks on drones where a threat model is typically established before the actual attacks are described. In [8], the threat model considers an attacker that is within Wi-Fi range of the drone but otherwise has no other direct physical access to the device. In their drone model, drones have Wi-Fi access points for their devices to be connected. According to the authors, it is very common in IoT systems that embedded devices with a Wi-Fi access point are used to communicate with an application in a smart phone.
The sensor regarding altitude, called an optical flow sensor, is used for the drone to attempt to keep the altitude constant. In [78], the authors demonstrate the effectiveness of their optical flow sensor input spoofing attack against certain drones.
On the other hand, in our TOE model, in addition to a wireless communication interface, there are more than one entry point. We also refer to recent research on drones [79, 80].
In Figure 5, TOE is defined as the area surrounded by the dotted line, which realizes communication control as a connection interface to the outside of the drone.
7.1.2. : Module Function List
The Module Function List in TOE is shown and the main security perspectives are Confidentiality (C), Integrity (I), and/or Availability (A) in order to provide security requirements as the goal of security design.
In a drone, sensors such as cameras, altitude sensors, geomagnetic sensors, and navigational systems are typically present. Drones use several kinds of sensors to fly safely. Many of them have Global Navigational Satellite Systems (GNSS) such as GPS receivers. A downward-facing optical flow camera is used to stabilize the drone.
The drone needs to use its sensors to react to its environment in a timely manner. Therefore, for instance, the cameras record video image and send this data to the drone’s navigation system to interpret the video image and trigger commands for navigation.
The battery is critical to flying safely. Lithium Polymer (Li-Po) batteries are known as good technology in terms of energy density, power density, and lifetime use.
7.1.3. : Lifecycle List
Table 26 shows the lifecycle list related to our drone system. This is similar to the case of the connected-car system except that the user performs the maintenance.
7.1.4. : Assumption List
Table 27 shows a description of assumptions. Compared to Case Study 1 (connected-car), the drone is conscious of autonomous movements, and attacks on sensors and the integrity of information necessary for flight are emphasized.
7.2. Results of Phase 3 in the CRSS Case
By applying TP 15002 (CRSS) and our method, our study identified 138 threats on our drone systems. In concrete, we applied our “Asset Container” method to the drone TOE as follows:(i)Figure 5 showed that the TOE had 7 “Where”(entry points).(ii)Table 25 showed that it had 26 combinations of “At” and “Asset.”(iii)Therefore, it was confirmed that the TOE could be subject to at least kinds of attacks.(iv) kinds of attacks were excluded because the attack from 4 entry points could only attack to each first module (see A.Attack_to_Sensor in Table 27).(v)52 kinds of attacks were added by applying STRIDE method which added the variants of “Who,” “Why,” and “What.”(vi)Finally threats were listed.
Table 28 shows a part of the prioritized threat list for our drone system () where there are 15 threats with risk value, ranging from 7.51 to 9.43, categorized as level III, the most important class of threats. From the perspectives of “Who” and “Why,” the assigned values are always the same. The “Who” is ‘Outsider,’ and the “Why” is ‘Maliciously,’ respectively, in all the 15 high risk threats. Hence these corresponding two columns are omitted from this table.
In these threats, the entry points for the attackers that come to TOE are mainly GPA and wireless communication, the remote network. Threat #26 corresponds to a lock-out attack [8] where the attacker prevents the legitimate owner of the device from connecting to it.
The threat to the electromagnetic wave and supersonic wave is ranked in, too. This fact indicates the importance of sensors. Also, the attacks targeting information necessary for autonomous flight are also ranked in the top rank. These are the features of our drone system.
7.3. On the Relevance of This Study to Real-World Drone Security
We believe that this study is of practical relevance to real-world Drone Security, so that Threat #27 and Threat #33 in Table 28 are related to the vulnerabilities in commercial drones [8].
In 2016, attacks via optical flow were reported in [78], where the authors demonstrate the effectiveness of this sensor input by mounting spoofing attacks on certain drones. From this point of view, additional relevance can be illustrated by our risk assessment comparison on Threat #6 that exploited sensor input shown in Table 29. It is interesting to us that, for this threat, our proposed RSS-CVSSv3 suggests a higher risk level, namely, three, than the other two methods.
8. Discussion
8.1. Number of Metrics Necessary for Risk Assessment Methods to Work Properly
It is worth discussing how many metrics a risk assessment method needs for the risk scores to be moderately distributed. Our intuition is that the number of these metrics may be about 5 or 6 based on what our case studies suggest. It seems to us that, if these metrics are classified in an appropriate manner, the resulting risk assessment method including the risk ranking would work properly.
8.2. Evaluation of Assets in the Asset Container Method
However, it does not appear that the risk scores are effectively distributed, in spite of the number of metrics. This ineffectiveness can be illustrated by our case-study example showing that there are six threats whose scores are the same, that is, 7.15 in CRSS, there are nine threats whose scores are the same, 6.80 in RSS-CVSSv3, and there are eight threats whose scores are the same 7.30 in Q-RSMA. One reason why several different threats tend to group in the same risk value could be due to the rough way of classifying the impact related to the asset.
Also, in Q-RSMA, where there are only four values of Effect Factor, the classification of CRSS and RSS-CVSSv3 are also rough, which can be explained because the Impact values are only a few at best, even though the asset values can be selected out of three values for each of the three metrics (Confidentiality, Integrity, and Availability).
It is possible that the other metrics concerning the attack route and method make a difference in the risk score, which does not mean that we can neglect to strictly evaluate the value of each asset.
Our proposed method of an “Asset Container” allows the system developer implementing JASO TP15002 to identify the threats in a systematic manner, without considering how the attack is mounted and what the attack means is. Although it is difficult to cover all attack variations, that is, to imagine the attacker’s potential, we can grasp all the assets and the routes to them. We think that a combination method is preferable, as it ensures comprehensiveness of threats with the “Asset Container” method, prioritizes them in risk assessment, and then attempts to cover means of them by the STRIDE method and so on. Therefore, it is important for us to classify and prioritize the assets at first. For the sake of the developers, different assets corresponding to similar values in the risk assessment phase should be avoided.
8.3. Evaluation of Routes in the Asset Container Method
We have confirmed that the other metrics concerning the attack routes, that is, the metrics of Attack Ease in CRSS and Exploitability in RSS-CVSSv3, were effective in identifying the important threats. Specifically, by comparing Tables 17 and 20, we notice that the rank of threats via OBD-II [31], which have received much attention in recent years, has become higher in RSS-CVSSv3 because the degree of attention to the metric has been changed.
8.4. On the Use of Different Methods for Threat Identification
In this paper, we mainly assume the same method threat identification by means of 5W is used for the existing methods in TP15002 and the proposed methods for risk assessment. It appears to us that the 5W method is suitable for these risk assessment methods in terms of risk score computation. On the other hand, it would be interesting whether we could find desired tradeoffs between time spent in the security design task and the granularity of the result. This could give the designers a variety of choices. We speculate that there are certain designers who would need a rough result with limited time.
9. Limitations
The scope of this research is limited in the sense that we only discuss the results up to Phase 3 in JASO TP15002. Our main focus is on the trend of how the risk values are taken and distributed through the comparison result of risk assessment methods. We are currently considering whether an efficient definition of the security objectives can be realized by efficiently distributing risk values at Phase 4: define security objectives or phase 5: security requirement selection in JASO TP15002. Our discussions in this paper regarding by which operation the risk score is derived from the multiple metrics, addition or multiplication, are limited. The only point we make clear is that the same technical paper TP15002 specifies the different approaches, namely, CRSS and RSMA. CRSS uses addition while RSMA uses multiplication.
10. Conclusion
We have investigated a promising security JASO TP15002 guideline. We have identified the difficulties in carrying out the actual design tasks based on JASO TP15002 and we have proposed a systematic way for implementing the 3-phase risk assessment methodology in the guideline.
For the systematic risk assessment, we introduced the idea of an asset container and proposed extending CRSS to a novel Risk Scoring System (RSS)s, RSS-CVSSv3, by appropriately replacing the CVSSv2 vulnerability scoring system on which CRSS is based with CVSSv3. We performed a comparative study on CRSS, RSMA, and RSS-CVSSv3 for multiple use cases such as a CGW and a drone to examine the efficiency and usefulness of our methods. For this comparison purpose, we devised an interesting approach of refinement of RSMA to the obstacles in comparing CRSS and RSMA. As a result, we have provided some interesting data so that the system developers can have a better understanding on original and proposed risk assessment methods in a comparative manner. We expect that our research will offer an information basis on which the system designers and developers can choose suitable security risk assessments for the future system designers of connected vehicles.
Appendix
A. Inputs for Case Study 1
A.1. : Impact for RSS-CVSSv3
Table 30 shows the metrics and the criteria related to impact of asset if it will be attacked. This impact metric indicates importance of asset from the viewpoint of the security CIA and the seriousness when the attack succeeds, the same as the Effect Factor (EF) at CRSS method.
B. Inputs for Case Study 2
B.1. : Attack Ease
See Table 31.
B.2. : Effect Factor for CRSS
See Table 32.
Abbreviated Terms
| ADAS: | Advanced Driver-Assistance Systems |
| AE: | Attack Ease |
| ASIL: | Automotive Safety Integrity Level |
| AT: | Attack Tree |
| BT: | Bluetooth (Communication) |
| CAL: | Cybersecurity Assurance Level |
| CAN: | Controller Area Network |
| CEM: | Common Evaluation Methodology for Information Technology Security Evaluation |
| CC: | Common Criteria |
| CERT: | Computer Emergency Response Team |
| CGW: | Central Gateway |
| CIA: | Confidentiality, Integrity, and Availability |
| CRSS: | CVSS Based Risk Scoring System |
| CVSS: | Common Vulnerability Scoring System |
| CWC: | Close-Range Wireless Communication |
| DFD: | Data Flow Diagram |
| DSRC: | Dedicated Short Range Communications |
| ECU: | Electronic Control Unit |
| EF: | Effect Factor |
| FT: | Fault Tree |
| FTP: | File Transfer Protocol |
| GPA: | GPS Antenna |
| GPS: | Global Positioning System |
| JSAE: | Society of Automotive Engineers of Japan, Inc. |
| JASO: | Japanese Automotive Standards Organization |
| IR: | Infrared (Communication) |
| ITS: | Intelligent Transport Systems |
| OBD-II: | On-Board Diagnosis second generation |
| OTA: | Over-The-Air |
| Q-RSMA: | Quantifiable-RSMA |
| RSMA: | Risk Scoring Methodology for Automotive system |
| RSS: | Risk Scoring System |
| STRIDE: | Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege |
| TOE: | Target of Evaluation |
| TPMS: | Tire Pressure Monitoring System |
| UAV: | Unmanned Aerial Vehicle |
| VMG: | Vehicle Mobile Gateway. |
Data Availability
The data used to support the findings of this study are included within the article.
Disclosure
This paper is an extended version of [1], which appeared in the proceedings of the DECSoS 2017 Workshop.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
We would like to thank the anonymous reviewers for their valuable comments and Edit Science, Inc. (http://www.editscience.com/) for English language editing.