Security and Communication Networks

Volume 2018, Article ID 2864873, 28 pages

https://doi.org/10.1155/2018/2864873

## Multistage Attack Graph Security Games: Heuristic Strategies, with Empirical Game-Theoretic Analysis

^{1}University of Oregon, Eugene, USA^{2}University of Michigan, Ann Arbor, USA

Correspondence should be addressed to Thanh H. Nguyen; moc.liamg@78hnahtgnohneyugn

Received 6 May 2018; Revised 14 October 2018; Accepted 8 November 2018; Published 13 December 2018

Academic Editor: Petros Nicopolitidis

Copyright © 2018 Thanh H. Nguyen et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

We study the problem of allocating limited security countermeasures to protect network data from cyber-attacks, for scenarios modeled by Bayesian attack graphs. We consider multistage interactions between a network administrator and cybercriminals, formulated as a security game. This formulation is capable of representing security environments with significant dynamics and uncertainty and very large strategy spaces. We propose parameterized heuristic strategies for the attacker and defender and provide detailed analysis of their time complexity. Our heuristics exploit the topological structure of attack graphs and employ sampling methods to overcome the computational complexity in predicting opponent actions. Due to the complexity of the game, we employ a simulation-based approach and perform empirical game analysis over an enumerated set of heuristic strategies. Finally, we conduct experiments in various game settings to evaluate the performance of our heuristics in defending networks, in a manner that is robust to uncertainty about the security environment.

#### 1. Introduction

*Attack graphs* are graphical models used in cybersecurity research to decompose complex security scenarios into a hierarchy of simple and quantifiable actions [1–3]. Many research efforts have employed attack graphs or related graphical security models to analyze complex security scenarios [4, 5]. In particular, attack graph models have been used to evaluate network hardening strategies, where a network administrator (the defender) deploys security countermeasures to protect network data from cybercriminals (the attacker) [6–13].

Attack graphs are particularly suitable for modeling scenarios in moving target defense (MTD) [14], where the defender employs proactive tactics to dynamically change network configurations, limiting the exposure of vulnerabilities. MTD techniques are most useful in thwarting progressive attacks [15–17], where system reconfiguration by the defender prevents the attacker from exploiting knowledge accumulated over time. Attack graphs naturally represent the progress of an attack, and the defense actions in our model can incorporate MTD methods.

We build on prior works that represent security problems with attack graphs, and we take a game-theoretic approach to reasoning about the strategic interaction between the defender and the attacker. Building on an existing Bayesian attack graph formalism [11, 18], we model the problem as a simultaneous multistage attack graph security game. Nodes in a Bayesian attack graph represent security conditions of the network system. For example, an SSH buffer overflow vulnerability in an FTP server can be considered a security condition, as can user privileges achieved as a result of exploiting that vulnerability. The defender attempts to protect a set of goal nodes (critical security conditions) in the attack graph. Conversely, the attacker, starting from some initial security conditions, follows paths through the graph to undermine these goal nodes. At every time step, the defender and the attacker simultaneously take actions. Given limited security resources, the defender has to decide for which nodes of the attack graph to deploy security countermeasures. Meanwhile, the attacker selects nodes to attack in order to progress toward the goals. The outcome of the players’ actions (whether the attacker succeeds) follows a stochastic process, which represents the success probability of the actions taken. These outcomes are only imperfectly observed by the defender, adding further uncertainty which must be considered in its strategic reasoning.

Based on our game model, we propose various parameterized strategies for both players. Our attack strategies assess the value of each possible attack action at a given time step by examining the future attack paths they enable. These paths are sequences of nodes which could feasibly be attacked in subsequent time steps (as a result of attack actions in the current time step) in order to reach goal nodes. Since there are exponentially many possible attack paths, it is impractical to evaluate all of them. Therefore, our attack heuristics incorporate two simplifying approximations. First, we estimate the attack value for each individual node locally, based on the attack values of neighboring nodes. Attack values of goal nodes, in particular, correspond to the importance of each goal node. Second, we consider only a small subset of attack paths, selected by random sampling, according to the likelihood that the attacker will successfully reach a goal node by following each path. We present a detailed analysis of the polynomial time complexity of our attack heuristics.

Likewise, our heuristic defense strategies employ simplifying assumptions. At each time step, the defense strategies: (i) update the defender’s belief about the outcome of players’ actions in the previous time step and (ii) generate a new defense action based on the updated belief and the defender’s assumption about the attacker’s strategy. For stage (i), we apply particle filtering [19] to deal with the exponential number of possible outcomes. For stage (ii), we evaluate defense candidate actions using concepts similar to those employed in the attack strategies. We show that the running time of our defense heuristics is also polynomial.

Finally, we employ a simulation-based methodology called* empirical game-theoretic analysis* (EGTA) [20], to construct and analyze game models over the heuristic strategies. We present a detailed evaluation of the proposed strategies based on this game analysis. Our experiments are conducted over a variety of game settings with different attack graph topologies. We show that our defense strategies provide high solution quality compared to multiple baselines. Furthermore, we examine the robustness of defense strategies to uncertainty about game states and about the attacker’s strategy.

#### 2. Related Work

*Attack graphs* are commonly used to provide a convenient representation for analysis of network vulnerabilities [1–3]. In an attack graph, nodes represent attack conditions of a system, and edges represent relationships among these conditions: specifically, how the achievement of specific conditions through an attacker’s actions can enable other conditions. Various graph-based security models related to attack graphs have been proposed to represent and analyze complex security scenarios [5]. For example, Wang et al. [21] introduced such a model for correlating, hypothesizing, and predicting intrusion alerts.* Temporal attack graphs*, introduced by Albanese and Jajodia [4], extend the attack graph model of Wang et al. [21] with additional temporal constraints on the unfolding of attacks.* Augmented attack trees* were introduced by Ray and Poolsapassit [22] to probabilistically measure the progression of an attacker’s actions toward compromising the system. This concept of augmented attack trees was later used to perform a forensic analysis of log files [23]. Vidalis et al. [24] presented* vulnerability trees* to capture the interdependency between different vulnerabilities of a system.* Bayesian attack graphs* [25] combine attack graphs with quantified uncertainty on attack states and relations. Revised versions of Bayesian attack graphs incorporate other security aspects such as dynamic behavior and mitigation strategies [18, 26]. Our work is based on the specific formulation of Bayesian attack graphs by Poolsappasit et al. [18]. The basic ideas of our game model and heuristic strategies would also apply to variant forms of graph-based security models with reasonable modifications. In particular, our work might be extended to handle zero-day attacks captured by zero-day attack graphs [27–30] by incorporating* payoff uncertainty* as a result of unknown locations and impacts of zero-day vulnerabilities. In the scope of this work, we consider known payoffs of players.

Though many attack graph models attempt to analyze different progressive attack scenarios without considering countermeasures of a defender, there is an important line of work on graph-based models covering both attacks and defenses [5]. For example, Wu et al. introduced* intrusion DAGs* to represent the underlying structure of attacker goals in an adaptive, intrusion-tolerant system. Each node of an intrusion DAG is associated with an alert from the intrusion-detection framework, allowing the system to automatically trigger a response.* Attack-Response Trees (ARTs)*, introduced by Zonouz et al. [13], extend attack trees to incorporate possible response actions against attacks.

Given the prominence of graph-based security models, previous work has proposed different game-theoretic solutions for finding an optimal defense policy based on those models. Durkota et al. [8, 9] study the problem of hardening the security of a network by deploying honeypots to the network to deceive the attacker. They model the problem as a Stackelberg security game in which the attacker’s plans are compactly represented using attack graphs. Zonouz et al. [13] present an automated intrusion-response system which models the security problem on an attack-response tree as a two-player, zero-sum Stackelberg stochastic game. Besides Stackelberg games, single-stage simultaneous games are also applied to model the security problem on attack-defense trees, and Nash equilibrium is used to find an optimal defense policy [6, 7, 10].

Game theory has been applied for solving various cybersecurity problems [31–38]. Previous work has modeled these security problems as a dynamic (complete/incomplete/imperfect) game and analyzed equilibrium solutions of those games. Our game model (to represent security problems with attack graphs) belongs to the class of partially observable stochastic games. Since the game is too complex for analytic solution, we focus on developing heuristic strategies for players and employing the simulation-based methodology EGTA to evaluate these strategies.

Similar to our work, the non-game-theoretic solution proposed by Miehling et al. [11] is also built on the attack graph formalism of Poolsappasit et al. [18]. In their work, the attacker’s behavior is modeled by a probabilistic spreading process, which is known by the defender. In our model, on the other hand, both the defender and the attacker dynamically decide on which actions to take at every time step, depending on their knowledge with respect to the game.

#### 3. Game Model

##### 3.1. Game Definition

*Definition 1 (Bayesian attack graph [18]). *A* Bayesian attack graph* is a directed acyclic graph, denoted by . (i) is a nonempty set of nodes, representing security-related attributes of the network systems including (i) system vulnerabilities; (ii) insecure system properties such as corrupted files or memory access permissions; (iii) insecure network properties such as unsafe network conditions or unsafe firewall properties; and (iv) access privilege conditions such as user account or root account.(ii)At time , each node has a* state*, where 0 means is* inactive* (i.e., a security state not compromised by the attacker) and 1 means it is* active* (compromised). The* initial state* represents the initial setting of node . For example, suppose a node (representing the root access privilege on a machine) is active; this means the attacker has the root access privilege on that machine.(iii) is a set of directed edges between nodes in , with each edge representing an* atomic attack action* or an* exploit*. For edge , is called a* precondition* and is called a* postcondition*. For example, suppose a node represents the SSHD BOF vulnerability on machine and node represents the root access privilege on . Then the exploit indicates that the attacker can exploit the SSHD BOF vulnerability on to obtain the root access privilege on . We denote by the set of preconditions and by the set of postconditions associated with node . An exploit is* feasible* when its precondition is active.(iv)Nodes without preconditions are called* root nodes*. Nodes without postconditions are called* leaf nodes*.(v)Each node is assigned a* node type*. An -type node can be activated via any of the feasible exploits into . Activating an -type node requires all exploits into to be feasible and taken. Root nodes (-type nodes without preconditions) have no prerequisite exploits and so can be activated directly. We denote by the set of all -type nodes and by the set of all -type nodes. The sets of edges into -type nodes and -type nodes, respectively, are denoted by and .(vi)The* activation probability* of edge represents the probability that the -node becomes active when the exploit is taken (assuming is not defended, is active, and no other exploit is attacked). -type nodes are also associated with activation probabilities; is the probability that becomes active when all exploits into are taken (assuming is not defended and all parent nodes of are active).

An example of a Bayesian attack graph based on this definition is shown in Figure 1. Our multistage Bayesian attack graph security game model is defined as follows.