Abstract
With the occurrence of cyber security incidents, the value of threat intelligence is coming to the fore. Timely extracting Indicator of Compromise (IOC) from cyber threat intelligence can quickly respond to threats. However, the sparse text in public threat intelligence scatters useful information, which makes it challenging to assess unstructured threat intelligence. In this paper, we proposed Cyber Threat Intelligence Automated Assessment Model (TIAM), a method to automatically assess highly sparse threat intelligence from multiple dimensions. TIAM implemented automatic classification of threat intelligence based on feature extraction, defined assessment criteria to quantify the value of threat intelligence, and combined ATT&CK to identify attack techniques related to IOC. Finally, we associated the identified IOCs, ATT&CK techniques, and intelligence quantification results. The experimental results shown that TIAM could better assess threat intelligence and help security managers to obtain valuable cyber threat intelligence.
1. Introduction
With the development of technologies, the scale of online devices has reached 6 billion [1]. Although the Internet has facilitated people’s lives in various aspects, it cannot be ignored that the risks of information exposure during data transmission are increasing day by day [2]. Attackers may explore cyber vulnerabilities, launch attacks to obtain private information, and finally conduct malicious attacks [3]. Therefore, it is vital to protect the privacy of users in a network environment [4]. Several traditional defence methods against network have existed: firewalls, system patches, authentication, information encryption, and intrusion detection systems, etc. Since cyber attacks are becoming more sophisticated, traditional security protection strategies based on passive defence measures are difficult to deal with various types of attacks. The main reasons are as follows: (i) Vulnerabilities are unpredictable, and the attack methods used by attackers are constantly changing. It is difficult to use a general method to deal with network attacks. (ii) For advanced persistent threats, feature detection-based protection technology has been failed, and traditional means have been unable to cope. Therefore, it is imperative to strengthen the cyber security capabilities based on vulnerability analysis and cyber threat intelligence (CTI) information extraction and improve the network’s active security defence performance [5]. CTI is able to describe the attack behavior, provide the context of the network attack, and guide how to defend against the attack, which can play a crucial role in network security protection.
IOC (Indicator of Compromise) describes the behavioral characteristics of cyber threats, including static information (such as signatures), and dynamic characteristics (such as the behaviors that malware takes on the victim’s computer). Meantime, it reveals the attack strategy adopted by attackers, and the strategy can be used to match existing network threats and discover variants or similar cyber-attack cases [6]. Existing CTIs usually come in the form of IOCs. Once these IOCs are collected by threat intelligence platforms and formatted according to threat information sharing standards such as Trusted Automated Exchange of Intelligence Information (TAXII), they can be automatically converted and imported into various defence mechanisms such as intrusion detection systems. The virtue of threat intelligence collection resides in that data used as a source of threat alerts can be extracted from cyber open source intelligence (OSINT) [7] based on the specific demand of the organization. Due to the uncertainty of threat intelligence data sources, the same category of threat intelligence might exist from multiple sources; hence, their quality and credibility must be assessed to avoid “data poisoning.” Reliability and quality are the key assessment factors of OSINT.
ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) is a knowledge base that reflects the attack life cycle and attack behavior. ATT&CK organizes the adversary’s tactics and techniques through real-world observations. It can analyze the TTPs potentially used to execute an attack from the attacker’s perspective and displays them in the form of a matrix [8]. Threat intelligence provides insight into attackers and their malicious activity, including context, indicators, and operating recommendations. Both ATT&CK and threat intelligence involve the study of attacker information. Therefore, we apply the ATT&CK matrix to threat intelligence assessment.
Extracting threat information from CTI reports faces two main difficulties: (i) Since CTI reports are written in natural language, extracting structured attack behaviors requires analyzing the content in unstructured CTI text (i.e., sparse text). (ii) Attack information is scattered in the report, which makes it difficult to fully analyze the techniques attackers may use. For the above two reasons, we propose TIAM, a model for automatically assessing CTI. The contributions of this paper are mainly reflected in three aspects: (i)Propose a quality assessment model of threat intelligence, which processes highly sparse text and automatically calculates the score of intelligence(ii)Propose an automatic threat classification method based on feature extraction, which takes dictionary word density and feature word weight as classification criteria(iii)Correlate the ATT&CK attack matrix with CTIs for associating threat intelligence with attacking organizations. Experiments show TIAM can automatically identify the tactics and techniques used by the attacker
In this report, we first discuss existing methods and related research work in Section 2, describe the currently widely used threat intelligence sharing standards and platforms in Section 3, and then propose a new threat intelligence assessment methodology in Section 4. Subsequently, the relevant experimental verification and results analysis are presented in Section 5. Finally, Section 6 provides the concluding remarks.
2. Related Work
CTI plays an important role in cyber security. However, most of the threat intelligence is inaccurate, incomplete, and out of date. Therefore, low-quality CTI is identified to be a pivotal issue [9]. In contrary, high-quality CTI can promote identifying potential threats at early stage or even prevent network attacks.
The quality of threat intelligence can be assessed from two aspects: threat intelligence quantification and threat information extraction, which will be discussed in detail in the following.
2.1. Quantification of Threat Intelligence
Large numbers of works have been done on threat intelligence quality assessment. Some scholars quantified threat intelligence from the user’s perspective. Li et al. [10] introduced five-dimension criteria, namely, availability, reliability, usability, relevance, and presentation quality, and formulated a hierarchical data quality framework from the perspective of data users. They constructed a dynamic big data quality assessment method based on a feedback mechanism. Jaikrit et al. [11] proposed a quality assessment model for Internet products and services. The model divided the assessment indicators into two categories, which were those that met the design requirements and those that exceeded the requirements, and introduced several Internet-related assessment factors. Qiang et al. [12] proposed a threat intelligence assessment framework, which realized a multidimensional quantitative assessment of threat intelligence services from the perspective of users.
Other scholars attempted to assess the quality of threat intelligence from its source. Andrea et al. [13] implemented an automated scheme to quantify the CTI sources to understand the relevance between various sources of cyber threats and proposed a signature-driven approach to assess OSINT sources. Schaberreiter et al. [14] utilized quantitative parameters to assess cyber threat information sources. They introduced such parameters as extensiveness, verifiability, false positives, and intelligence and proposed a method for quantitative assessment.
Furthermore, some scholars assessed the quality of threat intelligence directly. Magee et al. [15] developed a threat intelligence collection system to identify threats by classifying threat intelligence according to its type, maliciousness, and credibility. Botega et al. [16] proposed IQESA for assessing information quality. This strategy consisted of three stages: capturing data and information quality requirements, defining metric functions and quantifying quality dimensions, and instantiating contextual information using ontology.
However, while assessing the quality of intelligence, the above works only considered the information contained in the intelligence, but ignored the potential attack techniques and tactics that attackers may use in threat intelligence.
2.2. Threat Information Extraction
Recently, attack behavior extraction from CTIs has attracted attention from industry and academia and is considered an effective method to defend against network attacks [17]. NLP (natural language processing) technology, an increasingly mature tool, is used by some scholars to analyze CTI from multiple aspects. Liao et al. [18] presented iACE to collect threat intelligence and implement the automatic extraction of IOC. iACE can automatically locate the IOC token and context and analyze their relationship. Husari et al. [19] developed a TTPDrill tool, which combined NLP and information retrieval to automatically mine threat behaviors, and constructed STIX-formatted TTPs from threat reports. Zhou et al. [20] designed CTI View, a system that used NLP technology to automatically process CTI. CTI View can automatically extract and analyze CTI text information released by security vendors.
Other scholars applied machine learning technology to information extraction. Mulwad et al. [21] used the support vector machine (SVM) classifier to identify potential vulnerability description and then used the taxonomy in Wikipedia to extract vulnerabilities and attacks. But this classifier only identified and extracted two kinds of concepts, one is the attack means, and the other was the consequences. Yuma et al. [22] proposed a method to automatically generate interpretable IOCs by tracking malware processes. The main idea was to enumerate the key information of all potential IOCs, then continuously optimized and combined this information to maximize the interpretability and accuracy of threat intelligence, and finally generated reliable IOCs. Lv et al. [23] proposed a threat intelligence analysis method based on the detection of the attack chain and network traffic. The method detected the network anomaly traffic, analyzed the relationship of characters, and combined the characteristics of each stage of the kill-chain to restore the process of attack.
Besides, some scholars applied deep learning-related technologies such as the long-short-term memory (LSTM) [24] network, the convolutional neural network (CNN) [25], and the recurrent neural networks (RNN) [26] to information extraction. Luo et al. [27] proposed EDL-WADS, a deep learning-based network attack detection system that used all three models of CNN, MRN, and LSTM for network attack detection. Yan et al. [28] used two techniques of feature weighting and BERT-BiGRU and proposed a method to analyze IIoT threat intelligence, which could realize automatic identification of attack behavior and attack strategies. Wang et al. [29] improved the PCNN-ATT model and proposed a DRL-ETPCNN-ATT-based method for remote supervised relation extraction, which could extract threat intelligence from unstructured text.
However, these works mentioned above were limited to the extraction of threat intelligence and ignored the quality assessment of extracted information.
3. Threat Intelligence Sharing Standards and Platforms
IoT devices can generate a large amount of sensitive and private information while working, which is extremely attractive to attackers [30]. The sharing of CTI is an effective measure to strengthen cyberspace security collaboration and improve cyber security. The threat intelligence community can leverage this information to better understand the situation and share intelligence with communities, organizations, and the public.
3.1. Threat Intelligence Sharing Standards
Today’s network attack behaviors are becoming increasingly complex and attack patterns are constantly evolving, which lead to the requirement of automatic information processing, and rapid sharing and responding to changes in cyber attacks [14]. Since threat indicators can specify information such as threat actors, vulnerabilities exploited, attack programs, and threat activities related, a unified threat indicator becomes a prerequisite for automation. STIX is a standardized language widely used to represent cyber threat information.
STIX is used as a serialization format to exchange CTI, and it is one of the most widely used threat intelligence sharing languages [31]. STIX can describe various characteristics of threat intelligence, such as threat signatures, threat activities, and security incidents. It increases the threat intelligence exchanging efficiency and accuracy, improves the responsiveness of security managers to threats, and helps organizations effectively realize the automation of cyber threat management and application. Given that, we convert the obtained threat intelligence into STIX format. The data conversion method will be introduced in Section 4.1 as an example.
3.2. Threat Intelligence Sharing Platforms
Dandurand [32] proposed that a CTI sharing platform should include three aspects: (i) enabling information sharing, (ii) automating information exchange, and (iii) facilitating the generation and updating of threat intelligence data. Today, a growing number of threat intelligence sharing platforms have begun to enter the public’s vision. In Table 1, some threat intelligence sharing platforms are listed.
Some threat intelligence sharing platforms can not only realize the sharing of information but also conduct online analysis and detection of threat intelligence text, identify the IOC information contained in the text, and then give relevant warning information. For example: Qi-Anxin and ThreatBook, we compare them with our model in Section 5.1 of the paper.
4. Cyber Threat Intelligence-Automated Assessment Model (TIAM)
How to assess the quality of a large scale of threat intelligence from many enterprises and organizations has become a key research problem. The vector space is a simple and effective text representation model, generally used in discrete text [33]. The vector representation is characterized by high dimensionality and sparsity due to the nature of text. The vector space contains feature words extracted from a large amount of text. If the feature words in the vector space do not exist in the corpus, the value of feature word is set to 0; otherwise, the value is set to the number of times it appears. Figure 1 presents one example of word vector representation for sparse text.
The model proposed in this paper is called Cyber Threat Intelligence-Automated Assessment Model (TIAM), which can realize automatic assessment of CTI. TIAM analyzes the content in CTI by extracting IOC with a specific format, which can help security managers assess the value of CTI. IOC can assist security managers in finding specific types of characteristic data in system or network logs, which leads to the identification of infected targets. These characteristic data include IP address, domain name, malicious file hash associated with C2 servers (Command and Control Server), malware download, and other characteristics. In the process of CTI assessment, TIAM is combined with the ATT&CK matrix, which can help security managers analyze attack techniques an attacker might use and then find mitigations to prevent it. Figure 2 presents the TIAM framework, which consists of TICA, TIE, and TIQA modules.
TICA is responsible for collecting and aggregating threat intelligence, converting them to STIX format, and storing them into a database. TIE classifies sparse text-based threat intelligence, identifies, and filters IOCs. TIQA correlates the IOCs extracted from TIE with the intelligence collected by TICA and also assesses the intelligence quality and identifies the ATT&CK attack techniques related to the assessed intelligence.
4.1. TICA Module
TICA is primarily composed of two parts: data collection and database. Data collection is an automated data collection system, which collects and converts community-sourced threat intelligence (e.g., CVE [34], CWE [35], and CAPEC [36]), security website source intelligence (e.g., Lvmeng and X-Force), and open-source intelligence (e.g., ATT&CK knowledge base) to STIX format.
TIAM can dynamically assign intelligence collection tasks to a python agent that crawls data based on the corresponding data collection environment. TIAM converts the data into STIX format and stores them in the database. We show a partial mapping of ATT&CK concepts to STIX objects in Table 2. Through data normalization, different databases can be connected to each other.
4.2. TIE Module
TIE categorizes intelligence obtained from various sources into IOC and non-IOC intelligence. The information obtained from various sources is of uneven quality, some of which may not be relevant to CTI. For example, some data only contains information related to product advertisements and news, which can be defined as non-IOC intelligence.
4.2.1. Threat Intelligence Categorization
For distinguishing IOCs from non-IOCs, TIE considers the feature word weight and non-dictionary word density as classification criteria as follows.
(1) Feature Word Weight. To ensure quick and efficient classification performance. The text needs to be transformed into an intermediate form, thus filtering out redundant and irrelevant features. In text vectors, feature values are usually used to represent the weights of feature words, which also reflect their importance. TIE calculates the weight of feature words through the TextRank [37], as shown in where represents the weight from node to node . For a given vertex , represents the set of vertices that point to , represents the set of vertices out of , and is a damping factor that can be set between 0 and 1.
(2) Non-dictionary Word Density. Dictionary words are common English words in dictionaries. IOC text contains information related to cyber threat behavior characteristics and the analysis codes of malicious samples; thus, it involves plenty of non-dictionary words. TIE uses the method proposed in [38] to calculate the non-dictionary word density.
4.2.2. IOC Identification
As the information contained in threat intelligence is unstructured, traditional natural language processing techniques have difficulty in identifying IOCs. Our study reveals that most of the information in threat intelligence shows a certain structure, such as malicious IP (xxx.xxx.xxx.xxx) and CVE number (CVE-xxxx-xxxx), which can be matched by regular expressions. The ability to match substrings in preprocessed text using regular expressions is available in many applications [39]. Therefore, TIE constructs some regular expressions [40] for different types of IOCs (Table 3).
TIE uses a whitelist to filter the obtained IOCs. Diverse typical whitelists are applied in the security industry, such as Alexa top 100W, Google Safe Browsing, and Cisco Umbrella. TIE uses the whitelist filter to process the received IOCs. If the IOC is found in the whitelist, then the intelligence information record for that IOC will be filtered out.
4.3. TIQA Module
TIQA takes IOCs as an input for assessing threat intelligence and correlates with TICA to analyze possible attack techniques.
4.3.1. Threat Score
By viewing the threat description, it is found that cyber-attacks that can cause great damage typically utilize multiple vulnerabilities, most of which are marked as high-risk vulnerabilities by CVE, while those exploit single and less threatening ones will cause less harm [41]. The threat scoring follows three principles: (i) the closer the attack time is to the threat intelligence release time, the less likely the attack will fail and the more damage it will cause; (ii) the more threat alerts are generated in an attack, the greater damage is caused to the system; and (iii) the more external threat activities are associated with a threat entity, the more significant impact will be exerted on the enterprise. Therefore, the threat can be quantified by exploring the attack time, the number of alerts, and the number of threat activities related to the threat entity. Threat intelligence score is a good measure of the quality of information in threat intelligence.
TIQA quantifies threat intelligence based on the above three threat features. It assigns values to the obtained IOCs (Table 4) and assesses the quality of threat intelligence with a scoring function defined as
Here, represents the number of IOCs extracted from one threat intelligence, represents the score of -th threat feature in -th IOC, and is the weight of . The threat score (TIS) ranges from 1 to 10 (), and the higher the TIS value, the higher the quality of threat intelligence.
When a CVE vulnerability is published, it will include information about the vulnerability description, date, and comments. CVSS (Common Vulnerability Scoring System) [42] is an open standard for measuring the impact of vulnerabilities and can assess the severity of vulnerabilities. TIQA places the extracted complete CVE number into CVSS to assess the threat level and subsequently assigns score according to Table 4. The threat level caused by exploiting the vulnerability can be calculated with
Among them, is the correction factor with a value of 10.41; VL is the difficulty of utilizing vulnerability, which is divided into four grades: critical, high, medium, and low. indicates the confidentiality, integrity, and availability hazards caused by the vulnerability to the system.
TIQA calculates the weight of with where denotes the score of IOC creation time, denotes the number of threat activities scores, denotes the score of alerts, and is the number of alerts.
4.3.2. Related ATT&CK Attack Technology
ATT&CK abstractly describes a framework composed of sequential attack tactics, each of which covers abundant attack techniques [43]. ATT&CK framework can help organizations predict the adversary’s attack behavior, gain a comprehensive understanding of the attack techniques that attackers may use, and provide mitigation measures.
TIAM uses TIE to parse threat intelligence and extract IOCs. Then, TIQA correlates the extracted IOCs with ATT&CK knowledge base in TICA and automatically identifies attack techniques related to IOCs. Identifying attacks in threat intelligence can help security managers gain a more complete picture of the attack event.
5. Experiments and Assessment
5.1. Assessing One IOC Intelligence as an Example
We analyzed an APT attack on India’s cyber space reported by Indian Infosec consortium [44]. First, 50 articles from an existing dataset were manually selected to label entities, and the occurrences of each entity were counted. Then, the top 800 entities with the most occurrences were selected as dictionary words. Finally, the feature words of the remaining articles were calculated and compared with the dictionary words. If a number of feature words appeared in the dictionary words, TIE considered the article to be threat intelligence. For those articles that were not considered threat intelligence, TIE used regular expressions to extract the IOCs and calculated the non-dictionary word density. If a large number of IOCs could be extracted and the density of non-dictionary words was high, TIE considered the intelligence to be threat intelligence as well.
TIAM forwarded this report into TIE, which removed useless stop words and characters. After feature word calculation, this report was considered threat intelligence. Performing the whitelist filtering, 15 out of 32 IOC indicators features, which extracted by TIE, related to threat attacks were finally identified, and 10 of these are presented in Table 5.
TIQA associated the first threat indicator in Table 5 with the database in TICA. After correlation analysis, the following were found: (i) this threat indicator was created in September 2021; (ii) five alerts were detected, namely, “recon_beacon,” “persistence_auto,” “allocates_rwx,” “network_http,” and “antivm_memory_available,” and the database in TICA defines them as high-risk, high-risk, medium-risk, medium-risk, and low-risk, respectively; and (iii) this threat indicator was associated with three kinds of threat activities: attacks at US polling places, attacks against the US government, and yarex-related malware campaigns.
TIQA assigned values to the identified information according to Table 4 and repeated the above analysis process for the remaining threat indicators. Through the threat scoring function, this report had a final score value of 7.43.
In the process of association analysis, three alerts were reported through the database in TICA which were “persistence_autorun,” “antivm_memory_available,” and “recon_beacon” and then were associated with the ATT&CK. TIQA found three attack techniques, namely, Registry Run Keys/Start Folder, Software Packing, and Automated Collection, which might be used by attackers for launching attacks.
Registry run keys/start folder:attackers add compromise code to the startup folder or use registry run keys to ensure persistence. The “run key” added to an entry when a user logs in will execute automatically [45]. It is possible for attackers to gain account-level privileges and execute the malicious code in the context of the user.
System information discovery:attackers can obtain hardware and operating system details such as patches, service packs, hotfixes, and architectures. Attackers can exploit this information to enhance their own operability, for example, to determine whether the target is fully infected [46].
Automated collection:after a successful implementation of the attack behavior, the attackers can automatically collect data inside the infected host. Attackers may use a script interpreter or command-line operations to search for information that matches the set criteria, including location, file type, and name [47].
Through analysis, it is found that these three techniques are in line with the characteristics commonly used in cyber-attack against cyberspace. Figure 3 visualizes the threat intelligence quality score, identified threat indicators, and the ATT&CK attack techniques.
We compared some of the threat intelligence platforms (ThreatBook, Qi-AnXin) in Table 1 against our model in five aspects: (i) the number of threat alerts generated, (ii) the number of ATT&CK attack techniques, (iii) whether focus on the time of IOC creation, (iv) IOC-related threat activity can be identified, and (v) whether to include visual analysis of results. The comparison results are shown in Table 6.
From Table 6, we can find that the TIAM proposed in this paper is superior compared with the other two in terms of overall performance. The reason for its advantage, we believe, is that TIAM integrates more vulnerability libraries and connects with ATT&CK. TIAM focuses on assessing the possible impact of each IOC in threat intelligence and quantifying threat intelligence, thus generating more comprehensive alert information, while the other two are more focused on analyzing the attacker’s attack path, and network attack behavior. The reason why Qi-Anxin did not identify the ATT&CK attack techniques is that its database for file detection is not connected to ATT&CK.
5.2. IOC Intelligence Quality Assessment and Analysis
TIAM assesses threat intelligence from APTnotes [48], which contains APT reports over the past 13 years. Figure 4 shows the statistics on the volume of threat intelligence by year. Most of the reports are from the top security vendors, such as McAfee, Aurora, Kaspersky, SecureWorks, Cisco Talos, and FireEye. The reports can be grouped into three categories: analysis of APT organizations, analysis of the current raging malware, and analysis of a specific attack.
We used TIAM to assess APT reports and found that an average of 65 threat indicators appeared in each article and 15% of articles had more than 100 threat indicators. TIAM extracted 39,090 pieces of threat indicators from 632 reports. By quantifying the quality of these reports, TIQA found that there were 312 threat intelligence with a score between 1 and 4, 106 threat intelligence with a score between 4 and 7, 146 threat intelligence with a score between 7 and 10, and 68 threat intelligence identified as non-IOCs. Among them, 2016 contained the most low-scoring threat intelligence, while 2022 contained the most high-scoring threat intelligence. Figure 5 shows the quantitative assessment results of CTI for each year. Meanwhile, TIAM identified 5505 attack techniques (including recurring techniques) from these reports. Table 7 shows the 10 most frequently used attack techniques and the number of occurrences in the reports.
Shared Modules can instruct the window template loader using NTDLL.dll to load DLLs from arbitrary local and Universal Naming Convention (UNC) network paths. NTDLL.dll is included in the Windows Native API, which is called by functions such as CreateProcess and LoadLibrary of the Win32 API [49]. After analysis, we found that this technique was difficult to monitor with the current tools. Redundant DLL made monitoring and detection efforts pointless. Obviously, the commonly used techniques come from different tactics. In fact, this experimental result reflects our intuitive understanding of the attack lifecycle [50], where typical attacks must consist of a unique set of strategies that can be implemented with different techniques.
6. Conclusions
In order to tackle the problem of high sparsity and uneven quality of the information in threat intelligence, this paper proposes TIAM to analyze unstructured CTI, classify and identify threat intelligence through text features, and extract IOC information automatically. TIAM introduces the existing attack technology knowledge in ATT&CK into automated assessments of threat intelligence. In the end, it provides the quantitative assessment results of unstructured threat intelligence. In the future, we will expand the scope of information extraction, beyond IP, hash, URL, and features that have a fixed format and introduce technologies such as machine learning.
Data Availability
The data used to support the findings of this study are included within the paper.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work was supported in part by the Key Program Research Fund of Higher Education of Henan, China (grant number No. 21A520053 and No. 19A520048).