Abstract
The prosperity of mobile networks and social networks brings revolutionary conveniences to our daily lives. However, due to the complexity and fragility of the network environment, network attacks are becoming more and more serious. Characterization of network traffic is commonly used to model and detect network anomalies and finally to raise the cybersecurity awareness capability of network administrators. As a tool to characterize system running status, entropybased timeseries complexity measurement methods such as Multiscale Entropy (MSE), Composite Multiscale Entropy (CMSE), and Fuzzy Approximate Entropy (FuzzyEn) have been widely used in anomaly detection. However, the existing methods calculate the distance between vectors solely using the two most different elements of the two vectors. Furthermore, the similarity of vectors is calculated using the Heaviside function, which has a problem of bouncing between 0 and 1. The Euclidean DistanceBased Multiscale Fuzzy Entropy (EDMFuzzy) algorithm was proposed to avoid the two disadvantages and to measure entropy values of system signals more precisely, accurately, and stably. In this paper, the EDMFuzzy is applied to analyze the characteristics of abnormal network traffic such as botnet network traffic and Distributed Denial of Service (DDoS) attack traffic. The experimental analysis shows that the EDMFuzzy entropy technology is able to characterize the differences between normal traffic and abnormal traffic. The EDMFuzzy entropy characteristics of ARP traffic discovered in this paper can be used to detect various types of network traffic anomalies including botnet and DDoS attacks.
1. Introduction
The prosperity of network technologies, such as mobile networks and social networks, brings revolutionary changes to our daily lives. However, due to the complexity and fragility of the network infrastructures, network anomalies and attacks frequently cause serious problems and significant loss to people. Researchers are studying various cybersecurity awareness technologies to help people understand the security status and trend of networks. Characterization of network anomaly traffic is one of the key technologies commonly used to model and detect network anomalies and then to raise the cybersecurity awareness capability of network administrators. The existing approaches of network anomaly detection can be mainly classified into six categories [1]: classificationbased methods [2–4], clusteringbased methods [5–9], statistical methods [10, 11], stochastic methods [12, 13], deeplearningbased methods [14–17], and others [18–21].
Network anomaly detection via traffic feature distributions is becoming more and more popular these days. As the measure of uncertainty, entropy can be used to summarize feature distributions in a compact form [22]. There are many forms of entropy, but only a few have been applied to network anomaly detection [23–27]. On this basis, we apply a Euclidean DistanceBased Multiscale Fuzzy Entropy (EDMFuzzy) algorithm which we proposed to detect abnormal network traffic as a useful supplement of other approaches.
Investigation irregularity of signals generated by complex systems is valuable to predict the future states as well as detect abnormal behaviors [28]. In order to quantitatively analyze signal irregularity and diagnose system anomalies, researchers have proposed various signal complexity and uncertainty indicators, such as algorithmic complexity [29], Shannon Entropy [30], Approximate Entropy [31], Sample Entropy [32], Fuzzy Entropy [33], Multiscale Entropy (MSE) [34], and Composite Multiscale Entropy (CMSE) [35]. Entropybased technologies have been widely applied in diagnosing the anomalies of various systems. For example, Shannon Entropy was applied in detecting faults of mechanical systems [36], MSE was applied in fault diagnosis of power systems [37], and so on.
However, the existing methods calculate the distance between vectors solely using the two most different elements of the two vectors. Furthermore, the similarity of vectors is calculated with Heaviside function, which has a problem of bouncing between 0 and 1. To this end, we proposed a novel entropy technology named EDMFuzzy in the paper [38]. The EDMFuzzy technology uses the sum of the Euclidean distances of the elements corresponding to two vectors instead of the largest element difference between the two vectors and uses the hyperbolic function to calculate the similarity between the two vectors. Thus, the EDMFuzzy technology avoids the two disadvantages inherent in the other entropy technologies and measures entropy values of system signals more precisely, accurately, and stably. In this paper, we apply the EDMFuzzy algorithm to characterize network anomaly traffic. We first briefly introduce the EDMFuzzy algorithm and then introduce the botnet CTU13 dataset and the Distributed Denial of Service (DDoS) attack CICDDoS2019 dataset used in this paper. Then, the basic characteristics of these two datasets are introduced. Then, the EDMFuzzy entropy value analysis is performed on two datasets. Finally, we analyze the characteristics of the normal traffic and investigate the characteristics of the malicious traffic by comparing the differences between the normal and malicious traffic.
The rest of this paper is organized as follows. The related works are introduced in Section 2, and the EDMFuzzy entropy technology and network traffic traces are introduced in Sections 3 and 4, respectively. Section 5 is the analysis of network anomaly traffic with EDMFuzzy entropy. Section 6 concludes the paper and introduces the future work.
2. Related Works
2.1. Network Anomaly Traffic Detection Approaches
Network anomaly traffic detection approaches have been extensively explored. The existing approaches can be mainly classified into six categories [1]: classificationbased methods [2–4], clusteringbased methods [5–9], statistical methods [10, 11], stochastic methods [12, 13], deeplearningbased methods [14–17], and others [18–21]. A classificationbased approach is a supervised learning algorithm. Classification algorithms such as logistic regression, knearest neighbor algorithm, decision tree, and support vector machine are commonly used. More recently, several hybrid classification models were proposed [3–5]. However, in most cases, labeling data manually is highly timeconsuming and inefficient. Clustering techniques are used to identify clusters and outliers in multiple lowdimensional spaces. The evidence of traffic structure provided by these multiple clusters is then combined to produce an abnormality ranking of network traffic [39]. Several distancebased metrics are commonly used in anomaly detection, such as the Euclidean distance, Manhattan distance, and dynamic time warping (DTW) distance. However, the number of clusters is difficult to decide and different numbers of clusters would produce extremely different results. In a statistical method, an abnormality is often determined by checking whether the traffic complies with the assumed distribution model and whether the value is larger than a preset threshold. The most frequent assumptions are Gaussian distributions, Poisson distributions, multivariate Gaussian distributions, and so on. The model systematically analyzes abnormal behaviors of the network, but detection of such abnormalities is difficult since there will be cases that do not obey the presumed distributions. Stochastic processes like Hidden Markov Model and Conditional Random Field were also frequently applied in detection of traffic anomaly [12, 13]. Due to the success of deeplearning technologies in image processing and natural language processing, they have been intensively studied in network intrusion detection [14, 15], network traffic tracking [16], and network traffic abnormal behavior detection [17]. Besides, timeseries density analysis [18], wavelet [19], principal components analysis [20], and ensemble learning technologies [21] have been extensively investigated in network anomaly detection.
2.2. EntropyBased Technologies
Entropybased technologies are highly valued in detecting the degree of disorder or irregularity of a complex system. Thus, there have been a number of entropybased technologies being proposed and being widely applied in detecting anomalies of complex systems. Khan et al. [37] presented an entropybased approach for detecting faults in power systems. An entropybased methodology was proposed in paper [40] to extract characteristics from signals of smart meters to effectively classify power quality problems. The KullbackShannon Entropy was applied as a standalone feature to predict failure in lubricated surfaces [41].
Pincus [31], Richman, and Moorman [32] and Costa et al. [34, 42] proposed Approximate Entropy, Sample Entropy, and MSE to measure signal complexity, respectively. Although MSE has been widely applied, the variance of the entropy values increases significantly as the time series is coarsegrained for larger time scales [43]. In order to solve the problem, Wu et al. proposed CMSE [35] and introduced a composite averaging method to reduce the variance. Niu and Wang [44] applied CMSE to study the characteristics of stock market indices and found that CMSE is more stable and reliable than MSE. Chen et al. [33] proposed Fuzzy Approximate Entropy (FuzzyEn) and applied it in the study of surface muscle signal. Wang et al. [45] proposed fractional fuzzy entropy to study physics financial dynamics. Li et al. [46] integrated fractional fuzzy entropy with a binary tree support vector machine to perform early diagnosis of rolling bearing faults. Composite multiscale fuzzy entropy is proposed in paper [47] and is applied to extract the hidden features of vibration signals.
Entropybased network anomaly detection via traffic feature characterization is becoming more and more popular these days. Ranjan et al. [23] proposed a worm detection algorithm that measures Shannon Entropy values for traffic and alarms on sudden bursts. Gu et al. [24] applied Shannon maximum entropy estimation to draw the network baseline distribution and to build a multiperspective view of network traffic. Paper [25] presented a novel network intrusion detection system using Shannon Entropy and traffic distributions of the source port. Paper [26] proposed a hybrid DDoS detection method, which integrates Kernel Online Anomaly Detection (KOAD), Shannon Entropy, and Mahalanobis Distance. In this study, Shannon Entropy is utilized with an online machine learning method to detect malicious traffic including DDoS attacks and Flash Event traffic. Paper [27] presented anomaly detection in activities of daily living based on entropy measures.
However, there are still two disadvantages in the existing stateoftheart entropy algorithms, such as MSE, CMSE, RCMSE, MMSE, and FuzzyEn. That is, the existing methods calculate the distance between vectors solely based on the two most different elements of the two vectors. Furthermore, the similarity of vectors is calculated using Heaviside function, which has a problem of bouncing between 0 and 1. In order to address the shortcomings of existing stateoftheart entropy algorithms, we proposed novel entropy technology [38], named EDMFuzzy.
3. EDMFuzzy Technology
EDMFuzzy measures the distance of the two vectors with Euclidean distance taking all the corresponding elements in the two vectors into the computation. Furthermore, in order to solve the problem of instability, we choose the hyperbolic function as the fuzzy function instead of the Heaviside function to define the similarity between vectors with fullrange continuous values from zero to one based on the Euclidean distance of the two vectors. The computation process of EDMFuzzy is formally described in Algorithm 1.

The goal of the algorithm is to measure the complexity and irregularity of time series more accurately and stably. The input of the algorithm is a time series , time scale τ, vector dimension m, tolerance coefficient r, and standard deviation SD of time series X. The output of the algorithm is the EDMFuzzy entropy value of time series X at time scale τ. The general process of the algorithm is first to coarsegrain the time series with time scale τ, then split the time series into mdimensional vectors, move the vectors to its centroid, and finally, calculate the Euclidean distance of the two vectors and compute the Euclidean distance based on fuzzy sample entropy value of time series. For parameters m and r, m is usually set to 2 and r generally ranges from 0.1 to 0.2. In our experiments, r is set to 0.15; that is, the similarity tolerance is set to 0.15SD. Here, SD represents the standard deviation of the original time series.
4. Network Traffic Trace
A suitable network traffic trace is essential to the research of the characterization of network anomaly traffic. The traces used in this paper are publicly accessible, within which anomaly activities including botnet and DDoS attack were recorded. Through analysis of these public traces with EDMFuzzy algorithm, we can further discover the characteristics of such anomaly activities.
4.1. Botnet Traffic Trace
The botnet traffic trace used in this section is the CTU13 trace that was collected and provided by the Stratosphere Laboratory of CTU University in the Czech Republic [48, 49]. This trace contains botnet traffic as well as normal background traffic. The CTU13 trace contains 13 botnet samples in different scenarios. In each sample, a specific malware is executed and different operations were performed accordingly. The brief information of the trace is shown in Tables 1 and 2.
Table 1 shows the characteristics of 13 types of botnet scenarios. Each type of botnet has different characteristics of malicious behavior. In Table 1, IRC represents the network relay chat protocol, SPAM represents spam, CF represents malicious clicks, PS represents port scan, FF represents fast flux, P2P refers to endtoend, DDoS refers to Distributed Denial of Service, and US refers to a protocol that is controlled and completed by humans. The basic characteristics of each botnet can be seen in Table 1.
Table 2 shows the duration, the number of data packets, the type of malware, and the number of infected computers of these 13 types of botnet scenarios. The duration of botnet scenarios varies from 15 minutes to 66 hours. The number of infected hosts for most scenarios is 1 host. Neris3, Rbot3, Rbot4, and NSIS.ay scenarios have 10 and 3 infected hosts, respectively.
4.2. DDoS Traffic Trace
DDoS attack is an abnormal network behavior designed to exhaust server resources. It will cause server congestion and thus will be unable to provide services to users. The traffic trace used in this paper is the CICDDoS2019 which was published by the Canadian Cyber Security Institute (CIC) [50]. The CICDDoS2019 trace contains common and latest DDoS attacks. There are mainly two categories of DDoS attack methods involved in this trace, DDoS reflection attack and DDoS direct attack. DDoS reflection attack method utilizes routers, servers, and other facilities to respond to requests, thus reflecting the attack traffic to hide the source of the attack. The direct DDoS attack method is to directly attack the target using the controlled hosts. Compared with the reflection type attack, the direct attack method has a lower degree of anonymity. The specific attack types and attack duration time in the CICDDoS2019 dataset are shown in Tables 3 and 4.
Two days of traffic were collected in this trace, which were November 3 and December 1, as shown in Tables 3 and 4, respectively. There were 10 types of DDoS attacks on December 1, that is, NTP, DNS, LDAP, MSSQL, NetBIOS, SNMP, SSDP, UDP, UDPLag, and TFTP attack that lasted from 10 : 30 to 17 : 15. On November 3, there were 7 types of DDoS attacks including PortMap, NetBIOS, LDAP, MSSQL, UDP, UDPLag, and SYN attacks; the duration is from 9 : 40 to 17 : 35. The attack method of each type of DDoS attack in the CICDDoS2019 dataset is shown in Figure 1.
As shown in Figure 1, there are two types of DDoS attacks in the CICDDoS2019 trace, namely, reflection DDoS attacks and direct DDoS attacks. Both DDoS attacks are based on TCP/UDP protocol execution. As shown in the figure above, 9 types of DDoS attacks such as MSSQL, SSDP, DNS, LDAP, NetBIOS, SNMP, PortMap, NTP, and TFTP, are distributed reflective denial attacks, while SYN, UDP, and UDPLag Flood are direct DDoS attacks. In Figure 1, TCPbased attacks include MSSQL, SSDP, and SYN, and UDPbased attacks include NTP, TFTP, UDP, and UDPLag. The remaining types of attacks such as DNS, LDAP, NetBIOS, SNMP, PortMap, and other types of attacks are executed by using TCP or UDP.
5. Analysis of Network Anomaly Traffic with EDMFuzzy Entropy
Entropybased timeseries complexity measurement methods are widely used in fault diagnosis and anomaly detection of various complex systems. In this section, we apply EDMFuzzy in network traffic anomaly characterization and detection. The analysis of anomaly traffic characteristics based on MSE of Euclidean distance is an important part of the study of abnormal traffic. In this section, two anomalies of botnet and DDoS attack will be analyzed by Euclidean distance multiscale entropy. This section will calculate the entropy value of these two abnormal network protocol time series to obtain the entropy curves of the two and study the characteristics of the abnormal traffic by comparing the difference in the entropy curves.
5.1. Botnet Traffic in ARP
In this section, we will study the EDMFuzzy entropy characteristics of 13 types of botnets abnormal ARP traffic in the CTU13 dataset. According to the TCP/IP architecture, the ARP protocol is located in the IP layer of the network layer, and its main function is to provide address translation services and find the network physical address of the host corresponding to the IP address. We first calculate the entropy values for each type of botnet using ARP protocol traffic data in the CTU13 dataset at time scales from 1 to 40. The entropy curves of 13 types of botnets in the CTU13 dataset with scale factors from 1 to 40 are shown in Figure 2.
As can be seen from the figure, there are common trends shared by entropy curves of most types of botnet traffic. More specifically, there is a reflection point for 11 entropy curves (Neris1, Neris2, Rbot1, Rbot2, Virut1, Menti, Sogou, Murlo, Neris3, NSIS.ay, and Virut2) when the time scale is 20, and the second reflection point appears at the time scale of 30 for all entropy curves. For the above 11 types of botnet ARP traffic, the entropy values between the inflection points increase first and then decrease. The trend of the entropy curves of Rbot3 and Rbot4 is different from other types of abnormal behavior. Entropy curves of Rbot3 and Rbot4 are in a steady growth state when the time scale is around 20, but when the time scale is 30, there is also an inflection point. Moreover, entropy values of Rbot4 are significantly larger compared to those of other types of anomalies. The above results illustrate that the attack methods of Rbot3 and Rbot4 are different from the other types of botnets. This difference is caused by the way they infect hosts, and the complexity of the botnet is consistent with the complexity of the ARP protocol.
5.2. DDos Traffic in ARP
In this section, we will study the EDMFuzzy entropy characteristics of the malicious traffic of the distributed denial attacks on November 3 and December 1 in the CICDDoS2019 dataset. Through analysis of the trend of entropy values, it is possible to understand more characteristics of DDoS attack traffic. As introduced in the dataset, there were seven and ten types of distributed denial attacks launched on November 3 and December 1, respectively. In this section, we first calculate the entropy value of the ARP traffic of each type of DDoS attack in the CICDDoS2019 dataset at time scales from 1 to 40, and the entropy value curves are shown in Figures 3 and 4.
Figures 3 and 4 show the entropy curves of the ARP traffic for DDoS attacks on November 3 and December 1, respectively. As can be seen from Figures 3 and 4, there are three characteristics of the entropy values of DDoS attacks on November 3 and December 1. Firstly, for both November 3 and December 1, all of the entropy values of DDoS attacks are larger than 0.18 when the time scale is larger than 4. Secondly, the entropy values of most types of DDoS attacks gradually stabilized to a value between 0.3 and 0.5 when the time scale is larger than 10. There is only the NetBIOS attack that has a relatively big fluctuation that may exceed the upper bound. Thirdly, the entropy values of the same type of DDoS attack for the two different days are quite similar. The possible underlying principle of the characteristics is that, in DDoS attacks, attackers continuously use distributed attacks to attack hosts, and the attacked hosts continue to communicate during the attack. In the communication process, the ARP protocol continuously performs address resolution, while the number of resolved source IP addresses and destination IP addresses remains stable, so the entropy values of ARP traffic are gradually stabilized.
5.3. Normal Traffic in ARP
In this section, we will analyze the characteristics of network traffic under normal status. The normal traffic trace used in this paper is captured and published by the Stratosphere laboratory.
In order to study the entropy characteristics of normal traffic, the EDMFuzzy entropy values are calculated on the CTUNormal20 and CTUNormal23 traces with time scales from 1 to 40 and the results are shown in Figure 5.
As can be seen from Figure 5, the entropy values of normal ARP traffic exhibit different characteristics. The entropy values grow steadily when the time scale grows from 1 to 30, and then the entropy values grow slowly as the time scale increases. Furthermore, for all time scales, the entropy values of normal ARP traffic are smaller than 0.18.
Compared with the time series of the CTU13 dataset, Figure 5 shows that the entropy value of the ARP protocol in the CTU13 dataset exhibits its own unique laws. Compared with the CICDDoS2019 dataset, the basic law of the ARP protocol is that the entropy curve increases first and then gradually stabilizes.
5.4. Malicious versus Normal
In this section, we will compare the ARP traffic entropy curves between botnet, DDoS attack, and normal status and then characterize the differences between normal and abnormal traffic.
By comparing the entropy curves of ARP traffic of botnet, DDoS attack, and normal status, we find out the following main differences between normal traffic and malicious traffic. In the entropy curves of 13 types of botnets, 11 entropy curves (Neris1, Neris2, Rbot1, Rbot2, Virut1, Menti, Sogou, Murlo, Neris3, NSIS.ay, and Virut2) have a reflection point at the time scale of 20, and all entropy curves have a reflection point at the time scale of 30. In the entropy curves of DDoS traffic, all of the entropy values of DDoS attacks are larger than 0.18 when the time scale is larger than 4, and most types of DDoS attacks gradually stabilized to entropy values between 0.3 and 0.5 when the time scale is larger than 10. In contrast, the entropy values of normal ARP traffic grow slowly as the time scale increases and the entropy values are smaller than 0.18 for all time scales.
In order to be presented more intuitively, the main characteristics of entropy curves of ARP traffic of botnet, DDoS attack, and normal status are listed in Table 5.
On the basis of the above analysis, it is reasonable to summarize that the characteristics of entropy curves of ARP traffic of botnet, DDoS, and normal status are quite distinguishable. Thus, the characteristics are easy to be used to detect these types of network traffic anomalies. In the future, we will study characteristics of EDMFuzzy entropy curves of more types of network traffic anomalies and utilize the learned characteristics of network traffic anomalies in combination with intelligent algorithms to automatically detect network anomalies.
6. Conclusions
In order to raise the cybersecurity awareness capability of network administrators, it is necessary to develop new technologies for detecting network anomalies more accurately and efficiently. The basis of such network anomaly detection technologies is to understand the characteristics of abnormal network traffic. In this paper, we apply the EDMFuzzy technology as a tool to analyze the characteristics of abnormal network traffic such as botnet network traffic and DDoS attack traffic. The EDMFuzzy is a technology that we proposed for analyzing and diagnosing faults/anomalies of complex systems by measuring the complexity and regularity of their timeseries signals. The experimental analysis shows that the EDMFuzzy entropy curve is capable of characterizing the difference between normal traffic and abnormal traffic and the characteristics are easy to be used to detect various types of network traffic anomalies. In the current work, we have not investigated other types of network anomalies and have not finished the automatic detection of network traffic anomalies. In the future, we will investigate EDMFuzzy entropy characteristics for more types of network anomalies and then integrate the EDMFuzzy entropy and deeplearning technologies to propose the novel network anomaly detection method.
Data Availability
The botnet traffic trace used in this section is the CTU13 trace that was collected and provided by the Stratosphere Laboratory of CTU University in the Czech Republic [48, 49]. The traffic trace used in this paper is the CICDDoS2019 which was published by the Canadian Cyber Security Institute (CIC) [48].
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was in part supported by the National Key Research and Development Program of China under Grant 2019YFB2102100, the China Postdoctoral Science Foundation under Grant 2016M600465, the Key Research and Development Program of Zhejiang Province under Grant 2019C03134, and the National Natural Science Foundation of China under Grant 61772165.