Mathematical Problems in Engineering

Volume 2015 (2015), Article ID 923792, 26 pages

http://dx.doi.org/10.1155/2015/923792

## Network-Wide Traffic Anomaly Detection and Localization Based on Robust Multivariate Probabilistic Calibration Model

^{1}National Digital Switching System Engineering & Technological Research Center, Jianxue Street No. 7, Jinshui District, Zhengzhou 450002, China^{2}Air Defence Forces Academy of PLA, Zhengzhou, China^{3}Science and Technology on Information Transmission and Dissemination in Communication Networks Laboratory, Shijiazhuang, China

Received 29 April 2015; Revised 5 August 2015; Accepted 6 August 2015

Academic Editor: Mark Leeson

Copyright © 2015 Yuchong Li et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

Network anomaly detection and localization are of great significance to network security. Compared with the traditional methods of host computer, single link and single path, the network-wide anomaly detection approaches have distinctive advantages with respect to detection precision and range. However, when facing the actual problems of noise interference or data loss, the network-wide anomaly detection approaches also suffer significant performance reduction or may even become unavailable. Besides, researches on anomaly localization are rare. In order to solve the mentioned problems, this paper presents a robust multivariate probabilistic calibration model for network-wide anomaly detection and localization. It applies the latent variable probability theory with multivariate *t*-distribution to establish the normal traffic model. Not only does the algorithm implement network anomaly detection by judging whether the sample’s Mahalanobis distance exceeds the threshold, but also it locates anomalies by contribution analysis. Both theoretical analysis and experimental results demonstrate its robustness and wider use. The algorithm is applicable when dealing with both data integrity and loss. It also has a stronger resistance over noise interference and lower sensitivity to the change of parameters, all of which indicate its performance stability.

#### 1. Introduction

Network traffic anomalies are unusual and significant changes at network’s traffic level. Intrusions such as DDos attacks and zombie networks significantly jeopardize the Internet security, and network jams and malfunctions have unpleasant impact on service quality; therefore it is critical to detect and locate network anomalies for both network operators and end users. It is a challenging task to detect and locate them because one must extract and interpret anomalous patterns from large amounts of high-dimensional, intricate, and noisy background traffic data.

There are a great number of researches on anomaly detection. Host-based anomaly detection system monitors and analyzes the internals of a computing system by applying data mining of the system logs and audit records [1, 2]; another detection method based on performance measurement data such as end to end round-trip time and packet loss probability in a single path can be implemented by single variable time series analysis [3, 4]; network anomaly detection based on traffic measurements from single link can be implemented by applying machine learning and signal analysis [5, 6]. All of these methods have their limits, because they only concentrate on a part of the information and their detection area is limited. When the scale of network enlarges and data transfer rate speeds up, many network anomalies often exhibit strong network-wide characteristics [7, 8], and their impact may always spread to multiple links, while their local characteristics may not be that obvious. It is difficult to conduct network-wide analysis with above methods and their accuracy cannot be guaranteed.

In order to solve the problems mentioned above, Lakhina et al. come up with network-wide anomaly detection based on subspace construction via PCA [9]. This method employs network traffic of many Origin-Destination (OD) flows to establish a model of normal behavior and detects anomalies by measuring deviations from that model. With the concept of network-wide detection, researches are conducted in space-time expansibility [10–13], robustness [14–16], real-time processing [17, 18], and anomaly measure [19, 20], which enrich network-wide anomaly detection. This kind of methods uses whole-network traffic data, which has huge performance advantages over single point, single path, and single link. At the same time, compared with other methods, it sets up normal behavior model which enables us to avoid building anomalous feature library. Thus it can be implemented to detect known anomalies as well as unknown anomalies, and it can be used widely. Network-wide anomaly detection improves detection performance by introducing wider and multidimensional network information, but it also faces some real problems when implemented in large scale and high speed backbone network. Firstly, because of its wider collection area, more collection equipment, and faster network speed, it might not be applicable if collected data were lost during collection or transfer process [21]; secondly, traffic flows in backbone network continue to grow in volume and complexity, and hidden noise like anomalous traffic could degrade performance of the anomaly detection algorithms [22], while some of the attacks might even pollute the detectors [14–16]; thirdly, the above anomaly detection methods can only find when anomalies happen, but they still have some defects on locating those anomalies [22].

Therefore, we propose an approach named RMPCM based on robust multivariate probabilistic calibration model to overcome these problems discussed above. This anomaly detection and locating algorithm introduces a latent variable probabilistic model based on -distribution instead of a Gaussian distribution to establish a normal traffic model. By judging if the sample’s Mahalanobis distance from the normal model exceeds the threshold, traffic anomaly detection is achieved. Locating anomalies is attained with contribution analysis. RMPCM approach is more robust; not only can it be widely used for processing complete data as well as missing data, but also it acquires stronger robustness under the noise interference and lower sensibility of model parameters. The contributions of this paper consist of following 4 aspects; the RMPCM approach can

solve anomaly detection problem when data loss occurs by establishing a latent variable probabilistic model,

increase detection accuracy by introducing multivariate -distribution to relieve noise interference in modeling the normal traffic behavior,

correctly locate the underlying Origin-Destination (OD) flows being the source of the anomaly,

reduce the amount of work involved in implementing complicated parameter testing, because RMPCM has a better stability and lower sensibility for model parameters.

This paper is organized as follows. We begin in Section 2 with a discussion of the related work. We describe data source model and problems that need to be solved in Section 3. In Section 4, we describe the RMPCM approach in detail and solve three problems raised in Section 3. We validate our approach in three different ways of experiments and contrast our RMPCM with existing approach on traffic anomaly detection in Section 5. A discussion of several details is presented in Section 6. Concluding remarks and our ongoing work are presented in Section 7.

#### 2. Related Work

Back in 1987 Denning had demonstrated statistic model for detecting network anomalies [23]. And it is becoming more and more important with the development of the Internet. There are many traditional anomaly detection approaches based on host computer, single path, and single link. Researches in [7] indicated the generation and development of network anomalies have exhibited a tendency of network-wide characteristic. They found that the performance of the anomaly detectors increases with enlarging the range of detection beyond linear growth, which sets up the foundation for network-wide anomaly detection. The authors of [9] proposed network-wide anomaly detection algorithm based on network traffic for the first time, which illustrated low dimensionality of OD flows. They also integrated traffic statistics of multiple OD flows to build up a model of normal behavior and detected anomalies by measuring deviations from that model. The authors of [11] came along to improve the anomaly detection approach based on PCA by applying stochastic matrix perturbation theory and proposed a PCA-based distributed approach for network-wide anomaly detection. Reference [13] expanded the classical PCA and proposed the Karhunen-Loeve expansion for network-wide anomaly detection. Reference [17] proposed an online anomaly detection approach using kernel recursive least squares algorithm to solve the problem of online detection. All of them did not address the problems of how to detect anomalies in the condition of noise interference and data loss and how to locate the anomalies in the actual network environment.

The authors of [22] took in-depth study on the influence of anomalous traffic on the performance of detector and indicated that large anomalies may cause the offset of normal model based on PCA, which increased the false positive rate (FPR) of anomaly detection. References [14, 15] took further steps to study poisoning attacks on anomaly detectors and evaluated poisoning techniques and developed defense. The authors of [16] listed 3 mechanisms of poisoning attacks and proposed defense based on robust PCA with projection pursuit. All of the above only focus on the poisoning and defense techniques based on PCA detector, but there is lack of researches on common modeling approaches in the intricate and noisy environment.

Data loss is very common in many fields, and the question of how to get enough information from missing data needs to be answered. The authors of [24, 25] proposed an algorithm to solve the problem of goodness-of-fit test for varying coefficient models with missing data. The authors of [21] gave their opinion on data loss problem when network traffic flows were measured in large scale and high speed backbone network; they proposed an approach of sparsity regularized matrix factorization (SRMF) to make the data complete. This is applicable in traffic engineering, capacity planning, forecasting, and so forth, but it did not research deeply in network anomaly detection when facing data loss.

Network anomaly detection can determine when anomalies take place, but locating anomaly is an extremely challenging task. The authors of [22] point out the deficiency of network-wide detection algorithm based on PCA in locating anomaly. Reference [26] then proposed an approach of BasisDetect for network-wide anomaly detection and locating, but it could only locate anomalies to border router; it was unable to pinpoint the position.

In this paper we propose a network-wide anomaly detection algorithm based on RMPCM, which will later be proved to have a better performance in solving problems of noise interference, data loss, and locating anomalies.

#### 3. Data Model and Problems Description

##### 3.1. Data Model

Conventionally the researches of the Internet traffic flow mainly focused on temporal characteristics of data package on a single link, which help in developing concepts of self-similar stochastic processes, long-range dependence, and so forth. One ISP (Internet service provider), however, consists of hundreds of those links which are connected all over, and the Internet contains several thousand ISPs. In such a vast background the spatial characteristics of network traffic come to people’s attention inevitably. However, it is difficult to analyze traffic flow data of all links in the network simultaneously, because it amplifies the complexity of modeling traffic on a single link which is itself a complicated task. As compact and elegant descriptions of traffic flows between nodes in a certain network structure, traffic matrix is a constantly employed model to conduct explorations on the spatiotemporal component of network-wide traffic. Traffic matrix is an overview of network-wide traffic. Instead of studying traffic on all links, applying traffic matrix provides more straightforward and fundamental insights into network-wide traffic study [8]. We employ traffic matrix at PoP (point of presence) level as data source in our research.

Traffic matrix at PoP level: assume that an autonomous system (AS) has PoPs. Continuous measuring of the traffic between each PoP pair at a certain period can obtain the traffic of Origin-Destination (OD) flows. An OD flow denotes the collection of all traffic that enters the network from an ingress node and departs from an egress node. Arranging these point-to-point measured values in matrix can obtain this AS’s traffic matrix . As shown in Figure 1, denotes the number of measurement periods, and denotes the number of measured value of OD flows at each measurement . The element in denotes the volume of a certain traffic measure at the th period and the th OD. Traffic volume (the number of bytes, packets, or flows) is adopted as a traffic measure in this paper.