Knee Point Search Using Cascading Top-k Sorting with Minimized Time Complexity

Abstract

Anomaly detection systems and many other applications are frequently confronted with the problem of finding the largest knee point in the sorted curve for a set of unsorted points. This paper proposes an efficient knee point search algorithm with minimized time complexity using the cascading top-k sorting when a priori probability distribution of the knee point is known. First, a top-k sort algorithm is proposed based on a quicksort variation. We divide the knee point search problem into multiple steps. And in each step an optimization problem of the selection number k is solved, where the objective function is defined as the expected time cost. Because the expected time cost in one step is dependent on that of the afterwards steps, we simplify the optimization problem by minimizing the maximum expected time cost. The posterior probability of the largest knee point distribution and the other parameters are updated before solving the optimization problem in each step. An example of source detection of DNS DoS flooding attacks is provided to illustrate the applications of the proposed algorithm.

1. Introduction

Anomaly detection system and many other applications often rely on finding the largest knee point in the sorted curve to perform clustering, classification, anomaly identification, and so forth [1–6]. Here the largest knee point is targeted because the particular interests lie in finding the cluster of the largest points whose values differ significantly from their lower neighbors in the sorted curve.

Knee point is defined as the point whose value is close to its upper neighbor while far from its lower neighbor in the sorted curve and thereby taken as the boundary of the cluster of upper points. For an unsorted list, it is necessary to sort it to facilitate the knee point search. Due to time and space efficiency considerations, the method of first completely sorting and then searching the sorted list is often not the optimal one. An alternative approach is to perform search on the partially sorted, namely, top-k, list, hoping to save the cost of sort. Therefore the top-k sort algorithm is introduced to help minimize the time complexity of the knee point search in this paper. There have been many efforts for bounding and evaluating the time and space complexity of sort algorithms [7–12]. These works provide component algorithms for our work. But the problem of knee point search via top-𝑘 sorting has not been addressed by any of the previous works. We present in this paper a knee point search algorithm using top-𝑘 sorting with minimized time complexity.

This paper is organized as follows: some basic concepts and definitions on knee point search and top-k sorting are presented in Section 2; Section 3 will design a knee point search algorithm, including basic idea, top-k sort algorithm, time complexity, parameter updating, cascading top-k sorting with minimized time complexity, the knee point search algorithm, and the solution of the optimization problem; Section 4 will introduce source detection of DNS DoS flooding attacks as an application example of the proposed algorithm; Section 5 will conclude this paper.

2. Knee Point Search and Selection Sort

Assume there are 𝑛 points 𝑎₁,𝑎₂,…,𝑎_𝑛 whose values are 𝑞₁,𝑞₂,…,𝑞_𝑛. Let 𝑎₁,𝑎₂,…,𝑎_𝑛 be sorted by their values and let their differential values be 𝛿₁,𝛿₂,…,𝛿_𝑛, where 𝑞₁≥𝑞₂≥⋯≥𝑞_𝑛, and 𝛿_𝑖=𝛿₁𝑞𝑖=1_𝑖−1−𝑞_𝑖𝑖=2,3,…,𝑛. (1) As illustrated in Figure 1, there is usually a notable gap of value between points on the upper left side and those on the lower right side of the sorted curve. We define a knee point as the one whose neighboring differential values differ significantly.

Figure 1 

Sorted graph for first knee point search.

Definition 1. Point  𝑎_𝑖,𝑖=1,2,…,𝑛−1, is a knee point if it satisfies 𝛿_𝑖+1𝛿_𝑖>𝜃, (2) where 𝜃 is the threshold, whose value ranges from 10 to 50 in the practice of anomaly detection.

Note that there may be more than one knee point for a list, and the goal of the algorithm is to find the largest one in the sorted curve. For an unsorted list, we should first partially sort the list to find the sorted top-k list and then search the sorted top-k list for the largest knee point.

Definition 2. A top-k sort problem of selection number k for an unsorted list 𝐿 is a problem that finds k largest elements of L sorted in descending order.

Apparently, total sort is often not optimal for the problem as knee point search may be successful on a partially sorted top-k list if it contains the largest knee point. Therefore it is preferable to selectively sort first using the top-k sort algorithm and then search in each step. The procedure may go through many recursive steps until finding the largest knee point for the search may fail in the previous steps. There is a tradeoff between the time cost and expected hit probability of knee point search in each step, both determined by the selection number and both contributing to the expected overall time cost. In this paper, the optimization problem of the selection number is solved by minimizing the maximum expected time cost.

3. The Knee Point Search Algorithm

3.1. Basic Idea

The knee point search algorithm is based on cascading top-k sorting. In each step, top-k sorting segments the list left to be searched. The optimal selection number is determined by minimizing the expected time cost on the list left. If the search successfully finds the knee point in the sorted top-k list, the algorithm ends there. Otherwise, the residual list excluding top-k requires further checking. It becomes the objective list for the next step and the function of the expected time cost using top-k sorting should also be updated according to the a priori knowledge that the search fails in the previous step. Thus the new top-k sort problem, likewise, holds for the next step. The algorithm runs in this way recursively until the knee point is found. There are two cases that bring the algorithm to the end.(1)The knee point is found in the sorted top-k list in a step.(2)The optimal selection number in one step equals the length of the objective list. This means the optimal option is total sorting. Therefore the knee point is certain to be found in the completely sorted list.

3.2. Top-k Sort Algorithm

We design a quicksort variation as the top-k sort algorithm. Quicksort is a very efficient sort algorithm invented by Hoare [7]. Quicksort has two phases: the partition phase and the sort phase, which makes it a good example of the divide and conquer strategy for solving problems.

Top-k sorting only aims at treating the largest k elements, and thus it can be facilitated by the divide and conquer strategy. The intermediate results of quicksort, namely, the pivot positions, can be leveraged to possibly cut off one of the smaller problems divided from the bigger problem and to be conquered. For the strategy to be effective, the partition phase runs recursively only for the lower part if the pivot falls below position k, because there is no need to sort the upper part, which only consists of elements larger than top-k. This is the major distinction from the original quicksort algorithm and brings sorting efficiency.

At the same time, the pivots located after position 𝑑_𝑖 (the optimal selection number) at step 𝑖 in Section 3.6 are potentially useful for the afterwards steps, while they are actually not helpful for the inner top-𝑑_𝑖 sorting. Therefore we record those pivots via a stack. A stack is a data structure featured by last in, first out (LIFO). Recalling that the recursive partitions with their pivots after position 𝑘 produce their pivots in a sequential descending position order, we push these pivots into the stack resulting in a stack of pivots ordered by their positions. At the afterwards step 𝑖+1, if the optimal selection number 𝑑_𝑖+1 is larger than the position of the pivot at the top of the stack, the pivot is popped from the stack used for an inner pivot of top-𝑑_𝑖+1. Since this pivot is no longer needed for the afterwards steps, it should not be maintained in the stack. When the stack is empty or the pivot at the top of the stack (so do all of the other pivots) is located after the selection number, the partition has to run by itself to find a pivot without the help of the pivot stack.

The top-k sort algorithm, namely, QuickSortTopK, can be expressed as in Algorithm 1.

Input: 𝐿: an unsorted list of length 𝑛 FirstIndex: the first index of 𝐿 to be sorted LastIndex: the last index of 𝐿 to be sorted 𝑘: top-𝑘 elements of 𝐿 are to be sorted Output: 𝐿𝑘: Sorted top-𝑘 elements of 𝐿 ranging from FirstIndex to LastIndex 𝑆: a stack of pivot positons useful for the afterwards selection sorts (1)  if FirstIndex < LastIndex then (2)  if !isempty(𝑆) & top(𝑆) < LastIndex then  /* the stack is available */ (3)   pivotpos = pop(𝑆)  /* use the top element of the stack as the pivot */ (4)  else (5)   /* Partition without using pivots from the stack */ (6)   pivotpos=Partition(L, FirstIndex, LastIndex) (7)  if  pivotpos− FirstIndex +1>𝑘  then /* The pivot falls after position 𝑘 */ (8)   /* The pivot may be useful for the afterwards steps */ (9)   push(pivotpos, 𝑆) (10)   /* QuickSort for top-k in the upper part */ (11)     QuickSortTopK(L, FirstIndex, pivotpos −1, k) (12)  /* The pivot is located exactly at position 𝑘 */ (13)   elseif(pivotpos− FirstIndex +1==𝑘) (14)   /* QuickSort for top-𝑘−1 in the upper part */ (15)   QuickSortTopK(L, FirstIndex, pivotpos −1, 𝑘−1) (16)   else  /* The pivot falls prior to position k */ (17)    /* QuickSort the upper part */ (18)    QuickSortTopK(L, FirstIndex, pivotpos −1, pivotpos− FirstIndex) (19)    /* QuickSort for the residual elements in the lower part */ (20)    QuickSortTopK(𝐿, pivotpos +1, FirstIndex, 𝑘-pivotpos+ FirstIndex −1)

Algorithm 1 

QuickSortTopK(𝐿, FirstIndex, LastIndex, 𝑘).

The input of QuickSortTopK is the objective unsorted list 𝐿 indexed from FirstIndex to LastIndex. For all steps, LastIndex is fixed at 𝑛, whereas FirstIndex is progressively increased to exclude the sorted part of 𝐿 in all previous steps. The output includes the sorted top-𝑘 elements of 𝐿 indexed from FirstIndex to LastIndex and the stack 𝑆 containing all pivots falling after position 𝑘 obtained in all previous steps.

The termination condition of the recursion is checked in Line 1 of the algorithm. If the stack is nonempty and the top element of the stack falls into the objective range (see Line 2), the top element is used as the pivot for the partition (see Line 3). Otherwise, the pivot is obtained by a partition (see Line 6). Once the pivot is presented, different recursive steps are to be taken depending on the position of the pivot. If the pivot falls after position k, it should be pushed into the stack and then run further sorting on the original list subtracting the pivot, hoping to help afterwards steps (see Lines 9, 11). If the pivot is located exactly at position k, the pivot itself is the last element of the output and thereafter only top-k-1 sorting on the original list subtracting the pivot is needed (see Line 15). If the pivot is located prior to position k, both the upper and lower parts should be treated. The action on the upper part is equivalent to Quicksort, while the action on the lower part is actually the recursive running of QuickSortTopK with diminished selection number k and the shrinking objective list (see Lines 18, 20).

3.3. Time Complexity

Let totally sorting of a list of length 𝑛 require 𝑆(𝑛) time. Calculated by the number of comparisons, the average time complexity of 𝑆(𝑛) is 𝑂(𝑛log(𝑛)) following some efficient algorithms, for example, Quicksort [7].

Let top-k sorting of selection number k require 𝐶(𝑘,𝑛) time, where 𝑛 is the length of the list. The QuickSortTopK algorithm requires an expected time of 𝑂(𝑛+𝑘log𝑘). So 𝐶(𝑘,𝑛) equals 𝑂(𝑛+𝑘log𝑘).

Let the time complexity of finding the knee point in the sorted list of length 𝑛 be 𝐷(𝑛). Recalling (2), 𝐷(𝑛) takes 𝑂(𝑛).

3.4. Parameter Updating

For a list of length 𝑛, the algorithm divides the overall procedure into 𝑚+1 steps by a sequence of selection numbers, 𝑑₁,𝑑₂,…,𝑑_𝑚, 1≤𝑑₁<𝑑₂<⋯<𝑑_𝑚≤𝑛. Additionally, to facilitate the formulation, we let 𝑑₀=0 and 𝑑_𝑚+1=𝑛. Let the length of the objective list in step 𝑖, 𝑖=1,2,…,𝑚+1 be 𝑛_𝑖; we have 𝑛_𝑖=𝑛−𝑑_𝑖−1,𝑖=1,2,…,𝑚+1. (3)

In the first step, 𝑛_𝑖=𝑛. Let top-𝑘_𝑖 sorting for the objective list of length 𝑛_𝑖 be performed in step 𝑖, 𝑖=1,2,…,𝑚+1, and we have 𝑘_𝑖=𝑑_𝑖−𝑑_𝑖−1,𝑖=1,2,…,𝑚+1. (4)

Particularly, 𝑘_𝑖=𝑛_𝑖 in step 𝑚+1. And thus top-𝑘_𝑖 sorting for the objective list of length 𝑛_𝑖 is actually total sorting of the objective list. If the search of the knee point is successful in step 𝑖 for the sorted top-𝑘_𝑖 list, the algorithm ends at step 𝑖. Otherwise, the algorithm continues with the next step. The algorithm lasts until step 𝑚+1 if the search misses in all of the previous steps during the progressive search. Since step 𝑚+1 takes no further selection of the objective list, the algorithm finishes in it.

Let 𝐴 be the position variable of the knee point and 𝐴=1,2,…,𝑛. Let 𝑝(𝑖) represent the probability that 𝐴=𝑖, 𝑖=1,2,…,𝑛, and thus ∑^𝑛_𝑖=1𝑝(𝑖)=1. The value of 𝑝(𝑖) is assumed to be known at the beginning of the algorithm. Let 𝐴_𝑖 be the position variable of the knee point in step 𝑖 and 𝐴_𝑖=𝑑_𝑖−1+1, 𝑑_𝑖−1+2,…,𝑛. Let 𝑝_𝑖(𝑗) represent the probability that 𝐴_𝑖=𝑗, 𝑗=𝑑_𝑖−1+1, 𝑑_𝑖−1+2,…,𝑛, 𝑖=1,2,…,𝑚+1, and thus ∑^𝑛_{𝑗=𝑑𝑖−1+1}𝑝_𝑖(𝑗)=1.

Lemma 3. The probability distribution of the knee point in step 𝑖  𝑝_𝑖(𝑗) can be written as 𝑝_𝑖𝑗=𝑝𝑗∑^𝑛_{𝑘=𝑑𝑖−1+1}𝑝(𝑘),𝑗=𝑑_𝑖−1+1,𝑑_𝑖−1+2,…,𝑛,𝑖=1,2,…,𝑚+1. (5)

Proof. At the first step, all knowledge about the probability distribution of the knee point is only given by 𝑝(𝑖). But the search in the afterward steps should make use of the posterior distribution of the knee point for it is confirmed not to exist prior to the selection number in the previous steps; for example, when the algorithm comes to step 𝑖, the knee point is already checked to be not present in the top-𝑑_𝑖−1 list, 𝑖=2,3,…,𝑚+1. Therefore 𝑝_𝑖(𝑗) should be updated in step 𝑖 as 𝑝_𝑖𝑗=𝑃𝐴_𝑖𝑑=𝑗=𝑃𝐴=𝑗∣𝐴∈_𝑖−1+1,𝑑_𝑖−1=𝑃𝑑+2,…,𝑛𝐴=𝑗,𝑗∈_𝑖−1+1,𝑑_𝑖−1+2,…,𝑛𝑃𝐴∈𝑑_𝑖−1+1,𝑑_𝑖−1=𝑝𝑗+2,…,𝑛∑^𝑛_{𝑘=𝑑𝑖−1+1}𝑝(𝑘)𝑗=𝑑_𝑖−1+1,𝑑_𝑖−1+2,…,𝑛,𝑖=1,2,…,𝑚+1. (6)

Let the hit probability of search in step 𝑖 be 𝐻_𝑖.

Lemma 4. The hit probability of search in step 𝑖  𝐻_𝑖 can be written as 𝐻_𝑖𝑑_𝑖=^𝑑𝑖_{𝑗=𝑑𝑖−1+1}𝑝_𝑖𝑗,𝑖=1,2,…,𝑚+1. (7)

Proof. For the selection number 𝑑_𝑖 in step 𝑖, the search for the knee point is successful if and only if the knee point falls into the interval of position among 𝑑_𝑖−1+1 and 𝑑_𝑖. According to Lemma 3, the probability distribution of the knee point at step 𝑖  𝑝_𝑖(𝑗) should be updated as (5). Therefore we have 𝐻_𝑖𝑑_𝑖=^𝑑𝑖_{𝑗=𝑑𝑖−1+1}𝑃𝐴_𝑖==𝑗_{𝑑𝑖𝑗=𝑑𝑖−1+1}𝑝_𝑖𝑗,𝑖=1,2,…,𝑚+1. (8)

3.5. Cascading Top-k Sorting with Minimized Time Complexity

Lemma 5. Let the expected overall computational time cost in step 𝑖 be 𝑡𝑐_𝑖; 𝑡𝑐_𝑖 yields 𝑡𝑐_𝑖=𝐶𝑑_𝑖−𝑑_𝑖−1,𝑛−𝑑_𝑖−1+1−𝐻_𝑖𝑑_𝑖𝑡𝑐_𝑖+1,𝑖=1,2,…,𝑚. (9)

Proof. When the search succeeds in step 𝑖, tc_𝑖 comes only from top-𝑘_𝑖 sorting which requires 𝐶(𝑘_𝑖,𝑛_𝑖) time and the search in the top-𝑘_𝑖 sorted list which requires 𝐷(𝑛). However, recalling Section 3.3, the time complexity of 𝐷(𝑛) is negligible compared to that of 𝐶(𝑘_𝑖,𝑛_𝑖), so the summation of them can be approximated by 𝐶(𝑘_𝑖,𝑛_𝑖). For the failure of search in step 𝑖, the residual list of length 𝑛_𝑖−𝑑_𝑖 has to be further checked. Thus the overall computation time cost consists of top-k sort and search in the remaining list. Note that the computation time cost of sort and search in the remaining list is no other than tc_𝑖+1. Hence tc_𝑖 yields tc_𝑖=𝐻_𝑖𝑑_𝑖𝐶𝑘_𝑖,𝑛_𝑖+1−𝐻_𝑖𝑑_𝑖𝐶𝑘_𝑖,𝑛_𝑖+tc_𝑖+1=𝐶𝑘_𝑖,𝑛_𝑖+1−𝐻_𝑖𝑑_𝑖tc_𝑖+1,𝑖=1,2,…,𝑚. (10) Plugging (3) and (4) into (10), we get (9).

Lemma 5 tells us that the expected computational cost tc₁ can be calculated iteratively following (7), until reaching tc_𝑚+1 where there is no selection for step 𝑚+1. Thus tc_𝑚+1 only consists of the time cost of total sorting of the list of length 𝑛_𝑖 and the search in it. As total sorting takes 𝑆(𝑛_𝑖) and search takes 𝐷(𝑛_𝑖), thus we have tc_𝑚+1=𝑆𝑛_𝑚+1+𝐷𝑛_𝑚+1. (11)

Let 𝑁(𝑎,𝑏) (𝑎≤𝑏 and 𝑎, 𝑏 are integers) denote the set of integers {𝑖∣𝑎≤𝑖≤𝑏and𝑖isinteger}. In every step 𝑖, the algorithm calculates the current probability distribution of the knee point which determines the hit probability 𝐻_𝑖 and chooses 𝑑_𝑖 to be the solution of the following optimization problem: Min∶tc_𝑖s.t.𝑑_𝑖∈𝑁𝑑_𝑖−1+1,𝑛. (12)

We see in (12) that for any fixed 𝑑_𝑖 the minimum of tc_𝑖 is determined by tc_𝑖+1 under the optimal selection of 𝑑_𝑖+1 in the next step, 𝑑_𝑖∈𝑁(𝑑_𝑖−1+1,𝑛). And the optimal tc_𝑖+1 for any fixed 𝑑_𝑖+1 is also determined by the tc_𝑖+2 under the optimal selection of 𝑑_𝑖+2 and so on. This kind of iterative dependency finally extends to the last step which has no further selection. So tc_𝑖 is the function of 𝑑_𝑖, 𝑑_𝑖+1,…. The variation of any choice of selection in any number of steps makes the search space of optimization very huge, especially for the initial steps. Therefore it is not practical to evaluate all possibilities of selections in all of the afterwards steps when solving the optimization problem in step 𝑖. Thus it is necessary to constrain the variable of the objective function tc_𝑖 in (12) as mere 𝑑_𝑖.

Theorem 6. The upper bound of the minimum of 𝑡𝑐_𝑖 for a fixed 𝑑_𝑖 can be written as substituting 𝑡𝑐_𝑖+1 in (9) by 𝑆(𝑛_𝑖+1)+𝐷(𝑛_𝑖+1), such that min_{𝑑𝑖+1𝑑∈𝑁𝑖+1,𝑛,𝑑𝑖+2𝑑∈𝑁𝑖+1+1,𝑛,…}𝑡𝑐_𝑖≤𝐶𝑑_𝑖−𝑑_𝑖−1,𝑛−𝑑_𝑖−1+1−𝐻_𝑖𝑑_𝑖𝑆𝑛−𝑑_𝑖+𝐷𝑛−𝑑_𝑖,𝑖=1,2,…,𝑚. (13)

Proof. We only need to prove that min_{𝑑𝑖+1𝑑∈𝑁𝑖+1,𝑛,𝑑𝑖+2𝑑∈𝑁𝑖+1+1,𝑛,…}tc_𝑖+1≤𝑆𝑛−𝑑_𝑖+𝐷𝑛−𝑑_𝑖. (14) As total sort of the list of length 𝑛_𝑖+1 can be viewed as performing top-𝑛_𝑖+1 sorting, their time costs are both 𝑆(𝑛_𝑖+1)+𝐷(𝑛_𝑖+1). This is equivalent to the case when 𝑑_𝑖+1=𝑛. Because the algorithm ends with the total sorting and there are no afterwards steps, we have no definitions for 𝑑_𝑖+1, 𝑑_𝑖+2,…. Thus we have tc_{𝑖+1𝑑𝑖+1=𝑛(hence𝑑𝑖+2,𝑑𝑖+3…hasnodefinition)}=𝑆𝑛_𝑖+1+𝐷𝑛_𝑖+1. (15) Hence min_{𝑑𝑖+1𝑑∈𝑁𝑖+1,𝑛,𝑑𝑖+2𝑑∈𝑁𝑖+1+1,𝑛,…}tc_𝑖+1≤tc_{𝑖+1𝑑𝑖+1=𝑛(hence𝑑𝑖+2,𝑑𝑖+3,…hasnodefinition)}. (16) Plugging (3) into (15) and then (15) into (16), we have (14), and thereby (13) is proved.

Theorem 6 manifests that, for a fixed 𝑑_𝑖, tc_𝑖 is definitely bounded by the time cost of total sort of the residual list of length 𝑛_𝑖+1plus that of search in it. The optimization problem in step 𝑖 described by (13) can be isolated from all of the possible selections of the afterwards steps and becomes a function of mere 𝑑_𝑖. This minmax technique brings convenience to our analysis, such that (13) can be simplified as Min∶𝐶𝑑_𝑖−𝑑_𝑖−1,𝑛−𝑑_𝑖−1+1−𝐻_𝑖𝑑_𝑖𝑆𝑛−𝑑_𝑖+𝐷𝑛−𝑑_𝑖s.t.𝑑_𝑖∈𝑁𝑑_𝑖−1+1,𝑛. (17) Plugging (5) into (17), we have Min∶𝐶𝑑_𝑖−𝑑_𝑖−1,𝑛−𝑑_𝑖−1+1−^𝑑𝑖_{𝑗=𝑑𝑖−1+1}𝑝_𝑖𝑗𝑆𝑛−𝑑_𝑖+𝐷𝑛−𝑑_𝑖s.t.𝑑_𝑖∈𝑁𝑑_𝑖−1+1,𝑛. (18) Solving (14), we can obtain the optimal 𝑑_𝑖 in step 𝑖.

3.6. The Knee Point Search Algorithm

The knee point search algorithm runs iteratively using cascading top-𝑘 sorting. When the optimal selection number 𝑑_𝑖 is determined at step 𝑖, top-𝑘 sorting can be done via running QuickSortTopK  (𝐿,𝑑_𝑖−1+1,𝑛,𝑑_𝑖−𝑑_𝑖−1).  Specifically, the first step starts with QuickSortTopK   (𝐿,1,𝑛,𝑑₁).

The procedure can be described as follows.

Step 1. (1) According to (5), the probability distribution of the knee point for the optimization problem is initialized as follows 𝑝₁𝑗=𝑝𝑗,𝑗=1,2,…,𝑛. (19)(2) The optimal selection number 𝑑₁ is obtained by solving the following optimization problem as (18): Min∶𝐶𝑑₁,𝑛+1−^𝑑1_𝑗=1𝑝𝑗×𝑆𝑛−𝑑₁+𝐷𝑛−𝑑₁s.t.𝑑₁∈𝑁(1,𝑛). (20)(3) Perform top-𝑑₁ sorting on the list of length 𝑛.(4) Search for the knee point on the sorted top-𝑑₁ list. If successful or 𝑑₁=𝑛, the algorithm ends. Otherwise, go to Step 2.

Step 2. (1) The probability distribution of the knee point for the optimization problem is updated as follows: 𝑝₂𝑗=𝑝𝑗∑^𝑛_𝑘=𝑑1+1𝑝(𝑘),𝑗=𝑑₁+1,𝑑₁+2,…,𝑛. (21)(2)𝑑₂ is derived as the solution of the following optimization problem: Min∶𝐶𝑑₂−𝑑₁,𝑛−𝑑₁+1−^𝑑2_𝑗=𝑑1+1𝑝₂𝑗×𝑆𝑛−𝑑₂+𝐷𝑛−𝑑₂s.t.𝑑₂∈𝑁𝑑₁+1,𝑛. (22)Note that 𝑑₁ is inherited from Step 1.(3) Perform top-(𝑑₂−𝑑₁) selection sort on the list of length 𝑛−𝑑₁ subtracting top-𝑑₁ elements in Step 1.(4) Search for the knee point on the sorted top-(𝑑₂−𝑑₁) list. If successful or 𝑑₂=𝑛, the algorithm ends. Otherwise, go to Step 2.

Step i.(1)The probability distribution of the knee point for the optimization problem is updated according to (5).(2)𝑑_𝑖 is obtained as the solution of the optimization problem in (18), where 𝑑_𝑖−1 is known from step 𝑖−1. (3)Perform top-(𝑑_𝑖−𝑑_𝑖−1) selection sort on the list of length 𝑛−𝑑_𝑖−1 subtracting top-𝑑_𝑖−1 elements already sorted in the previous steps.(4)Search for the knee point on the sorted top-(𝑑_𝑖−𝑑_𝑖−1) list. If successful or 𝑑_𝑖=𝑛, the algorithm ends. Otherwise, go to step 𝑖+1.

The knee point search algorithm can be summarized as in Algorithm 2.

Input: Unsorted list 𝐿 of length 𝑛, probability distribution of the knee point in the sorted list of length 𝑛: 𝑃 Output: The knee point e or the non-existence of it (𝑒 = null). (1) /* initialize the parameters for the optimization problem */ (2) 𝑃1←𝑃, 𝐿1←𝐿 (3) 𝑖←1 (4) while (true) (5) solve the optimization problem of list 𝐿𝑖 in (18) to get 𝑑𝑖 (6)  if  𝑑𝑖=𝑛𝑖  then /* total sorting is the optimal option */ (7)  totally sort list 𝐿𝑖 (8)  /* successful if the knee point exists */ (9)  Search the sorted list for the knee point e (10)  break /* the algorithm is completed */ (11)  else /* top-𝑑𝑖−𝑑𝑖−1 sort is the optimal option */ (12)  QuickSortTopK(𝐿, 𝑑𝑖−1 + 1, 𝑛, 𝑑𝑖--𝑑𝑖−1) (13)  Search the sorted list for the knee point e (14)  if  e is found then (15)   break (16)  else /* Update the parameters for the next round of optimization problem */ (17)   update 𝑃𝑖+1 according to (5) /* the posterior probability */ (18)   update 𝐿𝑖+1 by exluding the top-𝑑𝑖−𝑑𝑖−1 item from list 𝐿𝑖 (19)   𝑖←𝑖+1 (20)  return  𝑒

Algorithm 2 

FindKnee(𝐿, 𝑃).

The knee point search algorithm can also be expressed recursively.

For each recursive step, we have an unsorted list 𝐿 of length 𝑛 and the probability distribution of the knee point in the sorted list of length 𝑛:𝑃. Thus we can modify the optimization problem in (18) as follows: Min∶𝐶(𝑑,𝑛)+1−^𝑑_𝑗=1𝑝𝑗(𝑆()(().𝑛−𝑑+𝐷𝑛−𝑑))s.t.𝑑∈𝑁1,𝑛 (23)

Solving (14), we can obtain the optimal 𝑑 in each recursive step.

When the knee point search fails after top-𝑘 sorting in each recursive step, the algorithm has to go to the next recursive step. First, we need to update the probability distribution of the knee point as well as the residual unsorted list as two parameters for the recursive function. According to Lemma 3, the update of the probability distribution of the knee point yields 𝑝𝑗⟵𝑝𝑗∑^𝑛_𝑘=𝑑+1𝑝(𝑘),𝑗=𝑑+1,𝑑+2,…,𝑛. (24)

In each recursive step, top-𝑘 sorting can be done via running QuickSortTopK(𝐿, 1, length(𝐿), 𝑑), where 𝐿 is the objective list in the current step and length (𝐿) denotes the length of𝐿.

A recursive version of knee point search algorithm can be summarized as in Algorithm 3.

Input: Unsorted list 𝐿 of length 𝑛, probability distribution of the knee point in the sorted list of length 𝑛: 𝑃 Output: The knee point e or the non-existence of it (𝑒 = null). (1) solve the optimization problem of list 𝐿 in (20) to get 𝑑 (2)  if  𝑑=𝑛 then /* total sort is the optimal option */ (3)  totally sort list 𝐿 (4)  /* successful if the knee point exists */ (5)  Search the sorted list for the knee point 𝑒 (6)  return 𝑒 /* the algorithm is completed */ (7)  else /* top-𝑑 sorting is the optimal option */ (8)   QuickSortTopK(𝐿, 1, length(𝐿), 𝑑) (9)   Search the sorted list for the knee point e (10)    if 𝑒 is found then (11)    return e (12)    else /* Update the parameters for the next round of optimization problem */ (13)    update 𝑃 according to (5) /* the posterior probability */ (14)    update 𝐿 by exluding the top-𝑑 item from list 𝐿 (15)     FindKnee(𝐿, 𝑃)

Algorithm 3 

FindKnee(𝐿, 𝑃).

3.7. The Solution of the Optimization Problem

In this section, we will assume two forms of the probability distribution of the knee point and discuss the solutions of (18) under these presumptions.

(1) Uniform Distribution. 𝑝(𝑖) is equal for all 𝑖∈𝑁(1,𝑛), and thus 𝑝(𝑖)=1/𝑛. Plugging 𝑝(𝑖) into (23), we have ()+𝑑Min∶𝐶𝑑,𝑛1−𝑛(𝑆()(().𝑛−𝑑+𝐷𝑛−𝑑))s.t.𝑑∈𝑁1,𝑛 (25)

Let 𝑓(𝑑)()+𝑑=𝐶𝑑,𝑛1−𝑛(𝑆(𝑛−𝑑)+𝐷(𝑛−𝑑)). (26) Although (26) is a discrete function, we still utilize the method of derivation to find the extremum, which can be only applied to the continuous and derivable function. Here we treat the discrete variable 𝑑 as continuous ones, and thereafter (26) turns into a continuous and derivable function. This is a rational approximation of the problem, which facilitates our analysis and solving. The final solution should be the round-off of 𝑑 obtained by solving the continuous function.

For simplicity, we let 𝐶(𝑘,𝑛)=𝑐₁𝑆(𝑛)𝑛+𝑘log𝑘=𝑐₂(𝑛)𝐷(𝑛)𝑛log=𝑐₃𝑛, (27) where 𝑐₁,𝑐₂, and 𝑐₃ are all constants.

By choosing 𝑑 such that 𝑑𝑓(𝑑)/𝑑=0, we have the optimal 𝑑 which satisfies 𝑐₁1+log𝑑𝑛=(𝑛−𝑑)2𝑐₂log(𝑛−𝑑)+2𝑐₃+𝑐₂. (28)

For large 𝑛 and 𝑑, we have the approximation of (28) as 𝑐₁𝑛log𝑑=2𝑐₂(𝑛−𝑑)log(𝑛−𝑑). (29)

To get the explicit mathematical expression of the solution of the nonlinearity equation (29), we used a heuristic approach to simplify the problem. We assume that 𝑑 is a proportion function of 𝑛, such that 𝑑=𝑝𝑛,0<𝑝<1, (30) where 𝑝 decides the optimum of 𝑑.

Thus (30) yields 𝑐₁𝑛log𝑝𝑛=2𝑐₂1−𝑝𝑛log1−𝑝𝑛, (31) then 𝑐₁𝑛log𝑝+𝑐₁𝑛log(𝑛)=2𝑐₂1−𝑝𝑛log1−𝑝+2𝑐₂1−𝑝𝑛log(𝑛). (32)

Leave out the low order item in both sides of (32), and we get 𝑐₁𝑛log𝑝+𝑐₁𝑛log(𝑛)≈𝑐₁𝑛log(𝑛),2𝑐₂1−𝑝𝑛log1−𝑝+2𝑐₂1−𝑝𝑛log(𝑛)≈2𝑐₂1−𝑝𝑛log(𝑛). (33)

Plugging (33) into (32), the optimum of 𝑝 yields 𝑐𝑝=1−¹2𝑐₂. (34)

We can see that the optimum of 𝑝 that takes the form of a proportion function of 𝑛 exists only when 𝑐₁<2𝑐₂. Particularly, if 𝑐₁=𝑐₂=1, the optimum of 𝑝 is 1/2, which means that the optimum first selection number is half of the length of the list. As the search algorithm may run recursively if the knee point search misses, the optimal method is the so-called binary search or logarithmic search method.

Theorem 7. If the probability distribution of the knee point follows uniform distribution and the optimal selection number takes the form of (30), the optimal method for the knee point search algorithm is binary search or logarithmic search method.

Proof. We first prove using an inductive method that in each recursive step of the search algorithm, 𝑝(𝑖) is equal for all 𝑖∈𝑁(1,𝑛).
We know that for the first recursive step, 𝑝(𝑖) is equal for all 𝑖∈𝑁(1,𝑛) as the starting point of the induction. Then for the second recursive step, we can derive according to (24) that 𝑝(𝑖) is equal for all 𝑖∈𝑁(1,𝑛).
Suppose that for the 𝑙th recursive step, 𝑝(𝑖) is equal for all 𝑖∈𝑁(1,𝑛). Similarly, for the (𝑙+1)th recursive step, 𝑝(𝑖) is equal for all 𝑖∈𝑁(1,𝑛) following (24).
Therefore we can form the concept inductively that in each recursive step of the knee point search algorithm, 𝑝(𝑖) is equal for all 𝑖∈𝑁(1,𝑛). So the optimization problem in each recursive step can be written as no other than (25), whose solution of number of selection is half of the length of the list discussed above. Here the list is the one left from previous recursive step. Hence the optimal method for the search algorithm is binary search or logarithmic search method.

(2) Inverse Proportion Distribution. 𝑝(𝑖) is in direct inverse proportion to 𝑖, 𝑖∈𝑁(1,𝑛), and thus 𝑝(𝑖)=𝑐/𝑖, where 𝑐=1/∑^𝑛_𝑖=1(1/𝑖).

As an approximate treatment of the summation ∑^𝑏_𝑖=𝑎(1/𝑖), we consider ^𝑏_𝑖=𝑎1𝑖≈^𝑏_𝑎1𝑥𝑑𝑥. (35)

Particularly, for 𝑎=1 and a large 𝑏, we have ^𝑏_𝑖=11𝑖=ln𝑏+𝐸. (36)

And for a large 𝑛, we have 𝑐=1/(ln𝑛+𝐸), (37) where 𝐸 is the Euler constant and has the approximate value as 0.5772.

Plugging 𝑝(𝑖) and then (36) and (37) into (23), we have Min∶𝐶(𝑑,𝑛)+1−ln𝑑+𝐸(𝑆()(().ln𝑛+𝐸𝑛−𝑑+𝐷𝑛−𝑑))s.t.𝑑∈𝑁1,𝑛 (38)

Let 𝑔(𝑑)=𝐶(𝑑,𝑛)+1−ln𝑑+𝐸(𝑆()(ln𝑛+𝐸𝑛−𝑑+𝐷𝑛−𝑑)). (39)

Theorem 8. If the probability distribution of the knee point 𝑝(𝑖) is in direct inverse proportion to 𝑖,  𝑖∈𝑁(1,𝑛), the optimal top-k sorting for the knee point search algorithm is full sorting of the list for a large length of the list.

Proof. We only need to prove that 𝑔(𝑑) is a monotony increase function of 𝑑, and the equivalent condition for it is 𝑑𝑔(𝑑)𝑑=𝑐₁1+log𝑑−𝑐₂(𝑛−𝑑)log(𝑛−𝑑)−𝑐₃𝑛()𝑑−𝑐ln𝑛+𝐸₂(ln𝑛−ln𝑑)1+log(𝑛−𝑑)ln𝑛+𝐸<0, (40) that is, 𝑐₁1+log𝑑(ln𝑛+𝐸)𝑑<𝑐₂(𝑛−𝑑)log(𝑛−𝑑)+𝑐₃𝑛+𝑐₂(ln𝑛−ln𝑑)1+log(𝑛−𝑑)𝑑. (41)
We assume that 𝑑 is an exponential function of 𝑛, such that 𝑑=𝑛^𝑞,0<𝑞<1. (42)
Thus the left and right sides of (41) can be, respectively, written as 𝑐₁1+log𝑑(ln𝑛+𝐸)𝑑∼𝑂log𝑛²𝑛^𝑞𝑐₂(𝑛−𝑑)log(𝑛−𝑑)+𝑐₃𝑛+𝑐₂(ln𝑛−ln𝑑)1+log(𝑛−𝑑)𝑑∼𝑂(𝑛). (43)(44) Comparing (43) and (44), we obtain (41) for a large 𝑛. That means the optimal selection number is 𝑛 for the first step, or the optimal top-k sorting for the knee point search algorithm is full sorting.

4. Source Detection of DNS DoS Flooding Attacks: An Application Example

4.1. DNS DoS Flooding Attacks

The Domain Name System is a fundamental and indispensable component of the modern Internet [13, 14]. The availability of the DNS can affect the availability of a large number of Internet applications. Ensuring the DNS data availability is an essential part of providing a robust Internet.

In the past few years, some important DNS name servers on the top level of the DNS hierarchical structure were targeted by the DoS or DDoS attackers, and some of these attacks did succeed in disabling the DNS servers and resulted in parts of the Internet experiencing severe name resolution problems [15–18]. Particularly, DNS DoS flooding attacks are the attacks launched by the attackers towards the DNS name servers with an overwhelming traffic flux in order to disrupt the DNS service for the legitimate clients. However, it is usually not easy to efficiently detect and defend the DoS flooding attacks because the attacking traffic is blended with the legitimate ones, which complicates the distinguishing efforts. Moreover, the detection mechanism should be implementable or should not add heavy computational load. Here we focus on the source-based detection method and show that the problem of source detection of DNS DoS flooding attacks can be addressed by the knee point search in the sorted curve discussed in this paper.

4.2. Detection Using the Knee Point Search

Generally, DNS name servers may receive queries coming from thousands of DNS clients (mostly DNS cache servers), whose traffic volumes are expected to remain far below those of the DoS flooding attacks. The real-time query rates for all incoming sources can be counted by the traffic monitoring system residing at the border gateway in front of the DNS name server. The goal of the DoS attack defense is to realize real-time attacking source detection and then filter out the attacking traffic from these sources accordingly. Therefore the detection problem is equivalent to knee point search in the sorted curve, where all points above the largest knee point are identified as the attacking sources. Moreover, time efficiency is also the key requirement for the problem, for timely attacking detection means timely defending action. Applying the knee point search algorithm proposed in this paper, the expected detection time is minimized.

4.3. Leaning the Knee Point Distribution

The assumption on probability distribution of the knee point is the prerequisite for the knee point search algorithm. However, in the initial rounds of detection we have hardly any a priori knowledge about the knee point. But the distribution estimation of the knee point can be learned based on the empirical data obtained in all previous rounds of detection.

First, suppose that the knee point largely follows stationary random distribution; hence its distribution exhibits almost the same probability model in all rounds of detection. We can fit a statistical model to data and provide estimates for the model's parameters. Here we apply the method of maximum likelihood for the estimation.

Let the count variable of detected knee points so far at position 𝑖 in the ordered list of length 𝑛 be ℎ_𝑖,𝑖=1,2,…,𝑛. Let the value of ℎ_𝑖 be 𝑥_𝑖, 𝑖=1,2,…,𝑛. If the number of rounds of detection is 𝑅, we have ^𝑛_𝑖=1𝑥_𝑖=𝑅. (45)

Let the probability vector of the knee point at different positions be 𝑃=(𝑝₁,𝑝₂,…,𝑝_𝑛)^′. The likelihood function of 𝑃 can be written as lik(𝑃)=𝑓_𝐷ℎ₁=𝑥₁,ℎ₂=𝑥₂,…,ℎ_𝑛=𝑥_𝑛∣𝑃, (46) where 𝑓_𝐷(⋅) is the density function. To calculate lik(𝑃), we have lik(𝑃)=𝑓_𝐷ℎ₁=𝑥₁∣𝑃𝑓_𝐷ℎ₂=𝑥₂,…,ℎ_𝑛=𝑥_𝑛∣ℎ₁=𝑥₁,𝑃=𝑓_𝐷ℎ₁=𝑥₁∣𝑃𝑓_𝐷ℎ₂=𝑥₂∣ℎ₁=𝑥₁,𝑃𝑓_𝐷×ℎ₃=𝑥₃,…,ℎ_𝑛=𝑥_𝑛∣ℎ₁=𝑥₁,ℎ₂=𝑥₂⋮,𝑃=𝑓_𝐷ℎ₁=𝑥₁∣𝑃𝑓_𝐷ℎ₂=𝑥₂∣ℎ₁=𝑥₁⋯𝑓,𝑃_𝐷ℎ_𝑛−2=𝑥_𝑛−2∣ℎ₁=𝑥₁,ℎ₂=𝑥₂,…,ℎ_𝑛−3=𝑥_𝑛−3,𝑃×𝑓_𝐷ℎ_𝑛−1=𝑥_𝑛−1,ℎ_𝑛=𝑥_𝑛∣ℎ₁=𝑥₁,ℎ₂=𝑥₂,…,ℎ_𝑛−2=𝑥_𝑛−2,𝑃=𝑓_𝐷ℎ₁=𝑥₁∣𝑃𝑓_𝐷ℎ₂=𝑥₂∣ℎ₁=𝑥₁⋯𝑓,𝑃_𝐷ℎ_𝑛−2=𝑥_𝑛−2∣ℎ₁=𝑥₁,ℎ₂=𝑥₂,…,ℎ_𝑛−3=𝑥_𝑛−3,𝑃×𝑓_𝐷ℎ_𝑛−1=𝑥_𝑛−1∣ℎ₁=𝑥₁,ℎ₂=𝑥₂,…,ℎ_𝑛−2=𝑥_𝑛−2,𝑃×𝑓_𝐷ℎ_𝑛=𝑥_𝑛∣ℎ₁=𝑥₁,ℎ₂=𝑥₂,…,ℎ_𝑛−1=𝑥_𝑛−1,𝑃, (47) where the last item in (47) is actually not an independent one given that all ℎ_𝑖 other than ℎ_𝑛 are known due to the constraint in (45), such that 𝑓_𝐷ℎ_𝑛=𝑥_𝑛∣ℎ₁=𝑥₁,ℎ₂=𝑥₂,…,ℎ_𝑛−1=𝑥_𝑛−1,𝑃=1. (48) Plugging (48) into (47), we get (𝑃)=𝑛𝑥lik₁𝑝^𝑥1₁1−𝑝₁^𝑛−𝑥1𝑛−𝑥₁𝑥₂𝑝₂1−𝑝₁^𝑥2×𝑝1−²1−𝑝₁^{𝑛−𝑥1−𝑥2}⋯𝑛−∑^𝑛−2_𝑖=1𝑥_𝑖𝑥_𝑛−1×𝑝_𝑛−11−∑^𝑛−2_𝑖=1𝑝_𝑖^𝑥𝑛−1𝑝1−^𝑛−11−∑^𝑛−2_𝑖=1𝑝_𝑖^{𝑛−∑𝑛−1𝑖=1𝑥𝑖}. (49) Thus let 𝜕ln(lik(𝑃))𝜕𝑝_𝑖=0,𝑖=1,2,…,𝑛−1. (50) We obtain the maximum likelihood estimation of 𝑝_𝑖, 𝑖=1,2,…,𝑛−1: 𝑝_𝑖=𝑥_𝑖𝑅𝑝,𝑖=1,2,…,𝑛−1,_𝑛=1−^𝑛−1_𝑖=1𝑝_𝑖=𝑥_𝑛𝑅. (51)

At the beginning of each round of detection, if the previous round finds the knee point at position 𝑖^∗, 𝑅 and 𝑥_𝑖, 𝑖=1,2,…,𝑛, are updated as follows: 𝑥𝑅⟵𝑅+1,_𝑖∗⟵𝑥^𝑖∗𝑥+1,_𝑖⟵𝑥^𝑖,𝑖∈𝑁(1,𝑛),𝑖≠𝑖^∗. (52)

The knee point distribution may evolve over time; thus the position of the knee point detected in recent rounds provides more reliable information for the estimation than earlier rounds. Taking the chronological order into consideration, we assign more weight to recent rounds than earlier rounds. This can be done by decreasing the detection results in previous rounds progressively. The deceasing is performed in updating 𝑅 and 𝑥_𝑖, 𝑖=1,2,…,𝑛, and sums up the current detection and the previous ones at a discount 𝛽, 0<𝛽<1. Formally, the updating of 𝑅 and 𝑥_𝑖, 𝑖=1,2,…,𝑛, can be modified as follows: 𝑅⟵𝛽^∗𝑥𝑅+1,_𝑖∗⟵𝛽^∗𝑥^𝑖∗𝑥+1,_𝑖⟵𝛽∗𝑥^𝑖,𝑖∈𝑁(1,𝑛),𝑖≠𝑖^∗. (53)

5. Conclusion

Knee point search in the sorted curve is often used in the practice of anomaly detection and many other applications. Due to the inefficiency of total sorting, top-k sorting should be adopted for the knee point search. In this paper, a knee point search algorithm using cascading top-k sorting is proposed. The expected time complexity is minimized via optimizing the selection number 𝑘 in each step.

Acknowledgments

This work was supported in part by the National Key Technology R&D Program of China under Grant no. 2012BAH16B00 and the National Science Foundation of China under Grant no. 61003239.