Abstract
A fundamental issue of 6G networks with aerial access networks (AAN) as a core component is that user devices will send highvolume traffic via AAN to backend servers. As such, it is critical to load balance such traffic such that it will not cause network congestion or disruption and affect users’ experience in 6G networks. Motivated by the success of softwaredefined networkingbased load balancing, this paper proposes a novel system called Tigris, to load balance highvolume AAN traffic in 6G networks. Different from existing load balancing solutions in traditional networks, Tigris tackles the fundamental disruptionresistant challenge in 6G networks for avoiding disruption of continuing flows and the controlpath update challenge for limiting the throughput of updating load balancing instructions. Tigris achieves disruptionfree and lowcontrolpathcost load balancing for AAN traffic by developing an online algorithm to compute disruptionresistant, perflow load balancing policies and a novel bottomup algorithm to compile the perflow policies into a highly compact rule set, which remains disruptionresistant and has a low controlpath cost. We use extensive evaluation to demonstrate the efficiency and efficacy of Tigris to achieve zero disruption of continuing AAN flows and an extremely low controlpath update overhead, while existing load balancing techniques in traditional networks such as ECMP cause high load variance and disrupt almost 100% continuing AAN flows.
1. Introduction
The nextgeneration cellular networks (i.e., 6G networks) can interconnect devices in space, air, and ground networks with the help of aerial access networks (AAN) to provide users Internet access with a substantial higher coverage than traditional network technologies (e.g., 5G networks) and hence have drawn much attention from both academia and industry [1–5]. Figure 1 gives an abstract architecture of 6G networks with AAN as a key component. After receiving data traffic from end devices, AAN forwards traffic to edge servers, which process and forward traffics to backend servers.
A fundamental issue of this 6G architecture is that with higher coverage of user devices and more ultrahighbandwidth, ultralowlatency applications such as VR/AR and remote medical, AAN needs to deliver highvolume data traffic involving a high number of flows from user devices to the corresponding backend servers via edge and backbone networks. As such, it is critical to load balance such traffic before they enter the backbone networks, such that it does not cause any congestion or network disruption in 6G networks.
Specifically, motivated by the recent success of flexible load balancing using softwaredefined networking (SDN), e.g., [6–14], in this paper, we investigate the feasibility and benefits of load balancing AAN traffic in 6G network using a softwaredefined edge (SDE), which we term as SDELB. Specifically, continuing the trend of moving from dedicated load balancing appliances to native network switches for load balancing (e.g., [6–8, 12, 15–19]), SDELB takes advantage of the flexibility of a logically centralized controller to collect load statistics from both the network and the servers to compute load balancing flow rules and install them on programmable switches (also called load balancing switches), whose TCAM flow tables allow highspeed packet processing [20–22] to forward packets to different backend servers, based on the matching results [12, 16, 18, 23, 24].
Despite the potential of SDELB, key challenges remain. In particular, we found that directly using existing load balancing solutions in SDN to SDELB encounters substantial challenges under dynamicity, where dynamicity can happen either on the data path or on the control path.
The key dynamicity challenge on the data path is the disruptionfree challenge. Specifically, as the load of AAN from the flows contained in each flow rule changes over time, imbalance occurs and rebalancing is needed. Although one may apply previous studies [16, 18, 25] to conduct rebalancing, since they do not consider existing assignments of continuing flows from AAN, i.e., flows with open TCP connections, they can result in unacceptable disruptions of continuing flows. Although one may use migration to reduce the impacts of disruption, migration is considered adding substantial system complexity and hence is not preferred by many operators. Another possibility is to install exact machines to pin the assignment of continuing flows, but flow table size constraints make this approach infeasible. For example, one commodity edge server equipped with Dell Z9100 switch has only 2304 flow entries [26], while the number of data flows with different source IP addresses is much larger. Utilizing TCAM wildcard rules to aggregate flow rules may reduce the number of rules (e.g., [16, 18, 21, 24, 27–29]) but would again result in a mass disruption of continuing flows.
The key dynamicity challenge on the control path is the controlpath update challenge. Specifically, rebalancing AAN traffic load among servers requires an SDELB controller to send flowmod instructions to update flow tables on the load balancing switches. Unfortunately, stateoftheart SDN controllers have limited throughput in sending flowmod instructions, putting a limit on controlpath update frequencies.
In this paper, we cope with both dynamicity challenges by proposing the first disruptionresistant, lowcontrolpathcost, dynamic SDE load balancer for AAN traffic in 6G networks. Tigris provides two key novel insights for addressing the dynamicity challenges: (1) it shifts a small number of incoming flows among servers to achieve load balancing without disrupting continuing flows and (2) it leverages the small number of shifted flows, the decomposition of aggregation of a large rule set into parallel aggregation of multiple smaller rule sets, and the cached intermediate rule aggregation results from previous time slots to substantially improve the efficiency of flow rule aggregation and reduce the controlpath update cost. This work sheds light for future research on system and protocol design in AAN and 6G networks, such as traffic engineering, resource orchestration, and networkapplication integration [30].
The main contributions of this paper are as follows. (i)We study a fundamental problem for 6G networks with AAN, the disruptionfree load balancing problem for AAN traffic, identify the disruptionresistant and the controlpathcost challenges, and design a novel dynamic load balancer at the edge called Tigris to address these challenges(ii)We design DRLB, an online algorithm, as the first phase of Tigris to compute perflow disruptionresistant load balancing policies and prove that DRLB achieves a competitive ratio of for load balancing, where is the number of backend servers(iii)We develop TreeAgg, an incremental, bottomup rule aggregation algorithm as the second phase of Tigris to compile perflow load balancing policies into a highly compact, disruptionresistant rule set which also has a low controlpath update cost(iv)We conduct extensive evaluations to show that Tigris achieves zero disruption of continuing flows in AAN of 6G networks, a closeto1 competitive ratio on load balancing, a less than 5% rule update ratio between time slots, and up to 53x flow rule compression ratio, while other stateoftheart approaches such as ECMP can cause 24x load variance and disrupt almost 100% continuing flows in AAN
The remainder of this paper is organized as follows. We discuss related work on different load balancing approaches in Section 2. We present the system settings and problem definition in Section 3. In Section 4, we propose the Tigris load balancer, introduce its overall workflow and the details of its key components, and discuss its generality and overhead. We evaluate the performance of Tigris with extensive simulation in Section 5 and finally conclude our work in Section 6.
2. Related Work
Data traffic load balancing is a wellstudied problem for which not only many algorithms have been designed and thoroughly analyzed [31–33] but also many systems have been developed in supporting scalable, reliable network services [6–8, 12, 15–19]. Modern networks increasingly rely on switches to develop load balancing solutions. In this section, we give a brief review of existing studies on load balancing and discuss the key limitations of these techniques for load balancing traffic from AAN.
2.1. HashBased Load Balancing
Among various load balancing systems, hashbased solutions such as EqualCost Multipath (ECMP) [25] and WeightedCost Multipath (WCMP) [7] are the most widely used ones. ECMPbased systems [25] evenly partition the flow space into hash buckets and use a hardware switch to redirect incoming packets to different software load balancers based on the hash value of the packet header. WCMPbased systems [7] achieve an unevenly weighted partition of the flow space by repeating the same software load balancers as the next hop multiple times in an ECMP group. However, these hashbased solutions split the traffic based on the size of flow space rather than the actual volume of flows, resulting in load imbalance due to the unequally distributed and dynamically changing traffic contribution from different flow spaces. Because ECMP hash functions are usually proprietary [18, 25], users have a limited customization capability in rebalancing to adapt to such dynamic flow statistics.
2.2. SDN Load Balancing
Compared with hashbased load balancing, SDN load balancing is a more powerful and flexible load balancing technique. As such, it is a more suitable solution to AAN load balancing, because balancing the dynamic traffic in AAN and 6G networks requires more flexible load balancing policies. SDN supports the programming of the flow rule table on switches using wildcards and hence enables a more flexible, finegrained load balancing service [12, 16, 18, 23, 24]. The select group table defined in OpenFlow specification [21] can be used for load sharing, but it requires the controller’s guidance to respond to events not detectable by the switch, e.g., server failures. Solutions in [23, 24] send the first packet of every flow back to the controller for calculating the flow rules. Benson et al. [12] integrate the perflow load balancing rule generation with WCMP to minimize the traffic congestion. Sending the first packet of each flow to the controller would add extra delay for these packets. Kang et al. [18] and Wang et al. [16] study the TCAM sizeconstrained traffic splitting problem and design an algorithm to generate efficient flow rules to partition the flow space into weighted subspaces for different servers. In addition, TCAM wildcards are also utilized to reduce the number of flow rules used for expressing network policies (e.g., TCAMRazor [27] and CacheFlow [28]), including the load balancing policy.
One important limitation of applying existing SDN load balancing solutions to load balance AAN traffic at an SDE of 6G networks is that they do not consider the key dynamic SDE load balancing challenges, i.e., the disruptionresistant challenge and the controlpath update challenge, leading to the unacceptable disruption of continuing flows and a high controlpath update cost. Miao et al. [34] design a stateful data plane load balancer, but it requires the support of expensive, customized programmable hardware. On the contrary, Tigris addresses the above challenges to design the first disruptionresistant, lowcontrolpathcost SDE load balancer using commodity SDN switches.
3. Dynamic SDE Load Balancing: System Settings and Problem Definition
We consider a dynamic SDELB system shown in Figure 2: it provides a service using multiple servers, indexed by . Clients access the service through a single public IP address, reachable via AAN. For any incoming packet, the load balancing switch at SDE finds the matched flow rule based on its source IP address, rewrites its destination IP addresses to that of the assigned server, and forwards it correspondingly to achieve load balancing. One may extend our system to multiple switches and to match on other fields for more complex load balancing scenarios for better scalability and fault tolerance, but we focus on the single load balancing switch and the source IPv4 address matching for clarity.
We consider that our system operates in discrete time where time is divided into slots, indexed by . Different time slots can have different duration to support a combination of periodic operations and eventdriven operations, where events can include server up/down and the burst of data flows. At the beginning of each slot , the SDE controller computes a set of load balancing flow rules denoted as and update the flow table of load balancing switch accordingly.
Given a flow rule , it has three basic attributes: the set of source IP addresses it matches, the forwarding action, and priority, denoted as , , and , respectively. Given a packet, its source IP may be matched by multiple rules. In this case, the switch will rewrite its destination IP and forward it following the action of the rule with the highest priority, as specified in OpenFlow specification [21]. We use an attribute named to denote the total estimated load of the flows that followed the action of . Our system allows the load metric to be a generic metric, considering factors such as data volume and CPU. The controller collects related data from servers and switches to compute the load metric.
A set of flow rules can be aggregated into a single rule using wildcard. We call the source rules of , denoted as . We also define some parameters to assist rule aggregation. We use to record the aggregation level of rule , where the first matching bits of must not contain any wildcard. As we will show in Section 4.3, can be increased even if it fails to be aggregated with another rule.
An efficient SDELB system must satisfy multiple constraints, which we separate into two categories: (1) traditional constraints (i.e., table size, fullcoverage, and zeroambiguity) and (2) dynamicity constraints (i.e., disruption resistance and low controlpath update).
3.1. Traditional Constraints
The first traditional constraint is the table size constraint. Specifically, the set of flow rules installed on the load balancing switch in any time slot must not exceed its flow table capacity , as expressed in Inequality
Secondly, for any incoming packet, there must be at least one flow rule that matches its source IP address and does not have punt as its action. This is the fullcoverage constraint. It ensures that no packet will be punted back to the controller, eliminating the switchtocontroller delay for all incoming data traffic, and is expressed as
Thirdly, for any incoming packet, if there exists more than one flow rule matching its source IP, only one rule’s action will be taken. This is the zeroambiguity constraint. We use to denote the set of source IP addresses that belong to but will not follow and express this constraint as where is the logical conjunction.
3.2. Dynamicity Constraints
Other than the traditional constraints, SDELB under dynamicity also introduces additional constraints on both data path and control path. For the data path, any flow whose TCP connections to the servers are currently open should not be shifted to another server unless the current server fails. We call it the disruptionresistant constraint. Denoting a continuing flow as and its server time slot as , this constraint can be expressed as
For the control path, the number of rules updated (i.e., deleted or inserted) in each time slot should be within the capacity of the controller, which is called the controlpath update constraint. Using to denote this constraint, we have
Denoting the data flow forwarded to server in time slot as and the number of time slots that server is working till time slot as , we formulate the disruptionresistant, lowcontrolpathcost, dynamic load balancing problem as subject to constraints (1), (2), (3), (4), and (5). We prove the NPhardness of this problem via a transformation from the classic multicore balancing problem [31]. Note that in Equation (6), we use the timeaveraged load balancing as the system objective. This is the typical objective of load balancing systems [31–33], and it reflects the requirement of dynamic SDELB, i.e., longterm, online load balancing. As we will discuss in Section 4.4, we design the Tigris system to be modular so that it provides users the flexibility to define different load balancing objectives and methods while maintaining the benefits of Tigris, e.g., lowcontrolpathcost and disruption resistance.
4. Tigris: A DisruptionResistant, LowControlPathCost, Dynamic SDE Load Balancer
We now fully specify Tigris, in which a controller computes disruptionresistant load balancing policies and generates compact flow rules with low controlpath cost correspondingly. We will start with an overview of Tigris in Section 4.1. Then, we will give the design details of its key components: policy generation in Section 4.2 and rule aggregation in Section 4.3. We also discuss the generalization of Tigris in Section 4.4.
4.1. Overview of Tigris
In practice, Tigris operates in both periodic mode and eventdriven mode in response to events such as the up/down of servers and the burst of data flows. For simplicity of presentation, we assume a periodical operation mode, where the controller executes Tigris at the beginning of every slot . Figure 3 presents the workflow of Tigris. Specifically, each invocation of Tigris can be divided into two phases: (1) policy generation and (2) rule compilation.
4.1.1. Policy Generation
During the policy generation phase, Tigris first collects load statistics and related flow information, e.g., TCP connection status, from each server , and estimates the target load of , i.e., the estimated load of incoming flows which should be assigned to in time slot for load balancing purpose (Step 1). A flow is identified by a TCP/IP 5tuple. It then uses the DRLB algorithm to compute a set of disruptionresistant load balancing policies for all servers available in time slot and generates a set of perflow rules to express the computed load balancing policies (Step 2). In particular, DRLB keeps the forwarding action of continuing flows on available servers and adopts an online, greedy approach to shift incoming flows from overloaded servers to underloaded servers based on the target load of each server and the similarity of source IPs between continuing flows and incoming flows. This design is disruptionresistant. It achieves load balancing by shifting a small number of flows among servers and increases the chance of reusing rule aggregated results from last slot during the rule aggregation phase, substantially increasing the aggregation efficiency and reducing the controlpath update cost. And we prove that it achieves a competitive ratio for Equation (6) in certain cases.
4.1.2. Rule Compilation
In the second phase, Tigris adopts a bottomup rule aggregation algorithm TreeAgg to iteratively aggregate the large perflow rule set into a highly compact rule set along a binary IP tree (Step 3). TreeAgg has three novel design points: (1) only traversing from IP leaves of shifted flows to the root of the IP tree, (2) decomposing the aggregation of a large rule set into parallel aggregation of multiple smaller sets, and (3) using cached intermediate rule aggregation results from slot during aggregation. These design points substantially increase the efficiency of rule aggregation and yield a compact rule set with a high similarity with , which reduces the controlpath cost for updating the data plane. After getting , Tigris updates the data plane by deleting the set of obsolete rules from the switch and installing the set of new rules (Step 4).
4.1.3. Addressing the SDELB Constraints
We show in the next subsections that the compact rule set expresses the same policies as the perflow rule set and satisfies the disruptionresistant constraint, the fullcoverage constraint, and the zeroambiguity constraint. Because it is NPhard to decide if a rule set can be aggregated into a smaller set of a given size [35], Tigris uses a software switch as a safety measure to install extra rules if the compact rule set still exceeds the table size of the hardware switch. In this way, when packets arrive, the hardware switch first tries to find a matching rule. If no matching rule is found, the packet is forwarded to the software switch (e.g., OpenvSwitch) for matching and processing. We show through extensive evaluation in Section 5 that this measure is rarely needed in practice since the aggregated rule set computed by Tigris is highly compact and outperforms the stateoftheart rule aggregation solution, i.e., TCAM Razor. Furthermore, we also show that the compact rule set computed by Tigris has an extremely low controlpath update cost.
4.2. Online, DisruptionResistant Policy Generation
We now give the details of the policy generation phase of Tigris. It involves two steps: statistics collection and target load adjustment and online, disruptionresistant load balancing, marked as Step 1 and Step 2 in Figure 3, respectively.
Step 1 (statistics collection and target load adjustment). At the beginning of time slot , Tigris first collects , the actual load of each server in slot , and related info about its flows, e.g., TCP connection status. In practice, such statistics can be retrieved from the log or the monitor process of the servers. It then estimates the total incoming load for all servers in slot as , the load of continuing flows (i.e., flows with open TCP connection) at each server in slot as , and the load of incoming flows at each server in slot as when the load balancing policy stays the same in as in , using methods adopted in [21, 28]. Next, Tigris calculates a key metric , the “target share” of each available server at each slot till the current time slot , in where is the number of time slots server is available among all slots. Note that one can extend it to the case where different servers have different capacities.
With , Tigris then computes the target load assigned to individual server which is available in slot , denoted as in Specifically, according to , should serve in all slots until . Since it has already served and will serve in slot for continuing flows, these “credits” are deducted. Note that the way Tigris computes depends on different load balancing objectives. This paper focuses on Equation (6).
Step 2 (online, disruptionresistant load balancing DRLB). With the targeting load for every server , we design the DRLB algorithm, summarized in Algorithm 1, to compute a set of disruptionresistant, perflow load balancing policies and the corresponding perflow rule set .
4.2.1. Basic Idea
We design DRLB as an online algorithm to achieve load balancing by shifting a small number of incoming flows among servers. This design ensures disruption resistance of continuing flows. It also increases the chance of reusing cached aggregation results from the last time slot during the rule aggregation phase (Section 4.3), substantially increasing the aggregation efficiency and reducing the controlpath cost to update the flow table on the switch.
At the beginning of the whole system, i.e., , it sets the load balancing policies to evenly divide flows from the whole source IP space to all servers and generates a perflow rule set (Line 1). At each time slot , it takes the collected load statistics from Step 1 and , the perflow rule set for slot as input (Line 3). It then iteratively shifts flows from overloaded servers to underloaded servers to achieve the target load for each server (Lines 418). During this process, it replaces the perflow rule for every shifted flow with a new perflow, eventually getting the new perflow rule set (Lines 1923).

4.2.2. Categorizing Overloaded and Underloaded Servers
DRLB first categorizes servers into overloaded and underloaded based on their target load (Lines 411). Given an available server at time slot , it is overloaded (underloaded) if its target load is smaller (larger) than the estimated load of incoming flows when load balancing policies stay the same in slot as in (Lines 48). For any server that is unavailable in slot , we set its target load as 0 and consider all the incoming flows destined to incoming (Lines 910). Hence, every unavailable server is an overloaded server (Line 11).
4.2.3. Shifting Incoming Flows from Overloaded Servers to Underloaded Servers
Next, DRLB adopts a greedy approach to balance the load among servers. For every overloaded server , it iteratively finds the incoming flow with the largest estimated load and shifts it to an underloaded server which is the assigned server for the flow with the largest IP prefix match with (Lines 1418). This process stops when is no longer overloaded or there is no underloaded server (Line 13). The rationale of this approach includes that (1) it achieves the load balancing among servers by shifting a small number of flows among servers and (2) it increases the chance of rule aggregation for the flow rules for shifted data flows. For instance, suppose has the source IP 10.0.0.0 and was forwarded to an overloaded server . And there exists a flow whose source IP is 10.0.0.1 and was forwarded to an underloaded server . DRLB will make the load balancing decision to forward to and generates a flow rule to express this decision. In this way, the rules for and can be aggregated into during the rule aggregation process.
During flow shifting, DRLB also generates the perflow rule set for slot . It initializes as the perflow rule set for the last slot (Line 3). For each shifted data flow, DRLB generates a flow rule with a priority of 32, a level of 0 (Lines 1922). is also assigned a property as 1 to indicate that represents a newly generated load balancing policy. Then, it finds the flow rule in who has the same match field as and replaces it with (Line 23). In the end, we get the perflow rule set representing the load balancing policies for slot .
4.2.4. Performance Analysis of DRLB
It is easy to see that satisfies the disruptionresistant and the fullcoverage constraints. We then propose the following proposition on the performance of DRLB.
Proposition 2. When all servers are available, repeatedly generating load balancing policies via DRLB at every time slot achieves a competitive ratio of on the objective function in Equation (6).
Proof. When all servers are available, any instance of our load balancing problem can be transformed into an instance of the classic load balancing problem, which is aimed at minimizing the maximal task completion time across servers. And in the transformed instance, all tasks arrive at the same time. With this transformation, we can see that the load balancing policy computed by DRLB is in the set of all possible policies computed by greedy Graham’s algorithm [31]. Hence, the competitive ratio of of DRLB for Equation (6) is a direct result of applying the technique in proving the competitive ratio of Graham’s algorithm.
Although the complexity of DRLB depends on the number of arriving flows, which can be large due to mouse flows, in practice, we can only focus on elephant flows for computing the load balancing decisions and assign mouse flows evenly across different servers.
4.3. Incremental, BottomUp Rule Compilation
We now give the details of the rule compilation phase of Tigris. It involves two steps: incremental, bottomup rule aggregation and data plane update, marked as Step 3 and Step 4 in Figure 3, respectively.
Step 3 (incremental, bottomup rule aggregation (TreeAgg)). Directly installing the initial disruptionresistant perflow rule set computed by DRLB into the load balancer switch is infeasible because the size of is much larger than the size of the switch flow table, tens of thousands vs. a few hundreds or thousands. Next, we develop the TreeAgg algorithm which adopts a bottomup approach to aggregate into a highly compact flow rule set expressing the same load balancing policies as does.
4.3.1. Basic Idea
We design TreeAgg to iteratively aggregate rules in along the 32level binary tree for IPv4 addresses, starting from the leaf nodes, i.e., level 0. At a first glance, this approach is impractical since the complete IPv4 address tree has leaf nodes, an extremely large number to traverse. However, we propose three novel design points in TreeAgg. First, and typically only have small different flow rules, i.e., rules where is set to in DRLB, due to the online feature of the DRLB. Hence, at each time slot , TreeAgg only needs to traverse from the leaves representing the new rules in to the root of the binary tree. Secondly, TreeAgg decomposes the aggregation of a large rule set into parallel aggregation on multiple disjoint subsets. Thirdly, TreeAgg uses the cached intermediate aggregation results from the last time slot to aggregate with the new rules from . These design points have two major benefits: (1) they substantially reduce the traverse and aggregation scale, hence significantly increasing the efficiency of aggregation, and (2) they reuse the cached intermediate aggregation results from at maximum to ensure that would have a high similarity with , the installed rule set for time slot , hence substantially reducing the controlpath cost to update the flow table of the switch.
4.3.2. Rule Aggregation Operations
Before we present the details of the TreeAgg algorithm, we first introduce some basic operations for rule aggregation, which are summarized in Algorithm 2. To aggregate a set of flow rules into one rule , we first need to decide the matching field of . This is computed in the matchFieldAgg function, where a wildcard is used at every matching bit that is not the same across every rule , and an exact bit 0 or 1 is used otherwise (Lines 17). In this way, it is guaranteed that every flow that matches at least one rule in will also match .
After computing the matching field of , we use the ruleAgg function to set other properties of , including action and priority. Note that during the whole rule aggregation process, we only consider aggregating rules that have the same forwarding action to avoid altering the original forwarding policies (Line 10). And the priority of the aggregated rule is set as the number of exact matching bits in its matching field (Line 11). The aggregated source and are set based on the definition in Section 3, and the aggregation level of is also increased by 1 (Lines 1215).

Given an aggregated rule and a set of rules , we can also compute the set of rules that direct a subset of the flow space of to a different server, i.e., , using the findDom function. This process is straightforward by checking the matching field, the action, and the priority of every rule (Lines 1922). In addition, as we will show next in the TreeAgg algorithm, if an aggregated rule cannot be inserted in the flow rule set due to the violation of the noambiguity constraint, the aggregation level of every rule in still needs to be increased by 1, by using the ruleLevUp function.
4.3.3. BottomUp Rule Aggregation along an IP Tree
Having introduced the basic operations for aggregating rules, we now present the details of the TreeAgg algorithm. TreeAgg iteratively aggregates rules in along the 32level binary tree for IPv4 addresses, starting from the leaf nodes, i.e., . As stated earlier, TreeAgg is an incremental aggregation algorithm. At each time slot, , it only traverses from the leaves representing the new rules in , i.e., rules with , to the root of the binary tree. During the traverse, it uses , the cached intermediate aggregation results from last slot , to aggregate with the new rules from .
The pseudocode of TreeAgg is shown in Algorithm 3. In the beginning, we construct a rule set by adding all newly generated rules in (Lines 24). We use this initialization of to ensure that the rule aggregation process only traverses from the leaves representing the new rules to the root of the binary IP tree. And because the load balancing policies for time slot are different from slot , intermediate aggregated rules from that conflict with the load balancing policies of slot need to be removed (Line 5). For instance, suppose a rule with is in and a rule with is in . Because and have the same location on the IP tree, i.e., the same leaf node, needs to be removed from . And we define the function to remove all the rules with that have the same location as rules in in the IP tree.
4.3.4. Decomposition of Rule Set into Disjoint Subsets Using 3Node Subtree Representation
In each iteration of the main loop (Line 649), we aggregate the set of flow rules at the current level of the IP tree into a more compact set and send it to the next level. In particular, every time, we randomly select a remaining flow rule from the current rule set (Line 9). And we define the function to return all the rules whose first matching bits are the same as but the th bit is different, from the union of and . Using this function, we construct two subsets and and remove their overlapping rules from and (Lines 1012). We can view and as two nodes with the same parent in the IP tree. Figure 4(a) gives an example of this subtree. Denoting the aggregated rule set of and as and placing it in the parent node of this 3node subtree, we can see that the aggregation process in each iteration of the main loop can be decomposed into the aggregation process of multiple 3node subtrees. We prove the following property on the IP tree.
Proposition 4. If and are two rule sets belonging to different 3node subtrees on the same level of the IP tree, and , .
Proof. For any and , they only share the same first bits and differs on the . And we see that and only contain rules with lower levels. Therefore, it is impossible to have a rule in to intersect the flow space of both and .
(a)
(b)
(c)
(d)
With this proposition, the aggregation of each subtree on the same level of the IP tree can be performed in parallel without violating the noambiguity constraint. Hence, such a decomposition substantially reduces the problem scale and improves the efficiency of rule aggregation. Next, we describe the aggregation process for this 3node subtree.
4.3.5. Aggregating a 3Node Subtree When Is Empty
If the set is empty, i.e., has no sibling with the same parent in the IP tree, it means that there are no flows with nonzero estimated load coming to the load balancing switch in the next time slot. In this case, we generate a new set of aggregated rules by changing the th bit of the matching field of all rules in to the wildcard (Lines 1415). It is straightforward that the newly generated aggregated rules do not increase the size of the rule set, cause no violation of the noambiguity constraint, and help ensure the fullcoverage constraint by covering the flow space that has zero expected loads. As a result, it is a successful aggregation and we insert the updated with an increased level into the aggregated rule set (Line 16).

4.3.6. Aggregating a 3Node Subtree When Is Nonempty
If the set is not empty, we move on to aggregate rules with the same forwarding server in and (Lines 1845). To do this, we randomly select from and from whose forwarding actions are the same and generate the aggregated rule (Line 22). Note that in TreeAgg, we select and from different sets. This is because we can prove.
Proposition 6. Given any two rules and both from or whose forwarding actions are the same, they cannot be aggregated to reduce the number of rules without violating the nonambiguity constraint.
Proof. Without loss of generality, assume that there exist such two rules and with the same forwarding action and both from . If they can be aggregated to reduce the number of flow rules, they should have been done during the aggregation of the 3node subtree rooted at . The only reason they are not aggregated is that they cannot reduce the number of flow rules, i.e., aggregating them requires extra rules to eliminate ambiguity. So, at itself, they should not be aggregated. Now, assume that there is another rule that shares the same forwarding action. Aggregating , , and together would cause the same issue because only shares the same first bits with and . Hence, and cannot be aggregated to decrease the number of flow rules.
After generating , we then compute using the findDom function defined in Algorithm 2 (Lines 2324). Leveraging Proposition 4, we only search in the union of , , and instead of the whole rule set to find efficiently. Having computed , we can check if directly inserting will lead to any violation of noambiguity constraint. To this end, we first check if there exists a rule from which violates this constraint with and . If so, cannot be inserted into the aggregated rule set since such ambiguity cannot be removed even if is later aggregated with another rule (Lines 2526). If such an does not exist, we then check if any ambiguity will happen between and other newly aggregated rules in (Line 27) and take different actions in different cases.
Case 1. If such a rule does not exist, can be directly inserted into as no ambiguity will be introduced by this insertion (Lines 2831).
Case 2. If there are more than two such rules, we do not insert into because at least 2 rules in have to be unaggregated to avoid ambiguity, which would increase the size of the rule set (Lines 3334).
Case 3. If there is only one such from , we compare the size of and and only keep the one with a smaller set of flowspace intersection rules (Lines 3539). The rationale of this strategy is that an aggregated rule with a smaller set would have a smaller chance to cause ambiguity with future aggregations in and .
If the aggregated rule eventually cannot be inserted into the aggregated rule set, we move on to select the next pair of rules from and for aggregation trial. For any rules and that has failed all possible aggregations, we increase their aggregated level by 1 and insert them directly into (Lines 4045). The aggregation process of a 3node subtree stops when both and are empty. We insert the resulting aggregated rule set for this subtree into a temporary set and move on to select the next 3node subtree for aggregation. The aggregation process for the current level stops when every nonempty 3node subtree on this level has been aggregated. And we then repeat the whole aggregation process for the next level on the complete aggregated set (Lines 4649) until we reach the root of the IP tree, i.e., . Note that before each iteration of the main loop, the function is invoked to remove obsolete, conflicting rules from (Line 48). And we also cache to assist the future rule aggregation for time slot (Line 49). In the end, TreeAgg returns as the compact rule set (Line 50).
4.3.7. An Example
We use the example in Figure 4 to illustrate the whole flow rule aggregation process on a subtree. The original 3node subtree is shown in Figure 4(a), and we omit the first 29 bits for simplicity. We start by aggregating the two rules forwarding to server and get with priority 1. Because this rule does not cause any ambiguity violation with either or , we insert it into in Figure 4(b). Next, we generate another aggregate rule for server , i.e., with priority 1. Though this rule does not cause any ambiguity violation with , it conflicts with the newly inserted rule in . To decide which rule to keep, we compare the size of their set. contains both and from , while only contains . Therefore, we keep the latter to reduce the probability of ambiguity conflict with future aggregated rules, and unaggregate back to its sources, and get the new shown in Figure 4(c). Last, we generate an aggregate rule for server and found that it has no ambiguity conflict with other rules. In this way, we get the minimal aggregated rule set with only 5 rules in Figure 4(d). Readers may find that if we keep in , the minimal size of will increase to 6, causing unnecessary waste of limited flow rule space.
4.3.8. Performance Analysis of TreeAgg
From the previous propositions, we see that TreeAgg does not change the action of any rule in when compressing it into . Hence, expresses the same load balancing policies as does, i.e., it satisfies the disruptionresistant and the fullcoverage constraints. And Propositions 4 and 6 ensure that satisfies the zeroambiguity constraint. One may notice that this algorithm has a polynomial complexity of the number of shift flows. However, because the number of the shifted flow is usually small, the decomposition of flow rule aggregation and the cached intermediate rule aggregation results from slot ensure that TreeAgg is computationally efficient and that has a high similarity to , substantially reducing the controlpath update cost of Tigris.
Step 4 (data plane update). After computing the compact rule set , Tigris deletes the set of obsolete rules from the switch and then installs the set of new rules . It is proved NPhard to decide if a given set of flow rules can be aggregated into a smaller set of a given size [35]. Hence, Tigris uses a software switch as a safety measure to install extra rules if the compact rule set still exceeds the flow table size of the hardware switch. As we will show in Section 5, however, it is rarely needed in practice since is highly compact. And we will also show that is highly similar to , i.e., a low rule update ratio, yielding an extremely low controlpath update cost.
4.4. Generalization of Tigris
4.4.1. Supporting Heterogeneous Servers and Other Load Balancing Objectives
For simplicity, we assume identical server machines in this paper, and the DRLB algorithm makes online, disruptionresistant load balancing policies that achieve a good competitive ratio on the load balancing objective in Equation (6). However, Tigris can also be applied to scenarios where servers have different computation resources, e.g., CPU and memory. This is because Tigris adopts a modular design which separates the load balancing decision process from the rule compression process. With this design, users have the flexibility to define and implement different load balancing algorithms, while still leveraging the high rule set compressing capability of TreeAgg.
4.4.2. Prefix vs. Suffix and PerFlow Load Balancing vs. PerNetwork Load Balancing
Though in the TreeAgg algorithm we assume an aggregation process based on the IP prefix, it is straightforward to apply it for suffixbased aggregation. In addition, Tigris supports not only perflow load balancing but also pernetwork load balancing. In the latter case, the DRLB algorithm can make load balancing decisions for each network, i.e., the traffic from the same network will be grouped and forwarded to the same server. With this type of load balancing decision, the TreeAgg algorithm starts the aggregation from the network level, instead of from the leaf level of the IP tree. The computation overhead for pernetwork load balancing hence can be substantially reduced, with the tradeoff of finegrained load balancing policies.
5. Evaluations
We implement a prototype of Tigris and carry out extensive simulations to evaluate the efficiency of Tigris in achieving sizeconstrained, disruptionresistant load balancing. The evaluation is performed on a MacBook Pro with 4 2.2 GHz Intel i7 cores and 16 GB memory.
5.1. Methodology
In our evaluations, we assume the setting of Figure 1, where a load balancer switch direct clients’ requests to a set of servers. We generate flows for the slot. Each flow has a source IP address chosen uniform randomly in the whole IP address space. Among the flows, a ratio of flows are considered continuing flows with randomly chosen forwarding servers in the previous slot and hence cannot be disrupted (i.e., cannot be redirected to another server). We assume that the load of each flow is its traffic volume and the volume is uniformly distributed between 1 MB and 1000 MB. We evaluate different settings of flows, servers, and ratio of continuing flows. We compare Tigris with the stateoftheart ECMP [25] system on the following load balancing metrics: (i)Load variance: the standard derivation of servers’ load over the load balancing target (ii)Disruption ratio: the percentage of disrupted continuing flows
We also compare Tigris with the stateoftheart TCAM Razor system [27] on the following rule aggregation metrics: (i)Rule compression ratio: the ratio of the size of perflow rule set generated by DRLB and that of the aggregated rule set computed by TreeAgg(ii)Rule set size: the size of the compact flow rule set computed by the Tigris
For these metrics, we run Tigris for time slots and summarize the average results. In addition, we also study the following metric to measure the controlpath cost of Tigris: (i)Rule update ratio: the ratio of compact flow rules in slot that are different from slot
5.2. Results
Due to the limited space, we only present the results with , which are more representative with a larger load balancing solution space and a more stringent requirement on rule aggregation, and omit the similar results of servers. Figure 5 summarizes the load balancing performance of Tigris under different numbers of flows and ratios of continuing flows. From Figure 5(a), we see that Tigris achieves a stable, small load variance, i.e., between 1000 MB and 2500 MB, and the competitive ratio of the Tigris algorithm in all settings is close to 1, as shown in Figure 5(b). This is consistent with our theoretical finding in Proposition 2. Taking the case of 8000 flows as an example, we also compare the load variance of Tigris with that of ECMP in Figure 5(c). We see that the load variance of ECMP is between 24x higher than Tigris. These observations demonstrate the cumbersomeness of hashbased load balancing solutions in adapting to the dynamics of flow statistics, the necessity for developing a perflowbased load balancing system, and the efficiency of Tigris in generating dynamic load balancing policies. We then plot the ratio of disrupted flows of ECMP and Tigris in Figure 5(d). We observe that Tigris achieves a 0 disruption ratio of continuing flows in all cases of evaluations, while ECMP has a closeto100% disruption ratio of continuing flows in all cases. This is because continuing flows are randomly distributed across all flows with random current servers, but ECMP simply divides the flow space evenly to different servers. This huge difference in the disruption ratio between Tigris and ECMP demonstrates the efficacy of Tigris for disruptionresistant load balancing.
(a)
(b)
(c)
(d)
We then study the capability of Tigris in generating highly compact flow rule sets in Figure 6. We see from Figure 6(a) that in most cases, Tigris is able to compute an aggregated rule set of less than 400 rules. Even in the worst case with 2000 continuing flows and a total of 8000 flows, Tigris yields an aggregated rule set of less than 800 rules. Figure 6(b) shows that Tigris achieves a high rule compression ratio, i.e., between 8 and 52. These observations show that Tigris is capable of computing a highly compact flow rule set that fits into typical commodity switches without the need to truncate any rules and hence achieves efficient utilization of the limited TCAM resource on commodity switches. Taking the case of 8000 flows as an example, we also compare the compression ratio of Tigris with that of TCAM Razor in Figure 6(c). We see that when the ratio of continuing flows is small, i.e., 0.05, Tigris and TCAM Razor give almost the same compression ratio. As the ratio of continuing flows increases, Tigris outperforms TCAM Razor by yielding a higher rule compression ratio. This is because the design of Tigris shifts a small number of flows during rebalancing and leverages the cached rule aggregation results from the previous time slot for better rule aggregation performance, while TCAM Razor has to start from the perflow rule set for a cleanslate aggregation. Furthermore, we plot the rule update ratio of Tigris over from to for the setting of flows and in Figure 6(d). We observe that Tigris yields a less than 5% rule update ratio, implying an extremely low controlpath update cost of Tigris.
(a)
(b)
(c)
(d)
In summary, we demonstrate the efficiency of Tigris in providing disruptionresistant, lowcontrolpathcost, dynamic load balancing service. Results show that Tigris achieves zero disruption of continuing flows and a closeto1 competitive ratio on load balancing objective. It also has a higher flow rule compression ratio, i.e., up to 53x and higher than TCAM Razor, with an extremely low controlpath update cost, while the stateoftheart hashbased solution ECMP can cause 24x load variance and disrupt almost 100% continuing flows.
6. Conclusion and Future Work
In this paper, we explored the key challenges for dynamic SDELB of AAN traffic in 6G networks, i.e., the disruptionresistant challenge and the controlpath update challenge. We address these challenges and design Tigris, the first disruptionresistant, lowcontrolpathcost dynamic SDE load balancer. Tigris computes disruptionresistant load balancing policies on a perIP basis and transforms the original large flow rule set into a highly compact rule set with a small size and a low controlpath cost, while remaining disruptionresistant. Evaluation results show that Tigris simultaneously achieves a closetooptimal load balancing performance, a high rule compression ratio, a low controlpath update cost, and zero disruption of continuing flows. This work sheds light on future research in traffic management in AAN and 6G networks, such as traffic engineering, resource orchestration, and applicationaware networking.
Data Availability
The synthesized and real evaluation data used to support the findings of this study have not been made available because it is proprietary due to nondisclosure agreements with research partners.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This research is supported in part by NSFC grants #61976064 and #5207050731.