Abstract
Virtualization has been an efficient method to fully utilize computing resources such as servers. The way of placing virtual machines (VMs) among a large pool of servers greatly affects the performance of data center networks (DCNs). As network resources have become a main bottleneck of the performance of DCNs, we concentrate on VM placement with TrafficAware Balancing to evenly utilize the links in DCNs. In this paper, we first proposed a Virtual Machine Placement Problem with TrafficAware Balancing (VMPPTB) and then proved it to be NPhard and designed a Longest Processing Time Based Placement algorithm (LPTBP algorithm) to solve it. To take advantage of the communication locality, we proposed LocalityAware Virtual Machine Placement Problem with TrafficAware Balancing (LVMPPTB), which is a multiobjective optimization problem of simultaneously minimizing the maximum number of VM partitions of requests and minimizing the maximum bandwidth occupancy on uplinks of Top of Rack (ToR) switches. We also proved it to be NPhard and designed a heuristic algorithm (LeastLoad First Based Placement algorithm, LLBP algorithm) to solve it. Through extensive simulations, the proposed heuristic algorithm is proven to significantly balance the bandwidth occupancy on uplinks of ToR switches, while keeping the number of VM partitions of each request small enough.
1. Introduction
As virtualization technology [1] becomes the mainstream way to multiplex various physical resources in modern cloud data centers, the effective and efficient placement of virtual machines (VMs) becomes an important issue. Mechanisms such as VMware Capacity Planner [2] and Novell PlateSpin Recon [3] consolidate VMs such that the consumption of CPU, memory, and power are optimized. Owing to the increasing deployment of communicationintensive applications like MapReduce [4], the data center network (DCN) is becoming the bottleneck of applications performance and scalability. Thus, those mechanisms without considering network resources are not feasible in cloud data centers.
Threelayer treelike architecture is prevalently used in modern data centers [5], as shown in Figure 1. This kind of architecture, however, inherently suffers scalability issue due to the fact that links connected to the core layer switches usually transfer more traffic from lower layers. Physical machines (PMs) connected to the same Top of Rack (ToR) switch can communicate at full line speed, and the traffic between PMs connected to different ToR switches has to traverse across links of the core layer, which is often the bottleneck of data center networks (DCNs). In this paper, we placed VMs on PMs by effectively balancing traffic in the core layer of DCNs to minimize the maximum bandwidth occupancy on uplinks of the Top of Rack (ToR) switches.
The issue on scalability of DCNs has attracted great attention from academia recently. Several recent works address this issue by designing new DCN architectures, aiming at maximizing the network bisection bandwidth [6–8] and reducing the overall oversubscription ratio of the DCN. Other papers [9–11] address this issue by optimizing VM placements on PMs with different optimization goals. The bisection bandwidth can be improved; however, the available bandwidth in the core layer cannot be fully utilized. The number of highly utilized links in the core layer never exceeds 25% [12], which means a great number of links in the core layer are underutilized. Our proposed VM placement with TrafficAware Balancing can evenly spread traffic across links of the core layer to utilize the high bisection bandwidth. Since link utilizations in the core/aggregation layers are higher than that in the ToR layer and a significant fraction of the core links appear as hotspots persistently [12, 13], we only consider the traffic across the core layer of DCNs, and the traffic between VMs of the same request under the same ToR switch does not contribute any traffic towards the core layer.
We formally defined our Virtual Machine Placement Problem with Traffic Balancing (VMPPTB) as an optimization problem, which minimizes the maximum bandwidth occupancy on uplinks of each ToR switch. We proved its NPhardness by reducing from the Multiprocessor Scheduling Problem [14] and designed a Longest Processing Time Based Placement algorithm (LPTBP algorithm) to solve it. The LPTBP algorithm provides an optimum solution to the VMPPTB problem; however, the generated placement schema tends to evenly place VMs of each request under every ToR switch. As the fact that VMs of a tenant’s request only communicate with other VMs within the same request (communication locality property), we should reduce the number of VM partitions of each request at the same time (i.e., placing VMs of the same request on as few PMs as possible). We further proposed LocalityAware Virtual Machine Placement Problem with Traffic Balancing (LVMPPTB), which aims to minimize the maximum number of VM partitions of each request and the maximum bandwidth occupancy on uplinks of ToR switches simultaneously. We also proved it to be NPhard by reducing from VMPPTB and designed a LeastLoad First Based Placement algorithm (LLBP algorithm) to solve it.
We summarize our contribution as follows:(i)We formally formulated the Virtual Machine Placement Problem with Traffic Balancing (VMPPTB), proved its computation complexity, and designed a Longest Processing Time Based Placement algorithm (LPTBP algorithm).(ii)To take advantage of the communication locality property, we further proposed the LocalityAware Virtual Machine Placement Problem with Traffic Balancing (LVMPPTB), proved its NPhardness, and designed a LeastLoad First Based Placement algorithm (LLBP algorithm).(iii)We designed a Greedy Based Placement algorithm (GBP algorithm) as the expected baseline, and took the LPTBP algorithm as the optimum solution. We evaluated the performance of LLBP through extensive simulations, compared with GBP algorithm and LPTBP algorithm.
The rest of this paper is organized as follows. We briefly present related work in Section 2. Problem formulation and computation complexity proofs are presented in Section 3. In Section 4, we describe the LPTBP and LLBP algorithms. We evaluate our algorithms in Section 5. Finally, we make a conclusion in Section 6.
2. Related Work
In [9], the authors addressed the network scalability issue by using trafficaware virtual machine (VM) placement. They defined the placement problem as an optimization problem to minimize the communication costs of all the VMs, where communication cost is defined as the hops between each VM pair. They assumed that the traffic matrices between virtual machines are known in advance. The generated placement scheme can reduce the communication distance between VMs with large traffic and reduce aggregated traffic into the higher level of the data center network (DCN) architecture.
In [10], the authors proposed jointly optimizing virtual machine placement and route selecting. They strived to minimize the averaged congestion rate of every link in DCNs. Their placement strategy performs better in topologies with rich connectivity and path diversity. Although the averaged congestion rate is minimized, the traffic in DCNs may not be significantly balanced. The traffic matrix is also assumed to be known in advance.
In [11], the authors presented a MinCut RatioAware VM Placement (MCRVMP) problem. They tried to minimize the maximum ratio of the demand and capacity across all cuts in the network, where the cut is defined as a set of links that partition the hosts into two disjoint connected components, the capacity is the sum capacity of the links, and the demand is the total traffic from either side of the hosts. In this way, each network cut may have spare capacity to absorb unpredicted traffic bursts. This work is only used in medium sized data centers, and traffic rates are assumed to be known in advance.
Knowing the traffic matrix in advance is a very strong assumption. In [15], the authors proposed to adopt the product traffic pattern model to characterize the traffic rates between VMs, where the traffic rate is defined as the product of activity levels of two communicating VMs. Similar to the objective of [9], they tended to place more active VMs into physical machines (PMs) with less communication cost. The proposed product traffic pattern is unrealistic to some degree. The generated placement scheme may result in hotspots, as VMs with higher activity levels are placed in hosts connected to the same switch; the uplinks of that switch may become hotspots.
In [16], the authors formulated the placement problem as an optimization problem to minimize the total cost caused by network traffic and utilization of PMs. When the number of PMs is fixed, the authors proposed three different traffic cost functions to solve the optimization problem. The data center topology, however, is not taken into consideration when designing the traffic cost functions.
3. Problem Formulation
In this section, we define the VM placement problem based on treelike architectures such as fattree [6] and VL2 [7]. Data centers usually leverage threelayer treelike architectures, in which physical machines (PMs) are directly connected with Top of Rack (ToR) switches, ToR switches are connected with aggregation switches, and aggregation switches are further connected with core switches. The treelike topology, however, is often oversubscribed, due to the fact that a higher level link has to carry traffic from several lower level links. As the higher level links are often the bottlenecks and hotspots [12], we explore to balance traffic on links of the core layer of the data center networks (DCNs).
In the VM placement problem, the traffic between VMs connected to the same ToR switch does not transfer across the core switch layer of DCNs. We only need to consider traffic on the uplinks of ToR switches, since the traffic can be dynamically load balanced among the aggregation layer and the core layer using load balancing mechanisms like Valiant Load Balancing [6]. By properly placing VMs of multiple requests, we can better evenly and effectively utilize every link of core layer of the DCNs and thus minimize the maximum bandwidth occupancy on uplinks of ToR switches.
3.1. VMPPTB Problem
We define the VM placement problem in the scenario where the DCN contains ToR switches, represented as a set . For each ToR switch, we can place up to VMs in one PM connected with it. Let be the uplink of the th ToR switch, and let be the accumulated bandwidth occupancy on uplink . Suppose there are requests from different tenants in the cloud data center, and the corresponding numbers of requested VM are . If , the requested VMs could be placed under the same ToR switch, and when , the requested VMs must be partitioned into several PMs. The bandwidth requirement of VM of request is denoted as . If VM was placed under a ToR switch , it would contribute bandwidth occupancy on uplink . We should properly place all the VMs of requests under ToR switches to minimize the maximum bandwidth occupancy on the uplinks of ToR switches with the constraint that each VM should be placed under some ToR switch and the number of VMs placed under one ToR switch should be no more than .
Let be a binary indicator of whether VM of request is placed under ToR . Consider
The descriptions of the symbols used in this paper are summarized in Notations.
We formally define the Virtual Machine Placement Problem with Traffic Balancing (VMPPTB) as follows:
The objective function (2) minimizes the maximum bandwidth occupancy on uplinks of ToR switches. Constraint (3) ensures that each VM of requests is placed under some ToR switch. Constraint (4) guarantees that the number of VMs under every ToR switch is not more than the capacity of one PM. Constraint (5) updates bandwidth occupancy on uplink when we place a VM under ToR switch .
We prove VMPPTB is a NPhard problem.
Theorem 1. For the Virtual Machine Placement Problem with Traffic Balancing (VMPPTB) defined above, finding its optimal solution is NPhard.
Proof. This can be proven by a reduction from the Multiprocessor Scheduling Problem (MSP). Given a set of independent jobs and a number of processors and given that job has length , what is the minimum possible time required to schedule all jobs in on processors such that none overlap? The MSP is known to be a NPhard problem [14].
For example, we make the set as a request of some tenant, job as of request , the length of job as bandwidth requirement of of request , and processors as ToR switches. It turns out to be an instance of VMPPTB, except that the number of VMs placed under a ToR switch is limited to in VMPPTB. However, we can set extremely large such that the constraint holds in any placement schema. In this way, the MSP can be reduced to the VMPPTB. Thus, we prove the VMPPTB is a NPhard problem.
Since we have proven that VMPPTB is NPhard by reducing from MSP, we can solve VMPPTB by approximation algorithms designed to solve MSP. A simple but classical algorithm called Longest Processing Time (LPT) algorithm can achieve an upper bound of [17]. It is feasible to design a Longest Processing Time Based Placement (LPTBP) algorithm to solve VMPPTB. We first sort bandwidth requirements of VMs of all requests in nonincreasing order. Then we place the VM with the maximum bandwidth requirement under the ToR switch, of which the uplink has minimum bandwidth occupancy. We repeat the process until VMs of all requests are placed on PMs.
3.2. LVMPPTB Problem
The LPTBP algorithm can generate an approximate optimal solution to VMPPTB; however, it cannot take the communication locality of VMs of a request into consideration (i.e., VMs of a request only communicate with other VMs within the same request). Thus, if all VMs of a request are placed under as few ToR switches as possible, it could essentially reduce the traffic forward to the core layer and further mitigate hotspots in the core layer.
Let be a subset of , which contains the ToR switches under which the VMs of request are placed. denotes the number of VM partitions of request , which should be minimized to effectively take advantage of the communication locality property. The multiobjective optimization problem named LocalityAware Virtual Machine Placement Problem with Traffic Balancing (LVMPPTB) aims to minimize the maximum bandwidth occupancy on the uplinks of ToR switches and minimize the maximum number of VM partitions of all the requests simultaneously, which is formally defined as follows:
We also prove LVMPPTB is a NPhard problem.
Theorem 2. For the LocailityAware Virtual Machine Placement Problem with Traffic Balancing (LVMPPTB), finding its optimal solution is NPhard.
Proof. Since we have proven that the VMPPTB is a NPhard problem, we can prove LVMPPTB’s NPhardness by reducing it from the VMPPTB. A special instance of LVMPPTB is that there is only one request in the DCN. Therefore, the number of VM partitions of this request equals the number of VMs of this request divided by the maximum number of VMs placed under a ToR switch. It turns out to be the same with the VMPPTB. If we can find an optimal solution to the VMPPTB, we can also find an optimal solution to this special instance of LVMPPTB and vice versa. As the VMPPTB is NPhard, the LVMPPTB’s NPhardness is proven.
4. Algorithms
We have proven that the Virtual Machine Placement Problem with Traffic Balancing (VMPPTB) is a NPhard problem. As the VMPPTB is the reduction from the Multiprocessor Scheduling Problem, we designed a Longest Processing Time Based Placement (LPTBP) algorithm to solve the VMPPTB, as shown in Algorithm 1.

The LPTBP algorithm aims to achieve balanced bandwidth occupancy of uplinks of all ToR switches by placing the VM with maximum bandwidth requirement under the ToR switch with minimum bandwidth occupancy every time. First, are put into a twodimensional array . and are initially . For the VM with maximum bandwidth requirement, the LPTBP algorithm selects the ToR switch with minimum bandwidth occupancy, places the VM under it, and then updates , , and . The LPTBP algorithm repeats the process until no VMs can be placed under any ToR switches and outputs a virtual machine placement schema.
Though the LPTBP algorithm has an approximation ratio [17], the output placement schema does not take advantage of the communication locality property. Hence, we propose a heuristic algorithm named LeastLoad First Based Placement (LLBP) to solve the LocalityAware Virtual Machine Placement Problem with Traffic Balancing (LVMPPTB), as shown in Algorithm 2.

The LLBP algorithm places the requests in the nonincreasing order of their numbers of VMs. For each request, LLBP tries to find a minimum empty ToR switch set that can hold all its VMs. The ToR switch(es) in an empty ToR switch set is(are) fully available and connected to none of VMs. If the empty ToR switch set exists, LLBP places all VMs of the request in the nonincreasing order of bandwidth requirement under the empty ToR switch set in the leastload first way (placing the VM with maximum bandwidth requirement under the ToR switch with minimum bandwidth occupancy). If no such empty ToR switch set exists for a request, LLBP tries to find a minimum ToR switch set to hold its VMs and also place the VMs under the ToR switch set in the leastload first way.
LLBP first places VMs of the request with maximum number of VMs, and an empty ToR switch set exists to meet the communication locality property, and the number of VM partitions would be small. LLBP repeats the process until no such empty ToR switch set exists. At this time, for these requests with smaller numbers of VMs, LLBP tries to find some ToR switch sets to hold their VMs, which are placed under several ToR switches, and the numbers of VM partitions would rise. The worst case is that each VM of a request is placed under one ToR switch, in which the communication locality property is hard to meet. However, as the number of VMs is smaller, the number of VM partitions would be smaller and the overhead of communication between these VMs is smaller. The VMs of all requests are placed in a leastload first way, and the bandwidth occupancy on uplinks of ToR switches can be balanced significantly. Therefore, LLBP achieves a better tradeoff between the number of VM partitions of requests and bandwidth occupancy of uplinks of the ToR switches.
We also design a Greedy Based Placement (GBP) algorithm, as shown in Algorithm 3. The GBP algorithm places VMs of all requests sequentially under ToR switches in the ToR switch order, which means that each VM of a request is placed as close as possible to other VMs of the same request.

We illustrate the three algorithms by an example, as shown in Figure 2. There are 3 requests, , , and , which are placed under 5 ToR switches, , , , , and . A ToR switch can be connected with up to three VMs. The numbers of VMs of 3 requests are 4, 5, and 6, respectively. The sets of bandwidth requirements of VMs of requests are , , and , respectively. The results of the three algorithms are shown in Figures 2(a)–2(c). The output placement schema of LTPBP algorithm achieves the best balance on the bandwidth occupancy of uplinks of ToR switches (775825). However, the numbers of VM partitions of requests are all more than 3. The output placement schema of LLFBP algorithm achieves a tradeoff between bandwidth occupancy (6501100) and locality (the numbers of VM partitions of requests are all less than 2). Although the GBP algorithm guarantees the locality (the numbers of VM partitions are all less than 2), it is the worst in balancing the bandwidth occupancy (3751500), which varies significantly. Let be the ratio of maximum bandwidth occupancy over minimum bandwidth occupancy. As shown in Figure 2, is , is about , and is about . We originally set infinity as a default. If we set a value, the output placement schema might be changed. If in LPTBP, the output VM placement schema would be not changed. When , more ToR switches are needed. If in LLBP, the new output VM placement schema is shown in Figure 2(d), in which the schema swaps a VM of for a VM of . Though has more VM partitions, the bandwidth occupancy of uplinks is more balanced.
5. Evaluation
In this section, we evaluate the performance of the proposed heuristic algorithm (LeastLoad First Based Placement, LLBP algorithm) by extensive simulations with different settings. We implement a Greedy Based Placement (GBP) algorithm and make it as the expected baseline of the algorithm performance. The Longest Processing Time Based Placement (LPTBP) algorithm can be taken as the optimal solution of the algorithm performance. We compare the performance of LLBP heuristic algorithm against the GBP and LPTBP algorithms. We first present the simulation settings and then show the evaluation results and corresponding analyses.
5.1. Simulation Settings
In our simulations, we set the five parameters as follows: the number of ToR switches , the number of VMs that can be placed under each ToR switch , the number of requests , the number of VMs of each request , and bandwidth requirements of each requested VMs . The scale of DCNs is denoted as and . In the simulations, we set , , , and . is drawn from a uniform distribution between and . is drawn from a normal distribution with a mean of and a variance of , where is drawn from a uniform distribution between and and is drawn from a uniform distribution between and . is determined by other parameters. We assume that all available VM slots are occupied by the VMs of requests.
5.2. Simulation Results
We run the GBP algorithm, LPTBP algorithm, and LLBP heuristic algorithm on the same randomly generated dataset under each scale of DCN for 100 rounds. We record the minimum bandwidth occupancy and maximum bandwidth occupancy on the uplinks of all the ToR switches for each round and get the average values shown in Figure 3. It is clear that the performance of the GBP algorithm is the worst, the LPTBP algorithm is the best, and the LLBP algorithm is in the middle. According to the simulation results, is larger than , is nearly , and is about . values are close to the results of example in Section 4.
(a) oR = 1000; Cap = 80
(b) ToR = 1500; Cap = 60
(c) ToR = 2000; Cap = 40
(d) ToR = 3000; Cap = 30
Although the GBP algorithm places as many as possible VMs of the same request under the same ToR switch, fully taking advantage of the communication locality property, the bandwidth occupancy on uplinks of all the ToR switches varies significantly. The LPTBP algorithm can spread the traffic of VMs evenly across all the uplinks of the ToR switches; however, it distributes the VMs of a request under several ToR switches. The proposed LLBP algorithm simultaneously balances the bandwidth occupancy on uplinks of all the ToR switches and takes advantage of the communication locality property. From the simulation results and analyses, we can conclude that LLBP algorithm performs effectively under different scales of DCNs.
6. Further Optimization
6.1. Locality
The classification of the topologies of data center networks (DCNs) falls under switchcentric, servercentric, and other topologies, according to their structural features. Switchcentric topologies consist of treelike [6, 7, 18, 19], flat [20], and optical topologies [21, 22]. Servercentric topologies are divided into topologies designed for mega data centers [23, 24] and modular data centers (MDCs) [25, 26]. Other topologies include unstructured [27, 28] and wireless topologies [29, 30]. In DCNs, the locality can be defined as different levels. For example, in the fattree [6] in Figure 4, we can logically categorize locality into ToR level ( and ), pod level ( and ), and tree level ( and ). Locality is different physically, such as server level, rack level, and row level. We can set different values for various locality levels in optimization.
6.2. Different Optimization Goal
In Section 3, we proposed LocalityAware Virtual Machine Placement Problem with TrafficAware Balancing (LVMPPTB), which is a multiobjective optimization problem of simultaneously minimizing the maximum number of VM partitions of requests and minimizing the maximum bandwidth occupancy on uplinks of ToR switches. We also can optimize one objective while the other objective is fixed. For example, we can restrict the maximum bandwidth on uplink of a ToR switch to a fixed value to optimize the number of VM partitions of requests.
6.3. Online Balancing
VM placement problem occurs in the scenario of running applications from the very beginning, which is addressed by an offline algorithm. Once the VMs of applications are running, various problems could occur at different time, while we should use an online algorithm to perform VM migration for load balancing and fault tolerance. We also can consider the correlations between several VMs to optimize the VM migration [31].
6.4. Fault Tolerance
In LPTBP and LLBP, we assumed that all requests of tenants were running normally. If the event of requests being lost occurs, we need to design a fault tolerant mechanism to retrieve the lost request. Timeout retransmission may be a way of solving the problem.
7. Conclusion
In this paper, we formulated Virtual Machine Placement Problem with Traffic Balancing (VMPPTB) to balance traffic in data centers. We proved its NPhardness and designed a Longest Processing Time Based Placement algorithm. To take advantage of the communication locality property, we proposed LocalityAware Virtual Machine Placement Problem with Traffic Balancing (LVMPPTB) which simultaneously minimizes the maximum number of VM partitions of each request and minimizes the maximum bandwidth occupancy on uplinks of ToRs. We proved its NPhardness and designed a LeastLoad First Based Placement heuristic algorithm. We conducted extensive simulations to evaluate the performance of algorithms.
Notations
:  Set of ToR switches 
:  Maximum number of VMs under a ToR switch 
:  Number of VMs under ToR switch 
:  Uplink of ToR switch 
:  Accumulated bandwidth occupancy on uplink 
:  Maximum bandwidth capacity of uplink 
:  Set of requests from tenants 
:  Set of number of requested VMs of each request 
:  Bandwidth requirement of VM of request 
:  Indicator of whether VM of request is under 
:  Set of ToR switches connected with VMs of request . 
Disclosure
The opinions, findings, conclusions, and recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the funding agencies or the government.
Competing Interests
The authors declare that they have no competing interests.
Acknowledgments
This work has been supported in part by the China 973 Project (2014CB340303), China NSF Projects (nos. 61472252 and 61133006), the Opening Project of Key Lab of Information Network Security of Ministry of Public Security (the Third Research Institute of Ministry of Public Security) (Grant no. C15602), the Opening Project of Baidu (Grant no. 181515P005267), and the Open Project Program of Shanghai Key Laboratory of Data Science (no. 201609060001). The authors also would like to thank Tao Liang for his contributions on the early versions of this paper.