Abstract
For embracing the ubiquitous Internet-of-Things (IoT) devices, edge computing and Network Function Virtualization (NFV) have been enabled in branch offices and homes in the form of virtual Customer-Premises Equipment (vCPE). A Service Provider (SP) deploys vCPE instances as Virtual Network Functions (VNFs) on top of generic physical Customer-Premises Equipment (pCPE) to ease administration. Upon a usage surge of IoT devices at a certain part of the network, vCPU, memory, and other resource limitations of a single pCPE node make it difficult to add new services handling the high demand. In this paper, we present IoT-B&B, a novel architecture featuring resource sharing of pCPE nodes. When a pCPE node has sharable resources available, the SP will utilize its free resources as a “bed-and-breakfast” place to deploy vCPE instances in need. A placement algorithm is also presented to assign vCPE instances to a cost-efficient pCPE node. By keeping vCPE instances at the network edge, their costs of hosting are reduced. Meanwhile, the transmission latencies are maintained at acceptable levels for processing real-time data burst from IoT devices. The traffic load to the remote, centralized cloud can be substantially reduced.
1. Introduction
Customer-premises equipment (CPE) devices, such as routers, switches, residential gateways, and set-top boxes, have been deployed at the subscriber’s premises to originate, route, and terminate communications between the customer premises and the central office [1]. In the wake of cloud computing and Network Function Virtualization (NFV) [2, 3], Service Providers (SPs) leverage virtual Customer-Premises Equipment (vCPE) as Virtual Network Function (VNF) instances on top of generic physical Customer-Premises Equipment (pCPE), in search of rebuilding a dynamic revenue stream [4]. There can be enough resources for pCPE to deploy VNFs locally [5], while pCPE can also coordinate with the cloud if VNF scale-out is needed to accommodate heavier usage.
Taking advantage of centralized cloud services in the core networks has benefits [6] because of scalable and flexible computing capabilities. However, large-number deployments of Internet-of-Things (IoT) devices bring challenges to VNFs running in a centralized cloud, as the network traffic load would be drastically increased by transmitting data between the core and the edge of the network. Such traffic overhead can become unacceptable with excessive data transmission, causing high processing delay or even service outage due to the congestion of the network. Meanwhile, high usage of the cloud networks would jack up the price per usage, resulting in a higher-than-expected operating expense (OPEX).
Recent research has been aware of the explosive growth of devices in the edge of the networks. The concepts of fog computing [7] and edge computing [8] were proposed to move the initial handling of raw data to the edge for IoT devices. Although the fog can mitigate the load of the core network, the power of the Customer Edge (CE), namely, the computational capabilities of CPE, is buried. While a single pCPE node has limited resources and typically serves a designated location, the aggregated computing capabilities of pCPE nodes across the edge of a network can be powerful: pCPE nodes have time-varying resource usage that does not always reach full workloads. For instance, the home gateways typically have significantly lower usage in business hours as their users leave for work, while office gateways become idle after hours. If the spare resources of pCPE can be shared within the network edge, VNFs will be able to roam around the edge. Both SP and users will benefit from the considerable capabilities of the sharable resources. Meanwhile, the VNFs deployed on the pCPE nodes keep most traffic within the edge and reduce the traffic to the core network.
It certainly sounds interesting to utilize the time-varying computational resources. However, CPE resource sharing faces challenges:(i)It is unclear how much SP can benefit from spare resources of pCPE nodes compared to traditional virtualization without resource sharing.(ii)Service availability becomes a concern. A pCPE node’s availability can be jeopardized if it no longer has enough resources to host VNFs. It can also be down due to power outages or user pulling the plugs. The availability of offloaded VNFs must be ensured by enforcing proper redundancy.(iii)The users need to be motivated to consent to contributing their pCPE nodes for resource sharing. They would not do so without incentives or discounts. Incentives returned to users are required in order to benefit both the SP and its end users.
In this paper, we propose an architecture to allow sharing resources of pCPE within the network edge, namely, IoT-B&B. We discuss the scenario that SP deploys VNFs, which are vCPE instances, to both the cloud and the available pCPE nodes participating in the resource sharing program. As Figure 1 shows, when a sharable pCPE node is not actively used by its owner, it will be treated as a “bed-and-breakfast” place for vCPE instances to “stay.” SP will have the permission to utilize free resources through the resource manager to deploy VNFs of other users from the same edge network. The following contributions are made for enabling crowdsourcing at the network edge by utilizing resources of pCPE nodes:(i)We propose an architecture extended from ETSI NFV architecture and interfaces [9] to support resource sharing of pCPE nodes. The pCPE nodes at the network edge are treated as compute hosts which can have VNF instances deployed. They are grouped together and abstracted into the NFV Infrastructure (NFVI) layer. A resource manager is embedded in the NFVI and can leverage placement algorithms to make placement decisions.(ii)A model is presented to evaluate the cost of assigning a VNF instance to a pCPE node and to the remote cloud. Multiple factors are considered to determine the cost, including remaining resources, network transmission delay, and availability requirements.(iii)A placement algorithm called “IoT-B&B Algorithm” is also presented to assign vCPE instances to pCPE nodes with the goal of finding a cost-efficient pCPE node for each VNF.(iv)We implement a system with the IoT-B&B architecture with steps to set up the NFVI and the system’s life cycle. Numerical results are shown to demonstrate the placement algorithm’s effectiveness.
We divide the contents into the following sections. Section 2 formulates the problem. Section 3 proposes the IoT-B&B Algorithm based on the problem formulation. Then, Section 4 is presented, covering the actual implementation of the system, followed by the numerical results in Section 5. The related work is illustrated in Section 6. Section 7 concludes the paper and lists future work items.
2. Problem Formulation
In this section, we formulate the problem by modeling the resource properties and constraints of the network edge. The resource types we discuss are limited to CPU, memory, and network bandwidth. We believe these three types of the resources are most representative for cost estimation and optimization. Adding consideration of more resource types will not necessarily change the optimization model. Then, the properties of the VNF instances are defined and annotated. All notations defined and used are also summarized in the Notations Used in Problem Formulation. Note that the terms “VNF”; “VNF instance”; and “vCPE instance” in this paper are interchangeable.
2.1. Connected pCPE at Network Edge
We discuss one particular network edge, which includes all pCPE nodes under it. A network edge is defined as the networks connecting all pCPE nodes under it. We model a network edge to have a topology such that each pCPE node within it can communicate with another. An example with two network edges can be found in Figure 2. The nodes of pCPE 1A, 1B, and 1C group as one network edge connected to each other via Edge Network 1, while the other one consists of the nodes 2A, 2B, and 2C, connected by Edge Network 2.
Based on the definition above, the pCPE nodes and their links at the network edge can be modeled as a directed graph . Set represents all pCPE nodes at the network edge, while is the set of all connected links from one vCPE node to another. A pCPE node in is denoted by , and there is . Define the total number of pCPE nodes to be . Let be a specific pCPE node in , such that Since all pCPE nodes are connected to each other, is strongly connected. For any data transmitted from one node to another node in , there exists a link , such that The network edge is connected to the core network via a logical link, denoted by . The capacity of is limited due to budget reasons: the network edge has a certain bandwidth quota. The total bandwidth to the cloud must be kept within the quota. In reality, can be a group of links connecting the remote cloud.
2.2. VNF Types and Resource Requirement Profiles (Flavors)
The network functions serving the IoT networks have been encapsulated into various types of VNFs. Each type provides a user with a specific service. When the demand increases for a certain type of VNF to a certain level that it exceeds the maximum capacity of current VNF instances, the VNF scales out by increasing the number of Virtual Machine (VM) instances, so that more requests can be processed at the same time. Depending on the purpose, different types of VNF have different resource requirements. The term “flavor” is used to describe the resource requirements profile of an instance of the VNF, including the number of vCPUs, the amount of memory, the size of the disk. We use to define a VNF instance.
Assign to identify a specific type of network function and to be the number of network function types. A VNF instance with type “” can then be represented by . The CPU, memory, and bandwidth requirements of are then denoted by , , and .
2.3. The User of a VNF Instance
Another property of a VNF instance is the user who owns and uses it. Suppose each pCPE node has one unique owner. This is a valid assumption as the actual device on customer premise is typically not shared and belongs to the entity who pays for the service. For a pCPE node , its owner uses a set of VNF instances to satisfy its needs, regardless of where the instances are deployed. When referring to a specific node , we represent the node’s owner by , where .
Figure 3 illustrates an example of VNF instances grouped by the user . There is 1 instance of and 5 instances of . The only instance of , denoted by , is on , along with . Instances and are on . Instances and are on . The numbers and types of VNF instances grouped by their users are determined by user activities and can change dynamically according to user demands.
Based on the user of VNF instances, the annotation of can be extended from to , where is the user who owns and uses .
2.4. Places to Deploy VNF Instances
Placement decisions are made based on the flavors, which are the resource requirement profiles, of VNF instances and the resource capacities of pCPE nodes. A VNF instance of a certain user , denoted by , can be deployed at either of the locations below.
2.4.1. B&B Deployment
For a VNF instance , any pCPE node within the same network edge can be considered as a candidate place to deploy, also known as B&B deployments. A B&B deployment is performed when is deployed on a pCPE node . For a B&B deployment of , its notation can be extended to , where is the place to deploy . We use the set to define all VNF instances deployed on the pCPE node .
Particularly, when there are enough resources on the pCPE node of , can be deployed on locally. For a local deployment of , its notation becomes , where is the place to deploy .
2.4.2. Cloud Deployment
Besides B&B deployments, the remote cloud can be chosen as an alternative place to deploy VNF instances. For a cloud deployment of , its notation can be extended to , where is the place to deploy , and stands for the remote cloud location. We use the set to define all VNF instances deployed on the cloud.
2.5. Factors to Impact Placement Decisions
The following factors will impact the placement decisions.
2.5.1. pCPE Resource Capacity
Every pCPE node has its own resource capacity to host a limited number of VNF instances. As a compute node, the resources for VNF instances are as follows: virtual CPUs (vCPUs) and memory. We assume that there is a plenty of disk space on each pCPE node for any virtual instances deployed. Therefore, the disk space of the pCPE nodes is not in the scope of discussion. Let denote the number of vCPUs can provide. Let be the total amount of memory for VNF instances from . Figure 4 provides an example of the resource capacity for a pCPE node with 20 vCPUs and 10 GB memory in total. There are currently 2 instances of VNF and 3 instances of deployed on it.
2.5.2. Edge Network Transmission Delay
Comparing with the core network transmission delay to be discussed in the next section, the transmission delay between pCPE nodes at the network edge can be ignored as the bandwidth between edge nodes is considered plenty and the transmission delay is small enough to be ignored in the discussion.
2.5.3. Core Network Transmission Delay
The core network transmission delay of a VNF instance offloaded to the cloud is defined as the time consumed by offloading the VNF instance to the cloud and is denoted by , while is the maximum allowed network delay for a specific network function instance. The actual delay must not exceed this limit, or the requests would eventually overflow the buffer and cause malfunction of the VNF.
There are many factors that may affect the core network transmission delay. As stated in [10], the transmission delay to the cloud can be calculated by For the same message, the less available bandwidth left from the network edge to the cloud, the longer transmission delay will be. During the peak hours, more VNF instances are requested by users concurrently across the network and would congest . The severity of direct oversubscription is reflected by the residue bandwidth of the link from the edge switch to the cloud, denoted by , which is the link’s total bandwidth less the mean bandwidth usage for all remote VNF instances. The smaller residue bandwidth there is, the bigger core network transmission delay we should expect.
Because all network function instances offloaded to a remote cloud share the same link that connects the edge network to the remote cloud and the same cloud environment, we assume the core network transmission delay is the same for all offloaded network function instances to simplify the modeling process. Let be the total bandwidth of and be the transmission delay of . VNF instances take up the bandwidth of to communicate with the pCPE nodes. The latency of the core network is therefore highly correlated to the bandwidth consumption of .
Let denote the set of VNF instances deployed in the remote cloud. can be calculated by
Even if is not congested, the remote cloud environment may be degraded by other sources. For example, overloaded VNFs from other users or applications of a shared cloud could affect other VNFs on the same host because of overcommitting. To better utilize the resources on a host, overcommitting is enabled by default [11]. However, the performance could be jeopardized if some VNFs are taking up most of the resource [12]. We define the delay not directly caused by the network edge as a , where the value of changes according to the load of the cloud environment.
Oversubscription will result in a higher core network delay . For all VNF instances offloaded to the cloud, there must be
The equation above ensures the functionality of all VNFs offloaded to the cloud with the existence of . It draws a limit of how much VNF offloading can be done, since an oversubscribed cloud environment would increase . Based on the calculation of transmission delay, we further model to be inversely proportional to and proportional to with as a constant scoping the maximum core network delay when the bandwidth of is depleted:Combining (5) with (6), we have
When gets higher or becomes lower, the value of would exceed of one or more offloaded VNF instances.
2.6. Cost of Offloading to Edge Network
By enabling the resource sharing of pCPE nodes across the network edge, SP benefits from extended containers hosting the VNFs. The costs of leveraging these resources include the following.
2.6.1. Incentives Returned to End Users
By encouraging the users to participate in the resource sharing program and to consent to share, it is necessary to give incentives to the users based on the amount of resource shared. We denote the unit incentive of CPU, memory, and bandwidth usage for pCPE node as , , and , respectively. One exception is that when the pCPE node is hosting VNF instances used by its own user, the incentives do not apply.
2.6.2. Extra Redundancy
Since the availability of the pCPE nodes is lower than the cloud, more standby VNF instances are needed. We define the redundancy factor to be the mean number of standby VNF instances needed for one VNF instance on B&B nodes.
Based on the two factors above, the cost of offloading a VNF instance to any of the pCPE nodes, denoted by , is calculated as below:
2.7. Cost of Offloading to Cloud
As defined earlier, we use to represent the remote cloud in general to deploy VNF instances, to make it distinct from B&B deployments. Although the resources in the cloud, especially in the public cloud, can be considered infinite [13] due to the elasticity of the amount of resources available, for a specific edge network, there are budgets for resources assigned to it. Therefore, resources in the cloud to an edge network are limited when we model them.
The total amounts of vCPUs, memory, and network bandwidth assigned to the edge network we discuss in the cloud are denoted by , , and , respectively. The cost of offloading to the cloud depends on its usage. In general, the less resources left in the cloud for the edge network, the higher unit price our model gives, because the cloud needs to be available as an alternative place to host vCPE instances. Allowing the cloud resources to be drained too early will jeopardize the flexibility of placement and do harm to the service availability.
Let stand for the unit cost for consuming the cloud’s CPU resource. We model to be inversely proportional to the cloud’s remaining vCPUs with the constant of proportionality . The remaining number of vCPUs is denoted by . The total cost of vCPUs for a VNF instance to be offloaded to the cloud, denoted by , is thenIn (9), is a small positive number to avoid dividing by zero.
Also, let represent the unit cost of the memory resource in the cloud. Like vCPUs, is modeled to be inversely proportional to the cloud’s residue memory with the constant of proportionality . The residue memory resource of the cloud is denoted by . Similar to the induction of (9), we have the total cost of cloud memory for a VNF instance denoted by , where is a small positive number to avoid dividing by zero:
Let denote the unit cost of the remote cloud’s network bandwidth. The variable is defined to be proportional to the core network delay with the constant of proportionality . We define the total cost of bandwidth used between the VNF instance and the cloud as . As defined previously, is a constant representing the maximum core network delay when the bandwidth of is depleted. Then we have
From (9), (10), and (11), the cost of offloading a VNF instance to the cloud, denoted by , is then calculated as
2.8. Objective and 0-1 Integer Programming Formulation
SP would like to reduce the total cost of deploying and running VNF instances for all users across the network edge. The VNF instances can be deployed either to the remote cloud location or to the pCPE location . Based on where the VNF instances are offloaded, we identify two portions of costs offloading the VNF instances: to the cloud and to B&B nodes. The objective of the optimization is to minimize the total offloading cost of the SP.
Variables(i): a group of Boolean variables representing if each VNF instance is deployed on the B&B node .(ii): a group of Boolean variables representing if each VNF instance is deployed on the remote cloud
Objective
Constraints
Remarks(i)Function (14) is the objective function. It minimizes the total cost of offloading VNFs instances to the cloud and to B&B nodes.(ii)Constraint (15) ensures that every VNF instance is only deployed at one place.(iii)Constraints (16) and (17) are the capacity bounds of the CPU and memory of the cloud. Each type of the three resources leveraged by all VNF instances offloaded to the cloud must not exceed the cloud’s allocated resource capacity for the network edge.(iv)Constraint (18) sets the bottom line of the residue bandwidth for between the network edge and the cloud, which is essentially setting a limit for the number of VNFs offloaded to the cloud.(v)Constraints (19) and (20) are the capacity bounds for CPU and memory of every pCPE node. Each type of the resources used by all VNFs offloaded to the vCPEs must not exceed these bounds.
3. IoT-B&B Heuristic Placement Algorithm
From the 0-1 integer programming in the previous section, we design a heuristic algorithm to achieve a lower cost by choosing the first valid candidate place to deploy new VNF instances, after the candidate places are sorted by the remaining resources.
3.1. Preliminary Resource Check
For every request of deploying a new VNF instance, we first use Algorithm 1 to check the placement eligibility of every pCPE node, as well as the remote cloud. If the place does not meet the resource constraints of the instance, it will be excluded from the list of candidate places. By calling the function GETCANDIDATES, a list of candidate places will be returned from the input of a specific VNF instance and the current resource level. The list will be sorted by considering the lowest percentage of remaining resource type, in a descending order. For example, if a pCPE node has 90% of vCPU left but only 20% of memory left, then the remaining memory will be used for sorting.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
3.2. Cost Estimation
With the list of eligible candidate places for a VNF instance , we can further estimate the cost of deployed at each place. Algorithms 2 and 3 provide implementation of the cost model from Section 2. Algorithm 3 defines the function to choose the place for VNF instance at the lowest cost, namely, CHOOSEPLACE. The function first calls GETCANDIDATES in Algorithm 1 to get the places eligible for deploying . Then, for each eligible place, the cost is checked based on the type of the place based on Algorithm 2. If the place is the cloud, CLOUDCOST is invoked for cost; if the place is a pCPE node, the function BNBCOST is called instead. After iterating all eligible places, the place with the lowest cost is selected and returned.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||
3.3. Time Complexity
Algorithm 1 has the time complexity of because of sorting the pCPE nodes by remaining resources (assuming merge sort is used). Algorithm 2 has time complexity of . Algorithm 3 will always compare the first candidate pCPE node with the cloud and choose the destination with the lower cost, which has the time complexity of . Combining the three algorithms, the time complexity of IoT-B&B Algorithm is .
If the exhaustive algorithm is used, which does not sort the candidate pCPE nodes, it would have to check all candidates and find out the one with the lowest cost. Such algorithm would increase the time complexity to . Figure 5 shows the time consumed using the two different algorithms (Algorithms 2 and 3) with up to 50 pCPE nodes. From the results, we can see that VNF-B&B algorithm scales well compared to the exhaustive algorithm, where the time consumed is less than 1000 ms for 50 nodes, while the exhaustive algorithm takes more than 9000 ms.
3.4. Access to IoT-B&B Algorithms
We have implemented the Java version of the IoT-B&B Algorithm library. The library source code is published under the MIT License and is downloadable from the following URL: https://github.com/zhuheec/iot-bnb.
4. System Implementation
We have implemented a system following the architecture illustrated in the previous section. The system provides a platform to practice and evaluate the IoT-B&B algorithm.
4.1. Hardware Configuration of pCPE Nodes
For flexibility and scalability, we use VMs instead of bare metal machines as pCPE nodes. Up to 99 VMs are deployed, and each acts as a pCPE node with 8 Cores of CPU, 16 GB of memory, and 40 GB of disk space. Every pCPE node can communicate with any other one via a private virtual network, to mimic that these pCPE nodes are at the same network edge.
4.2. NFVI Setup
Each pCPE node has CentOS 7 [14] installed as its operating system. It has its essential functionalities running as CPE. We use the OpenStack Kolla Project [15] to deploy OpenStack services across multiple pCPE nodes as well as PE, such that(a)the OpenStack services on a pCPE node runs as Docker containers. They can be spun up and torn down with minimal overhead;(b)with container-based OpenStack services, a pCPE node can be converted into an OpenStack compute node and then register to the controller to be one of the available hosts;(c)when the pCPE is no longer available to be a compute node due to high usage, the OpenStack services on it can be stopped to free up resources.
Figure 6 reflects the architecture we use to provision IoT-B&B service upon container-based OpenStack.
4.3. IoT-B&B Algorithm as Filter Scheduler
To leverage the proposed IoT-B&B algorithm in the system, we implement it as a filter scheduler used by nova service, with the name VnfBnbFilter. The IoT-B&B algorithm matches the filtering-weighting mechanism of the filter scheduler.
Figure 7 shows how IoT-B&B algorithm works as a filter scheduler to rank places to deploy. Suppose there are four places to choose: , , , and . Algorithm 1 is first invoked to filter out ineligible places that do not have enough resources. In the example, is filtered out as a result of resource shortage. Then, Algorithms 2 and 3 are used to rank the places according to the cost to deploy the VNF instance. They determine that has the lowest cost to deploy the instance.
4.4. System Life Cycle
We define a set of system states and events for IoT-B&B. The system transitions its state based on the events according to the actual demand, in order to adjust the scaling of the VNF. As Figure 8 shows, the states and events are listed as follows.
4.4.1. Up State
The state indicates the system is up and running as expected. System load level is acceptable to satisfy the needs. The system is expected to stay in this state if it runs properly.
4.4.2. Load Check Event
This is the event to update the system load level. If the updated load level triggers a change, the system may enter Overloaded or Underloaded state or remain in Up state, depending on the threshold to determine them.
4.4.3. Overloaded State
The state indicates the system is overloaded by a higher volume of requests. A scale-out is pending. The placement scheduler will be invoked to determine the place to scale out: B&B or the cloud and then triggers the actual event to scale out.
4.4.4. Scale-out to B&B Event
This is the event to trigger a new VM to be deployed on a B&B, that is, a CPE deployment.
4.4.5. Scale-out to Cloud Event
This is the event to trigger a new VM to be scaled out in the remote cloud environment.
4.4.6. Underloaded State
The state indicates the system is underloaded because of a lower volume of requests. A scale-in is pending. If the number of the VMs has already reached the minimum number required, then the system would not enter this state.
4.4.7. Scale-in Event
This is the event to trigger an existing VM to be scaled in from either the B&B or the remote cloud.
4.4.8. Migration Event
This is the event that is triggered by pCPE availability changes. When a pCPE is no longer capable of hosting a VM because of higher usage from its user, it will cease to be a B&B and be removed from the list of available hosts. Meanwhile, the migration event will be added to the system to move the existing VMs deployed.
4.5. Typical Use Cases
With the definition of the system life cycle, we describe typical uses cases leveraging the IoT-B&B algorithm.
4.5.1. Launch of New Application
The dynamic nature of IoT-based services allows the user to launch new applications which are processed by new VNF instances. A new VNF instance does not automatically get deployed on-site, that is, the pCPE node of its user. The reason can range from lack of enough resources to higher cost being deployed on-site. The IoT-B&B algorithm will be called to determine the place to deploy the new VNF instances.
4.5.2. Scaling out due to Higher Load
When the user applications have more significant activities, resulting in a higher load of the existing instances, the system’s periodical load check daemon will detect the load increase. If the load is above the threshold raising flags of performance, extra VNF instances are needed for processing the larger amount of requests. The IoT-B&B algorithm will be called to determine the place to scale out new VNF instances.
4.5.3. Migration for Lower Cost
The VNF instances in the cloud may start with low cost. However, it does not last forever. As more VNF instances are deployed to the remote cloud, fewer resources are available and the unit resource becomes more expensive. At some point, a migration from the cloud to B&B nodes, or vice versa, is reasonable to lower considerable cost.
5. Numerical Results
The numerical results based on simulations are shown in this section. From the numerical results, our goals are to verify the benefits of leveraging B&B nodes, compared to using a centralized cloud alone. Table 1 lists the values of the constants used in the algorithms.
5.1. Host Nodes Setup
We create 99 pCPE nodes with random levels of initial resources. As all pCPE nodes and the cloud are used to deploy VNF instances, in our configuration, there is a total of 100 nodes for deployments. As seen in Figure 9, the nodes are arranged as a 10 × 10 matrix. Each node is represented by a cell, and is given a horizontal and a vertical coordinate from 1 to 10. The last element, which has the coordinates , represents the remote cloud. The colors of the cells reflect the remaining resource levels of the nodes. For the ease of demonstration, the CPU, memory, and bandwidth resources are all broken into 20 levels ranging from 1 to 20. Deploying VNF instances on a pCPE node follows the Law of the Minimum [16], meaning that the capacity of a pCPE node to host instances is determined by its scarcest resource. Therefore, we color the cells according to the resource type of the lowest level of a node. For example, if the remaining CPU level of a node is 20, while the memory level is 3, the cell representing the node will be colored at Level 3. From the initial resource levels, it can be learned that the pCPE nodes have various levels of resources available. Meanwhile, the cloud starts with the maximum level of resources for deployment.
5.2. VNF Resource Requirement Profile (Flavor) Types
We predefine 10 types of VNF resource requirement profiles (flavors), as shown in Table 2, with different requirements of resources and max delays allowed. A VNF instance to be deployed will have a flavor from the 10 predefined ones. Templating VNF flavors is based on the real use cases as users will need VNF instances from limited kinds of images for serving known functionalities.
5.3. Placement Configuration Modes
In order to compare the effectiveness of the IoT-B&B algorithm, we configure the simulated system to keep deploying new VNF instances of a specific flavor with one of the three modes below, until the resources are depleted:(i)Local mode: deploying only on the pCPE node the user owns. The cloud and B&B nodes are not allowed.(ii)Local + cloud mode: deploying locally on the pCPE node the user owns, as well as on the remote cloud.(iii)Local + cloud + B&B mode: local, cloud, and B&B deployments.
5.4. Extended VNF Instance Capacity
Figure 10 shows the numbers of instances of deploying each of the 10 predefined flavors with the 3 modes. From the results of Figure 10, we learn that the numbers of instances deployed for all 10 flavors have dramatically increased. Taking F3 as an example, in Local Mode, only 9 instances are deployed. When using Local + Cloud Mode, the number jumps to 85. For Local + Cloud + B&B Mode, the number skyrockets to 550. Therefore, the most beneficial part of the system is to extend the total capacity of hosting VNF instances. If using the remote cloud alone, the capacity of the cloud for VNF instances is limited by the core network delay, even if other resources are assumed to be unlimited. This bottleneck is greatly relieved by the B&B nodes hosting instances, because the instances on B&B nodes do not put extra traffic to the core network.
5.5. Cost Hike by Cloud Load Increase
We pick the three flavors: F3, F6, and F9, to investigate the trends of cost increase as more VNF instances are deployed in the system. Figures 11 and 12 demonstrate the changes of costs to deploy a new VNF instance on the cloud in two different modes, as the numbers of deployed instances go up.
Using Local + Cloud Mode, the cost is first 0 as the instances are deployed on the local pCPE nodes. As the loads increase, the remote cloud starts to be picked and the cost to deploy an instance increases as the numbers of deployed instances climb. For F6 and F9, the numbers of deployed VNF instances stop at 42 and 21, respectively.
Comparing Figure 12 with Figure 11, in Local + Cloud + B&B Mode, the costs are lower when deploying the same numbers of instances in the system. For instance, when deploying 20 instances with flavor F9, the cost using Local + Cloud Mode is about 140. Meanwhile, when deploying the same number of instances with the same flavor, the cost under Local + Cloud + B&B Mode is only around 50.
The results above have demonstrated the ability of the B&B nodes to redirect the load off the cloud and to reduce the overall cost, even if offering incentives to the users.
5.6. Impact from outside the Network Edge
As discussed in Section 2, when the cloud load from outside the network edge gets higher, that is, the value of is higher, the ability of the cloud hosting VNF instances may be reduced. To verify how much the impact will be, under Local + Cloud + B&B Mode and for the three flavors F3, F6, and F9, we increase the level of by 1 each time and repeat the deployment for 10 times. The numbers of VNF instances deployed are shown in Figure 13. From the results, the capacity of the system is affected by the increase of . However, the impact becomes less significant as the level of increases. With the considerable buffer of the B&B nodes, the impact from is reduced.
5.7. Remaining Resource Levels
In Local + Cloud + B&B Mode, all B&B nodes participate in hosting VNF instances. We examine the resource levels after the system resources are depleted. When all VNF instances deployed are of flavor F1, the resource levels after the maximum number of instances is deployed are displayed in Figure 14.
The dark colors of all cells indicate that the remaining resource levels are low across all pCPE nodes. The cloud resource levels are also low because of the link/delay bottleneck from the network edge to the core network. The results have demonstrated the ability of the IoT-B&B algorithm to extract the resources to deploy more instances following the best-effort basis.
6. Related Work
NFV has been a key role to accelerate usage-based changes and to reduce OPEX by both SPs and vendor. Much of the recent NFV research relies on cloud computing as the underlying infrastructure [17].
By leveraging generic cloud computing Infrastructure-as-a-Service (IaaS) frameworks, such as OpenStack [18] and VMWare [19], research on cloud-based NFV has been done to ensure that VNFs run at optimum levels in the cloud [20]. Soares et al. presented a platform for VNFs called Cloud4NFV [3], which is compliant with the ETSI [9] NFV architectural specification. Two approaches were discussed to virtualize NF: full virtualization moved all control and user plane functional entities to the cloud, while partial virtualization still forwarded user traffic to physical hardware. With the implementation of service provisioning and end-to-end inventory management, vConductor [21] was presented by Shen et al. that enabled users to plan the virtual network services using its data model. The systems and architectures above focus on deploying VNF instances into the generic cloud infrastructure, rather than the edge of the network.
Due to the nature of varying cost of resources in the cloud, cloud-based resource allocation problems have been studied to reduce the cost and to help evenly distribute the workload. We have analyzed vulnerability of mobile apps in [22] to keep sensitive information in local mobile device, while offloading secured computing-intensive modules to the cloud. Xiao et al. [23] presented a system leveraging virtualization to dynamically allocate resources in datacenter and to optimize the number of servers in use. While these solutions did help better use cloud resources, they keep the computing remotely in the cloud and will not move it to the edge of the network.
The concept of fog computing was proposed in [7] and was anticipated to become an essential part of cloud computing with the volume of Internet-of-Things (IoT) growing explosively. Vaquero and Rodero-Merino [24] proposed a comprehensive definition of the fog covering its features and impact, including device ubiquity, challenges on service and fog-based network management, levels of device connectivity, and privacy. Edge clouds were presented as entry points for IoT, which could be parts of the Enhanced Packet Core (EPC). The scenarios of fog computing in several domains were discussed in [25], including Smart Grid, IoT, and SDN, with topics about security, privacy, trust, and service migration. The work above has pointed the research direction of leveraging the edge of the network from high levels. Based on fog computing, crowdsourcing becomes an option as fog nodes can be updated dynamically with the participation of the third party. The security and privacy challenges were illustrated in [26], where a general architecture was presented to model crowdsourcing networks, including crowdsourcing sensing and crowdsourcing computing. The security concerns were captured from the characteristics of the architecture.
Virtualization in edge networks as a form of fog computing, including NFV, have been given a close look. Manzalini et al. visioned potential value chain shifts and business opportunities in [27] by emerging paradigms such as SDN and NFV. The paper pictured a massive number of virtualized network and service functions running at the edge of the network, making the processing power more distributed globally. The service chaining in the cloud-based edge networks was analyzed in [28] by programming actions into OpenFlow switches to achieve dynamic service chaining. A platform called Network Functions At The Edge (NetFATE) was proposed in [29] as a proof of concept (PoC) of an NFV framework at the edge of a telco operator networks. Each CPE node was realized with a generic-purpose computer installed with a hypervisor and virtual switches. This made the CPE node capable of deploying VNFs on itself. The focus of this paper was to prove that deploying VNFs on the edge of the network is feasible. However, the benefits of resource sharing across different CPE nodes are not mentioned.
7. Conclusions
In this paper, we have presented the architecture and the algorithms to share resources of pCPE nodes across the network edge. When a sharable pCPE node has enough resources, SP will utilize its free resources as a bed-and-breakfast place to deploy VNFs of other users from the same network edge for a certain period. The users can get incentives by allowing SP to leverage the free resources.
By applying the VNF-B&B architecture, the capacity of VNF instances for the network edge is greatly increased. The cost of offloading to the centralized cloud is reduced. By keeping the VNFs at the network edge, the delay is reduced for better processing of real-time data burst from IoT devices. Meanwhile, the traffic load to the core network is substantially reduced with the same number of VNF instances deployed.
Making better use of the network edge is an interesting topic and has a massive potential. While the paper ends here, we are continuing to beef up the architecture, including the following:(i)Explore the availability to factor in the service up and down of B&B nodes. This paper has used a constant factor to model the backup VNF instances. The modeling can be improved so that it is closer to the real-world scenario.(ii)Consider more factors impacting the deployment placement besides vCPUs, memory, and network bandwidth. Also, consider detailed factors that can indirectly impact the cost and the core network delay of the remote cloud.
Future work of this paper will consider the items listed above with the aim of obtaining a practical and effective framework of virtualizing and utilizing the network edge.
Notations Used in Problem Formulation
| : | is a pCPE node. is a specific pCPE node by its index, where . is the total number of pCPE nodes in the network edge |
| : | The remote cloud location to deploy |
| : | is a user. Each user owns one pCPE node. is a specific user by the pCPE index, where |
| : | is the link between pCPE nodes and , where , . is the link between the network edge and the core network. is the remaining network bandwidth of |
| : | is a VNF type/flavor. is the total number of VNF types |
| : | is a VNF instance. is an instance of type , |
| : | A VNF instance of type and user , , |
| : | A VNF instance of type , user , and deployed on pCPE node , , |
| : | is the set of all VNF instances deployed on pCPE node . is the set of all VNF instances deployed on the cloud |
| : | The number of vCPUs, the amount of memory, and the network bandwidth capacity that can be provided by the pCPE node , respectively |
| : | The number of vCPUs, the amount of memory, and the network bandwidth required by , respectively |
| : | is the core network delay by offloading to the cloud. is the maximum delay allowed by . is the core network delay not caused by the network edge |
| : | is the maximum core network delay when the bandwidth of is depleted. is a very small positive number used as part of the denominators in (9), (10), and (12) to avoid dividing by 0 |
| : | Cost of deployed on the cloud and the pCPE node , respectively |
| : | They equal 1 if is on and 0 if not. |
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this article.
Acknowledgments
The authors would like to thank Yiwen Wang who kindly reviewed the manuscript of this paper and shared comments.