Abstract
Developing specialized cloud-based and open-source testbeds is a practical approach to investigate network slicing functionalities in the fifth-generation (5G) mobile networks. This paper provides a comprehensive review of most of the existing cost-efficient and small-scale testbeds that partially or fully deploy network slicing. First, we present relevant software packages for the three main functional blocks of the ETSI NFV MANO framework and for emulating the access and core network domains. Second, we define primary and secondary design criteria for deploying network slicing testbeds. These design criteria are later used for comparison between the testbeds. Third, we present the state-of-the-art testbeds, including their design objectives, key technologies, network slicing deployment, and experiments. Next, we evaluate the testbeds according to the defined design criteria and present an in-depth summary table. This assessment concludes with the superiority of some of them over the rest and the most dominant software packages satisfying the ETSI NFV MANO framework. Finally, challenges, potential solutions, and future works of network slicing testbeds are discussed.
1. Introduction
The fifth-generation (5G) and beyond networks are expected to provide various services compared to the 4G and previous generations of networks. The Quality of Service (QoS) requirements can be quite different in terms of low latency (or even extralow latency), bandwidth, reliability, and availability. Remote surgery, autonomous driving, a massive number of sensors communicating with the network, and video streaming with extrahigh quality are just some of the numerous 5G services. The main concern here is that the physical infrastructure resources are limited and valuable, especially when data traffic demands from different operators increase. Therefore, efficient network sharing [1, 2] is considered as a conventional solution. Through network sharing, multiple operators can share infrastructure resources according to their agreed allocation plans. This approach can help an operator to reduce a significant amount of Capital Expenditure (CAPEX) and Operational Expenditure (OPEX).
As an evolution of network sharing, network slicing brings the flexibility and dynamicity of allocating the required and appropriate amount of physical resources to all service types mentioned above over the same physical infrastructure simultaneously. In fact, network slicing leverages the running of multiple logical networks on top of physical infrastructure. Network Functions (NFs) [3] are constructive operational components (physical networking devices) such as routers, firewalls, and load balancers that have specific functionalities in network infrastructure and hold distinct exterior interfaces for establishing communication between each other. An End-to-End (E2E) network slice [4] is a logical separated (isolated) network, created by chaining NFs, which delivers a particular network service according to QoS requirements via the underlying shared infrastructure in the (Radio) Access Network ((R)AN), Transport Network (TN), and Core Network (CN).
Network Function Virtualization (NFV), Software Defined Networking (SDN), and Cloud computing are considered as the three enabling technologies for implementing network slicing in 5G.(i)NFV [3] is a network architecture framework where NFs that traditionally used dedicated vendor-specific hardware, so-called Physical NFs (PNFs), are now implemented in software. There are two leading solutions towards softwarized PNFs: (1) Virtualized NFs (VNFs) deployed on virtual machines and (2) Containerized NFs (CNFs) deployed on containers. These VNFs and CNFs, in turn, are then implemented in data centers or on cloud environments that run on top of general-purpose (vendor-neutral) hardware.(ii)SDN [5] enables programmable and dynamic network configuration by separating the Control Plane (CP) and the Data Plane (DP), where a centralized entity (controller) in the CP configures the forwarding devices in the DP.(iii)Cloud computing [6] deploys remote network resources in shared pools that can be administered over the Internet. Cloud computing is based on two principal orientations: (1) Cloud-based applications that point to relocating legacy applications, which were established on end-users’ devices or on the companies’ IT infrastructure, to cloud-based servers in order to deliver the applications over web browsers, and (2) Cloud-native applications, which refer to those applications that are particularly created and developed to employ the advantages of the cloud environment such as constant development, modularity, Application Programming Interface (API) integration, and scalability.
As mentioned, one of the 5G objectives is to implement ultralow latency services and to serve many devices with different amounts of computing resources. Multi-access Edge Computing (MEC) [7] is an enhancement of cloud computing that reduces the latency in a mobile network by pushing the processing and computing tasks to the edges of the network (such as base stations) to be closer to the devices with a limited amount of resources. This yields in facilitating the operation of delay-sensitive applications in such devices. These enabling technologies bring flexibility, programmability, and efficiency, but at the cost of higher complexity in operating and managing the 5G networks. The necessity for the Management and Network Orchestration (MANO) framework [8], which performs efficient resource management and orchestration between all network elements in the whole architecture, is undeniable.
Figure 1 illustrates a multi-layered architecture of network slice provisioning in 5G.(i)In the first layer, there is a shared infrastructure layer, which includes heterogeneous hardware and software resources (base station, compute, storage, and networking) spanning over the RAN, TN, and CN domains to host multiple NFs in the second layer. In fact, these resources are sliced according to various service requirements and then will be allocated to different service types.(ii)In the second layer, there are various NFs (PNFs, VNFs, and CNFs) with certain capabilities, belonging to different network domains. This layer encapsulates the essential configuration and managing operations of the NFs to provide different service types in the third layer.(iii)In the third layer, according to service specifications, particular PNFs, VNFs, and/or CNFs (from the second layer) are chained in an explicit order with the appropriate amount of resources (from the first layer) to grant a distinctive service instance. The uniqueness of a service instance in this layer has a straightforward association with the business model, which indicates the reason for creating such a service that will be presented via a slice.(iv)In the fourth layer, the launched service instances from the previous layer constitute E2E network slices. Hence, controlling and management policies on each of the network slices can be achieved independently via the NFV MANO framework.(v)The NFV MANO framework is in charge of orchestrating all of the mentioned layers. Basically, the NFV MANO delivers all the monitoring, coordinating, controlling, and managing tasks of the available physical and virtual resources in order to maintain an efficient resource utilization between all types of NFs (PNF, VNF, and CNF) in the whole architecture. This results in producing network services that meet the specific service requirements over distinctive network slices.
Since the introduction of the network slicing concepts and specification by the 3rd Generation Partnership Project (3GPP) [9], network slicing has attracted a lot of attention in the past years. Apart from the theoretical aspects of different ways of achieving the 5G objectives, research communities in academia and industry have followed practical approaches to examine different features of 5G and to evaluate the network performance under various use cases. In this regard, practical research works in the 5G area have developed prototype system implementations of individual parts of the mobile network architecture, which are known as research testbeds. Recently, even more complex network architectures have been deployed on such testbeds that support network slicing. Research testbeds grant the possibility to evaluate and enhance network performance. Besides, while research testbeds keep the cost of network deployment low, their functionalities, with a fair approximation, are comparable to real networks. Such testbeds can usually be implemented on standard PCs or servers with a not very high amount of resources and without the need of purchasing specialized hardware and software. Moreover, the availability of open-source software packages provides opportunities for creating innovative solutions towards 5G [10].
Deploying testbeds with network slicing capabilities is a challenging and error-prone task as it involves development of a network equipped with fundamental enabling technologies and the ability of programming and configuring the physical infrastructure. Depending on the specific service requirements, the physical and virtual components of a network slicing testbed must satisfy performance requests such as the amount of hardware and software resources (CPU, memory), reliability, and failure rates (dependability analysis) [11]. Nevertheless, the complexity of the testbed deployment process sometimes impacts the utilization of open-source solutions and standard PCs.
Although some excellent surveys have been done on different aspects of network slicing such as [4, 12, 13], just a few works focus on network slicing implementations, in particular, [14–17] elaborate collaborative 5G network slicing research projects and the proposed large-scale testbeds as outcomes of these projects. Reference [14] presents a broad study of five main large-scale SDN testbeds by explaining their design purposes, essential technologies, slicing capability, and use cases. Reference [15] investigates the necessity of network slicing for facilitating the implementation of Internet-of-Things (IoT) intelligent applications and smart services. Bonati et al. in [16] describe open source utilities, frameworks, and hardware components that can be used to instantiate softwarized 5G networks. Barakabitze et al. [17] provide a comprehensive review of 5G networks, a tutorial of the 5G network slicing technology enablers including SDN, NFV, MEC, Cloud/Fog computing, network hypervisors, virtual machines, and containers, as well as an overview of collaborative large 5G network slicing implementations. Nonetheless, there is a lack of a comprehensive survey that presents and evaluates small-scale state-of-the-art 5G network slicing implementations. Small-scale network slicing testbeds are important for the research community in several aspects. Small-scale testbeds require a lower deployment budget compared to large-scale testbeds. Besides, small-scale testbeds, with a compact softwarized version of the required entities, are more effortless to deploy and launch than large-scale ones. Further, due to such testbeds’ small scaling, they are more manageable to troubleshoot, and resolving possible issues is faster than large-scale testbeds with multiple involved entities. Eventually, although the number the practical use cases that can be investigated on small-scale testbeds is lower than large-scale testbeds and real networks, small-scale testbeds can afford similar analogous results to large-scale solutions. The aforementioned aspects motivate the work in this paper. We summarize our contributions as follows:(i)We present the software packages and platforms that fit in the ETSI NFV MANO framework functional blocks for emulating RAN, CN domains, and MANO.(ii)We define primary and secondary design criteria for network slicing testbeds.(iii)We provide a detailed study of small-scale state-of-the-art testbeds for deploying network slicing. These testbeds are relatively easy to deploy and usually without requiring a huge financial investment, thus, suitable for university labs.(iv)We further evaluate the testbeds according to the defined primary and secondary design criteria.(v)We highlight the typical challenges while deploying such testbeds, and present possible solutions and directions for future work.
The rest of the paper is organized as follows. Section 2 explains the research methodology for this paper. Section 3, firstly, presents the ETSI NFV MANO framework along with possible open-source software solutions for each specific block in this framework, and secondly, outlines the desired criteria for designing network slicing testbeds in 5G. In Section 4, small-scale and cost-efficient state-of-the-art network slicing testbeds are detailed with their specific features. In Section 5, first, we compare the testbeds presented in the previous section, and then, we explain some of the main challenges while deploying such testbeds. Section 6 concludes the paper. Table 1 presents a list of the acronyms used in this paper.
2. Research Methodology
Network slicing has become a very hot topic both in academia and industry. This trend has resulted in research on various aspects of network slicing in 5G and a fast-growing number of publications. It is evident that only a portion of these publications introduces implementation solutions for network slicing, i.e., network slicing testbeds. In order to review such publications, we followed a research methodology and defined the procedure to search for related publications, the inclusion and exclusion criteria, and finally, the data collection method to extract pertinent publications. Inclusion and exclusion criteria are used to filter out nonrelevant collected papers. There is also an extra step for quality assessment regarding those publications that pass the inclusion criteria in the final systematization.
In the first step, we identified the databases for searching for potential relevant publications such as (1) ACM Digital Library, (2) IEEE Xplore, (3) Springer Link, (4) ScienceDirect, and (5) arXiv. Next, we started our searching process with relevant keywords to narrow down the selection area of the scientific publications into the network slicing field and, in particular, the deployment of network slicing. We employed some keywords such as <5G testbed>, <network slicing testbed>, <network slicing platform>, and < network slicing framework>.
In the second step, we defined the inclusion criteria, for the publications resulted from the first step, as follows:(i)Does the publication present a solution for network slicing deployment?(ii)How is the solution provided? Which software and hardware components are used?(iii)Is the presented testbed cost-efficient in terms of equipment and also human resources needed for the tested deployment?
We also defined the exclusion criteria as:(i)A publication that introduces a large-scale testbed for network slicing, which is not possible to be implemented with a small budget.(ii)A testbed, which is a result of national or international research projects, and those projects have been finished or are no longer active.
In the third step, the publications that meet the inclusion criteria are assessed for their quality. Following questions are applied for quality assessment:(i)Can the presented testbed be used to investigate different typical use cases in the 5G network slicing, or is the solution just an initial implementation of network slicing with limited capacity for providing few realistic scenarios?(ii)Does the publication include comprehensive information for the testbed architecture and deployment? Are there any extra and complementary sources included in the publication, that could help other researchers to deploy a similar testbed or a possible future extension?
In the end, we categorize the testbeds following the primary and secondary criteria defined in Section 3.
3. ETSI NFV MANO Framework and Design Criteria for Network Slicing Testbeds
3.1. ETSI NFV MANO Framework and Different Open-Source Software Solutions
ETSI introduces the NFV MANO architecture [8], which is comprised of three main functional blocks. Figure 2 illustrates these blocks with the reference points that connect them. This figure also summarizes some of the preeminent software solutions for each specific block. We focus on combining these solutions into the presented testbeds in Section 4 instead of explaining each one of these software modules individually.(i)Virtualized Infrastructure Manager (VIM) performs controlling mechanisms for the NFV Infrastructure (NFVI) resources within an infrastructure provider. VIM is also responsible for receiving fault measurement and performance information of NFVI resources. Consequently, VIM can supervise NFVI resources allocation to the available VNFs. OpenStack [18] and OpenVIM [19] (for VNFs) and Kubernetes [20] (for CNFs) are possible solutions for the VIM section.(ii)VNF Manager (VNFM) conducts one or several VNFs and does the lifecycle management of VNFs. VNF lifecycle management involves establishing/configuring, preserving, and terminating VNFs.(iii)NFV Orchestrator (NFVO) implements resource and service orchestration in the network. NFVO is split up into Resource Orchestrator (RO) and Network Service Orchestrator (NSO). First, RO collects the current information regarding possible physical and virtual resources of NFVI through the VIM. Second, NSO applies a complete lifecycle management of multiple network services. In this way, NFVO keeps updating the information about the available VNFs running on top of NFVI. As a result, NFVO can initiate multiple network services. As part of the lifecycle management, NFVO can also terminate a network service whenever no longer a service request is received for that specific service. In several solutions, NFVO and VNFM are integrated into the MANO section. Open Source MANO (OSM) [21], Open Networking Automation Platform (ONAP) [22], OpenBaton [23], Cloudify [24], SONATA [25], and Katana Slice Manager [26] are considered as the leading integrated solutions for the MANO section. Note that OSM can also perform management and orchestration tasks on PNFs.
Regarding VNFs/CNFs, several open-source software solutions can emulate RAN and CN domains:(i)RAN domain is emulated with Software Radio Systems LTE (srsLTE) [27], OpenAirInterface (OAI) [10, 28], or O-RAN in its Bronze release [29, 30](ii)CN domain is realized with OAI, Open5GS (previously known as NextEPC) [31], Open Mobile Evolved Core (OMEC) [32], or free5GC [33]
Then, via chaining these VNFs/CNFs in the RAN and CN by the NFVO, distinguished service instances, so-called network slice subinstances, are formed. Some solutions for the TN domain, such as Open and Disaggregated Transport Network (ODTN) [34], utilize disaggregated optical equipment and open-source software to create a TN slice subinstance. An E2E network slice instance is created by pairing the definite RAN and CN slice subinstances via the TN slice subinstance [35].
3.2. Design Criteria for Network Slicing Testbeds
Multiple features should be taken into consideration when designing a comprehensive testbed of 5G and beyond networks. We identify the key design criteria for creating a 5G testbed that can emulate a real network’s major features and allow us to develop and test new algorithms. They are divided into two groups.
3.2.1. Primary Criteria
These attributes are fundamental for creating a network slicing testbed.(i)Support of the main enabling technologies. The proposed testbed should be based on SDN, NFV, and cloud computing. Therefore, flexibility and dynamicity in the network are granted. SDN and NFV are complementary, hence, combined with cloud computing pave the way for the paradigms Software-as-a-Service (SaaS), Platform-as-a-Service (Paas), and Infrastructure-as-a-Service (IaaS) [3].(ii)MANO equipped with dynamic monitoring capability. The testbed should support management, orchestration, programmability, and dynamic monitoring of different network functions, network services, and network slices. Therefore, the role of the MANO entity is essential that is the result of SDN/NFV utilization in the network architecture [36].(iii)Multi-network domain with partial slicing support. A 5G testbed needs to provide connectivity across all network domains (air interface, (R)AN, TN, and CN) in order to show a practical ability that emulates the main functionalities of the 5G network. Multi-network domain support allows achieving E2E network slicing; however, it is worth noting that network slicing is a capability that can be implemented partially, and testbeds can deploy slicing only in one specific network domain.(iv)Multi-tenancy support. 5G network is expected first to enable the coexistence of multiple tenants that demand the same network functionalities, and second to administrate the cooperation and interaction between them. This capability represents the so-called multi-tenancy environment, which means that a single instance of the software and its supporting infrastructure serves multiple tenants. Multi-tenancy is one of the main aspects of the 5G networks and should be supported in the testbed implementation.
3.2.2. Secondary Criteria
These attributes add extra features to a network slicing testbed apart from those in the primary group. Testbeds with these extra features broaden the research scope in the network slicing field.(i)Multi-radio access technologies support. Different Radio Access Technologies (RATs) such as Long-Term Evolution (LTE), WiFi, and 5G New Radio (5G NR) should be deployed on the same platform [37]. Furthermore, Cloud-RAN (C-RAN), as a cloud computing-based architecture, brings cloudification benefits into the RAN domain. C-RAN consists of a cloud-Baseband Unit (BBU) pool and several Remote Radio Heads (RRHs). Since the 5G-RAN domain integrates the mentioned RATs with the corresponding frequency bands and provides them via the cloud, a solid platform should implement these capabilities.(ii)End-to-End network slicing. The slicing capability should be expanded upon all network domains. An E2E network slice consists of several network slice subnet instances, each belonging to a particular network domain. Therefore, all network slice subnet instances should be provided and chained together to form an E2E network slice.(iii)Cross-location support. One possible solution for experimenting with more realistic scenarios is deploying testbeds located in two geographical areas. In this case, RAN and CN domains are implemented and launched on two geographically separated infrastructures, and a backbone TN interconnects them. The cross-location capability becomes even more essential when evaluating network performance for providing delay-sensitive services in the 5G network. In real-world use cases, the RAN and CN domains are not necessarily located in the same geographical location, and, as mentioned, MEC is the technology answer to expedite the communication between the RAN and CN domains. Hence, cross-location testbeds facilitate measuring service delay and proposing possible solutions for those services that require low latency.(iv)Machine Learning(ML)-enabled. 5G testbeds equipped with ML toolkits enable users to design, verify, and operate machine learning models via a supervised user interface. One possible outcome of using ML techniques in network slicing is to predict wireless channel behavior in the RAN domain. As a result, the available radio resources can be scheduled in an optimized way to maximize the resource usage per end-user or slice in the next transmissions.(v)Open-source. Providing open-source 5G platforms with well-defined interfaces is considered as a huge advantage in deploying 5G testbeds because an open-source testbed can be deployed by other researchers to help foster research and innovation. It helps to reduce the hassle of setting up a working mobile network that on itself is a complicated and error-prone process
These design criteria explained above and outlined in Figure 3 are later used as an assessment for the state-of-the-art testbeds.
4. An Overview of the State-of-the-Art Network Slicing Testbeds
We describe most of the state-of-the-art testbeds designed for implementing network slicing. We exclude large-scale testbeds that are costly in terms of equipment and human resources as some of them are already explained in [16, 17].
4.1. 5G4IoT [38, 39]
This testbed (Figure 4) deploys OAI in containers to virtualize both Evolved Packet Core (EPC) and eNB. For scalability purposes, the testbed has been implemented in several containers. The testbed is created on a cloud infrastructure based on OpenStack, which is located at Oslo Metropolitan University. There are also a cisco switch and a cisco router in this testbed, separated into two VLANs, which establish the connection between EPC and eNB in two ways. The first method uses SDN Calico (https://projectcalico.org) for layer-3 packet exchange, which provides scalability and dynamic security on the cloud infrastructure for layer-3 routing for IoT. The second approach is Layer-2 Tunneling Protocol (L2TP) to encapsulate the traffic for those IoT applications that need a lower security level. The latter approach is granted by Open virtualSwitch (OvS) without using IPSec. In the testbed, OpenStack Heat templates, as an underlay networking policy, are used to integrate OAI EPC in the OpenStack Neutron. These templates manage cloud-based applications in a stack of containers, and various services via network slices can be created. The testbed is evaluated by producing two isolated network slices for eHealth and light Internet on the same infrastructure.
4.2. 5G Test Network (5GTN) [40]
In this testbed (Figure 5), located at Oulu University, the RAN operates on licensed LTE and 5G bands. The CN comprises EPC and IP Multimedia System (IMS). The CN components are implemented on cloud infrastructure, OpenStack and VMWare. The testbed is aimed at serving different use cases; thus, it includes MEC in the edge to provide low latency services. However, there is no RAN slicing, so only CN slicing is currently implemented. The testbed includes multiple CN domains, which result in sharing radio resources for different services. In this case, each base station in the RAN utilizes a single gateway to access a slice. The authors showcased two slices, slice A and slice B, provided in the CN domain. Slice A from the EPC domain (deployed on OpenStack and orchestrated by CloudBand which is the Nokia platform for NFV orchestration) provides enhanced Mobile Broadband (eMBB) services for IoT and content delivery applications, and slice B in the IMS domain (deployed on VMWare and orchestrated by OSM) provides critical communication and Voice over LTE services. By changing the Access Point Name between EPC (deployed on OpenStack) and IMS (deployed on VMWare), User Equipment (UE) switching between the two slices is possible. The testbed has been examined for CPU utilization, throughput, and delay for the two specific slices.
4.3. 5G Tactile Internet Platform [41]
This testbed (Figure 6) follows the Smart End-to-end Massive IoT Interoperability, Connectivity, and Security (SEMIoTICS) [42] architecture to create a 5G platform based on SDN, NFV, and MEC. The principal objective of SEMIoTICS is to build a framework to provide secure and reliable E2E services with submillisecond latency in actuation operations for IoT/Industrial IoT (IIoT) applications. The SEMIoTICS architecture consists of 3 layers: Backend/Cloud, Networking, and Field layers. The Backend layer is a cloud-based OpenStack platform. It creates several VMs and performs their lifecycle management. The services are provided in several containers and managed by OpenStack Tacker. Currently, there are two deployed VNFs: one for smart monitoring and one for actuating. The Networking layer manages the virtual domain on the testbed and creates tenants to chain VNFs by utilizing the SDN controller Neutron. The communication between separated tenants and also with external networks takes place by performing layer 3 routing. The Field layer is responsible for establishing a connection between smart sensors and actuators with the upper layers. This process is done by exchanging messages through IoT/IIoT gateways in the Field layer and virtual SDN switches in the Networking layer. The testbed performance has been assessed for performing E2E slicing and dynamically sharing the available bandwidth between the two VNFs.
4.4. Mosaic5G [43]
Mosaic5G platform (Figure 7) brings flexibility and scalability to service provision. The testbed architecture consists of five software modules along with hardware components: OAI, FlexRAN, LL-MEC, Store, and JOX. OAI emulates both the RAN and the CN domains of LTE networks. FlexRAN [44] is an open-source Software Defined RAN (SD-RAN) entity. FlexRAN delivers one of these two tasks, deployment of controlling mechanisms for several base stations in a centralized way or performing distributed controlling policies. These two actions are done as reconfigurable control functionalities in the RAN domain. LL-MEC separates the control plane and data plane traffic at the edge and the CN domain. In this way, the MEC functionality is achieved. Basically, FlexRAN and LL-MEC perform SDN functionality in the RAN, and in the edge and core domains, respectively. Store includes a set of modules, monitoring, and control applications for developing network applications for a specific use case. JOX plays the role of orchestration in the network to provide several E2E network slices according to NFV MANO platform. Therefore, network slices can be deployed and then optimized based on various service specifications. The Mosaic5G platform has been used for a few use cases such as critical e-Health, V2X communication for intelligent transportation systems, and multi-service management/orchestration for smart cities.
4.5. Orion [45, 46]
The proposed architecture of the Orion testbed (Figure 8) enables dynamic RAN slicing. Orion provides the sharing of RAN resources in addition to applying isolation between slices, and so, operation in one slice cannot degrade the performance of another slice. This is achieved by having an independent control plane in the RAN domain for each slice. As a result, Orion offers the opportunity to deploy different service characteristics in the RAN domain, and it is a concrete step towards realizing RAN-as-a-Service. The testbed consists of two main components: Base Station Hypervisor and virtual control plane. The Base Station Hypervisor performs slice isolation in the RAN domain, while the Hypervisor capability prepares an abstraction layer of available radio resources to the slices in the RAN. In this way, service providers build virtual base stations on top of the Hypervisor in order to create their RAN slices. A separated virtual control plane for each slice interconnects to the Hypervisor to exchange the required signaling messages. This independent deployment enables slice isolation in the RAN domain. Furthermore, the Orion architecture enables a virtual control plane of a slice to connect to multiple base stations via their Hypervisor layer. Several case studies regarding Orion’s performance evaluation have been done, such as testing slice isolation and possibilities for E2E- and multi-service provisions.
4.6. 5G Testbed for Network Slicing Evaluation [47]
This testbed (Figure 9) utilizes OAI for both RAN and CN domains. There are two CNs which share radio resources of a single eNB in the RAN. OAI RAN consists of OAISIM that allows simulation of UE and eNB. OAISIM acts like a real RAN domain and simulates the LTE protocol stack. A UE with a Network Slice Selection Assistance Information (NSSAI) capability has been implemented in the testbed. Deploying two CNs in containers provides CN slice isolation. In both CNs, the Access and Mobility Management Function (AMF), which is one of the entities in 5G architecture, has been integrated with the Mobility Management Entity (MME) of the LTE platform. The eNB in the RAN selects a CN according to the NSSAI information, which is provided via the S1-AP interface between the eNB and each CN. The testbed has been appraised for connection establishment for both normal LTE UEs and UEs with an implemented NSSAI capability. In the case of normal LTE UE, it includes required encoding messages during the attach process to the network. In the case of the modified UEs, NSSAI is implemented as an optional field in them. The related CN decodes this NSSAI via the S1-AP interface provided by the eNB in the attach process to the network.
4.7. POSENS [48, 49]
POSENS platform (Figure 10) provides efficient resource utilization for creating independent and customizable E2E slices. Different NFs in the network layer are chained by MANO to create network slices. Then, the slices for different tenants need to be multiplexed/demultiplexed over shared resources. This procedure requires enabling the capability of multiple CN connections to a single RAN domain and the possibility for a UE to benefit from more than one slice simultaneously. In POSENS, the possibility of implementing RAN slicing is discussed via three options: (1) slice-aware shared RAN (slicing protocol stack down to Radio Resource Control (RRC)), where the whole radio domain is shared but CNs are distinguished by the specific services they provide. and a UE can utilize different slices provided by the CNs; (2) slice-specific radio bearer (slicing protocol stack down to Radio Link Control (RLC)), where only cell-specific functionality is shared; and (3) slice-specific RAN (slicing protocol stack down to Medium Access Control (MAC)), which apart from the air interface, slices of different tenants are isolated in other protocol stack layers. The latter case needs specific synchronization policies among slices to be deployed efficiently. Each option holds its own level of performance and implementation complexity and POSENS implements the first option for RAN slicing in its first release. For CN, POSENS utilizes OAI CN with no modifications. In the case of MANO, the testbed provides per-slice management, and an orchestration mechanism deployed in customized version of OSM. The testbed has been evaluated in terms of independency between slices, throughput, and independent service function chaining.
(a)
(b)
4.8. UPC University Testbed [50]
UPC platform (Figure 11) implements RAN slicing via RESTful API automatically. The testbed applies the slice-aware policy in Radio Resource Management (RRM) functionalities for admission control and scheduling processes. 5G-EmPOWER [51], acting as a SD-RAN, allows RAN slicing management, and it also shares the available radio resources among the created RAN slices according to RRM descriptors. The interconnection between 5G-EmPOWER and eNB in the RAN is provided via an EmPOWER Agent for performing management policies in the data plane. The testbed utilizes OAI or Next EPC for the CN domain. The srsLTE emulates LTE eNB. The EmPOWER Agent, in turn, splits up into two sections: (1) Agent, which includes protocol parser for EmPOWER exchanged messages and manager entities for different message types. The message type is changed depending on the requested message originated from either the 5G-EMPOWER or the Agent, and activation/deactivation of a specific capability on the Agent side; (2) Wrapper, which converts EmPOWER messages to LTE protocol stack information. Several practical scenarios have been carried out for implementing RRM functionalities for admission control, scalability of the network, isolation among the existent slices.
4.9. Mobile-Central Office Rearchitected as Datacenter- (M-CORD-) Based 5G Framework [52, 53]
This work focuses mainly on OAI integration with the M-CORD framework (Figure 12) and different implementation procedures to deploy LTE network on top of M-CORD. The testbed in [53] extends the previous work further by deploying two CN instances connecting to the C-RAN architecture via the TN in order to slice and manage the TN domain. Notably, different phases of a slice lifecycle from provisioning, allocating a slice to a UE, and managing the slice are provided by this framework. Several entities are integrated into M-CORD which emulate a complete network. XOS performs service orchestration while OpenStack provides the infrastructure for deploying the services via chaining VNFs. Open Network Operating System (ONOS) [54] acts as an SDN controller and separates CP and DP functionalities. Available resources are modified and configured/reconfigured via Graphical User Interface (GUI). TN slicing is performed by running slicing policy via ONOS SDN to establish a connection flow between CN and RAN domains. In fact, ONOS inquires the OpenStack via REST API to receive the necessary information regarding the underlying platform to create TN slice between the CN and RAN domains. ONOS performs management mechanisms on TN slices via its Southbound Interface (SBI).
4.10. Dynamic Network Slicing for 5G IoT and eMBB Services [55]
This testbed (Figure 13) demonstrates sharing of the same RAN resources among eMBB and IoT services. A Northbound SDN application is designed in this testbed to create isolated RAN slices for IoT and eMBB devices according to their service requirements. IoT devices connect to the C-RAN via a gateway. The real-time slicing decision in C-RAN is performed by an SDN controller (FlexRAN) that connects via its Northbound Interface (NBI) to a Slicing app entity, which includes IoT and eMBB modules. With the help of the scheduling process conducted by the SDN controller, the slicing app determines the number of allocated radio resources to each specific slice. In the testbed architecture, the LTE scheduling mechanism is operated by the SDN controller, where CP of the MAC layer is administered as a Northbound SDN application on the cloud. An agent module is responsible for connection establishment between the slicing scheduler entity in DP and the SDN controller in CP via the SBI. Other actions, such as admission control decision, duration of allocating radio resources to a slice, are also performed by the SDN controller and the slicing app. The testbed has been evaluated in some scenarios, such as measuring average downlink throughput in IoT and eMBB slices.
4.11. Transformable Resources Slicing Testbed for Deployment of Multiple VIMs [56]
This testbed (Figure 14) concentrates on providing Slice-as-a-Service (SlaaS) considering Data Centers (DCs). In this case, a slice is composed of a combination of DC slices (compute and storage resources) attached by Network slices (networking resources) operating on their own VIMs and Network Infrastructure Managers (NIMs), respectively. This work is considered as a supplement for Clayman’s model [57]. Clayman’s model consists of three layers: (1) orchestration layer, which manages the slice lifecycle, optimizes resource allocation, and coordinates DC slices and Network slices of a particular slice; (2) DC slice and Network slice controllers layer; former creates DC slices and deploys upon request VIM and later creates Network slices between DC slices and deploys upon request NIM; (3) infrastructure layer, which includes all physical resources.
(a)
(b)
As an extension to the Clayman’s model, here, slices are created via so-called transformable resources, which are interpreted as physically isolated (bare metal) or virtually shared resources. The VIMs are responsible for controlling and managing the number of allocated resources to each slice. The testbed utilizes the DC slice controller to deploy VIMs according to general templates for each slice dynamically. As a result, selecting a specific VIM converts to be a choice for a tenant and not a monopolized feature assigned by the network provider anymore, and it can be set based on the service specification. Hence, each tenant can now operate its own VIM. The testbed comprises a server running as the DC slice controller and four nodes with the same hardware configuration and equipped with an Arduino to trigger on/off action for each node and inspect each node’s status. The testbed offers an evaluation scenario to determine the required time (loading, booting, configuration, and service startup times) to establish different infrastructures (VLSP, Kubernetes, and OpenStack).
4.12. Dynamic Slice Allocation Framework (DSAF) [58]
This testbed (Figure 15) is a practical implementation to evaluate the DSAF paradigm. Basically, DSAF is an efficient resource usage model for dynamic and real-time slice (de)allocation in the CN domain, which is based on minimum CPU utilization and finding links with the lowest delay. DSAF considers allocation policies for slice requests. DSAF also brings isolation between chained NFs of a slice. It is composed of five entities: (1) Orchestrator, which manages slice (de)allocation mechanism and all of the framework elements; (2) Optimization module, which monitors the available CPU and link delays while receiving slice requests; (3) database, which maintains slice request information, slice allocation policies, and the available resources; (4) Optimization Agent, which acts as a mediator entity between the Orchestrator and the Optimization module to exchange information regarding slice allocation approaches; and (5) Hypervisor Agent, which interacts with the Orchestrator by presenting slice state information and performs slice (de)allocation. DSAF performance has been compared with First Come First Serve First Available (FCFSFA) method for different number of VNFs of a slice hosted by a Hypervisor. In these scenarios, the total number of slice requests allocated in DSAF is greater or equal than the FCFSFA scheme.
4.13. SliceNet Platform [59]
This testbed (Figure 16) proposes a QoS-aware network slicing for multiple services with distinct QoS requirements. The testbed focuses on studying use cases for providing critical services with various reliability requirements. It introduces a novel SliceNet platform strategy to provide eHealth services via 5G network slicing. SliceNet offers a realistic network slicing with guaranteed QoS requirements by QoS-programmable policies in the data plane. This is done by implementing traffic engineering functions in both hardware and software levels. Moreover, SliceNet presents a plug and play control layer to let users demand customizable network slices in the network. SliceNet suggests E2E network slicing in both single and multi-domain providers. It also contributes to a cognitive network slice management functionality to enhance the QoS requirement for the services granted by the network slices. In addition to that, SliceNet also operates an ML-enabled method for the patient’s examination in critical use cases via real-time video streaming communication from an ambulance to the medical center. This testbed is an extended version of the Mosaic5G platform [43].
4.14. Iquadrat Informatica (IqInf) Testbed [60]
This framework (Figure 17) utilizes OAI for deploying RAN and CN domains, and SDN switches (composed of Open vSwitch and VLAN switch) for building TN. The separation between CP and DP in the RAN domain is achieved by implementing FlexRAN Agent API, which provides a centralized load balancing and handover mechanism while having more than one eNB in this network. The OpenDayLight (ODL) realizes SDN policy in the TN domain. With the help of PHY abstraction mode of oaisim in OAI RAN, emulating practical network scenarios with numerous UEs and eNBs is conceivable. In particular, the OAI Traffic Generator (OTG) delivers network traffic of multiple applications like Voice over IP and Machine Type Communication (MTC) in this testbed. Deploying an orchestration scheme between SDN controllers and within the entire network domains has been considered as a future enhancement for the testbed architecture.
4.15. Slice-Aware Service Assurance (SA) Framework [61]
This framework (Figure 18) examines Service Assurance (SA) in order to satisfy Quality of Experience (QoE) and QoS requirements in the context of network slicing. The framework integrates a novel SA-based architecture to the ETSI MANO platform to assure the services provided by different network slices in a network. Each component in the NFV MANO architecture has a counterpart in the SA-based NFV MANO platform: Slice Assurance, NS Assurance, NFV Assurance, and Infrastructure Assurance. These components operate four actions, including monitoring, analytics, management, and reporting, to guarantee the performance of the corresponding layer. This extended platform supports reporting information from all involved layers in service provisioning in the network slicing context. The platform also facilitates management and orchestration of various NFVs to assure that QoS and QoE requirements are fulfilled. The testbed evaluates the QoE of a service according to multiple service dependability Key Quality Indicators (KQIs). To this end, the testbed implements web content browsing and adaptive video streaming services to appraise infrastructure performance and the variation of KQIs for the service.
(a)
(b)
4.16. Simula Metropolitan Centre Testbed [62, 63]
This testbed (Figure 19) demonstrates the deployment of OAI-EPC as a VNF on a cloud environment (OpenStack), and it presents the LTE CN service instantiation via OSM. In this testbed, according to the defined descriptors at the VNF and network service levels, the internal components of OAI-EPC are firstly cloned from related repositories. Secondly, they are implemented and configured via special configuration files, Juju Charms, on four separate virtual machines (Virtual Deployment Units (VDUs) in descriptors). The goals of this implementation are to produce MEC services to EPC as well as to integrate EPC with the extended eNB software. Finally, the testbed functionality is evaluated for establishing TCP and SCTP connections in three scenarios: downloading from server to UE, uploading from UE to the server, and bidirectional communication between UE and server. In its recent release, Simula testbed implements a mobile network based on OAI-EPC deployed as a VNF using OSM, which is now integrated with C-RAN architecture with functional split capability for BBU processing functions.
4.17. 5GIIK [64, 65]
5GIIK is a cross-location testbed (Figure 20), which deploys OAI-EPC and srsLTE eNB as cloud-based VNFs via OSM on two OpenStack platforms launched at two geographically separated areas. In this testbed, the VNF-onboarding process takes place in three phases of the VNF lifecycle. In the first phase, management policies for establishing the VNFs are performed. In the second phase, configured VNFs grant the requested services. In the third phase, reconfiguration of VNFs and monitoring of their Key Performance Indicators (KPIs) in runtime operation are provided. This testbed performs E2E network slicing via a hierarchical process by defining specific descriptors at the VNF, network service, and network slice levels on CN and RAN domains. The testbed also integrates 5G-EmPOWER for RAN and M-CORD for TN and CN as SDN controllers. This results in supporting multi-tenancy in the RAN and also implementing slicing in the TN domains. It is worth noting that the 5G-EmPOWER assists common ML toolkits to facilitate realization and administration of machine learning models in this testbed. 5GIIK extends its capabilities by introducing Wireguard [66] to its architecture as a Virtual Private Network- (VPN-) based solution for providing slice isolation. In this solution, WireGuard-enabled VNFs operate on the NFVI via actions performed by OSM-NBI and Juju proxy charms. As a result, traffic isolation and security isolation, which are two essential features in network slicing, are granted via the integrated OSM-WireGuard framework.
4.18. Integrated Slice Management with ONAP Framework [67]
This testbed (Figure 21) investigates E2E network slicing lifecycle management (modeling, onboarding, instantiation, and operation) by integrating ONAP service orchestrator with a network slice manager entity. This integration grants a platform for (1) monitoring and collecting KPI reports that belong to the chained VNFs that create an E2E network slice and (2) evaluating the provided logs of information. In this way, multiple slices are inquired to trace whether the Service Level Agreement (SLA) between the service provider and service user is met or not. The testbed performs a use case by creating a private mobile network that affords services with best-effort and broadband QoS types via E2E network slices. Firstly, a slice is modeled as an ONAP-Network-Service composed of three VNFs (CP and DP for the CN domain, and a RAN emulator); besides, some policies for guaranteeing the SLA are defined. Secondly, the slice is deployed according to corresponding templates for each VNF. The testbed then utilizes the defined policies to perform slice management by modifying the allocated cloud resources to the two QoS types. Consequently, a dedicated channel grants a higher priority to broadband service compared to the best-effort service.
4.19. BlueArch [68]
BlueArch platform (Figure 22) includes a customized structure of some open-source tools. The testbed serves in three operational modes: simulation, emulation, and access to a physical network to communicate with other platforms. The testbed architecture comprises two main sections. Section one consists of (1) a Network Attached Storage (NAS) server representing shared storage that presents a private network; (2) a gateway router attaches a wireless access point operating on the same private network, an external OpenStack infrastructure, and the Internet; (3) Raspberry Pi accessories for MEC nodes implementation outcomes in producing IoT infrastructure in order to migrate VNFs. Section two consists of six VMs each encompassing a specific functionality: (1) an open-source PfSence (https://www.pfsense.org/) firewall for conducting regular firewall actions and also for traffic shaping, network monitoring, and load balancing; (2) employing ODL, Ryu, and HP-VAN SDN controllers hosted by a Citrix XEN server for yielding a cross-platform controlling of OvS devices in DP; (3) open MANO and RIFT.io hosted by another XEN server that operate as orchestrators and support VNF-onboarding process for network slicing; (4) an application server works as the SDN application layer hosting open-source operating systems clients, which in turn driving GNS3 UI, a hypervisor, and XEN center; (5) a network emulation server involves two types of Mininet for wired and wireless SDN, and GNS3 Compute for offloaded computation resulting from GNS3 UI; (6) a MySQL-based database server interfacing the testbed with an external platform.
The testbed is evaluated in three use cases: (1) real-time monitoring of resource utilization in disaster recovery by installing ShellMon client on IoT gateways; (2) hosting VNF as a docker container when a MEC node becomes overloaded by taking a self-triggered action to relocate to another MEC node (known as VNF migration); (3) modeling wireless channel and scheduling radio resources in RAN domain employing Matlab and using the testbed to perform SDN functionality.
4.20. MEC-Enabled 5G IoT Platform [69, 70]
This work (Figure 23) is a solid proposal for onboarding and scheduling aspects in VNF lifecycle management, and it presents a programmable and flexible MEC-enabled platform for IoT traffic. In this work, VNFs are categorized into Latency Critical VNFs (LCVNFs) and Latency Tolerant VNFs (LTVNFs). As a result, the applications are also divided into (1) real-time, provided by High Priority LCVNFs (HP LCVNFs), with resources in the MEC, (2) near-real-time, provided by Low Priority LCVNFs (LP LCVNFs), and (3) non-real-time, provided by LTVNFs. The LP LCVNFs and LTVNFs are deployed on the cloud instead of MEC since they do not provide real-time applications. The work improves the joint orchestration capability in the NFVO for the MEC and cloud resources for the mentioned VNF types via two methods: (1) an online placement scheme to deliver the required management tasks at the VNF level according to the data traffic and (2) a latency embedding structure that enables VNF migration and scalability to fulfill service requirements in real time. These two methods are accomplished by introducing (1) an algorithm for VNF Forwarding Graph (VNFFG) in chained VNFs for prioritizing delay-sensitive services and (2) a second algorithm for the real-time allocation of the MEC and cloud resources to the VNFs that takes into account scale-in/out features for diverse service requirements. The testbed is deployed on several physical servers for the functionalities of the core (cloud infrastructure and NFVO) and network edge (MEC) with lower computational resources compared to the core. OpenStack, as the VIM with its telemetry feature, conducts data collection, data monitoring for future resource utilization, and placement policy through its compute schedulers. Furthermore, the OSM provides the NFVO functionality in this testbed. There are some hypervisors located at the core and the edge that afford the computing tasks. The testbed is assessed by some autoscaling, VNF placement, and online VNF scheduling scenarios.
4.21. CAI Testbed [71]
This testbed (Figure 24) offers a cost-efficient virtualized and orchestrated 5G mobile network equipped with containers and distinct fronthaul and backhaul topologies. The testbed mainly concentrates on integrating Artificial Intelligence (AI) using Kubeflow tool [72] to the management tasks in the 5G RAN and TN domains in order to optimize network performance. The testbed, called Connected AI (CAI), with the help of Kubernetes as a container-orchestrator, presents a mobile network composed of OvS devices, Ryu as SDN controller, and the OAI FlexRAN controller. CAI expedites the deployment of various network topologies on the fronthaul and backhaul by creating an emulated TN using Mininet. An AI agent takes various actions in the network by employing the information granted via Ryu and OAI FlexRAN controllers to feed ML models in order to implement several slice configurations. CAI builds a containerized implementation of OAI for C-RAN and Free5GC for the CN using Docker. The CAI testbed is evaluated via two use cases: (1) monitoring the amount of allocated radio resource blocks to different slice requests and (2) VNF placement in a cluster of containers by means of the AI agent.
5. Discussion
5.1. Comparison between Different State-of-the-Art Network Slicing Testbeds
In this section, we compare the testbeds according to the design criteria for network slicing testbeds presented in Section 3. Table 2 summarizes the major characteristics of each testbed. The testbeds in Table 2 can be arranged into two categories.(i)The first category comprises those testbeds that partially achieve some of the primary or secondary attributes of the design criteria for network slicing testbeds. In this regard, the testbeds in [38, 40, 45, 46, 50, 55, 56, 58, 60–62, 68–71] present network slicing in a particular network domain, and they do not realize a complete E2E network slicing. Reference [56] applies network slicing within multiple-VIMs (DCs); however, this implementation is limited to one network domain, and it does not present E2E network slicing, which crosses all network domains (RAN, TN, and CN). Other testbeds such as [47] implements E2E network slicing; however, it does not offer MANO capability, multi-RATs, and multi-tenancy facilities in the architecture. The platform in [41] applies light employment of MANO entity and E2E network slicing in its design.(ii)The second category encompasses the implementations which satisfy all of the primary and the majority of secondary attributes from the design criteria explained in Section 3. The testbeds such as those in [43, 48, 49, 52, 53, 59, 64, 67] deliver E2E network slicing with MANO privilege in their architectures along with multi-tenancy and multi-RAT support. The testbeds in [59, 64] also incorporate ML-enabled capability in their architectures, and the testbed in [64] is open-source.
5.2. Implementation Challenges for Deploying Network Slicing Testbeds
This section presents some of the current challenges for deploying small-scale network slicing testbeds and summarizes proposed solutions that can slightly mitigate these challenges.(i)Monitoring frameworks for testbeds. 5G is expected to provide heterogeneous services with distinct QoS requirements via utilizing network slicing. In this regard, the dynamic monitoring of the launched services is essential. This becomes challenging when recognizing the issues of possible performance degradation of the services. In fact, the multi-layered architecture of the 5G network, as shown in Figure 1, causes such challenges. Intelligently identifying such issues requires analyzing multiple possible sources of the problem via particular frameworks to effectively monitor the deployment and performance of services. To partially address this problem, different types of monitoring capabilities are integrated in some of the elaborated testbeds. The testbeds in which OSM acts as an orchestrator in their architectures, such as [40, 45, 46, 59, 62, 64, 70], usually employ the interaction of the system monitoring module (MON) with a monitoring toolkit such as Prometheus [73] for collection of VNFs’ metrics and then utilize Grafana [74] to visualize the collected data. The testbeds with an ONAP orchestrator, such as [67], focus on SLA monitoring by exploiting Data Collection Analytics & Events (DCAE) and Virtual Event Streaming (VES) components. Reference [43] benefits from a monitoring application in the store component. The architecture in [61] offers monitoring functions in each layer of SA and also implements virtual monitoring agents or virtual probes at each point of presence to actively observe network services.(ii)Cross-location testbeds. Launching testbeds over separate areas impacts the service performance because of delay, jitter, and packet loss. This issue becomes even more challenging when providing delay-sensitive services. Consequently, discovering techniques to enhance service performance in cross-location deployment is exceptionally important. As mentioned in Table 2, the testbeds in [46, 64] deploy a cross-location architecture for C-RAN (RAN and MEC) and CN on two separate cloud-based infrastructures. In these two testbeds, the MANO entity (OSM), with the help of an SDN-assist feature, partially considers this issue by implementing application-aware traffic flow strategies to mitigate the generated latency because of the cross-location architecture, which results in enhancing connection reliability [46, 75].(iii)C-RAN deployment on testbeds. Implementing C-RAN architecture on a testbed using open-source software packages can be challenging since the interaction between BBU and RRHs entails extremely low latency. Some attempts, such as in [43, 46, 63] resolve this problem by deploying the BBU section with a combination of PNF and VNF. They split the protocol stack of BBU into two sections in their solution instead of launching the BBU completely in a cloud-based environment. In particular, the functionality of the PHY layer of the BBU is split into a lower-PHY as PNF (to run on a physical machine along with RRHs) and higher-PHY as VNF (to run on a cloud infrastructure). In this way, the communication between (lower-PHY layer of) BBU and RRHs fulfills the ultralow delay requirement while keeping the benefit of the cloud-based implementation of (higher-PHY layer of) BBU.(iv)Resource management on testbeds with limited infrastructure capacity. Resource management is considered as another possible challenge while deploying testbeds on infrastructures with limited physical and/or virtual resources. Since diverse services demand various amounts of networking, computing, and storage resources, it is essential to identify optimized methods to allocate available resources to service instances. To deal with this issue, testbeds that adopt OpenStack as VIM in their infrastructures, such as references [38–41, 45, 46, 62, 64], can enable Telemetry Data Collection to gather event and data for utilization statistics of the infrastructure resources.(v)Slice isolation on testbeds. The (intra/inter) slice isolation concept is a common concern while implementing network slicing, and it is not limited to research testbeds. It is worth stating that there are some endeavors to tackle the isolation issue. Testbeds, such as those in [43, 45, 46, 55, 59], which utilize FlexRAN in their architectures, present partial slice isolation in the RAN domain. The testbeds in [48, 49] perform isolation in the RAN domain by slicing the protocol stack down to RRC, RLC, and MAC layers. Nevertheless, introducing and realizing efficient and practical techniques to guarantee isolation in network slicing, especially in the RAN domain, is subject of future work. The work presented in [65] is one step towards providing traffic isolation and security isolation in network slicing.
6. Conclusion
Network slicing testbeds with dedicated management and orchestration entities endeavor to outline and emulate trial and real use cases to achieve network slicing. On this basis and according to pioneer technologies, this paper addresses the principal design criteria for creating and deploying experimental environments for network slicing in 5G. After that, the paper explains the most common small-scale state-of-the-art testbeds for network slicing with their characteristics. The presented testbeds are then reviewed and compared via the design criteria, followed by possible challenges while creating such experimental platforms. Although many efforts have been performed to create testbeds for examining and evaluating network performance under various use cases in network slicing, there are still open research questions in this field.
Data Availability
No data were used to support this study.
Conflicts of Interest
The authors declare that there is no conflict of interest regarding the publication of this paper.