Abstract
The massive deployment of small cell Base Stations (SBSs) empowered with computing capabilities presents one of the most ingenious solutions adopted for 5G cellular networks towards meeting the foreseen data explosion and the ultralow latency demanded by mobile applications. This empowerment of SBSs with Multi-access Edge Computing (MEC) has emerged as a tentative solution to overcome the latency demands and bandwidth consumption required by mobile applications at the network edge. The MEC paradigm offers a limited amount of resources to support computation, thus mandating the use of intelligence mechanisms for resource allocation. The use of green energy for powering the network apparatuses (e.g., Base Stations (BSs), MEC servers) has attracted attention towards minimizing the carbon footprint and network operational costs. However, due to their high intermittency and unpredictability, the adoption of learning methods is a requisite. Towards intelligent edge system management, this paper proposes a Green-based Edge Network Management (GENM) algorithm, which is an online edge system management algorithm for enabling green-based load balancing in BSs and energy savings within the MEC server. The main goal is to minimize the overall energy consumption and guarantee the Quality of Service (QoS) within the network. To achieve this, the GENM algorithm performs dynamic management of BSs, autoscaling and reconfiguration of the computing resources, and on/off switching of the fast tunable laser drivers coupled with location-aware traffic scheduling in the MEC server. The obtained simulation results validate our analysis and demonstrate the superior performance of GENM compared to a benchmark algorithm.
1. Introduction
The foreseen dense deployment of BS empowered with computing capabilities in order to meet the ultralow latency demanded by mobile users raises concerns related to energy consumption. Apart from the fact that BS energy costs account for a large part of the operating expenses of MN operators, there are also increasing concerns regarding their environmental impact in terms of high carbon dioxide (CO2) emissions. In an effort to minimize energy consumption and energy costs in 5G cellular networks within the MEC paradigm, this paper advocates for the integration of EH systems into network apparatuses and the use of container-based virtualization within computing platforms (i.e., MEC servers). The use of green energy mitigates the negative environmental impact of MN and enables cost-saving for mobile operators in terms of lowering operational energy costs. The motivation towards green energy is due to the fact that current trends in battery and solar module costs show a reduction in prices. The benefits of container-based virtualization is the reduction in energy drained in the computing platform due to their lower overheads when compared with VM [1, 2]. For a qualitative comparison of different virtualization techniques, interested readers are referred to [1].
In this paper, a group of EH BS placed in proximity to a EH-MEC server is considered as an edge system. The EH-MEC server manages the BS system, deciding upon the allocation of transmission resources and also handling the computing and communication processes. In general, renewable energy systems are dimensioned to guarantee the autonomous operation of the edge system. Thus, it is desirable that the utilization of green energy be made one of the performance metrics when designing load-balancing strategies [3, 4], instead of the network impact [5, 6]. As a result, the green-based load balancing is a promising technique for optimizing MEC performance since it exploits the spatial diversity of the available green energy to reshape the network load among the BSs [4]. In this case, MN can prioritize the utilization of BS with sufficient green energy to serve more traffic while reducing the traffic loads for those BS consuming on-grid power. For instance, in the MEC server, a trade-off between the green energy utilization and the amount of workload that can be computed locally should be carefully evaluated.
In this regard, it is worth noting that the energy consumption within the virtualized computing platform is due to (i) the active computing resources, i.e., VM or containers [7–9], and (ii) the network communications, communication related to transmission drivers [10, 11], and the intracommunications [12]. In order to alleviate this, this paper assumes that the container-based virtualization be deployed in the MEC server as containers are lightweight, i.e., demand less memory space, have shorter start-up time, and offer software portability. At each time instance, the containers are provisioned based on the forecasted server workloads, a technique referred to as autoscaling. In addition, the transmission drivers used for data transfers within the server are tuned by taking into account the mobile user’s location. The server is also capable of caching the frequently requested contents locally.
In densely deployed BS, the energy drained is due to the always-on design approach [5, 6, 13], yet traffic load varies during the day, e.g., the demand is low during the night. Therefore, in order to intelligently manage the BS towards minimizing the energy consumption, the green-based load balancing technique is employed, i.e., BS sleep modes are enabled in some BS using green energy as a performance metric.
1.1. Paper Contributions
This paper considers an energy cost model that takes into account the computing, caching and communication processes within the MEC server, and transmission-related energy consumption in BS. Here, the GENM algorithm is proposed for enabling green-based traffic load balancing, i.e., the BS are dynamically switched on/off based on their harvested energy, autoscaling, and reconfiguring the computing resources, and the tuning of transmission drivers. This entails using a minimum number of optical drivers for real-time data transfers, over a short-term horizon. In order to solve the energy consumption minimization problem, the GENM algorithm performs online supervisory control, utilizing the learned traffic load and the harvested energy patterns. Then, the output is utilized within a LLC policy [14] to obtain the optimal system control actions that yield the desired energy savings that guarantee the required QoS.
This work is an extension of [11], where energy savings and QoS guarantee were considered only within a virtualized computing platform placed in proximity to a cluster of BS. In [11], the role of the MEC server is to handle the offloaded computational workload only, which means that the energy cost model lacks the consideration of the BS management procedures, caching process, and the use of containers.
The summary of contributions are listed as follows: (i)The use of container-based virtualization is introduced as they are lightweight, i.e., demand less memory space, have shorter start-up time, and offer software portability(ii)The proposed GENM algorithm, which is an online edge management system, makes use of predictive optimization, specifically using the LLC, where green-based load balancing, container provisioning, and the tuning of the transmission drivers is performed based on the learned information(iii)The numerical results, obtained with real-world harvested energy and traffic load traces, show that the proposed optimization strategy is able to efficiently manage the edge network resources in order to minimize the energy drained under the guidance of the intelligent online-based resource manager and the energy-saving procedures
In order to achieve these, the remainder of the paper is organized as follows: Section 2 describes the related work. Section 3 explains the system model. In Section 4, the design and the implementation of the online algorithm are presented. Simulated results are discussed in Section 5. Lastly, the work is concluded in Section 6.
2. Related Work
2.1. Methods for Load Balancing in MN
Load balancing has been studied towards data center management whereby the data center servers employ temporal dependency strategies, i.e., the servers are turned on/off depending on the arrival rates of workloads. This significantly differs from our considered problem as we consider load balancing in SBSs powered by green energy. Towards load balancing, the dynamic BS switching on/off strategies have been used. However, this may have an impact on the network due to the load that is offloaded to the neighboring BS(s). To avoid this, the BS to be switched off is carefully identified within the BS cluster. In [5, 6], the network impact is used to identify the BS to be switched off, one at a time, with no significant network performance degradation. Taking into account daily traffic load variation, strategies for opportunistic utilization of the unexploited third-party small cell (SC) capacity is exploited towards energy savings in [15], in order to enable the switching off of some BSs. Here, an offloading mechanism is introduced, where the operators lease the capacity of a SC network owned by a third party in order to switch off their BS (Macro BS) and maximize their energy efficiency, when the traffic demand is low.
The use of green energy as a performance metric has been explored within the Radio Access Network (RAN) [3, 16]. Along the lines of MN softwarization, a distributed user association scheme that makes use of the SoftRAN concept for traffic load balancing via the RAN Controller (RANC) is proposed in [3]. Here, the user association algorithm runs on the RANC, and the users report their downlink data rates via the associated BS to the RANC, where the traffic loads from individual users and BS are measured. The algorithm enhances the network performance by reducing the average traffic delivery latency in BS as well as to reduce the on-grid power consumption by optimizing the green energy usage. Then, the authors in [16] proposed to optimize the utilization of green energy for cellular networks by optimizing the BS transmission power. The proposed scheme achieves significant on-grid power savings by scheduling the green energy consumption along the time domain for individual BS and balancing the green energy consumption among the BSs.
Along the lines of MEC, the authors in [4] proposed a framework for jointly performing load balancing, admission control, and energy purchase within a network of EH-powered BS with the goal of minimizing the computation delay and data traffic drops (i.e., increasing the locally computed workloads). This work uses green energy as a performance metric. To solve this problem, an online and distributed algorithm is proposed leveraging the Lyapunov optimization with the perturbation technique. Here, the algorithm makes the traffic load decisions without forecasting the future traffic load and harvested energy. In contrast, the work presented in this paper considers the short-term future traffic load and the harvested energy for decision making. Then, in our previous work [6], a supervisory online control algorithm that makes use of clustering and the network impact metric towards load balancing in MN is proposed. Here, the BSs are empowered with computation capabilities (with VM as computing resources), the LSTM neural network is used for forecasting, and the LLC policy handles foresighted optimization. Even though these works perform load balancing, the problem and scenario considered in this paper are different. Here, a MEC server manages the SBSs powered by green energy. Similar to [11], forecasting and foresighted optimization are used for edge system management.
Load balancing that follows the energy routing, i.e., more computational workload is offloaded to where more energy is available, is presented in [17]. To handle spatial uneven computation workloads experienced by the MEC-enabled BSs, the authors proposed a peer offloading scheme. Here, the BSs share their computing resources and energy costs.
2.2. Methods for Energy Saving within Computing Platforms
Green computing over data centers is an emerging paradigm that aims at performing the dynamic energy-saving management of data center infrastructures. Here, procedures for the dynamic on/off switching of servers have been proposed as a way of minimizing energy consumption in computing platforms. A novel postdecision state-based learning algorithm for server provisioning at the network edge is presented in [18]. This work incorporates green energy. At the beginning of the time slot, the servers are consolidated, i.e., the number of turned on physical servers are minimized, using the learned optimal policy for dynamic workload offloading and the autoscaling (or right-sizing). Then, in our previous works [6, 11, 19], VM soft-scaling (i.e., the reduction of computing resources per time instance) is employed towards energy saving in virtualized platforms either energized by only renewable energy or hybrid supplies (solar and power grid). This is achieved by forecasting the traffic load and harvested energy and then employing foresighted optimization to obtain the system control inputs. The work of [20] uses an iterative algorithm to obtain the number of computing resources (VMs) to be provisioned within a node that transmits to clients wireless. Then, the work of [21] considers a vehicular scenario where vehicles connect wireless to fog nodes and then develop an adaptive scheduler, which computes on-the-fly the solutions of both the resource reconfiguration and consolidation problems. For this purpose, the primal-dual algorithm is used.
In computing platforms, computation offloading strategies can be jointly exploited together with delay constraints towards energy savings. The authors in [22] proposed an offloading policy to find the optimal place where to offload and the amount of offloaded task data. In this work, the time taken for processing the offloaded task is reduced, at the same time consuming less energy. Then, in [23], an efficient scheduling for latency-sensitive applications is proposed towards energy and response time minimization. The achieved results show a reduction in delay and network usage and the energy consumption. In addition, the works of [24] jointly optimize the computing and communication resources, taking into account the local task execution delay and transmission delay. To meet the task delay requirements, in [25], the heterogeneous clouds, i.e., edge and remote cloud, are coordinated. Here, different policies are employed in the clouds. In this, the edge cloud handles tasks with loose delay bounds and drops tasks with stringent delay bounds when the traffic load is heavy.
Towards minimizing energy consumption induced by communication activities within a computing node, the idea of tuning transmission drivers, as one of the energy-saving strategies within the MN infrastructure, is first conceived in [10, 26] where a computing node (router) is considered. Here, it is observed that having the least number of data transmission drivers (fast tunable lasers) can yield a significant amount of energy savings. Motivated by the aforementioned works, within the MEC paradigm, the authors in [11] put forward a traffic engineering- and MEC LS-based algorithm that uses a location-aware procedure for provisioning the transmission drivers for data transfer towards target BS. Here, the MEC LS API is employed for retrieving the UE’s location and then passing the information to the authorized applications within the MEC platform, for decision-making.
Overall, these works numerically analyze and test the energy performance of some state-of-the-art schedulers for computing platforms, but do not attempt to optimize it through the dynamic joint scaling of the available communication-plus-computing resources. The joint analysis of the computing-plus-communication energy consumption within the MEC paradigm is still an open research topic.
2.3. Methods for Guaranteeing Quality of Service and Enabling Energy Savings (within the MEC Paradigm)
The mobile operator’s goal is to provide QoS Internet services for large populations of clients, while minimizing the overall computing-plus-communication energy consumption. Hence, a trade-off is required between QoS and energy savings. Future MNs are expected to learn the diverse characteristics of user behavior, as well as renewable energy variations, in order to autonomously determine good system configurations. Towards this goal, online forecasting using ML techniques and the LLC method can yield the desired system behavior when taking into account the environmental inputs, i.e., BS traffic load, server workloads, and energy to be harvested. Next, the mathematical tools that are used in this research work are reviewed, namely, the LLC method [14, 27, 28] and LSTM neural networks [29, 30].
The LLC has been used in [27] to address a resource provision problem within virtualized environments. The optimization problem is posed as a profit maximization problem under uncertainty and the LLC formulation models the cost of control. To address the uncertainty over the workload arrival, the Kalman filter is used. Then, in [31], an online supervisory control scheme based on LLC policies is proposed. Here, after the occurrence of an event, the next control action is determined by estimating the system behavior a few steps into the future using the currently available information as inputs. The control action exploration is performed using a search tree assuming that the controller knows all future possible states of the process over the prediction horizon. An online control framework for resource management in switching hybrid systems is proposed in [14], where the system’s control inputs are finite. The relevant parameters of the operating environment, e.g., workload arrival, are estimated and then used by the system to forecast future behavior over a look-ahead horizon. From this, the controller optimizes the predicted system behavior following the specified QoS through the selection of the system controls. In [6, 19], a LLC-based supervisory algorithm is proposed to obtain the system control actions yielding the desired trade-off between energy consumption and QoS. Here, the traffic load and harvested energy are forecasted and then used as input in the algorithm. The BS are densely deployed in [6], and each BS is empowered with computation capabilities. Then, in [19], a remote site powered by only green energy is considered.
LSTM can be used for multistep time series forecasting as it is able to handle the long-term dependencies due to its inherent capability of storing past information and then recalling it. The forecasting method is utilized in [6, 19] within an LLC-based algorithm to obtain the system control actions yielding the desired trade-off between energy consumption and QoS. The application of the LSTM network is extended to include ITS in [32]. A new ITS edge analytics architecture that makes use of deep learning techniques that run either on the mobile devices or on the intravehicle processors for data analytics is presented. A combination of LSTM networks and deep CNN is adopted, i.e., CNN-LSTM network, for path selection in autonomous vehicles, whereby the CNN is used for feature extraction, and then, the extracted information is fed into LSTM networks for driving path selection. Forecasting server workloads using the LSTM network can be beneficial for dynamic resource scaling and power consumption in cloud computing data centers. In [33], a forecasting model using the LSTM network for predicting future data center workloads is proposed, and then, the results are fed into the resource manager for decision-making, which involves either scaling up or down the computing resources (servers in this case).
3. System Model
In line with ETSI-proposed MEC deployment scenarios discussed in [34], the considered network scenario is illustrated in Figure 1 where the proposed model is cache-enabled, TCP/IP offload capable (i.e., enables partial offloading in the server’s NIC such as checksum computation [35]). The virtualized MEC server is assumed to be hosting containers deployed at an aggregation point, which is in proximity to a cluster of BS from the same MN operator. The BS are interconnected to the MEC server for computation workload offloading. Each network apparatus (BS, MEC server) is mainly powered by renewable energy harvested from wind and solar radiation, and it is equipped with an EB for energy storage. In this case, energy can only be purchased from the grid supply to supplement the renewable energy supplies. The BS coverage areas overlap so that green-based load balancing is possible. The EM is an entity responsible for selecting the appropriate energy source to fulfill the EB and also for monitoring the energy level of the EB. Then, the electromechanical switch (SW) aggregates the energy sources to fulfill the EB level. In the MEC server, there is the presence of a virtualized access control router which acts as an access gateway for admission control, responsible for local and remote routing, and it is locally hosted as an application. Also, the MEC platform is able to track user location via the MEC Location Service Application Programmable Interface (LS API). Lastly, a discrete-time model is considered whereby time is discretized as and each time slot has a fixed duration . The list of symbols that are used in the paper is reported in Table 1.
3.1. Communication Traffic and Energy Consumption
From a networking perspective, the understanding and characterization of energy consumption within the MN can pave the way towards more efficient and user-oriented networking solutions. This can be achieved through the use of historical mobile traffic traces such as Call Detail Records (CDRs) obtained from mobile operators, specifically in the EPC network. Due to the difficulties in obtaining relevant open-source datasets containing computing requests, real MN traffic load traces obtained from the TIM network (availed through the Big Data Challenge [36]) are used to emulate the computational load. In order to understand the daily traffic load patterns, the clustering algorithm X-means [37] has been applied to classify the load profiles into several categories. Here, each BS is assumed to have a related load profile which is picked at random as one of the four clusters in Figure 2. In addition, it is assumed that consists of 80% delay sensitive workloads and the remainder is delay-tolerant workloads. The total aggregate delay-sensitive workload per time instance is .
The virtualized router in the MEC server of Figure 1 determines the amount of workload that can be accepted by the input buffer at slot , and the aggregated (or admitted) workload is denoted by (measured in [Mbits]). is the maximum input buffer size. In addition, it is assumed that the input/output (I/O) queues of the MEC server are loss-free, and they implement the First-In First-Out (FIFO) service discipline, thus , where is the amount of the aggregate computed workload at the output buffer.
The total energy consumption ([J]) for the edge system at time slot is formulated as follows, inspired by [11, 12]: where is the energy consumption term induced by all BS communications and is the energy consumption term induced by the MEC server’s computing, caching, and communication processes.
3.1.1. BS Energy Cost
The overall energy consumption within the coverage area is defined as the sum of all the BS components: where is the BS switching status indicator (1 for active mode and for power saving mode); is a constant value (load independent) representing the operation energy which includes baseband processing, radio frequency power expenditures, etc.; is the load-dependent BS transmission power to the served users that guarantees low latency at the edge. It is obtained by using the transmission model in [38].
3.1.2. MEC Energy Cost
The energy drained due to the computing, caching, and communication processes is defined as: where is the energy drained due to the active containers, w.r.t CPU utilization, and is the energy drained due to containers adapting to new processing rates . The term is the zero speed of the container, e.g., deep sleep or shutdown, and is the maximum available processing rate for container . It is worth noting that actual containers are generally instantiated atop physical computing cores which offer only a finite set of processing speeds. The term is the energy induced by the TCP/IP offload on the NIC, and is the energy drained due to the virtual-links (to-and-from containers) communication cost. Then, is the amount of energy consumed by the active transmission drivers, and is the total energy cost incurred by the content caching process.
In this regard, it is assumed that real-time processing of computation workloads are performed in parallel over the containers interconnected by a rate-adaptive Virtual LAN (VLAN). In addition, it is also assumed that the CPU frequency is fixed at each user and may vary over users. The amount of energy consumed by the CPU is related to the provisioned computing resources, i.e., the CPU share allocated to each container, per time instance , named , index by . Thus, is defined as [19]: where represents the static energy drained by container in the idle state, is the utilization function of container [28], and is the maximum energy that container can consume. The quantity represents the dynamic energy component of container .
The intelligent resource manager implements a suitable frequency-scaling policy in real-time, in order to allow the containers to scale up/down their processing rates at the minimum cost. In this regard, it should be noted that switching from the processing frequency (the processing rate at the time instance) to the next processing frequency entails an energy cost, . This depends on the absolute processing rate gap ; thus, is defined as: where is the per-container reconfiguration cost caused by a unit-size frequency switching. Typically,is limited to a few hundreds of millijoule per square megahertz.
Before proceeding, it is worth noting the following: at the beginning of the time slot, the online algorithm adaptively allocates the available resources and then determine the containers that are demanded,; the size of the workload allocated to the container, denoted by; andfor containerthat will yield the desired or expected processing time,., whereis the maximum per-slot and per-container processing time ([s]). Note that is the amount of computational workload admitted in the MEC server, by the router. The amount of the workload to be admitted per slot shall be decided at the beginning of each time slot depending on the forecasted green energy, grid power to be purchased, and the expected computational workloads . Moreover, virtualization technologies specify the minimum and maximum amount of resources that can be allocated per container [40]; thus, the maximum amount is denoted by . Lastly, the container(s) provisioning and workload allocation is discussed in Section 4.2, Remark 1, and .
By implementing a TCP Offload Engine (TOE) in high-speed computing environments, some TCP/IP processing is offloaded to the network adapter for the purpose of reducing the CPU utilization. To obtain the energy cost incurred, the performance measure for the Broadcom (Fibre) 10 Gbps NIC [35] is considered here as an example of a TCP/IP offload-capable device. Note that is data volume-dependent and it is obtained as: where is the energy drained by the TOE when powered, with all links connected without any data transfer. This provides an opportunity for switching off the network adapter if there is no data transfer, making the energy drained to be zero. For this, is the switching status indicator (1 for active state and 0 for idle state) and is the maximum energy drained, where is a fractional value representing the amount of load computed in the network adapter and is the NIC best throughput performance, hereby obtained as a fixed value measured in [Gbit/].
In order to keep the transmission delays from (to) the scheduler to (from) the connected containers at a minimum value, it is assumed that each container communicates with the resource scheduler through a dedicated reliable link that operates at the transmission rate of [(bit/s)]. Thus, the energy needed for sustaining the two-way link is defined, inspired by [41]: where is the power drained by the communication link and . is the noise spectral power density, is the bandwidth, and is the (nonnegative) gain of the link. In practical application scenarios, the maximum per-slot communication rate within the intra-VLAN is generally limited up to an assigned value . Thus, the following hard constraint must hold: .
In this regard, a two-way per-task execution delay is considered. Here, there is a total of link connection delays, each denoted by , and where is the server’s response time, i.e., the maximum time allowed for processing the total computation load, and it is fixed in advance regardless of the task size allocated to container . Since parallel real-time processing is assumed in this work, the overall communication equates to . Therefore, the hard per-task delay constraint on the computation time is: , where is the maximum tolerable delay, which is fixed in advance.
Edge distributed devices utilize low-level signaling for information sharing. Thus, edge computing systems receive information from mobile devices within the local access network to discover their location. In return, for every client who offloaded their task into the MEC server associated with the radio nodes, i.e., BS, its location and the computation result are known through the LS (which is a service that supports UE location retrieval mechanism, and then passing the information to the authorized applications within the server), thus enabling the location-aware traffic routing and obtaining the number of transmission drivers to be active for data transfers. The term depends on the number of active laser (optical) drivers, named , where is the total number of drivers that are required for transferring in time slot ( is the downlink traffic volume ([bits] of the driver at slot ). is accumulated over a fixed period of time to form a batch at the output buffer. This means that a large number of drivers yield large transmission speed while at the same time resulting in high energy consumption [26]. Therefore, the energy consumption can be minimized by launching an optimal number of drivers for the data transfer.
The energy drained during the data transmission process consists of the following: a constant energy for utilizing each fast tunable driver denoted by , the target transmission rate [bits/s], and . Thus, the energy is inspired by [11]: where the parameter is obtained using the total number of target BS as , where is a controllable factor that determines the delay constraint of optical networks, ([ms]) is the reconfiguration cost for tuning the transceivers, is an integer value representing the total number of target BS at time slot , and is the number of time slots at which the computed workload is accumulated at the output buffer. Thus, the terms and are fixed values, and is equally distributed over the drivers.
The MEC server is able to cache contents from the Internet and store the contents closer to mobile users. The caching process also contributes to the energy consumption in the server. The caching process is restricted to only viral content. For example, when a video becomes viral, users watching it share and talk about that video, which will be then requested by other users after a response time. Taking into account the Internet users’ response time , this epidemic behavior can be modelled by the self-excited Hawkes condition Poisson process described in [42]: , where is the response time function and is the number of potential viewers who will be influenced after , which is the time when the user shared the video. The term is added as a component of the model to capture the views that are not triggered by the epidemic effect. The energy consumption is mainly contributed by content caching and data transmission processes, as such is defined as: where is the power consumption due to transmission and is the power consumption contributed by the caching process.
3.2. Energy Patterns and Storage
The rechargebale energy storage device is characterized by its finite energy storage capacity . At each time instance, the energy level reports are pushed from the BS sites to the MEC server. Thus, the EB level is known, enabling the provisioning of the required computation and communication resources, i.e., the required containers, transmission drivers, and BSs to be active. In this paper, the amount of harvested energy , per site BS site, in time slot is obtained from open-source solar and wind traces from a farm located in Belgium [43] (see Figure 3 above). The data in the dataset matches our time slot duration (30). The dataset is the result of daily environmental records for a place assumed to be free from surrounding obstructions (e.g., buildings, shades).
The harvested energy is obtained by picking a day at random in the dataset and associating it with one site. Here, the wind energy is selected as a source during the solar energy off-peak periods. The available EB level located at the BS site (BS ) or computing platform evolves according to the following dynamics: where is the energy level in the battery at the beginning of the time slot and represents either , the BS energy consumption of the communication site, or , the energy drained at the computing platform, over time slot , see Eqs. (2) and (3). is leakage energy, and is the amount of energy purchased from the power grid. It is worth noting that is updated at the beginning of time slot , whereas , , and are only known at the end of it. Thus, the energy constraint at the computing site must be satisfied for every time slot: .
For decision-making in the GENM application, the received EB level reports are compared with the following thresholds: and , respectively, termed the lower and the upper energy threshold with . corresponds to the desired energy buffer level at the BS site or computing site, and is the lowest EB level that any site should ever reach. If , then BS or the computing site is said to be energy deficient. The suitable energy source at each time slot is selected based on the forecast expectations, i.e., the expected harvested energy . If is enough to reach , no energy purchase is needed. Otherwise, the remaining amount up to , i.e., , is purchased from the electrical grid. Our optimization framework in Section 4.1 makes sure that never falls below and guarantees that is reached at every time slot.
4. Problem Formulation
In this section, the optimization problem is formulated to obtain reduced energy consumption through short-term traffic load and harvested solar energy forecasting along with energy management procedures. The optimization problem is defined in Section 4.1, and the edge system management procedures are presented in Section 4.2.
4.1. Optimization Problem
Our objective is to improve the overall energy savings of the edge system through BS power saving modes (i.e., green-based traffic load balancing), autoscaling of containers, contents caching, and tuning of the transmission drivers, and also to guarantee the QoS within the network. Note that at the end of each time slot, the EB states are updated depending on the harvested energy and the consumed energy, thereby linking per-time slot problems across time.
To achieve our objective, two cost functions are defined, one captures the edge system energy consumption and the other handles the QoS. This is defined as (F1), which weighs the energy consumption due to transmission in the BSs and the computing-plus-communication activities in the MEC server and (F2) a quadratic term, which accounts for the QoS. In this regard, it is worth noting that F1 tends to push the system towards self-sustainability solutions and F2 favors solutions where the delay-sensitive load is entirely admitted in the MEC server by the router application, taking into account the expected energy to be harvested in the computing site. A weight is utilized to balance the two objectives F1 and F2. The corresponding (weighted) cost function is defined as: where . Hence, starting from (i.e., ) as the current time slot and the finite horizon , the following optimization problem is formulated as:
subject to:
A1:
A2:
A3:
A4:
A5:
A6:
A7:
A8:
A9:
where the set of objective variables to be configured at slot in the BS system and MEC server is defined as . The setting handles the transmission and computing-plus-communication activities. Constraint A1 specifies the BS operation status (either power saving or active), and A2 forces the required number of containers, , to be always greater than or equal to a minimum number : the purpose of this is to be always able to handle mission-critical communications. A3 makes sure that the EB level is always above or equal to a preset threshold , to guarantee energy self-sustainability over time. Furthermore, A4 and A5 bound the maximum processing rate and workloads of each running container , with , respectively. Constraint A6 represents a hard-limit on the corresponding per-slot and per-VM processing time. A7 bounds the aggregate communication rate sustainable by the VLAN to , and A8 ensures that the energy consumption at the computing site (due to the admitted computational workload) is bounded by the available energy in the EB. A9 forces the server to process the offloaded tasks within the set value .
From the optimization problem P1, it could be noted that consists of a nonconvex component defined in Eq. (7), while the others are convex and nondecreasing. In this case, Eq. (7) can be converted into a convex function using the GP concept [44], by introducing alternative variables and approximations. In this case, fixed parameters and approximations are introduced, i.e., . In the sequel, the index is dropped to improve readability. Thus, letting and then obtaining in terms of by rearranging the Shannon-Hartley expression and substituting the value of . From the Shannon-Hartley expression, the presence of the log-sum-exp function is observed as it has been proven to be convex in [45] and recall that .
To solve P1 in (12), the LLC principles [28, 31], GP technique [44], and heuristics is used towards obtaining the feasible system control inputs for . Note that (12) can iteratively be solved at any time slot , by just redefining the time horizon as .
4.2. Edge System Management
In this subsection, a traffic load and energy harvesting prediction method and an online management algorithm are proposed to solve the previously stated problem P1.
4.2.1. Traffic Load and Energy Prediction
Given a time slot duration of , the time series prediction is performed, i.e., the estimates of and are obtained by using an LSTM developed in Python using Keras deep learning libraries (Sequential, Dense) where the network has a one-dimensional (D) subsequence of data, single feature, and multistep for an output. The dataset is split as 70% for training and 30% for testing. The efficient Adam implementation of stochastic gradient descent and fit the model for 20 epochs with a batch size of 4 is used. As for the performance measure of the model, the RMSE is used.
4.2.2. Edge System Dynamics
The system state vector at time is denoted by , which contains the number of active BS, ; number of active containers, ; transmission drivers for data transfers, ; and the EB level, . The input vector drives the MEC server behavior (handles the joint switching on/off of BSs, autoscaling and reconfiguration of containers, and the tuning of transmission drivers) at time . In this work, is obtained with CVXOPT toolbox, and is obtained by following the procedure outlined in Remark 1 (M. Andersen and J. Dahl. CVXOPT: Python Software for Convex Programming, 2019. [Online]. Available: https://cvxopt.org/).
The system behavior is described by the discrete-time state-space equation, adopting the LLC principles [14, 28]: where is a behavioral model that captures the relationship between and the next state . Note that this relationship accounts for the amount of energy drained that harvested and that purchased from the electrical grid , which together lead to the next buffer level through Eq. (10). The GENM algorithm finds the best control action vector that yields the desired energy savings within the edge network. Specifically, for each time slot , problem (12) is solved, obtaining control actions for the prediction horizon . The control action that is applied at time is , which is the first one in the retrieved control sequence. This control amounts to setting the number of active BSs, ; number of instantiated containers, (along with their obtained , , values); NIC status to either active or not, ; and the optimal transmission drivers, . The entire process is repeated every time slot when the controller can adjust the behavior given the new state information.
State and are, respectively, measured and applied at the beginning of the time slot , whereas the offered load and the harvested energy are accumulated during the time slot and their value becomes known only by the end of it. This means that, being at the beginning of the time slot , the system state at the next time slot can only be estimated, which is formally written as:
Remark 1 (container provisioning and load distribution). For a fair provisioning of the computing resources, , and the expected workload allocation, , a remark is presented. Firstly, each container can only compute an amount of up to , and to meet the latency requirements, is obtained as , where returns the nearest upper integer. Secondly, to distribute the workload among the containers, a heuristic process splits the computational workload to the first containers, and the remaining workload to the last one.
4.2.3. Edge System Management Framework
In order to perform traffic load balancing using the green energy as a performance metric, a framework is defined that will identify the BS to be dynamically switched off and then steer the traffic load towards those BS with sufficient green energy. To do this, the available operating interval is defined as the ratio of the next time slot available green energy and the expected total power consumption (recall that the BS load is forecasted), per BS site, as
If , the BS site will not have sufficient energy to handle the expected traffic and it becomes a potential BS to be switched off. In the case where , the site energy will be sufficient to handle the expected traffic. The potential BS to be switched off, denoted by BS , will offload its traffic load to a neighboring BS, denoted by BS . For BS to be able to handle the offloaded traffic, the energy must be sufficient; thus, the green-based operating interval is defined as where is the total energy consumption of the BS site when the traffic load from the neighboring BS is combined with the expected load of the BS, and is the next time slot energy. Next, the BS wake-up procedure is discussed.
(1) BS Wake-Up Procedure. To support BS reactivation commands, the UE location fingerprints that are obtained from the LS API are considered. The UE trajectory is assumed to be sequential, i.e., from BS-to-BS along the same direction (while still associated with the MEC server), and this is represented as , where refers to the serving/target BS node in association with user , at time slot . When a BS node is switched off, it goes into a discontinuous reception cycle and configures a timer to awake and listen. Here, the MEC server, as BSs manager, sends wake-up control information as wake-up signaling (the information is a single bit). The wake-up information is only sent during the listening period. A BS can be woken up only if it meets the following conditions: (i) and (ii) a group of UEs that are associated with the MEC server is expected to receive the computed results via BS (their trajectory is towards BS as reported by the LS in the MEC).
4.2.4. Green-Based Edge Network Management (GENM) Algorithm
In order for the algorithm to manage the BS system, deciding upon the allocation of their transmission resources and also handling the computing and communication process, the best control action, , that will yield the expected system behavior shall be obtained.
The edge network management algorithm pseudocode is outlined in Algorithm 1, and it is based on the LLC principles from [14, 28]. Starting from the initial state, the controller constructs, in a breadth-first fashion, a tree comprising all possible future states up to the prediction depth . The algorithm proceeds as follows: a search set consisting of the current system state is initialized (line 01), and it is accumulated as the algorithm traverse through the tree (line 03), accounting for predictions, accumulated workloads at the output buffer, mobile devices trajectory , past outputs and controls, and operating intervals. The set of states reached at every prediction depth is referred to as (line 02). Given , the workload and harvested energy are estimated first, then obtain the operating intervals (line 02), and generate the next set of reachable control actions by applying the accepted workload , energy harvested, and green-based operating interval (line 03). The energy cost function corresponding to each generated state is then computed (line 04), where take into account of as observed from . Once the prediction horizon is explored, a sequence of reachable states yielding minimum energy consumption is obtained (line 05). The control action corresponding to (the first state in this sequence) is provided as input to the system while the rest are discarded (line 06). The process is repeated at the beginning of each time slot .
| ||||||||||||||||||||||||||||||||||||||||||||||
Algorithm 1. GENM algorithm pseudocode. | ||||||||||||||||||||||||||||||||||||||||||||||
(1) Algorithm Complexity. The algorithm is executed at each time instance, and the corresponding time complexity is obtained as follows. The time complexity associated with the computation of the and is linear with the size of the BS group interconnected to the MEC server. Next, the complexity associated with updating the load allocation for the active BSs is , which leads to . In the worst-case scenario (no BS has been switched off), the total complexity is , which is linear in all variables, namely, number of BSs interconnected to the MEC server, number of system states, number of control actions, and time horizon .
5. Performance Evaluation
In this section, some selected numerical results for the scenario of Section 3 are shown. The parameters that were used in the simulations are listed in Table 2.
5.1. Simulation Setup
A virtualized MEC server in proximity to a group of BS is considered. The BS coverage areas overlap to enable load balancing. Our time slot duration is set to 30, and the time horizon is set to time slots. For simulation, Python is used as the programming language.
5.2. Numerical Results
Data preparation: The information from the used mobile and energy traces is aggregated to the set time slot duration. The mobile traces are aggregated from 10 observation time to . As for the wind and solar traces, they were aggregated from 15 observation time to . The used datasets are readily available in a public repository (see [39]).
In Figure 4, the real and predicted values for BS traffic load and harvested energy is shown. Here, the forecasting routing tracks each value and predicts it over one-step. The shown selected prediction results are for Cluster 3, Solar 3, and Wind 3. Then, Table 3 shows the average RMSE of the normalized harvested energy and traffic load processes, for different time horizon values, . In the table, the term represents the forecasted values for energy harvested from wind turbines, and is for the harvested energy from solar panels. From the obtained results, the prediction variations are observed between and when comparing the average RMSE. The measured accuracy is deemed good enough for the proposed optimization.
The GENM algorithm is benchmarked with another one, named Iterative-based Resource Manager with network impact Capability (IRMC), which is inspired by the iterative approach for computing platforms from [20] and the use of the network impact towards load balancing from [5]. Both algorithms make use of the learned information. Figures 5 and 6 show the average energy savings obtained by GENM in the MEC server. In Figure 5(a), the average results for GENM () show energy savings of 59%, while IRMC achieves 34% on average. As expected, the highest energy savings gain is observed in the early hours of the day (1-8) as the aggregated computational workload was at its lowest. In Figure 5(b), the average energy savings obtained by GENM is 68% () and for IRMC is 49%. Again, here, the highest peaks for energy savings are obtained from 1-8. The results are obtained with respect to the case where no energy management procedures are applied, i.e., the MEC server provisions the computing resources for maximum expected computation workload (maximum value of ). Comparing the results of Figures 5(a) and 5(b), we observed that when the BSs being manage by the MEC server are reduced (i.e., ), the aggregated delay-sensitive workload is also reduced and this translates to reduced computation process demands, which in turn results into high energy savings.
(a) Mean energy savings for
(b) Mean energy savings for
Figure 6 shows the average energy savings obtained when green energy is used as a performance metric towards load balancing within a group of BSs. Here, the group size is increased from , using an incremental step size of 5. The obtained energy savings are with respect to the case where all BSs are dimensioned for maximum expected capacity (maximum value of ). From the results, it is observed that the energy savings increase as the BS cluster grows, thanks to the load balancing among active BSs.
Then, Figure 7 shows the average energy savings for the edge system. Here, the BS group size is set to , and the obtained energy saving results are with respect to the case where no energy management procedures are applied, i.e., the BSs are dimensioned for maximum expected capacity (maximum value of , ) and the MEC server provisions the computing resources for the maximum expected computation workload (maximum value of , with ). The average results of GENM (, ) show energy savings of 51%, while IRMC achieves 44% on average. The effectiveness of the joint dynamic BS management, autoscaling and reconfiguration of the computing resources, and on/off switching of the fast tunable laser drivers, coupled with foresighted optimization, is observed in the obtained numerical results.
6. Conclusions
This paper envisioned an edge system where a group of BS is placed in proximity to a MEC server for ease of handling the offloaded computational workload and BS management, and also, the edge apparatuses are powered by hybrid supplies, i.e., green energy is used in order to promote energy self-sustainability within the network and as a performance metric for traffic load balancing. The extra energy can only be purchased from the grid supply to supplement the renewable energy supplies. The considered energy cost model takes into account the computing, caching, and communication processes within the MEC server, and the transmission-related energy consumption in BSs. To intelligently manage the edge system, an online algorithm based on forecasting, control theory, and heuristics is proposed with the goal of minimizing the overall energy consumption and to guarantee the quality of service within the network. The algorithm jointly performs (i) dynamic BS management using green energy as a performance metric; (ii) autoscaling and reconfiguration of the computing resources, workload, and processing rate allocation; and lastly, (iii) switching on/off of fast tunable drivers. Numerical results, obtained with real-world energy and traffic load traces, demonstrate that the proposed algorithm achieves energy savings of above 50% with respect to the allocated maximum per-container loads of 10 MB. The computing platform is able to achieve energy savings from 59% to 68%, depending on the size of the BS cluster. The energy-saving results are obtained with respect to the case where no energy management techniques are applied in the BS system and the MEC server.
Data Availability
In this paper, open-source datasets for the mobile network (MN) traffic load and solar and wind energy have been used. The details are as follows: (1) the real MN traffic load traces used to support the findings of this study were obtained from the Big Data Challenge organized by Telecom Italia Mobile (TIM) and the data repository has been cited in this article. (2) The real solar and wind traces used to support the findings of this study have also been cited in this article.
Conflicts of Interest
The authors declare that they have no conflicts of interest.