Abstract
This paper presents a framework to activate and deactivate micronodes in a heterogeneous multicell LTE network, based on load and energy efficiency consideration. The framework exploits historical data (i.e., per-macro-cell load curves) to select a set of candidate switch-on/switch-off instants of microcells, assuming a limited number of state changes is allowed in a day. The switching instants are instead determined online, by taking into account the actual traffic as well as the load curves. Moreover, intercell interference is fully accounted for. Our simulations show that this framework allows a multicell network to sustain peak-hour load when necessary and to reconfigure to a minimum coverage baseline whenever feasible, thus saving power (up to 25% in our scenarios). Moreover, the framework is robust, meaning that deviations of the actual traffic with respect to the prediction offered by the load curves can easily be handled.
1. Introduction
Computer networks have become a major contributor to electricity consumption all over the world [1], which has fostered a large amount of research on how to make them more energy-efficient. This problem can be targeted from several, concurrent points of view: on the one hand, working at the hardware level to make network equipment more energy efficient; on the other hand, designing algorithms that exploit the inherent redundancy of computer networks and temporarily switch off unused or unnecessary portions of the networks during off-peak hours. This allows networks to be designed for peak, while operating at lower-than-peak power most of the time. As far as cellular networks are concerned, energy efficiency in LTE networks has been the subject of several studies. Leaving aside those that investigate battery saving at the user side (e.g., [2–4]), many works concentrate on the infrastructure side, which is where energy consumption meets OPEX (see, e.g., [5–9] and the references therein and again [10–18]).
LTE power saving schemes exploit the fact that cellular coverage is overlapping, due to both reliability and (mostly) performance design issues and that traffic load is highly variable. Coverage is designed to carry peak-hour loads, which normally occur during working hours in business areas, while nodes use very few resources during (long) off-peak periods, for example, at night. Therefore, some nodes can switch off during off-peak hours, and nearby nodes will increase their transmission power accordingly to guarantee coverage. When the load is high, the first node will be reactivated and the second one will revert to normal radius. A similar concept can be applied to heterogeneous cellular network deployments, where a macro node presides over a relatively large area and micro nodes provide additional localized capacity to cover hotspots within the former. Micronodes can be switched on to offload the macro node (which is supposed to remain always on, lest coverage holes appear) when necessary or profitable from an energy efficiency point of view.
Switch-off schemes—for both macro nodes, in homogeneous settings, and micronodes in heterogeneous settings—can be either offline or online [5]. Offline schemes rely on knowing the evolutions of load in time (the so-called load curves, e.g., [19]) and decide the optimal switch-on/switch-off intervals based on the shape of these curves. They are often used for resource provisioning purposes and, being based on load predictions, which may always prove wrong, need an online counterpart to react to unforeseen circumstances (flash crowds, accidents, etc.).
Online algorithms look at the current load and decide when and where it is high enough to warrant a switch-on (or low enough for a switch-off) of some nodes. Online schemes normally work at coarse timescales, for example, 15-minute slots, because switching operations are not instantaneous (in fact, they may take up to minutes [10]). Moreover, the ensuing topology alterations may trigger avalanche effects, such as massive handovers, or cause interference ripples in neighboring cells, which are difficult to predict and manage. For instance, switching off a micro node at location x may indeed increase the load in nearby areas, because the macro node at location x takes over its load and hence generates more interference around itself.
Most works in the existing literature usually make some assumptions which render their scheme impractical: the first one is that the number of per-node switch operations is unlimited. While we can expect future base stations to be able to tolerate frequent power transitions, most of the ones that are currently operational have not been designed with this characteristic [6]. This means that frequent switches will decrease their energy efficiency and possibly decrease their mean time between failures (MTBF), something that has a significant impact on OPEX. This calls for algorithms that can maximize the power saved by the network while limiting the number of power transitions for the node to a predefined maximum. Another frequent assumption is to rely on a particular shape of the load curves (e.g., one with a clearly distinguishable peak-hour and off-peak valley). While this allows one to build an optimal algorithm, the same algorithm would make suboptimal decisions when the shape of the load curve is different. Several algorithms assume clairvoyance: for instance, they assume that they know the exact amount of resources that a micro, currently switched off, will use to serve a given input traffic. This cannot be known in advance in practice and depends on the user positions and their perceived interference. Finally, they often make local decisions neglecting their global effects: as anticipated, switching off a micro (based on local knowledge, e.g., few users connected to it) will save the network owner the power to operate it; however, the extra resources required to serve the same users at the macro will foster a higher interinterference in the surrounding area, whose net effect may well be to increase the power consumption. Therefore, the global effects of switching decisions should be considered.
In this paper, we design and evaluate an offline-online framework for heterogeneous cellular networks, whose job is to satisfy a given load demand with the minimum possible energy cost. Our framework exploits offline information, that is, load curves, to select which configurations are feasible (i.e., able to carry the nominal load) at a given point in time. An optimization algorithm is then run on the set of feasible configurations, to decide the optimal switch-on points for the micro nodes, constrained to a maximum number of switch-ons in a day. To the best of our knowledge, ours is the only work that takes the latter aspect into account. The only work we know of that considers the number of switch-ons, that is, [20], seeking to minimize the latter rather than taking it as a constraint. The online part of the framework, instead, measures the current load and enforces switch-on/off of micro nodes so as to satisfy the demand in an energy-efficient way. More specifically, it may bring forward or postpone switching instants with respect to the optimal ones computed offline, when it is more energetically efficient to do so. The scheme described in this paper is designed so as to require few hypotheses: it can work with load curves of arbitrary shape, it does not assume clairvoyance, and it takes into account global effects of switching decisions through an accurate estimation of intercell interference. We evaluate our scheme using both synthetic and real-life load curves taken from operator data. Our results show that power savings up to 25% are achievable over a whole day and that neglecting intercell interference leads to overoptimistic results. Moreover, we show that our scheme is robust against variation in the expected load.
The rest of the paper is organized as follows: in Section 2 we describe our assumptions and state our problem formally. The related work is discussed in detail in Section 3. Section 4 describes the proposed framework, which is evaluated in Section 5. Finally, Section 6 concludes the paper.
2. System Model and Problem Statement
In LTE, cell transmissions are arranged in time slots called Transmission Time Intervals, (TTIs), whose duration is 1 ms. In a TTI, the node allocates frames, that is, vectors of (Virtual) Resource Blocks (RBs) to its associated user equipment (UE) [21]. UE is associated with one node at a time. Each RB carries a fixed number of symbols, which translate to different amounts of bits depending on the modulation and coding scheme (MCS) used by the antenna on that RB. In general, more information-dense modulations (e.g., 64QAM, yielding 6 bits per symbol) are favored when a better channel to the UE is perceived (i.e., one with less interference). The quality of the wireless channel varies over both time and frequency. For this reason, UE reports their perceived downlink channel state to the eNodeB as a Channel Quality Indicator (CQI), computed according to the measured Signal to Interference and Noise Ratio (SINR).
We consider a large-scale multicell LTE network as shown in Figure 1. The network is logically divided into areas (called hexagons henceforth for simplicity), each one under the coverage of a macro node (M in the figure), that is, an eNodeB with high radiation capabilities. Macro nodes are anisotropic, as shown in Figure 1. A hexagon can also accommodate micro nodes (m), , , in the figure. The latter, usually located in hotspots or at hexagon edge, can be used to provide additional localized capacity at a lower power cost. The term “micro” will henceforth be used in a broad sense, to denote nodes with smaller power and coverage, thus encompassing everything from a micro to possibly femto, with the following provisos: micro nodes alone cannot provide full coverage of a hexagon (not even jointly), and all nodes are owned by the same entity (i.e., a telecommunication operator), which pays the electrical bill. Either macro or micro nodes are used to serve UE within the network. UE associates with the node from which they perceive the highest SINR, possibly using a Cell Range Expansion (CRE) boost for the micros.
All the nodes share the same spectrum and hence interfere with each other. We assume no coordination, either among or within hexagons. A node that has to allocate RBs to its users will pick them at random from a vector of RBs and schedule each UE on each RB with the same probability. Cell coordination and Coordinated Multipoint (CoMP) techniques are a promising avenue of research to cope with interference and have been shown to increase the power saving opportunities [2]. However, they work at a significantly faster pace than algorithms for node switching and are thus orthogonal to them.
We assume that the operator possesses the following information: a per hexagon historical load curve, detailing (at least) the overall bitrate requested in a hexagon over time in a sample day. There may be several such curves, of course, for a single hexagon, depending on the context. For instance, working days and holidays have different load curves (see, e.g., [19]), and other human activities do affect the context: the typical example is an hexagon covering a major sports venue, whose load curve on the game days (e.g., Sundays) depends on whether the local sports team plays home or away. Moreover, the operator possesses a similar curve (or set thereof) detailing the number of users per cell over time, so that the average per-user bitrate can be inferred (a possible method to obtain such information is using monitoring systems that measure the values of specific KPIs (e.g., number of types of UE) with a fixed time resolution [22]. These measurements are repeated in different days with the same operating conditions (e.g., working days) in order to produce a daily profile of the average values for the given KPI.). Note that these curves do not contain information about the position of single users within a hexagon. The latter may be known in a probabilistic sense, for example, uniform within a hexagon. We make no assumption on the shape of the load curves, and—in particular—we do not rely on their having a single peak during business hours. The time resolution at which these curves are plotted is that of a sufficiently large interval (e.g., 15 minutes), called a snapshot henceforth. We denote with the number of such snapshots in a day. Furthermore, we assume that we can retrieve the current amount of allocated RBs at each macro/micro node, averaged over a snapshot.
We only consider the downlink direction, which is the most critical from the point of view of both carried load and infrastructure power consumption. Figure 2 shows the power consumed by an active node (taken from [23]). The power is an affine function of the number of transmitted RBs on each TTI, that is, , where is a baseline power and is the number of allocated RBs. The power/RB curve has the same shape for all the nodes, whether macro or micro, whereas the values , , , and do depend on the type of node. According to equipment manufacturers and cellular operators, the value of the above parameters has changed over the years thanks to the benefits introduced by new technologies and will keep changing, following the progressive shift towards the fifth generation of mobile networks (5G). However, the affine shape of the power/RB curve has remained unchanged, and such trend is expected to persist [23].
A macro node cannot be powered off and hence, the variable part of its consumed power is . For a micro node, the variable part is instead .
Since load curves represent what happens in a snapshot, whereas power is consumed by allocating RBs at each TTI, a method to infer the latter from a snapshot’s load value is required. RBs are computed as follows: first we infer a per-user requested data rate from the load curves, then we show how to compute the number of RBs needed to meet that data rate given the UE’s SINR, and then we show how to compute the SINR of a UE undergoing interference using a statistical interference model.
Focus on a single snapshot and a single hexagon, and call and the load and number of users for that hexagon at that snapshot. The average demand per UE at a snapshot is . The data rate per RB achievable by UE associated with node is a function of its average SINR, , an example of which is represented in Figure 3 (curves like the one in Figure 3 are obtained through link-level simulations, see, e.g., [24]). is the maximum data rate that can be achieved for values of SINR equal or above . UE whose SINR is below is considered out of range. Thus, the average number of RBs required to satisfy ’s demand can be computed as
Note that may not be integer. This is not a problem, since is an average value, and the time span of a snapshot is large enough as to allow a fluid approximation. For instance, an allocation of RBs per TTI can be enforced in practice by reserving two RBs on even TTIs and three on odd ones and so on.
In order to compute , we need to quantify both the signal received from node and the interference from nodes . Call and the number of RBs allocated by nodes and , respectively. Given that they do not coordinate, the probability that the same RBs are allocated simultaneously by both is [25]with . Therefore, the expected value of the number of overlapping RBs between and is
Thus, the percentage of RBs where will interfere is , whereas the remaining will be exempt from interference from . Call the power received by from node (which depends on the distance and angle between them, the propagation model and the transmitting power of ). Then, the average SINR of UE in the snapshot iswhere is the Gaussian noise.
As far as switching is concerned, as already specified, macro nodes cannot be powered off, because this would create holes in the coverage. Micro nodes, instead, can be powered off, for example, during low-load periods. However, switch-on/off operations should be few, for several reasons: first, especially with older equipment, it takes a considerable amount of time before a micro node is fully operational (in the order of tens of seconds or minutes), due to the operating system boot procedure, control and management plane setup operations (e.g., path setup in the Evolved Packet Core network), and so on, and it takes some time to shut it down as well. Second, topology alterations often have unpredictable ripples in a multicell network: a switch-off may cause massive handovers (even when cell wilting is used, [6]), possibly overloading the corresponding macro and affecting nearby macros through a change in the interference pattern. Third, switching operations increase the wear and tear of equipment, hence reducing its MTBF [6]. For this reason, we assume that an upper bound of switching operations must be enforced on all micro nodes. In other words, micro nodes can be powered up in at most disjoint time intervals within a day.
The aim of our framework is to plan and enforce switching operations at each snapshot for each micro node, so that(1)user requests are satisfied, if it is possible to do so. In other words, no overload occurs that prevents UE from obtaining the required rate;(2)as little energy as possible is consumed by the network, provided that UE requirement is met;(3)micro nodes are switched on up to times per day.
Our framework plans switching operations in advance (i.e., offline), based on historical load curves, so as to keep the macro overload probability below a predetermined threshold, and enforces switching operations online either to avoid overload or when it is energetically efficient to do so, by comparing the current load conditions to the expected (historical) ones.
3. Related Work
As already outlined, several schemes have been proposed in the past few years to deal with macro and micro node deactivation. Surveys [5–9] provide a general overview and taxonomy, besides the customary wealth of references.
In the context of homogeneous networks, many works advocate cell breathing (e.g., [11]): the radius of some cells is shrunk during off-peak hours, and nearby cells increase theirs accordingly to guarantee coverage. If the scenario allows it, a node can be completely switched off, which increases the saving. Cell wilting and blossoming are procedures for graceful shut-down and power-up of nodes, whose aim is to avoid massive handovers during topology changes [12].
It must be observed that operators often consider hazardous to vary the cell radius, because it makes it difficult, at best, to ensure the absence of coverage holes [6]. Moreover, most of the current equipment can only be powered on or off, lacking the capability to switch among a discrete number of power levels that is instead required in such schemes. Paper [13] models the energy saving sleep problem assuming a single macro and several small cells, with symmetric traffic, no interference, and “1 : 1 convertible” resource allocation between nodes. A larger-scale scenario (cluster of 7 macros, with no micro) is considered in [10], where cell breathing is advocated. Authors of [14] describe an offline algorithm that does not allow multiple switching of the same node in a day. In [15] a planning tool is described, which also takes cell breathing into account. A similar problem is tackled in [26], where authors focus on positioning of both macro and micro stations to achieve more capacity at a small cost in power consumption. Finally, [16] determines the optimal set of cells to be switched at any time t, given the instantaneous load at that time.
Among the above works, those advocating offline schemes often make assumptions on the shape of load curves, meaning that they only work when the load curves match those envisaged. Our work does not make such an assumption, and can work with any shape of load curves. Moreover, these works seldom consider intercell interference, which—as shown in the previous section—clearly influences the switching decisions. None of the above works considers (or may easily incorporate) constraining the number of switching decisions at a node, something which—as we will demonstrate—changes the picture completely.
Finally, we observe that the problem of energy-efficient resource allocation in LTE base stations can be tackled at different timescales. Besides the resource planning timescale (i.e., hours or days), which is the one of all the above works, we can also work at the frame timescale, that is, few tens of ms, for example, by scheduling MBSFN subframes in a frame [27]. During off-peak periods, even short ones, the eNB pools the few RBs that it has to transmit into as few subframes as possible, leaving the other ones blank, thus trading a modest reduction of responsiveness for a remarkable power saving. These solutions are complementary, rather than alternative, to the ones dealt with in this paper.
4. Node-Switching Framework
We first explain some of the practical challenges for a switching algorithm, which also sets the rationale behind our framework. Then we describe the framework at a high level and each of its components in detail.
We will use the shorthand to indicate that all the micros in a group of hexagons are switched on and − to indicate that they are all off. We will often distinguish two groups of hexagons: a tagged one, whose performance we monitor, and the rest of the network. Accordingly, will indicate that micros are switched on in the tagged hexagon and off in the other ones.
4.1. Justification and Practical Challenges
As already explained, we consider two performance metrics: overload and energy consumption. We first assess the impact of a micro switch-on/switch-off decision on both metrics.
The interference of a micro is, by and large, negligible outside its own hexagon. This means that, if switching on a micro succeeds in offloading the macro of some RBs, then the effects on overload at large, that is, in the whole network, can only be beneficial, since the intercell interference which is mainly due to the macro will be reduced through (4). Unless micros are misplaced, it is usually true that switching on a micro offloads the macro of some RBs and that the overload probability in a hexagon decreases when the micros are switched on. This means that switching micros on when a hexagon is in overload are the right thing to do. In order to demonstrate these two phenomena let us consider the following example: suppose we are monitoring the channel quality perceived by the UE in hexagon placed at the center of a scenario composed of other nine hexagons, each of them with one macro and two micro nodes. We define four configurations: a first one () with all the micros turned off, a second one () with all the micros turned on, a third one () with all the micros turned on except within , and a fourth one () with the micro nodes turned on only in . In Figure 4 we show the perceived channel quality in : the color of each point represents the best SINR computed among the active nodes of the considered hexagon. As we can see if we compare and , the SINR in slightly increases even if we just turn on micros in the other hexagons, thus confirming that interference from the latter has decreased. On the other hand, turning on micros improves the SINR of in both and scenarios, a benefit that comes along with the added capacity in terms of RBs that are made available by each active micro. An additional analysis of the large-scale effects of interference will be given in Section 5.
From an energy point of view, switching on a micro incurs a baseline penalty given to the difference . However, RBs allocated at the micro are normally cheaper than those at the macro, since , where subscripts and superscripts are self-explanatory. When a micro is switched on, and some UE switches from the macro to the micro, there is a global decrease of intercell interference through (4), which is also beneficial energy-wise. From the above, we obtain that keeping a micro on for very few allocated RBs is inefficient. However, it is impossible to estimate the minimum number of RBs that the micro should allocate to pay for its switch-on cost from online information only. In fact, that number can only be inferred by solving the following equation:where the left-hand side is the increase of power when a micro is switched on and allocates RBs, and the right-hand side is the reduction of power because the macro allocates fewer RBs. However, and depend on which kinds of UE will switch to the micro and what their SINR would be if they were associated with the micro. This is of course unknown, both because you do not know where this UE is located and because the micro is switched off, which creates a chicken-and-egg problem.
Similarly, it is unrealistic to assume that we can infer when it is energetically efficient to switch a micro off based on online data only. In fact, this would require knowing the SINR of each UE to the macro after the micro has been switched off. The latter is not equal to the one measured before the switch-off, due to the mutated intra- and interhexagon interference.
Nevertheless, we can figure out an approximate threshold by simulating the network offline using load curves and space distributions. Thus, the insight gathered using offline simulations can surrogate the lack of SINR-omniscience required in an online-only approached.
Another challenge stems by the constraint on the number of switch-ons in a day. This implies that making the decision to switch off a micro based on the current traffic conditions only (in a purely online approach) is the wrong thing to do. This can be explained through an example.
Example 1. In Figure 5 we show an example of average per-snapshot RBs allocation over time in a scenario with one macro () and one micro node (), constrained by a maximum number of switch-on operations . Call and the number of RBs allocated by and , and the macro saturation threshold. We start with off. The macro saturates at , triggering the activation of . At node is no longer in saturation, and the number of allocated RBs starts to decrease on both nodes, reaching value on the microside: judging by the allocated RBs only, we switch off. The same occurs in : m is activated when M saturates (), so the second and last micro switch-ons are spent. This means that when the load decreases again later on, we cannot switch off, even though drops below , since doing so would prevent us from reacting to possible future saturation conditions. Micro must therefore be kept active until , where the value of rises again above . Thus, we are able to cope with the saturation that occurs between and . Note that, in the absence of switch-on constraints, the micro would be switched off at and on again at .
Using the solution described above, we can react to overload conditions at the macro node and to inefficient utilization of micro ones, by just looking at the instantaneous allocation of RBs. Two considerations are in order.
If more energy was consumed in than is saved in , it would be more efficient to switch the micro off in the former and keep it active in the latter.
In , grows above before reaches . In this case, we can exploit efficiently, energy-wise, even though the macro node is not in saturation yet.
The above two observations can only be made with hindsight, hence are precluded to an online algorithm. However, we can again exploit historical knowledge, harvested from the daily load curves, to endow an online algorithm with prediction capabilities and increase its efficiency.
4.2. Description of the Framework
Our framework obtains historical information from the network, processes it offline, and makes switching decisions online, based on the current load and the historical knowledge. The framework and its information exchange with the network nodes are shown in Figure 6. The offline part of the framework is composed of a trainer and an Offline Analyzer. The trainer takes as input the historical load curves (detailing the requested bitrate and number of users per hexagon) and simulates what happens at each snapshot when micros are switched on and off, respectively. The outputs of this simulation are, for each snapshot, an overload probability, that is, the probability that a given hexagon is unable to carry the requested load when the micros are switched off, and a resource consumption, that is, the number of RBs allocated within a hexagon when the micros are turned on and off, respectively (note that the trainer could also be realized in different way, which may not involve simulation at all, e.g., as a machine learning system trained with a large set of measurements from the network in various configurations. As long as the trainer is able to compute overload probabilities and resource consumption, the rest of our framework will be able to work with it).
The Offline Analyzer takes as input the above information, plus the power models of the macros and micros and the constraint on the number of switch-ons, and finds the energy-optimal activation pattern for each hexagon, that is, the intervals where switching micros off allows the highest power saving, given that the requested load is satisfied.
The above information, plus the current load at the hexagons, is used by the Online Decider to enforce switching decisions. Hereafter, we present the three blocks separately.
4.2.1. Trainer
The trainer considers one snapshot at a time. In a snapshot, it simulates the network with the given number of users per hexagon, their spatial distribution, and requested bitrate, in a scenario. A scenario is defined by the number of nodes that is switched on: for instance, a micro-off scenario is one where all micros are switched off, and a micro-on scenario is the one where all micros are switched on.
The simulation is done as follows: users are dropped in the hexagon according to the spatial distribution. Figure 7 shows a uniform (a) and hotspot-based (b) spatial distribution. In the latter, UE is dropped in the portion of hexagon that intersects a circle of center c and radius r. Note that spatial distributions are used only by the trainer, while the Online Decider totally does without them. At a snapshot boundary, UE is created or deleted to match the required number, and the position of the remaining ones is randomly changed within a configured area. This allows us to simulate random mobility within the cell or parts thereof.
(a)
(b)
UE is associated sequentially, that is, one by one, to the active node with the highest SINR, with a CRE boost for the micros.
The average SINR of each UE is computed through (4). Note that there is a circular dependence between the SINR and the RBs allocated to a UE. When a UE is allocated some RBs (say, in hexagon e), in fact, grows, thus increasing the interference suffered by UE attached to x and reducing their SINR. This in turn increases their RB demand and so on. This means that the average SINR must be computed iteratively, until convergence is reached. The algorithm that does this is shown in Algorithm 1.
| ||||||||||||||||||||||||||||||
The procedure is a cycle repeated for up to iterations or until convergence is reached. Each iteration cycles through every UE. We distinguish two phases.
(i) Association Phase. For the first iterations (lines 6-7) UE is allowed to select the serving node, according to a best SINR policy (line 7). While doing so, the procedure also allocates RBs according to (1) (line 10).
(ii) Convergence Phase. For all the remaining iterations, UE does not change their serving node, and the allocated RBs are updated (line 10).
In both cases, at the end of an iteration the interference is updated (line 13) through (3). If any changes significantly from one iteration to the next, the interfChanges flag is set, to signal that the convergence has not been reached yet. Both and from (4) are computed using the ITU pathloss and angular attenuation models [28].
Note that, at the first iteration of the Association Phase, no RBs have yet been allocated at any node, hence the interference is null and the association is distance-based rather than SINR-based. After the first iteration the interference is updated, hence the nearest node may not be the one with the best SINR anymore. This is the reason why the Association Phase is repeated times. However, has to be limited; otherwise some UE will end up oscillating indefinitely between two or more nodes. Figure 8 shows how the SINR decreases with the iterations, quickly converging to a stable value.
We say that a hexagon, during snapshot , is overloaded in a scenario if it cannot serve its requested traffic. This means that all the resources of the macro (and possibly of the micros, if they are switched on) are depleted. Our first concern is to ensure that the network is able to satisfy the requested load. Therefore, we need to take note of overloaded hexagons. Moreover, we need to assess whether a scenario is more energy-efficient than another, hence to compute the number of allocated RBs at all active nodes, which contribute to the variable part of the power.
Obviously, both the above depend on the way UE has been dropped. We recall that UE dropping is random, according to a preestablished distribution. Therefore, the trainer repeats the simulation of the same scenario for times, dropping UE independently at every replica. This way, we obtain an overload probability for each hexagon in the considered scenario, given by
Moreover, we obtain a vector of average allocated RBs, whose elements are the RBs allocated at each node in the hexagon, averaged over the n replicas: where is the identifier of the node within hexagon .
The above two quantities can be used to compare scenarios, specifically to identify the minimum-power one having the overload probabilities below a given threshold. Unfortunately, an exhaustive comparison of scenarios is impossible. This would in fact require simulating as many scenarios as the powerset of the set of micros. In a network with few tens hexagons, micros are easily in the hundreds, which makes the task impossible. We therefore settle for a simple trade-off, which proves to be effective nonetheless. We simulate only the two limit scenarios with all micros switched on and switched off. We denote these two scenarios using subscripts and . The rationale beyond this choice is that the two configurations are expected to achieve, respectively,(i)the minimum overload probability (which should be null, unless the network is underprovisioned), but a high power consumption;(ii)a larger overload probability, with a considerably smaller power consumption.
All the quantities defined above are computed for each snapshot, thus values at snapshot will be represented as and .
4.2.2. Offline Analyzer
Values , , and , and the node power models are then passed to a module called Offline Analyzer (OA), whose purpose is to process this information and to generate guidelines for online decisions. The OA makes independent decisions for each hexagon. It compares to a predefined overload probability threshold (which may also depend on the snapshot and hexagon) and computes a binary value that marks whether the overload probability is above or below the threshold. First it computes the expected power consumption of each of hexagons and in the two configurations as where is the number of micros in hexagon .
Note that each node may have its own power model parameters, thus accounting, for example, for different versions of the apparatus and different cost of energy. The same applies to the switching-off thresholds. However, for the sake of readability, we will assume that all the power models are the same and drop the superscript whenever this does not generate ambiguity. For instance, we will use to denote the switch-off threshold.
We can then obtain an activation pattern for micro nodes in a hexagon during a day, as a vector of binary values , where indicates that all micros in the hexagon are switched off and if they are all on. We compute the energy-optimal pattern separately for each hexagon as the optimum of the following problem, where hexagon superscript is omitted for the sake of readability: is a binary variable that counts switch-on operations. The objective function to be minimized is the expected overall power consumed by a cell throughout the whole day. Constraint (i) states that micros should be on when saturation may occur, (ii) forces to one when the micro is switched on at , (iii) takes care of the wrap-around at the end of the day, and finally (iv) constrains the number of switch-ons to . The above is an Integer-Linear Problem (ILP) with constraints and variables and can be easily solved using commercial solvers such as CPLEX [29].
In Figure 9, we show an example of computation of the energy-optimal activation pattern, based on the same scenario of Figure 5. Each vector represents the values of , , and , respectively, throughout a day of snapshots within hexagon . Marked elements in the second and third vectors identify the lowest-power configuration (all off or all on) at that snapshot. Note that the same time instants as the ones in Figure 5 are reported. The dashed line indicates the pattern that will be chosen by our optimization problem if . As we can see we have three snapshots where , where keeping micros on is thus mandatory. The optimal solution is to keep the micros on in and switch them off in as the power saved in snapshot three is , whereas in snapshots 5 to 8 is . Moreover, note that, as observed in Example 1, the optimization problem brings forward the second switch-on event to .
4.2.3. Online Decider
The output of the OA is fed to the Online Decider that monitors the system utilization and takes decisions on a per-snapshot timescale. More specifically this module decides which micro to switch on/off, based on both current and historical information. It includes an activation and deactivation procedure, performed sequentially. Figure 10 reports the flow diagrams for such procedures. Each box represents a processing step, whereas each diamond represents a decision. Deactivation takes as an input a list of micro nodes, ordered by increasing load. Each of those micro nodes m is switched off if all the following conditions hold simultaneously at snapshot k:(i)no node in the hexagon is overloaded,(ii) for the given hexagon,(iii) is underloaded, that is, .
(a)
(b)
Conversely, we activate all the micros of a hexagon if either of or both the following conditions hold:(i)at least one node in the hexagon is overloaded;(ii).
Example 2. Consider as an example the RB utilization shown in Figure 11 where a hexagon with one macro and one micro is considered and the energy-optimal pattern of Figure 9 is applied. At the activation procedure detects saturation and switches the micro on. At , it is kept on since . The first switch-off opportunity occurs at : in this case the micro is kept on until , that is, at . The last switch on is at , as in the energy-optimal pattern.
4.3. Implementation Issues
The software framework described in this paper is meant to be run at a centralized element, which performs resource planning. The location and nature of this resource planner depend on the operator choices for RAN deployment. In a Distributed RAN, it should be a monitoring server in the operator’s network. In a Centralized RAN (C-RAN), it would naturally be a process running in the same data center that hosts the virtualized Baseband Units (BBUs) of the nodes in an area. A software system that supports a decision framework such as the one described in this paper is being designed and deployed in a testbed as part of the Flex5Gware EU 5G-PPP project [30].
Regarding the processing cost, our measurements show that a single-day run (i.e., 96 snapshots) of the trainer occupies approximately 1.5 minutes of a single core on a desktop PC. Solving the optimization problem at optimality takes less than one second and hence is negligible. The number of runs should be calibrated based on the required overload probability threshold: more runs allow one to estimate small probabilities with greater accuracy. Now, the planning normally occurs on a daily basis, and a single core can run the trainer up to ~950 times per day. This means that our framework can be made to work with arbitrary accuracy on off-the-shelf hardware in a day. More to the point, this very fact can also be exploited to compensate for unforeseen context mismatches. Suppose that an unforeseen event occurs, such as a large traffic jam, which predictably changes the context for the rest of the day (e.g., by postponing by 1-2 snapshots the onset of the daily peak in all the cells of a business district). One can then run the offline trainer again, starting from the current time up to the end of the day, discounting the switching operations that have already occurred from , and come up with an optimized planning for the rest of the day in a reasonable time, still executing tens of training runs on a multicore machine within a snapshot’s worth of computation time.
The communication overhead is also small to negligible. A vector of 96 binaries must be communicated daily (or on demand, in the above cases) to the macros, and the macros themselves can issue the switch-on/off commands to their micros according to the daily plan. Communications among macro and micro nodes belonging to the same hexagon can be realized exploiting the X2 communication and in particular the X2 Application Protocol (X2AP) [31]. The latter defines a set of standard messages that eNBs can exchange. For example, an overload condition can be signaled using the Load Indication Information Element, and an eNB can be switched on using the cell activation procedure.
5. Performance Evaluation
In this section we analyze the performance of the system described in the previous section. We feed our simulator with both synthetic and realistic load curves, specifying for each snapshot and for each hexagon, the number of active types of UE together with their traffic request in Mbps. Realistic curves are taken from data collected by an operator management system from real deployments. Data are from three hexagons of two different sites, namely, Site-A and Site-B, and include the average number of active UE and the bandwidth due to data communications during a snapshot. We also consider each UE as performing a VoIP call 1 minute long at 20 kbps. To obtain the total data volume we sum the bandwidth due to data and VoIP communication, and we scale the result by a factor of 1.25 and 1.5 to obtain low, medium, and high system loads. As an example, the resulting average data rates for Site-A in the three cases are, respectively, equal to 35, 44, and 53 Mbps. The overall traffic request for the medium load configuration for the whole set of snapshot is represented in Figure 12 by means of curves. Note that their shape is irregular and clearly different from the one assumed in most of the related work.
We first analyze the effects of interference on the system, using synthetic load curves with artificially unbalanced load among the hexagons. Then we compare the performance of our algorithm in various configurations against two baselines: a first one with all the micros always off and a second one with all the micros always on. The above analysis is performed both in case of uniform and nonuniform spatial user distribution within each hexagon. Finally, we will test our framework for robustness, verifying how it reacts to changes in the expected load.
System performance will be evaluated in terms of the power variation over . We will consider only the variable part of the power consumption and assess the saving of configuration over baseline as .
The main simulation parameters are summarized in Table 1.
5.1. Synthetic Load
In this section, we demonstrate the importance of keeping large-scale effects (such as intercell interference) into account. For this purpose we define a set of synthetic load curves, as depicted in Figure 13: hexagon 0 ()—the one being observed— has a constant requested load, while all the others (1 to 56) have a variable one. Figure 14 shows the RB occupancy of in the configurations previously defined ( and ) plus a third one () with all the micros on, except the ones in hexagon 0. Two considerations are in order: first, the number of allocated RBs in , thus its power consumption, varies proportionally with the load of neighboring hexagons, despite the requested load in the former being constant (see curve ). The interference from the other hexagons increases in fact with their load.
Second, using micros offloads the macro node, thus avoiding saturation and using resources more efficiently at high loads, as shown by curve . Note that the interference suffered by , thus its power consumption, drops also when activating micro nodes in neighboring cells only ().
5.2. Realistic Load
We now analyze the behavior of the system with realistic traffic curves with 57 hexagons, each one composed of one macro and two micro nodes. Each set of three load curves is applied to the central triplet of hexagons and is then replied in the rest of the system using the same pattern, as represented in Figure 15. With reference to the latter, all the hexagons of the same color will request the same data rate. However, UE is dropped randomly within each hexagon, thus hexagons will differ—possibly in a significant manner—in terms of requested resources (which also depend on the UE SINR). This approach allows us to have diverse levels of request resources throughout the system, while still using realistic values of requested data rates.
As a first step we analyze the performance of our system with various values of the threshold , then we will investigate the effects of . Figure 16 shows the power variation in the central triple of hexagons. As we can see the latter is not affected by the value of , regardless of the number of switches . This happens because our algorithm tries to follow the suggested energy-optimal pattern, thus mitigating the effects of erroneous settings of .
Figure 17 instead shows the power variation in Site-A for various configurations of . As we can see the highest savings are achieved at low loads, when micros are switched off most of the time. As the load increases, the impact of increases as well, but not in a significant manner. Similar things occur with Site-B, as shown in Figure 18, with an impact of that is even lower. This effect is due to the “smoothness” of the load curves and to the uniform distribution of UE among the system. When comparing Site-A and Site-B loads, higher saving is obtained with the former, as it has lower load requests throughout the day, thus allowing for longer periods with the micros turned off.
To better understand the effects of , in Figure 19 we report the cell load of in Site-A with high load. Moreover in the lower part of the figure we show three binary curves, each one representing the optimal energy-activation pattern for the micros, for three values of , in the shape of three binary curves indicating the values of . There are two saturation zones (circled peaks at the top of the figure). In these cases every micro must be turned on so as to provide additional capacity. There are two more zones (boxed areas) where the macro is not saturated, but the amount of requested resources is such that having the micros on is energetically profitable. In the remaining snapshots the best solution is having the micros off. The first two conditions (saturation) are satisfied with every value of , while the last one is fully achieved only for , when micros are switched only when necessary, regardless of the number of switches.
In Figures 20 and 21 we show instead the total number of overloaded RBs during the whole day, that is, the number of RBs, beyond those available, which would be required to cope with the traffic request of a day. This allows us to measure the ability of the system to satisfy the traffic request itself. As we can see, configuration performs way better than ; thus its performance has to be considered as the target of our algorithm. The latter in fact obtains the same number of overloaded RBs as , regardless of the choice of , thus enforcing energy saving with no impact on the carried load.
5.3. Nonuniform User Distribution
In this section, we will analyze the performance of our algorithm in case of nonuniform and time-varying distribution of UE within the system. For each hexagon, users are randomly dropped inside one hotspot of center c and radius r, as presented in Figure 7(a). Value c is changed every five snapshots, alternating between two positions, closer to the macro () and to the micro (), respectively. The value of r is instead picked randomly in the interval , every snapshot.
In Figures 22 and 23 we show the power variation obtained with respect to , for both Site-A and Site-B. The impact of is in this case greater than with a uniform user distribution. Moreover, this effect increases with load, as the energy-wise efficiency of micros is higher with larger traffic demands. As in the previous scenario, the performance of our algorithm from the carried load point of view, is equal to those obtained with and is thus not shown.
The analysis in Figures 17–23 suggests that a larger (which, as we remark, is an input datum and not a decision variable) yields diminishing returns in terms of power savings. Most of the power, in fact, is saved with the first few switch-offs—sometimes even a single one—beyond which topology alterations get progressively less useful. However, while there must be a point where increasing stops yielding tangible benefits (unless load curves are collections of IID points, which is not what appears from measurements), its location depends on several factors: the shape of the load curves, the distribution of the position of UE, the power profiles of the nodes, and so on. In fact, Figures 22 and 23 show that increasing from one to three in a hotspot scenario yields considerable benefits, unlike with a uniform distribution (see Figures 17 and 18).
5.4. Resilience Analysis
In this section, we will test how our solution reacts to variation in the expected load. More in detail, we want to check if the Online Decider is able to recover from wrong suggestions coming from the Offline Analyzer or to react to unexpected changes in the actual load of the system.
As a first step, given a load curve of an hexagon for a certain site, for every snapshot we take the original bitrate request, calling it , and we compute the modified bitrate request as , where is a random value extracted from a uniform distribution in the interval . Note that we are not varying the number of UE within a hexagon, but only their load requests. Figure 24 shows a portion of the original load curve in for Site-A load, compared against three modified load curves obtained with a extracted in with three different seeds.
Although the shape of the curve is roughly the same, pointwise variations can be clearly observed. We repeated the above procedure in order to obtain various modified load curves. For each of them we first fed the trainer with the original load curve, and then we perform simulations using the modified ones. We used this method for both Site-A and Site-B and for each load level. As we can see in Figures 25 and 26, the effects of the variations on the power saving are negligible in all cases. This means that, as far as the shape of the load curve is maintained, the Online Decider is able to react to variation in the requested load.
6. Conclusions
This paper presented a framework for the activation/deactivation of nodes in a heterogeneous network, where the number of switch-ons in a day is constrained. We first analyzed the limits of both online- and offline-only algorithms, demonstrating how optimality requires unrealistic hypotheses. We then proposed and evaluated an online-offline method consisting of three phases: training, offline analysis, and online decision. The training phase is done offline and allows one to evaluate the energy consumption and saturation probabilities of alternative configurations, taking into account intercell interference at large. The offline analysis provides the optimal (constrained) switching pattern, and the online decision algorithm evaluates the current network load and makes switching decisions, erring on the safe side as far as saturation probability is concerned. Our framework works with arbitrary load curves and power figures, and it takes a predefined maximum number of switch-offs as an input. Simulation results show performance savings between 10 and 25% with both synthetic and real traffic loads and with both uniform and nonuniform user distributions. Future work includes developing and analyzing switching algorithms that takes into account more detailed information on the spatial distribution of UE, as well as the possible benefits given by enhanced Intercell Interference Coordination (eICIC) techniques at the eNBs. Both frequency-based (e.g., coordinated scheduling or partial frequency reuse) and time-based (e.g., provisioning of Absolute Blank Subframes) can easily be taken into account into our framework. All it takes is to modify expression (4), by correctly accounting the number of overlapping RBs between couples of eNBs depending on the eICIC technique.
Finally, minimizing the number of topology alterations (i.e., switch-ons and switch-offs) required to achieve a target power saving in a network is a related problem that we plan to investigate in the future.
LTE-Related Acronyms
| CoMP: | Coordinated Multipoint |
| CQI: | Channel Quality Indicator |
| CRE: | Cell Range Expansion |
| eNB: | Evolved Node-B |
| LTE: | Long-Term Evolution |
| LTE-A: | Long-Term Evolution Advanced |
| MCS: | Modulation Coding Scheme |
| MTBF: | Mean Time between Failures |
| MBSFN: | Multicast Broadcast Single Frequency Network |
| RB: | Resource Block |
| SINR: | Signal to Interference and Noise Ratio |
| TTI: | Transmission Time Interval |
| UE: | User Equipment. |
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
The subject matter of this paper includes description of results of a joint research project carried out by Telecom Italia and the University of Pisa. Telecom Italia reserves all proprietary rights in any process, procedure, algorithm, article of manufacture, or other results of said project herein described. A preliminary version of this paper was presented to the 5th InfQ Workshop, Torino, Italy, October 2014. The authors would like to thank Alessandro Donati, former M.S. student at the University of Pisa, now with Telecom Italia, for his help with the simulations. This work was carried out while Dario Sabella was with Telecom Italia, Turin, Italy.