Abstract
Multimedia communications are attracting great attention from the research, industry, and end-user communities. The latter are increasingly claiming for higher levels of quality and the possibility of consuming multimedia content from a plethora of devices at their disposal. Clearly, the most appealing gadgets are those that communicate wirelessly to access these services. However, current wireless technologies raise severe concerns to support extremely demanding services such as real-time multimedia transmissions. This paper evaluates from QoE and QoS perspectives the capability of the ad hoc routing protocol called BATMAN to support Voice over IP and video traffic. To this end, two test-benches were proposed, namely, a real (emulated) testbed and a simulation framework. Additionally, a series of modifications was proposed on both protocols’ parameters settings and video-stream characteristics that contributes to further improving the multimedia quality perceived by the users. The performance of the well-extended protocol OLSR is also evaluated in detail to compare it with BATMAN. From the results, a notably high correlation between real experimentation and computer simulation outcomes was observed. It was also found out that, with the proper configuration, BATMAN is able to transmit several QCIF video-streams and VoIP calls with high quality. In addition, BATMAN outperforms OLSR supporting multimedia traffic in both experimental and simulated environments.
1. Introduction
Multimedia transmissions over packet-switched networks are becoming more and more popular among end users [1]. In addition, these types of communications are opening a new wave of unimaginable services little time ago. However, multimedia traffic poses strict requirements to meet in order to provide these services with acceptable levels of quality due to their intensive use of bandwidth. Delay-sensitive services such as Voice over IP (VoIP) or video-streaming, which are some of the most prevalent applications, are also strongly impacted by other impairments such as packet loss or signal distortions. Therefore, mechanisms to ensure the appropriate quality of these services are indispensable [2]. The classical approach to evaluate the quality of a given service, that is, Quality of Service (QoS), is being replaced by a user-centric approach known as Quality of user Experience (QoE). Unlike the former, which focuses on monitoring network metrics, such as delay, jitter, or Packet Loss Ratio (PLR), the latter quantifies what the user actually perceives when consuming the service. The regular subjective QoE evaluation is performed throughout a quality survey passed to a group of observers, which rate the service in a scale ranging from 1 (poor quality) to 5 (excellent quality) (please see ITU-T Rec. P.800 [3]). This quality metric, so called Mean Opinion Score (MOS), is extensively used by the research community to assess the quality level of multimedia services. However, quality surveys are expensive and time-consuming and cannot be carried out in real time; for these reasons, some mathematical models have been derived aiming at mimicking/characterizing the quality perception of a human being by means of analyzing technical parameters extracted from both the multimedia flow and the delivery system [4, 5].
Focusing on the customer side, it is unquestionable that wireless networks have brought a new era of experiences and facilities to the end users. Mobility or ease-of-access, among others, is good example of greatly valued characteristics by users. Nevertheless, achieving the desired service level of quality in wireless networks is not as simple as it is in wired environments. In the latter, most of the system variables are under control and many correcting actions can be taken when the quality of the service drops under certain low values. On the contrary, wireless systems have inherent characteristics that make them unpredictable and difficult to maintain under perfect control. New impairments that do not exist in wired networks, for example, interferences, fading, and channel access contention, together with the mobility of the nodes composing the network, pose new difficulties to the successful establishment and support of demanding transmissions, such as multimedia communications. Currently, the most widespread wireless standard to configure Wireless Local Area Networks (WLANs) is the IEEE 802.11 family (Wi-Fi). This technology defines two different operational modes. The most extended is the infrastructure mode that consists of a central node (i.e., access point) that provides network services to the rest of the network nodes, for example, routing and IP configuration. The remaining nodes act as hosts sending and receiving data traffic through the central access point. This Wi-Fi working mode has been deeply investigated, showing acceptable conditions to support multimedia traffic [6, 7]. The other configuration mode for IEEE 802.11 networks does not employ a central access point, and all the nodes composing the network operate in an ad hoc fashion. In this scheme all the nodes must assume both host and router roles, helping each other to relay packets in their path to the destination. That feature allows the quick deployment of self-configurable networks without a preexistent infrastructure. This type of network, known as Mobile Ad Hoc Network (MANET), is considered to be of great interest due to its multiple benefits such as network coverage extension, scalability, and mobility capabilities provided to the nodes. However, MANETs pose additional challenges, especially those related to node mobility. The dynamic multihop topology of the network may cause constant route changes, hidden/exposed terminals problems, and so forth. Thus, efficient routing protocols for MANETs are necessary, capable of providing enough reliability to the network and with the minimum impact on the service data flows through the nodes [8].
This paper evaluates the performance of the ad hoc routing protocol, called BATMAN (Better Approach to Mobile Ad Hoc Networking) [9], supporting multimedia traffic (both VoIP and video-streaming). This IETF draft has drawn great attention lastly due to its simple routing algorithm requiring low computational load. The performance of BATMAN is compared with that obtained by the deeply studied OLSR (Optimized Link State Routing) routing algorithm. This protocol, whose status is denoted as experimental RFC by the IETF, has shown better performance than other proactive routing protocols in multimedia-delivery scenarios [10] and a number of works analyzing its pros and cons can be found in the literature. However, to the best of the authors’ knowledge, a performance evaluation of both protocols in a real testbed composed of a significant number of nodes has not been published. Additionally, most of the works addressing quality attained by multimedia services over MANET scenarios obtained their results either only by simulation or only by using small real testbeds, separately. This is an important point, as the simulation studies should be corroborated by experiments in realistic scenarios in order to evaluate the correlation between the results obtained by both methodologies. For those reasons, all the results of this work are obtained through computer simulation and from a real test-bench by making use of the open-access platform Emulab [11] for a considerable number of nodes composing the network. All results are provided in terms of both QoE and QoS.
The experiment procedure was as follows. First, the testing network was defined and its main parameters and target metrics were designated. After the overall network design, the resulting scenario was evaluated by using computer simulation. Thereafter, the same network was built up and tested in the real test-bench. Finally, the attained outcomes from both testbeds were statistically compared aiming at finding inconsistencies between them. Furthermore, when one network-configuration parameter was modified, the previous steps were repeated a significant number of times in order to avoid nonrepresentative results.
Therefore, the contributions of this work are the following:(i)A performance evaluation from QoS and QoE perspectives of BATMAN and OLSR routing protocols with video transmission traffic as well as VoIP calls using the default routing parameters as set in their corresponding RFCs.(ii)A rigorous study of the impact of tuning the BATMAN OGM interval and the OLSR HELLO interval on the quality achieved by the multimedia service flowing through the ad hoc network.(iii)An assessment of the effect of using different bit-rates in video-streams on the QoS/QoE provided when using BATMAN or OLSR in MANETs either with their default configuration or with the tuned parameters.(iv)A statistical comparison of the results attained in both test-benches that allows studying the correlation between the simulation framework and the realistic testbed.
The rest of the paper is organized as follows. Section 2 reviews the related literature, focusing on those works that evaluate the performance of BATMAN and OLSR routing protocols supporting multimedia services. A description of the testbeds employed in this work is included in Section 3. Section 4 shows the results achieved and discusses the quality provided by the two routing protocols under study supporting different types of multimedia traffic. The paper ends summarizing the most important facts.
2. Related Work
The performance of BATMAN supporting multimedia traffic in real scenarios has been barely evaluated in the related literature. Two relevant works addressing this topic are those presented by Kulla et al. [12] and Seither et al. [13]. In the former, the performance of BATMAN supporting multimedia traffic (audio and video) in small testbeds was evaluated. In their results, the authors demonstrated the capability of BATMAN-managed MANETs formed by 5 nodes to support VoIP traffic; however, they also showed some issues to support video traffic, which is much more bandwidth demanding than VoIP. In turn, the work in [13] presented only a QoS comparison between BATMAN and AODV (Ad Hoc On-Demand Distance Vector) operating in more populated real testbeds. The results showed a noticeably better performance of BATMAN in terms of PLR, throughput, and stability than AODV. However, the traffic patterns employed seem rather limited as the authors made use of ping and constant TCP traffic to test the network. Another work employing CBR (Constant Bit-Rate) as testing traffic (UDP in this case) is that presented in [14]. Mozumder et al. conducted a simulation experiment aiming at comparing the scalability performance of the BATMAN and HWMP (Hybrid Wireless Mesh Protocol) routing protocols. The relevance of this work lies in the use of the HWMP, which is defined in the IEEE 802.11s standard. However, the authors found out that, in large networks, BATMAN presents better performance than HWMP in terms of packet delivery ratio, throughput, and route maintenance.
Using different simulation frameworks, several works have also addressed comparative performance evaluations of other ad hoc routing protocols against OLSR supporting VoIP and video traffic [15, 16], but not including BATMAN. Adam et al. [15] presented a comparison among OLSR, AODV, and DSR (Dynamic Source Routing). In this work, the authors carried out a performance evaluation of these protocols supporting CBR traffic simulating a multimedia service. They found out that AODV beats OLSR in terms of jitter and Packet Delivery Ratio (PDR), but at the expense of higher routing overhead. On the other hand, OLSR showed better performance than AODV in terms of delay for a high number of simultaneous connections in the network. In [16], the performance of OLSR supporting CBR traffic was also compared with that obtained by the routing Protocol for Unified Multicasting through Announcements (PUMA). Several QoS metrics, namely, throughput, latency, and PLR, were assessed. In every performed experiment, OLSR beats PUMA, obtaining promising results, although just one simultaneous multimedia flow in the network was considered. The good results in terms of QoS in comparison with other routing algorithms attained by OLSR in these works and others [10] have led us to choosing it as the reference scheme to compare BATMAN with.
In a previous work [17], the authors evaluated through computer simulation the capacity of BATMAN and OLSR to support VoIP traffic for several network topologies and configurations. The MOS reached by a variable number of simultaneous VoIP calls in ad hoc chain and mesh scenarios was studied. Furthermore, the impact of node-motion following a pedestrian mobility pattern on mesh topologies was also analyzed. The results showed the need of properly tuning the interval between BATMAN Originator Messages (OGMs) in order to obtain better quality for the calls. For instance, in high-dense mesh scenarios (50 m × 50 m, 50 nodes), a notable improvement was observed on the number of VoIP flows supported by the network reaching a minimum quality level of 3.2 out of 5 on MOS scale. The number of simultaneous VoIP streams supported by the BATMAN network was increased from 2 VoIP calls, using the OGM interval by default (1 s), up to 10 simultaneous VoIP connections, by tuning the OGM interval to 5 s. For the sake of clarity, Table 1 presents a comparison of the reviewed works.
3. Test-Benches
Two different frameworks have been employed to obtain the results shown in the following sections. First, the Omnet++ v4.4.1 simulator has been used with the InetManet 2.2 framework [18]. These libraries include several network devices, for example, wireless hosts, routers, and the implementation of the different-layer elements used in this work, namely, 802.11, BATMAN, OLSR, traffic generators, and so forth. On the other hand, it is assumed that it is also necessary to validate the results obtained through simulation by comparing them with measurements taken from real test-benches. Thus, the Emulab platform [11] has been selected to run the mesh network experiment under study. Emulab is a network test-bench that provides both wired and wireless nodes with high level of software customization. This platform has shown accurate results and fidelity [19], especially when no delay emulation, that is, delays introduced by the management system to simulate congestion situations or link-introduced delays, is set. This is the case of this work, in which additional delays to those induced by the network have not been included.
The evaluated scenario is an IEEE 802.11g wireless network at 54 Mbps in ad hoc mode. Channel access is done according to the Distributed Coordination Function (DCF) scheme without employing the RTS/CTS mechanism. A mesh network composed of 15 nodes was configured (Figure 1) and its performance was evaluated with a variable number (1, 2, 3, 5, and 7) of simultaneous VoIP calls and video-streams flowing through it. The floor, in which the Emulab test-bench is deployed, has the following dimensions: 110 m × 65 m, and it has been recreated in the Omnet++ workbench (Figure 2) as well.
In the real scenario, five different random groups of transmitters (TX) and receivers (RX) were employed, establishing three independent communications between each TX-RX pair. In the simulation environment, 10 independent simulation instances with different seeds have been run for every evaluated scenario. Hence, in order to generate the results, the average value for every measured parameter was taken, avoiding nonrepresentative singularities. The variance and the 95% confidence intervals for every measurement have been also calculated, although they have not been shown in the presented figures due to their low significance.
As part of the results is obtained from simulated scenarios, it is important to model the transmission channel as realistic as possible. For that reason, the effect of the fading channels on the wireless transmission medium is introduced in the simulation by characterizing it with the Nakagami-m propagation model. This characterization for the physical (PHY) layer has been widely used in the related literature [20, 21] proving its accuracy. After shaping the Nakagami-m model with different values, the most precise results according to the real ones were obtained by fixing the value to 5. This value is consistent with the indoor channel characterization performed by Oestges et al. in a similar campus environment for the 2.4 GHz band [22].
In order to study the greatest number of hops between source and destination nodes in the real test-bench, the wireless card’s transmission power was set to 1 mW, since this is the minimum power value supported by the card [23]. The IEEE 802.11 channel 10, which was free of other transmissions during the experiment measurements, was employed, thus avoiding undesired interferences and collisions. The rest of Emulab nodes characteristics and the protocols versions are specified in Table 2. Regarding the nodes characterization in the simulation environment, the wireless card transmission power was set to 1 mW, with a sensitivity of −72 dBm and a SNIR (Signal to Interference and Noise Ratio) threshold of 4 dB, therefore being consistent with the features of the cards used in the real-world experiment. The test traffic used in the Emulab test-bench was generated by the Distributed-Internet Traffic Generator v2.8.1 [24], which allows sending VoIP traffic following the typical pattern of G.711 codec and video traffic by generating bursty traffic characterized by a given distribution and bit-rate. In the simulation experiment, and as part of the InetManet framework, the following were used: (i) the VoIP stream generator, which generates a flow of VoIP packets from an arbitrary sound file using different audio codecs, and (ii) the UDPVideoStreamCli2 and the UDPVideoStreamSvr2 modules. These UDP apps emulate a video-streaming communication in which the video server sends to the client UDP-traffic following a typical pattern for a video transmission, allowing setting the transmission bit-rate. The video packets were fixed to 512 bytes that is a widely employed value for encapsulating video traffic [25, 26]. Regarding VoIP traffic, the same codec as in the real test-bench (G.711) was used. The starting time for each multimedia communication was generated randomly according to a Poisson distribution in a time interval of (0, 10 s), after the convergence time for each routing protocol. Each multimedia transmission lasted 60 s.
Regarding the quality metrics of interest, both QoS and QoE figures of merit were examined. For video quality evaluation, the latter has been calculated from the parametric model presented in [27]. This model obtains accurate video quality estimations in terms of MOS, using the PLR and the video-coding bit-rate (Br) parameters, as shown inwhere , , , and are experimental factors that depend on the coding scheme applied. In this case, the widely extended H.264 codec has been used for video-streaming transmission. Thus, the values for these factors are 3.8, 4.9, 3.6, and 3.5, respectively, consistent with the figures recommended by the model’s authors. The VoIP MOS was obtained using the E-model as defined in the ITU-T Rec. G.107. In order to evaluate the protocols performance from a QoS point of view, delay, jitter, PLR, and Packet Delivery Ratio (PDR) measurements have been also gathered. The selected values for the remaining 802.11g parameters, such as DIFS (DCF Interframe Space), SIFS (Short Interframe Space), slot time, contention window (CWMIN), and different-layer headers, are written in Table 3, which additionally includes the main routing protocols time-out.
4. Performance Analysis and Results
In this section, the outcomes obtained from the test-benches described above are presented and discussed. The capability of both routing protocols under study to support VoIP and video-streaming traffic is deeply investigated. To this end, the MOS, delay, jitter, PLR, and PDR for each multimedia stream are evaluated, comparing the results obtained through computer simulation with those attained in the real testbed. It is also explored how to improve the QoE/QoS performance of the multimedia transmissions by properly adjusting different system’s parameters, such as the protocols’ control-information delivery time intervals or the multimedia-signal’s bit-rate.
As described in the ITU-T Rec. G.114 and G.1010, real-time multimedia transmissions pose stringent constraints regarding the maximum accepted delay and PLR in order to provide acceptable levels of quality to the end users. Thus, the maximum accepted PLR for VoIP services is 5% and for video services is just 1.5%; regarding delay, the maximum accepted value for multimedia real-time communications is 150 ms. From a QoE perspective, a minimum MOS value of 3.6 is established in order to consider a multimedia transmission as valid. For VoIP, a MOS value between 3.6 and 4.03 means that some users are dissatisfied with the service. Above this value, users are satisfied or very satisfied. Although there is not such standardized MOS specific limit for video transmission, the same strict value has been chosen to be consistent in simulation and experimentation findings (please see ITU-T Rec. G.108 and G.109). The advantage of employing MOS as the reference metric is that it allows joining the effect of all the different impairments affecting the multimedia stream into one single parameter. In what follows, different subsections analyzing the performance of BATMAN and OLSR for several configurations are presented.
4.1. Regular Performance Evaluation
First, an evaluation of the system described above without any modification to the routing protocols has been performed; that is, all protocol parameters were set to their default values as described in their corresponding RFC (please see Table 3).
Figure 3 illustrates the MOS attained for a variable number of simultaneous multimedia streams in the network for BATMAN and OLSR routing protocols, respectively. This figure includes the results for both test-benches, namely, the real and simulated environments. Observing the VoIP case (codec G.711), the only acceptable MOS values (over 3.6) were achieved for a low number of VoIP calls flowing through the network (Figures 3(a) and 3(b)). Thus, using BATMAN (Figure 3(a)), the MOS falls below the established threshold when three or more simultaneous calls flow in the network. With less number of calls, excellent MOS values around 4.5 and 3.7 for 1 and 2 simultaneous calls, respectively, have been obtained. Note that although the shown trend is the same for the real and the simulation measurements, the latter tends to overestimate the quality of the calls. Even so, in both test-benches, the same number of simultaneous calls over the established MOS threshold of 3.6 is obtained, namely, 2 VoIP calls. For OLSR (Figure 3(b)), the results are worse than those attained by BATMAN: the system just provides enough quality support for only one VoIP connection. Under OLSR, the results obtained in both test-benches are slightly closer than in the BATMAN case. However, now the real environment offers better results.
(a) BATMAN
(b) OLSR
Regarding video traffic, a Constant Bit-Rate (CBR) stream of 5 Mbps (HD resolution) has been used. Note that, for both protocols under study, the MOS obtained is unacceptable in all the evaluated scenarios (Figures 3(a) and 3(b)). Consequently, not even one video communication has been successfully established in the studied examples.
In order to analyze these outcomes, but from a QoS perspective, Figures 4 and 5 represent, respectively, the jitter and PLR obtained in the previously discussed scenarios. Note the high level of jitter affecting the video-streams in the real test-bench no matter the routing protocol in use (Figure 4). That fact causes continuous communication interruptions that provoke an important drop in the quality perceived by the users. Analogously, the minimum PLR suffered for both protocols under consideration supporting video traffic is 40%, which is clearly unacceptable for such highly demanding service (Figures 5(c) and 5(d)). The PLR evolution in both BATMAN and OLSR scenarios is similar and significant differences between the results obtained in the real or the simulation test-bench are not perceived. For VoIP traffic, the jitter is more controlled than in the video-streaming case (Figure 4). Notice that its value is always below 80 ms, which is a quite acceptable figure. Hence, the VoIP MOS trend shown in Figures 3(a) and 3(b) is justified by the PLR suffered by the voice flows (Figures 5(a) and 5(b)). Observe that just 2 simultaneous VoIP calls in the case of BATMAN and 1 VoIP call employing OLSR exhibit a PLR below the established threshold of 5%. This result is therefore consistent with the number of simultaneous calls accepted as valid from the QoE perspective discussed above (Figures 3(a) and 3(b)). In the light of these results, it is concluded that neither BATMAN nor OLSR in their default configurations is capable of supporting heavy real-time traffic such as HD video-streaming. On the other hand, they reach acceptable quality levels just when few concurrent VoIP calls are present in the network.
(a) BATMAN, VoIP traffic
(b) OLSR, VoIP traffic
(c) BATMAN, video traffic
(d) OLSR, video traffic
With the aim of comparing both test-benches’ results from a statistical perspective, a correlation analysis of the PLR obtained in both testbeds has been performed. As regards BATMAN, Pearson correlation coefficients () of 0.84 and 0.86 for VoIP and video traffic, respectively, have been obtained. In the case of OLSR, for the same type of traffic, values of 0.99 and 0.94, respectively, have been attained. These statistical results confirm the high similarity between the outcomes from both testbeds.
4.2. Tuning of Routing Messages Intervals: Impact on the VoIP Service
As aforementioned, the need of tuning the BATMAN OGM interval to adapt it to the actual network conditions was suggested in an authors’ previous work [17]. This preliminary idea is going to be tested now in a real test-bench rather than by simulation, as originally done before. In order to provide a more complete study, the OLSR HELLO interval has also been tuned. Other works [28–30] have previously investigated the effect of tuning the OLSR intervals but from a computer simulation assessment [28, 29] or in a real test-bench just using 5 nodes [30]. Therefore, the performance of OLSR when tuning the OLSR HELLO interval from both methodologies’ simulation and real tests composed of a significantly greater number of nodes is included here.
Figure 6 shows the impact of tuning the BATMAN OGM interval on the network PDR. This metric has been evaluated for a different number of simultaneous VoIP streams in the network. Observe that, as the system load increases, the best results are attained for longer OGM intervals. In the case of the lowest load (1 VoIP call, Figure 6(a)), quite similar results in both test-benches are obtained, with acceptable levels of PDR reached no matter the value of the OGM intervals employed. This situation changes when more traffic is present in the network; for 3 and 5 simultaneous VoIP transmissions, Figures 6(b) and 6(c) show that the best results are obtained for an OGM interval of 2 s, that is, a larger time interval than that set by default of 1 s. Moreover, for the highest system load (7 calls, Figure 6(d)), the maximum PDR is attained with the longest OGM interval, namely, 5 s. Notice that in some scenarios, for example, 3 or 5 simultaneous VoIP streams, a proper OGM interval tuning satisfies a PDR over the ITU-T limit of 95%, which is not possible to reach when employing the default OGM interval. Regarding the accuracy of the measurements, observe that, as in the previous subsection, the simulator tends to overestimate the BATMAN performance. However, the resulting tendency in both test-benches is the same for every scenario under evaluation.
(a) 1 VoIP call
(b) 3 VoIP calls
(c) 5 VoIP calls
(d) 7 VoIP calls
From a QoE perspective, Figure 7 represents the MOS attained for 3 and 7 simultaneous VoIP calls by tuning the OGM interval. Observe how in both cases the maximum MOS are obtained for different OGM intervals: with 3 VoIP calls in the network, the interval has to be tuned to 2 s or 3 s in order to fulfill a MOS over the established threshold of 3.6. In turn, for 7 VoIP streams in the system, the OGM interval needs to be set to 5 s to obtain the maximum quality. Nevertheless, in the last case, the OGM interval tuning seems to be insufficient and the MOS does not reach the established minimum acceptable level (MOS = 3.6), despite the OGM interval employed. Again, the simulation environment yields quality values above those obtained in the real test-bench. Therefore, from both QoS and QoE analyses, it is deduced that, as the traffic load increases, it is advantageous to reduce the amount of control information in the network to improve the quality of the multimedia connections. This control-packets’ reduction is accomplished by enlarging the OGM interval, which shows a great influence on the performance of the service flowing through the network.
(a) 3 VoIP calls
(b) 7 VoIP calls
Focusing on OLSR, a clear improvement in the system performance by tuning the HELLO interval is not perceived. Observe in Figure 8, which depicts the PDR for a variable number of simultaneous VoIP calls in the system, that the relationship between this metric and the HELLO interval is not straightforward. In [28], the authors obtained greater throughput values by decreasing the HELLO interval in a network composed of nodes with high mobility. In the static scenario under evaluation, this effect is not noticeable. This outcome is consistent with those attained by Demers and Kant [29]. In this work, the authors stated that, in static nonclustered networks, there is no significant change in the total traffic sent/received by the ad hoc nodes as a function of the HELLO interval. Furthermore, this fact also agrees with the results presented by Hiyama et al. [30]. Although the authors claimed in their conclusion that, in static scenarios, the throughput is higher as the HELLO interval increases, their results present a great variability. Furthermore, this work does not provide any statistical analysis to prove the relationship between the HELLO interval tuning and the network performance, which in the authors’ humble opinion detracts consistency for their statement. It is also worthwhile to remark the poor overall performance of OLSR delivering VoIP packets in comparison with BATMAN (please compare Figures 6 and 8); with the exception of the lowest traffic load case (1 VoIP call, Figure 8(a)), the remaining scenarios show low PDR values, clearly inadequate to guarantee the established quality minimums. These outcomes are further confirmed from a QoE perspective (Figure 9). Observe how, no matter the HELLO interval in use, the MOS is always less than the minimum value of 3.6. With 3 simultaneous VoIP calls in the network, the best MOS is attained by tuning the HELLO interval to 1 s. In turn, loading the system with 5 VoIP streams, the best result is obtained with a HELLO interval of 3 s, despite being far below the established threshold of 3.6. As in the previous subsection, and on the contrary to the case of BATMAN, the simulator underestimates the OLSR performance in the majority of the scenarios evaluated.
(a) 1 VoIP call
(b) 3 VoIP calls
(c) 5 VoIP calls
(d) 7 VoIP calls
(a) 3 VoIP calls
(b) 5 VoIP calls
In order to provide statistical evidence of the aforementioned outcomes, a series of ANOVA (Analysis of Variance) tests at 95% level of confidence has been performed. The statistics and values obtained in the ANOVA tests are shown in Table 4. The null hypothesis is that OGM/HELLO intervals do not have an effect on the average number of received VoIP packets. The intervals used are 0.5 s, 1 s, 2 s, 3 s, and 5 s. Results are grouped by the number of simultaneous VoIP calls in the system. First, observe that no significant differences have been obtained in any OLSR scenario, revealed by values higher than 0.05 (see the last column of Table 4). On the other hand, BATMAN scenarios with low traffic load (1 and 2 VoIP calls) seem to be unreactive to the OGM tuning, too. However, for a higher traffic load (3, 5, and 7 simultaneous calls), values below the significance level of 0.05 are obtained. Therefore the null hypothesis (the OGM interval does not impact on the number of received VoIP packets) is rejected, and it is concluded that there is statistical evidence of differences in the VoIP packets received according to the OGM interval employed, corroborating the previous discussion.
Thereby, it is determined that, in a static situation, the higher the traffic load in the network the larger the OGM interval in order to provide better quality levels. On the contrary, no clear relationship has been found between the tuning of the OLSR HELLO interval and the quality achieved by the service flowing through the network.
Regarding the fidelity of the employed test-benches (simulation and real testbed), both of them reveal a similar behavior when tuning the routing intervals, although the simulator tends to overestimate the performance of BATMAN and to underestimate the OLSR operation. The Pearson correlation coefficients () attained for BATMAN and OLSR experiments corresponding to this subsection are 0.74 and 0.96, respectively.
A similar study injecting in the network the same video traffic pattern as in previous subsections (bit-rate of 5 Mbps) has been conducted for both protocols. These results are not presented here due to the poor performance of the system that, regardless of the time intervals employed, did not reach acceptable quality levels. For that reason, the maximum video-stream bit-rate accepted by the network in order to provide this service with minimum tolerable quality levels is studied in the next subsection.
4.3. Establishing the Maximum Video Bit-Rate
In order to provide an overview of the network capability to support video traffic, a simulation study varying the bit-rate of the video-flows has been performed. With this aim, the system evolution has been evaluated by changing the video-flows bit-rate from 5 Mbps down to 70 Kbps. In particular, typical bit-rates matching widely used video resolutions have been tested: 70 Kbps and 100 Kbps (QCIF), 250 Kbps and 500 Kbps (CIF), 750 Kbps and 1 Mbps (D1), and, finally, 3 Mbps and 5 Mbps (HD). Figure 10 shows the PLR attained in the video-streams set to bit-rates up to 1 Mbps. Higher bit-rates corresponding to HD resolution lead the system to unacceptable packet loss levels, so it has been considered not necessary to present these results here. BATMAN (Figures 10(a) and 10(c)) noticeably exhibits better performance than OLSR (Figures 10(b) and 10(d)) in almost every evaluated scenario. Observe that, for the former, the system accepts 1 video-stream between 500 and 700 Kbps (D1). Additionally, 3 CIF video communications of 250 Kbps and 3 QCIF streams (100 Kbps) attain acceptable levels of packet loss. With more streams in the system, the bit-rate should be decreased and, in the worst case scenario (7 video-streams), the bit-rate should be extremely reduced in order to keep the PLR down to acceptable values. On the other hand, OLSR presents rather disappointing results supporting video traffic (Figure 10(b)): PLR values below the established limit of 1.5% are obtained just for the case of one video-flow streamed at the lowest considered bit-rates (70 and 100 Kbps). In order to evaluate the fidelity of the obtained results, please observe that the obtained PLR in both test-benches (Figures 10(a) and 10(b) versus Figures 10(c) and 10(d)) is quite similar. Although some particular values that differ in both testbeds are found, the Pearson correlation coefficient () attained for both BATMAN and OLSR tests is 0.84.
(a) BATMAN, simulation test-bench
(b) OLSR, simulation test-bench
(c) BATMAN, real test-bench
(d) OLSR, real test-bench
4.4. Tuning of Routing Messages Intervals: Impact on the Video-Stream Service
Finally, for the real test-bench, a similar study to that shown for the case of VoIP traffic has been conducted (Figure 6), regarding the impact of a proper control-packet’s time interval tuning on the quality of the video service flowing through the ad hoc network. Three different video traffic patterns, namely, QCIF (70 Kbps), CIF (250 Kbps), and D1 (750 Kbps), and 5 different values for the OGM/HELLO interval (0.5 s, 1 s, 2 s, 3 s, and 5 s) have been employed.
Network statistics have been collected for every scenario under study, focusing on the average PLR suffered by the receiving nodes. Table 5 shows the average PLR for each video traffic pattern when the network is loaded with different number of simultaneous video-streams. BATMAN is the routing protocol tested and the best OGM interval has been used according to network conditions. Additionally, the attained results employing the OGM interval by default (1 s) are also presented for comparison purposes.
First, observe an overall improvement of the system performance, comparing with that attained with the OGM interval set by default. Furthermore, notice the evolution of the best OGM interval for each scenario: as the traffic load in the network increases, the best results are attained with greater values of the OGM interval. This confirms the statistical analysis presented for VoIP traffic (Table 3) from which the need of increasing the OGM interval when the network load grows was extracted. For the lightest video traffic pattern (QCIF), the lowest PLR values are obtained by employing an OGM interval of 1 s. In turn, for heavier traffic (CIF), the best figures are attained for OGM intervals of 2 s and 3 s. Finally, for the heaviest video traffic (D1), the best results are achieved by tuning the OGM interval to even greater values: 3 s and 5 s.
On the other hand, a similar study performed for OLSR ratifies the outcomes shown previously for the VoIP traffic case. The HELLO interval tuning does not have any impact on the quality of the multimedia service, in terms of good put. As a sample, Table 6 presents the average PLR attained in the network supporting a video-streaming service of 100 Kbps (QCIF) for a variable number of flows in the network and applying the HELLO interval tuning. Observe that the poor results are quite similar no matter the HELLO interval employed: PLR figures under the established threshold of 1.5% are only obtained for just one video-stream in the network.
In view of these outcomes it is concluded that OLSR is unable to support heavy traffic services such as video-streaming, at least in its configuration by default. In turn, BATMAN has been shown to be more robust managing video traffic and, even with a proper tuning, could be a real alternative to provide support to multimedia communications with good quality in MANETs.
5. Conclusion and Future Work
In this paper, a QoS and QoE exhaustive performance evaluation study about the capability of the ad hoc routing protocol BATMAN to support multimedia traffic (including both VoIP and video-streaming) was performed. The results were obtained through simulation as well as real test-benches, showing a high correlation between them. The achieved outcomes employing BATMAN were compared with those attained by the well-accepted routing protocol OLSR. The first conclusion derived from this work is the superior performance of BATMAN supporting demanding traffic in comparison with OLSR. Furthermore, it was revealed that, with an appropriate tuning of BATMAN configuration intervals, its performance can be additionally improved achieving quality levels above 3.6 out of 5 in terms of MOS. From the presented outcomes, it is also deduced that enlarging the OGM interval as the multimedia traffic load increases benefits the overall system performance. On the other hand, OLSR was shown to be unreactive to the modification of the HELLO interval. Finally, the maximum video-stream bit-rate supported by each protocol was also studied in order to provide acceptable levels of quality, again by simulation and by real experimentation. The obtained results reveal that BATMAN is capable of managing video traffic with an adjustment of its bit-rate according to the number of streams in the network. In contrast, OLSR showed a poorer performance supporting video-streaming and the quality attained for this service seemed to be clearly insufficient.
As future work, the results obtained will be exploited to implement and test cognitive networking techniques enabling network nodes to control in-stack protocol parameters as a function of the current observed traffic conditions of the network in order to improve the QoE of multimedia services in MANETs.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This work was supported by the MINECO/FEDER Project Grant TEC2013-47016-C2-2-R (COINS) and “Programa de Ayudas a Grupos de Excelencia de la RM, de la Fundación Séneca, Agencia de Ciencia y Tecnología de la RM,” by the Instituto de Telecomunicações, Next Generation Networks and Applications Group (NetGNA), Portugal, by the Government of Russian Federation, Grant 074-U01, and by the National Funding from the Fundação para a Ciência e a Tecnologia (FCT) through the UID/EEA/500008/2013 Project.