Abstract
Quality of Experience (QoE) of video streaming services has been attracting more and more attention recently. Therefore, in this work we designed and implemented a real-time QoE monitoring system for streaming services with Adaptive Media Playout (AMP), which was implemented into the VideoLAN Client (VLC) media player to dynamically adjust the playout rate of videos according to the buffer fullness of the client buffer. The QoE monitoring system reports the QoE of streaming services in real time so that network/content providers can monitor the qualities of their services and resolve troubles immediately whenever their subscribers encounter them. Several experiments including wired and wireless streaming were conducted to show the effectiveness of the implemented AMP and QoE monitoring system. Experimental results demonstrate that AMP significantly improves the QoE of streaming services according to the Mean Opinion Score (MOS) estimated by our developed program. Additionally, some challenging issues in wireless streaming have been easily identified using the developed QoE monitoring system.
1. Introduction
Internet Protocol Television (IPTV) services [1] have been becoming increasingly popular among telecommunication companies. Network/service providers are concerned with not only the cost of constructing an IPTV service system [2] but also its service quality [3, 4]. In the past, Quality of Service (QoS) of network services is the main concern. However, QoS cannot accurately characterize the users’ perception. Therefore, the concept of Quality of Experience (QoE) was proposed and studied recently [5–13].
The most accurate method to obtain QoE is doing subjective tests [14] that are time-consuming and costly. Therefore, we proposed a cost-effective and real-time QoE evaluation method in [5], where a hybrid QoE assessment scheme was employed. In the hybrid QoE assessment, subjective tests only need be performed occasionally for updating the parameters of QoS-QoE mapping functions. Hence, the cost of a hybrid QoE assessment scheme reduces significantly. In [7], the QoE measurement for P2P-based IPTV services was studied. Another paper [8] proposed a QoS/QoE mapping and adjustment model for the cloud-based multimedia infrastructure. Additionally, since streaming services over wireless networks have been becoming popular, the QoE enhancement algorithms, protocols, and evaluation methods for wireless streaming services were proposed [9–12]. In [13], Chen et al. presented a comprehensive survey of the evolution of video quality assessment methods and analyzed their characteristics, advantages, and drawbacks.
Since the QoE evaluation for streaming services becomes more and more important, several QoE evaluation or monitoring tools were developed [15–19]. A QoE monitoring tool for multimedia quality assessment in NS-3 network simulator was presented in [15]. However, since the objective and full-reference QoE assessment method is used in [15], it is costly and cannot accurately indicate the users’ perception in a real commercial video streaming system. The paper [16] developed a testbed, which allows obtaining conclusive results regarding QoE in video-mediated group communication, for controlled experiments. In [17], Jeong and Ahn designed the QoE monitoring function for video services by extending ITU-T IPTV QoS/QoE metrics and integrating to OMA-DM protocol agent. Although the monitoring service can improve the user experience, the presented QoE factors are still not good enough. In [18], the proposed concept, Monitoring of Audio Visual Quality by Key Performance Indicators (MOAVI), is able to isolate and focus investigation, set up algorithms, increase the monitoring period, and guarantee better prediction of perceptual quality. In [19], Ickin et al. added functionality to the VideoLAN Client (VLC) media player [20] to record a set of metrics from the user interface, application-level, network-level, and the available sensors of the device. Based on these metrics, the proposed VLQoE, a QoE instrumentation for video streaming, can infer the QoE of streaming services on a smartphone.
To improve the playout quality of video streaming services, Adaptive Media Playout (AMP) schemes [21, 22] have been used. In this work, an AMP playout system is implemented into the VLC media player, which is a free and open source cross-platform multimedia player. Since the playout rate of the AMP system can be dynamically adjusted during video playback, the effects of AMP on QoE must be considered as well. However, almost all the developed QoE monitoring tools in the literature [15–19] did not take the effects of AMP on QoE into account. Thus, in [23] we proposed the QoE monitoring system that takes the effects of AMP on QoE into consideration. Additionally, the most important feature of our proposed QoE monitoring system is that it can derive an overall QoE for several QoS metrics using the product form presented in [5]. Our previous work [5] has shown that the product form performs much better than the conventional averaging techniques in deriving an overall QoE of multiple QoS metrics. Using the developed QoE monitoring program, network providers can identify network troubles and fix them in real time to improve the system performance and qualities.
The novelties and contributions of this paper are summarized as follows. First, the implemented QoE monitoring system takes the effects of AMP on QoE into consideration. Second, the developed QoE monitoring system derives an overall QoE from several QoS metrics such as the initial playout delay, packet loss rate, underflow time ratio, and normalized playout rate. Thirdly, we implement the AMP scheme, which was proposed in [22], into the VLC media player. Finally, the Group of Pictures- (GOP-) based cumulative average jitter , which is used to determine the playback threshold of the proposed AMP, is proposed to eliminate the effect of diverse frame sizes on the estimation of network jitter.
The rest of this paper is organized as follows. Section 2 describes the implementation of the AMP playout system. Implementation of the QoE monitoring system is then introduced in Section 3. Section 4 conducts several experiments and evaluates the QoE performance of video streaming services using the developed programs. Finally, the concluding remarks are given in Section 5.
2. Implementation of AMP
This section first describes the principles of AMP in Section 2.1. Then the implementation schemes, available programming tools, and some challenging issues on measurements or estimations of key parameters are addressed in Section 2.2.
2.1. Principles of Proposed AMP
In this work, we employ the freeware VLC media player to realize the AMP. The AMP is implemented according to our proposed scheme [22] whose principle is described briefly as follows. In addition to the conventional slowdown threshold , a dynamic playback threshold and a speedup threshold are designed. If the buffer fullness is below , the playout rate is reduced to avoid buffer underflow. When the buffer fullness exceeds the speedup threshold , the playout rate is increased to reduce the buffer fullness for avoiding buffer overflow. The Dynamic Playback Threshold Algorithm (DPTA) in [22] is designed to dynamically adjust the playback threshold , which affects the initial playout delay, under various network conditions. Whenever the current buffer fullness surpasses or equals , the playback starts immediately with the proper playout rate which is determined by the following equation:where is a random process and is estimated by the Arrival Process Tracking Algorithm (APTA) presented in [22]. That is, depends on the frame arrival process at the client buffer and is limited between and , as shown in Figure 1. Anyone interested in the detailed derivation of and can refer to [22]. The quadratic playout rate functions and in [22] are defined as follows:where is the uppermost threshold correlated to the buffer size at the client. Notably, the playout rate must be limited such that rate variations are unnoticeable or acceptable by users. The perception of a slowdown video is usually different from that of a speedup video for users. Therefore, two different restricted deviation ratios, denoted by and for and , respectively, are set for playout rates. The corresponding playout rate function of the proposed AMP is given by Figure 1. Numerical results in [22] have shown that the proposed AMP with quadratic playout rates performs much better than several conventional playout systems with linear playout rates and can adapt to different network load conditions for reducing the initial playout delay.
2.2. Realization of AMP
VLC media player provides a useful VLC ActiveX package whose architecture is given in Figure 2. The main component of VLC ActiveX package is LibVLC which supports several properties such as input, audio output, and video output. The input property collects all related parameters of the received video stream from the Network Interface Card (NIC) (Access), the network demultiplexer (Demux), and the video decoder (Decoder), as shown in Figure 2. Several useful statistical parameters in input.item():stats() are listed in Table 1. Based on these collected parameters from the input property, one can set and control the playout rate of the VLC media player to realize the AMP function.
According to the descriptions in Section 2.1, the playout rate mainly depends on the buffer fullness (in frames). To accurately measure the number demux_frames of demultiplexed video frames, one can directly analyze the packets arriving at the NIC card. By reading the “payload unit start indicator” bit in the header of each 188-byte MPEG-2 TS (Transport Stream) packet, as shown in Figure 3, one can easily calculate the number demux_frames of video frames received. When the “payload unit start indicator” bit equals 1, a new MPEG-2 TS video frame is arriving. Hence, the buffer fullness can be computed by the following equation:After deriving the buffer fullness , the playout rate of the AMP media player can be dynamically adjusted according to (1). The command to dynamically set the playout rate of the VLC media player is defined by the function var.set(input, “rate”, ). Although the currently developed program is designed for the MPEG-2 TS format, there is no difficulty to include other advanced video encoding technologies such as the H.264/AVC and H.265/HEVC [24] in the future.
The dynamic playback threshold [22] is determined by where and are design parameters and is the average frame interarrival time. is the cumulative average jitter and is defined bywhere is the frame interarrival time between the -st and the th frames. However, video frames are categorized into I, B, and P frames and different types of frames usually have very different frame sizes. For example, the frame size of an I frame is usually much larger than that of a B or P frame. Hence, the frame interarrival time between two successive I and B (or P) frames may become relatively large because of the long transmission time of an I frame, compared with that between two successive B and P frames. Such variance of frame interarrival times maybe is not due to the network jitter but the diversity of frame sizes. To reduce the impact of frame size variation on the estimation of network jitter and estimate more accurately, in this work the average frame interarrival time within a GOP is used. Hence, the variable in (5) is replaced by , where , is the number of frames in a GOP, and is the interarrival time between the I frames of the -st and the th GOPs.
3. Implementation of QoE Monitoring System
The procedures to implement the QoE monitoring system are described as follows. First, we propose an architecture for the QoE monitoring system. Next, several network-based QoS metrics are defined and their measurement methods are presented. Finally, the overall QoE evaluation method for the QoE monitoring system is introduced.
3.1. Architecture of QoE Monitoring System
The architecture of our proposed QoE monitoring system is plotted in Figure 4. It includes the QoS measurement engine, QoE evaluation function, and performance reporting window. The QoS measurement engine is designed to collect the QoS metrics from the media player and the NIC card. These obtained QoS metrics are used by the QoE evaluation function for producing the overall QoE of video streaming services. All instantaneous QoS metrics and QoE results are reported in real time and the curve of overall QoE versus time is also displayed with a chart in the Graphic User Interface (GUI) for the monitoring purpose.
3.2. Measurements of QoS Parameters
In the proposed QoE monitoring system, the considered QoS metrics include underflow time ratio , packet loss rate , initial playout delay , and normalized playout rate . The measurements of these parameters are described as follows.(a)Underflow time ratio : whenever an underflow event occurs, the underflow time starts to accumulate until the playback restarts again. At the same time, the number of underflow events is increased by 1. The underflow time ratio equals the ratio of accumulated underflow time to the total playback time duration. The accumulated underflow time can be computed according to the value of Status() provided by VLC Lua extension program, as shown in Figure 5. When the value of Status() changes into 2, the underflow timer turns ON until the value of Status() becomes 3. The sojourn time at the state with Status() = 2 is the underflow time. Whenever the underflow timer turns OFF, the accumulated underflow time is updated.(b)Packet loss rate : the packet loss rate is obtained by analyzing the number of receiving packets at the NIC card or using the network protocol analyzer such as Wireshark.(c)Initial playout delay : initial playout delay strongly depends on the buffering time set in VLC media player. However, in our implemented AMP this parameter is directly determined by the DPTA algorithm proposed in [22].(d)Normalized playout rate : whenever the playout rate is adjusted, the normalized playout rate is computed and saved. However, within each minute only the maximum and minimum normalized playout rates are used in our design [5]. That is, the corresponding QoE of during every minute equals the mean of these two individual QoEs resulting from the maximum and minimum normalized playout rates.
3.3. Overall QoE Evaluation Function
The overall QoE evaluation function adopts the integrated multivariate QoE function and the individual QoS-QoE mapping functions , , , and proposed in [5] to derive the overall QoE. Based on the results of [5], they are given as follows:
The scheme used in [5] consists of three steps to be done beforehand. First, several Key Performance Indicators (KPIs) or QoS metrics, such as the initial playout delay, packet loss rate, underflow time ratio, and playout rate, were selected. Then several distorted videos were generated based on these selected QoS metrics. Second, subjective tests were conducted using human observers to rate the distorted videos in terms of Mean Opinion Score (MOS) [25]. The ACR subjective test method was adopted in [5]. However, the length of each distorted video clip is about 64 s rather than 10 s. Each distorted video clip was viewed and rated by no less than 30 viewers. The title of the used video clip, which is with the frame rate 30 fps, resolution 1920 1080 p, and video codec MPEG-2, is “Avatar.” Thirdly, using the regression approach, the QoS-QoE mapping functions for individual QoS metrics were obtained. Finally, the product form of individual QoS-QoE mapping functions [5, 6] is used to derive the overall QoE of a video streaming service. Notably, although only four QoS metrics are considered in (6), the number of QoS metrics can be arbitrarily extended as needed.
4. Experimental Results
In this section, we first briefly introduce the GUIs and functions of developed AMP and QoE monitoring programs. Subsequently, using the developed QoE monitoring program, several experiments are conducted to demonstrate the performance of AMP. Finally, we use the QoE monitoring program to identify some challenging issues of wireless streaming.
4.1. QoE Monitoring System
The developed AMP is based on the VLC media player and its add-ons such as the Lua extension. Figure 6 shows the VLC media player and its Lua extension program. One can directly set the multicast IP of the streaming source and activate the AMP function on the VLC Lua extension program. Although in this paper the AMP is implemented into the VLC media player, there is no difficulty to implement it into other popular media players such as the MS Windows Media Player or YouTube Player. The QoE monitoring program, which is developed using the C++ Builder and is shown in Figure 7, evaluates and displays the overall QoE. The measured QoS and synthetic QoE/MOS are displayed in real time on the monitoring window, as shown in Figure 7. The corresponding QoEs of individual QoSs are displayed on the left side of the monitoring window while the curve of the overall QoE versus time is displayed on the right side of the monitoring window, as shown in Figure 7. By clicking the “Tools” on the menu bar of the QoE monitoring program, other detailed QoS metrics such as the interarrival time of video frames, normalized playout rate, number of packet losses, buffer fullness, and the jitter observed at the client can be displayed, as shown in Figure 8. By clicking the interested QoS parameter, such as the “MoveAvg Interarrival,” the curve of the QoS versus time is immediately displayed on the monitoring window of Figure 8.
4.2. Performance Evaluation of AMP
In order to demonstrate our AMP design and QoE monitoring system, the first experimental network scenario is constructed, as shown in Figure 9. The server pumps the video stream into the first router, followed by a bottleneck link and the second router. Finally, the disturbed video stream arrives at the receiver. All link capacities in Figure 9 are set to be 10 Mbps. The Pareto ON-OFF cross-traffic is injected into the bottleneck link using the Distributed Internet Traffic Generator (D-ITG) tool [26]. Therefore, the video stream is disturbed by the cross-traffic in the bottleneck link and thus packet losses and jitters may occur. The video is encoded into the MPEG-2 TS at the streaming server and the RTP/UDP streaming protocol is employed. Table 2 lists the detailed information of the first test video and Table 3 lists the parameters of AMP. The other parameters not mentioned here are set the same as those in [22].
First, the QoEs of two media playout systems with and without AMP are compared. The video server pumps a multicast stream into the network and two independent clients simultaneously receive the same multicast stream at the remote side. One client uses the media player with AMP, while the other one adopts the media player without AMP. The observed QoEs of these two systems with and without AMP under different cross-traffic rates 0, 4, and 8 Mbps are given in Figure 10. When the cross-traffic rate is not larger than 4 Mbps, the overall QoE is almost over 4, no matter whether the AMP is enabled or not. This is because the jitter, burstiness, and losses of video packets are very small under a low cross-traffic rate. Additionally, under a low cross-traffic rate the observed QoE on the media player without AMP remains very stable while the QoE on the media player with AMP slightly oscillates in the first minute. The reason is explained as follows. The media player without AMP can start playback only when the buffer fullness exceeds , yielding the initial playout delay about 4 s, and always plays the video with the normal playout rate. However, the media player with AMP can start playback earlier but with a lower initial playout rate. Since the initial playout rate of the media player with AMP is smaller than the normal playout rate, the observed QoE on the media player with AMP is slightly degraded in the first minute, as shown in Figure 10. Table 4 shows the measured initial playout delays of the media players with and without AMP. Because the initial playout delay in the media player with AMP is much smaller than that without AMP, the corresponding QoE of of the media player with AMP becomes better. When the cross-traffic rate becomes 8 Mbps, the video stream is seriously disturbed by the cross-traffic so that the observed QoE at the client becomes unacceptable (MOS < 3). The serious QoE degradation is mainly due to the increase of the number of underflow events and underflow time. However, the observed QoE on the media player with AMP is still much better than that without AMP, as shown in Figure 10.
The corresponding buffer fullness and the playout rate at the client under the cross-traffic rate 6 Mbps are shown in Figure 11. According to Figure 11, the buffer fullness of the system with AMP is more stable than that of the system without AMP. For the media player with AMP, the underflow events never happen because the buffer fullness never drops to an unacceptable low level after the playback. Although the playout rate of the media player with AMP varies, the deviation of the playout rate is never over 25% so that the QoE degradation resulting from it is still acceptable. However, for the media player without AMP, the buffer fullness during the time interval (10 s, 66 s) decreases almost linearly to zero and leads to an underflow event, yielding a significant QoE degradation. All results in Figures 10 and 11 demonstrate that the AMP significantly improves the QoE of streaming services.
4.3. Performance Monitoring for Wireless Streaming
Video streaming over wireless networks has become very popular recently. However, it is still challenging to offer streaming services over wireless networks. To clearly identify the challenging issues of wireless streaming services, in this subsection the AMP function is not enabled in the media players of all receivers. To evaluate the QoE of wireless streaming services, the second network scenario shown in Figure 12 is considered and the first test video given in Table 2 is used. The video stream is multicast to two independent receivers at the last hop: via the IEEE 802.11n wireless link to Receiver 1 and via the wired link to Receiver 2. Receiver 1 is away from the WiFi AP by 2 m and no obstacle between them. Figure 13 shows the observed synthetic MOSs at Receivers 1 and 2 under different cross-traffic rates, using the QoE monitor program developed in this paper. By Figure 13, the QoE at Receiver 2 using a wired link is better than that at Receiver 1 using a wireless link. Even though no cross-traffic exists, the MOS of the video streaming service at Receiver 1 via the wireless link is less than 4. According to the QoS parameters measured at Receivers 1 and 2, the difference of QoE between them is mainly due to a higher packet loss rate in the wireless link.
Subsequently, to compare the performance of multicast and unicast, the network scenario 3 shown in Figure 14 is considered. The second video clip with the resolution 720 480 p and bit rate 1936 kbps is used, as shown in Table 5. Different numbers of similar unicast streams are pumped from the video server to the receivers using a wireless link at the last hop. The synthetic MOS of the streaming service at the receivers decreases significantly as the number of unicast streams increases, as shown in Figure 15. This is because the jitter, burstiness, and packet loss rate increase as the number of unicast streams in the network grows, yielding the QoE degradation at the receivers. Note that if the multicast scheme is used, only one video stream is required to be pumped into the network no matter how many receivers exist. That is, the QoE of video streaming services is irrelevant to the number of receivers if the multicast scheme is employed. Obviously, under the scenario that multiple users watch the same video stream, the multicast scheme performs much better than the unicast one in terms of QoE.
Finally, to investigate the impact of transmission distance and obstacles on the QoE of wireless streaming services, the wireless network scenario 4 plotted in Figure 16 is considered and the second test video given in Table 5 is used. The video stream is multicast to several wireless receivers at different positions in our labs. No cross-traffic is injected into the network. No obstacles exist between the wireless AP and the receivers at Locations 1 and 2. However, the receiver at Location 1 is much closer to the AP than the receiver at Location 2 is. There are cupboards and a wall between the AP and the receiver at Location 3. The receiver at Location 4 is outside the door of our lab and is isolated from the AP by a wall. The other receiver at Location 5 is also isolated from the AP by a wall and cupboards. All receivers receive the same multicast stream simultaneously and their individual MOSs are estimated using the developed QoE monitoring program. The results are shown in Figure 17, where the MOSs of the receivers located at Locations 1 and 2 are the best because no obstacle blocks the wireless signals for them, while the MOSs of the receivers located at Locations 4 and 5 are the worst because there exist cupboards and a wall between them and the AP. The MOS of the receiver located at Location 3 is better than those located at Locations 4 and 5 because the receiver at Location 3 is much closer to the AP than those at Locations 4 and 5 are. Accordingly, the distance and the number of obstacles between the WiFi AP and the receivers strongly affect the QoE of wireless streaming services.
5. Conclusions
In this work, an AMP and a real-time QoE monitoring system for video streaming services are realized. The AMP design is implemented into the freeware VLC media player. The QoE monitoring system can measure several key QoS metrics such as the packet loss rate, underflow time ratio, initial playout delay, and the playout rates of a video streaming service and derive the overall QoE in real time. Several experiments are conducted in various scenarios including wired and wireless networks. All numerical results demonstrate that the QoE monitoring system works well and the proposed AMP significantly improves the QoE of video streaming services. Using the real-time QoE monitoring system, network/content providers can find the network troubles and resolve them timely to maintain good QoEs for users. In the future, more advanced video codecs such as H.264/AVC and H.265/HEVC will be implemented into the developed system.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work was supported by Ministry of Science and Technology of Taiwan under Grant NSC101-2221-E-182-004 and Grant MOST106-2221-E-182-012-MY2.