Abstract
To improve the playout quality of video streaming services, several adaptive media playout (AMP) mechanisms were proposed in literature. However, all performance evaluations and comparisons for AMPs were made in terms of quality of service (QoS) metrics. As one knows, there may exist a trade-off between QoS metrics, such as buffer underflow and overflow performance. Thus, it is not sufficient to only evaluate the performance of AMPs in terms of QoS metrics. In this paper, we will evaluate and compare the performance of several AMPs from the aspect of quality of experience (QoE). Numerical results will show that some existing AMP systems do not perform better than the nonadaptive playout system from the point of view of overall QoE.
1. Introduction
Recently, multimedia streaming applications such as IPTV [1] have been increasing rapidly due to the significant growth of bandwidth in access networks, such as xDSL, FTTH, 3G/4G, and WiMAX. However, due to the random delay/jitter encountered in Internet, at the client side the video playout interruption, block distortions, and nonpredictive preroll time may occur during a playout session. To counteract the effects of network jitter on the quality of video streaming, adaptive media playout (AMP) techniques, which can control the media playout rate dynamically, have emerged [2–10]. Some AMP schemes, such as [2–7], adjust the media playout rate dynamically according to the buffer fullness. Another AMP based on buffer variation rather than buffer fullness was proposed in [8]. The other content-aware AMPs that take into account the content of a video sequence and motion characteristics of different scenes were presented in [9, 10]. The content-aware AMP only slows down the low-motion scenes such that the perceived effect is lower.
The quality of service (QoS) refers to several related aspects of telephony and computer networks that allow the transport of traffic with special requirements. QoS metrics belong to quantitative metrics that can be measured objectively by using network equipment. Therefore, they are usually called objective QoS metrics. Several objective QoS metrics, such as underflow probability, overflow probability, variance of distortion of playout (VDoP) [3, 7], initial playout delay, playout curve, and mean playout rate have been used for evaluating the performance of AMP mechanisms. Detailed definitions of most of these objective metrics can be found in [7]. Several works had shown that there may exist a trade-off among these metrics mentioned above [4, 7]. For example, there exists a trade-off between underflow and overflow performance. Thus, it is sometimes difficult to definitely judge whether an AMP scheme is better than the other one in terms of QoS metrics. Most importantly, these objective QoS metrics cannot directly reflect the users’ perception on the playout quality of an AMP system. Accordingly, it is necessary to develop another effective method for evaluating the performance of AMP schemes from the aspect of users’ perception.
The users’ perception of a service is expressed by the quality of experience (QoE) which is a subjective measure of a customer’s experiences with a service. Generally, the schemes of assessing the user’s perception are called QoE techniques [11–18] and measured in terms of MOS (mean opinion score) [19]. The MOS is expressed as a single number in the range 1 to 5, where 1 is the lowest perceived quality and 5 is the highest perceived quality. However, the average MOS of measurements can range from 0 to 5 in real applications. QoE assessment includes three different approaches: subjective, objective, and hybrid. Subjective QoE assessment is the most accurate method to get the user’s perception. However, subjective approach is time consuming and has a high evaluation cost. Objective QoE assessment [15, 16] may be performed in real time but the accuracy of the user’s perception could be a problem. As to the hybrid approach, it combines the subjective and the objective methods to obtain the QoE [13, 14]. The hybrid scheme can assess the QoE in real time while reducing the cost in the subjective test. However, all the QoE evaluation methods mentioned above do not consider the effects of AMP schemes. Therefore, a real time and cost-effective QoE assessment scheme which takes the impact of AMP into account was proposed in [20]. In [20], several QoS-QoE mapping functions for individual QoS metrics were derived based on subjective tests. The considered QoS metrics include the initial playout delay, packet loss rate, underflow time ratio, and playout rate. These objective QoS metrics can be obtained by using network measurement equipment. In addition, in [20] an integrated multivariate QoS-QoE mapping function was presented to evaluate the overall QoE of a video streaming service. In this paper, the QoE evaluation method proposed in [20] will be utilized for assessing the QoE performance of AMP systems.
The rest of this paper is organized as follows. Section 2 gives the overview of AMP schemes. In Section 3, the QoE evaluation method in [20] will be summarized. Section 4 evaluates the QoE performance of various AMP systems. Finally, Section 5 makes the concluding remarks.
2. Overview of Adaptive Media Playout Schemes
In [2, 3], several different playout rates are allowed for the AMP system to compensate for the effects of network delay jitters and reduce underflow events. If the number of buffered packets exceeds a given threshold, the AMP system uses a normal playout rate . Otherwise, the AMP system employs proportionally reduced rates. We call such an AMP scheme the linear slowdown scheme. Its playout rate function is depicted in Figure 1. Another novel AMP based on buffer variation for determining the playout rate was proposed in [8]. The scheme proposed in [8] can keep the video playout as smooth as possible while adapting to the network conditions. The playout rate adjustment is based on the buffer variation (in frames) and the parameter . When , a playout adjustment order is issued and the playout interval is adjusted linearly with time. For the AMPs mentioned above [2, 3, 8], the playout rates cannot be faster than the normal playout rate . When only slowing down the playout rate is allowed, the playout latency of the whole video stream may be extended significantly. In addition, slowing down the playout rate may increase the overflow probability when the client buffer size is finite. To eliminate the additional playout latency and reduce the overflow probability, speeding up the playout rate is possible in [4, 6]. The paper [6] proposed an algorithm to dynamically calculate a slow or a fast playout factor based on the current buffer state, target buffer level, past history of media data reception, estimate of future data arrival, and the estimated network conditions.
Most AMP schemes do not take the buffer overflow issue into consideration. To improve this drawback, the AMP with multiple thresholds, the slowdown threshold , the speedup threshold , and the dynamic playback threshold , were presented in [7]. In addition to the conventional slowdown threshold , a dynamic playback threshold and a speedup threshold were designed. Whenever the buffer fullness exceeds the speedup threshold , the playout rate must increase to reduce the buffer fullness for avoiding buffer overflow. The dynamic playback threshold algorithm (DPTA) in [7] is designed for dynamically adjusting the playback threshold , which determines the initial playout delay under various network conditions. The proposed AMP in [7] is called DPTA+APTA and is summarized as follows. DPTA estimates the network jitter from the most recent frame arrivals to decide the proper playout delay. First, , the cumulative average jitter of the first arrivals during a preroll period, is updated at each frame arrival as follows: where is the frame interarrival time between the th and the st video frames and the time is the normal playout duration of each video frame. Then the dynamic playback threshold is computed according to the following piecewise linear equation where and are design parameters. According to our experiments in [7], there exist larger fluctuations for at smaller values of . Thus, to increase the estimation reliability of , a lower bound is set for so that the false start of playback can disappear at . Also, the value of must not be set too large, or the initial playout delay cannot be reduced. According to our experiments in [7], is an appropriate setting. The parameter relates to linearly. Based on our experiments in [7], the value of should be set within the range of 1.5 to 2.5 so that the initial playout delay can be acceptable for most cases.
Whenever the current buffer fullness surpasses or equals , the playback can start immediately with the proper playout rate which is determined by the following equation where is a random process and is determined by the arrival process tracking algorithm (APTA) presented in [7]. That is, is dependent on the frame arrival process at the client buffer and is limited between and , as shown in Figure 2. Anyone interested in the detailed derivation of can refer to [7]. and are defined as follows: where is the uppermost threshold correlated to the client buffer size. Notably, the playout rates should be limited such that rate variations are unnoticeable or acceptable by users. And the perception of a slowdown video is usually different from that of a speedup video for users. Therefore, two different restricted deviation ratios, denoted by and for and , respectively, are set for playout rates. The corresponding playout rate function of DPTA+APTA is given by Figure 2.
3. QoE Evaluation Method for AMP Systems
According to the experiences on video streaming applications (e.g., YouTube videos) over Internet, nonpredictive buffering time before playout, playback interruptions due to buffer underflow, and video block distortions owing to packet losses are frequently encountered phenomenons during watching a video streaming session. Hence, in this work, the objective QoS metrics, such as the initial playout delay, loss rate, and buffer underflow time, are selected for assessing the QoE of video streaming services. Additionally, if the AMP scheme that can dynamically adjust the playout rate is employed in the media player for improving the video playout quality, the impact of AMP on the QoE must also be considered. Thus, the playout rate metric must be included in the QoE evaluation as well. The definitions and illustrations of these aforementioned QoS metrics are given as follows.(i)Initial playout delay: in order to compensate for the effects of network jitters, a playout buffer is usually allocated at the client. For alleviating the jitter effect on the video playout quality, a streaming video can start playback only when the buffer fullness exceeds a certain threshold. For a video stream, at the start of streaming, the time duration between the first frame arrival at the client buffer and its displaying is defined as the initial playout delay. Nonpredictive or long initial playout delay is annoying to users.(ii)Loss rate: video packets may be discarded in the network or at the client buffer due to buffer overflow. Such packet losses can lead to block distortions in the video playout and degrade the users’ perception.(iii)Underflow time ratio: the case that no frame exists in the client buffer for display is called buffer underflow. The occurrence of buffer underflow will intermit the video playout so that the streaming quality is seriously degraded. Therefore, the incidence of buffer underflow is a key metric for evaluating the quality of video streaming services. The ratio of total underflow duration to the overall playback time of a video streaming is defined to be the underflow time ratio.(iv)Playout rate: the playout rate may vary in the media player with AMP. When the playout rate variation is over a threshold, such as 25%, it may be easily perceived by users. So the effects of playout rate variation on the QoE must also be considered and evaluated.
The QoS parameters mentioned above can be obtained by using network measurement equipment. How to assess the overall QoE of a video streaming service from these measured QoS parameters, however, becomes an issue. To solve this problem, in [20], a product form is utilized to create an integrated multivariate QoE function for evaluating the overall QoE of a video streaming service. According to our previous work [20], the overall QoE of a video streaming service can be assessed based on these four QoS metrics mentioned above as follows: where , , , and represent the underflow time ratio , loss rate ( %), initial playout delay ( seconds), and normalized playout rate () metrics, respectively. In other words, the normalized integrated QoE or MOS, , equals the product of all normalized QoEs of individual QoS metrics. Note that since the QoE ranges from 0 to 5, the normalized QoE must be divided by 5. The individual QoS-QoE mapping functions in (5) are given as follows: All equations (6) to (9) are derived based on subjective tests conducted in [20] and the nonlinear regression approach. The resulting mapping functions for the underflow time ratio, loss rate, and initial playout delay metrics conform to exponentially decaying functions, while the mapping function for the normalized playout rate metric follows the probability density function of Gamma distribution. Since the maximum value of QoE is 5, the maximum values of individual QoEs in (6) to (9) cannot be larger than 5 for all . Thus, in (6) to (9) the coefficients before the exponential functions must be set to 5. Other constants in these mapping functions are determined by using the nonlinear regression approach. Related subjective tests and detailed derivations of these QoS-QoE mapping functions can be found in [20]. Next, (5) will be used for evaluating the integrated QoE performance of an AMP system.
4. QoE Assessment Results of AMP Systems
In this section, we will use simulations to obtain the QoS performance of different AMP mechanisms. Next, (5) will be employed to compute the QoEs of different media playout systems, including nonadaptive, linear slowdown [2, 3], linear slowdown+speedup, DPTA+APTA [7], and buffer variation [8] schemes. These playout systems are described briefly as follows:(i)Nonadaptive: the normal playout rate is always employed by the media player.(ii)Linear slowdown: it is similar to the scheme presented in [2, 3]. However, the media player will reduce the playout rate linearly between and only when the buffer fullness is less than the threshold , as shown in Figure 3(a).(iii)Linear slowdown+speedup: this AMP will slow down or speed up the playout rate linearly when the buffer fullness is below or over , respectively. Its playout rate function is depicted in Figure 3(b).(iv)DPTA+APTA: it is the AMP presented in [7] with parameter and has the playout rate function shown in Figure 2.(v)Buffer variation scheme: it is the AMP scheme presented in [8]. In our simulations, the adjustment algorithm for this scheme follows the method described in [8] except that the initial buffer reference level is set to rather than half of the buffer size. In addition, after an underflow event, the playout can resume only when the buffer fullness is accumulated up to again. The parameter defined in [8] equals 12 in our simulations.
(a) Linear slowdown
(b) Linear slowdown + speedup
Notably, all the startup playback thresholds of the above media playout systems except DPTA+APTA () are fixed at . Additionally, the playout rates are limited between and for all considered AMP mechanisms, as shown in Figures 2 and 3.
First, we adopt NS2 network simulator [21] to generate jittered video traffic patterns. Then the jittered video traffic patterns are employed as the arrival process of various media playout systems which are implemented using the C++ programs developed by ourselves. The network architecture for simulations is shown in Figure 4. In our simulations, we use the MPEG-4 encoded video, Jurassic Park I [22], as the video streaming source. This MPEG-4 video is VBR and has average bit rate 770 kbps, and its total playback time is one hour. The frame rate of this MPEG-4 video is 25 fps. Thus, the normal playout duration of each frame, , equals 40 ms. Moreover, the average frame size of this video is about 3800 bytes. Since the encoded frame size is variable in reality, the buffer size in our simulations will be set in number of bytes. But the playout rate adaptation is still based on the number of frames. In our simulations, the client buffer size is set to 2 Mbytes. Since the average frame length is close to 4000 bytes, the corresponding uppermost threshold in Figure 2 is set to about 500. All video frames are transmitted from the server (node 0) to the receiver (node 5) and are jittered by the cross-traffic when they traverse the network. The Pareto ON-OFF traffic type with parameters (burst time, idle time) = (400 ms, 600 ms) is chosen as the cross-traffic. In all our simulations, the cross-traffic rates of all links in Figure 4 are set to be equal. At the receiver node, all the frame arrival times of the simulated video stream are saved in a file to be the video frame arrival process for media playout systems. Various cross-traffic loads are conducted in our simulations. And under each traffic load, 10 distinct jittered video traffic patterns are produced by starting streaming at different times. In addition, no frame loss in the network is assumed in our simulations, that is, in this study the frame losses are only induced by the buffer overflow at the client buffer. Related parameter settings in our simulations are listed in Table 1.
Subsequently, the QoS parameters, including initial playout delay , underflow time ratio , loss rate , and normalized playout rate of each media playout system are measured. The measurements of these QoS values are similar to those in [7]. After obtaining these QoS parameters, we apply the QoE mapping functions in (6)–(9) to transform each QoS value into its corresponding QoE/MOS. The mapped QoE/MOS of each playout system in the individual QoS metric is shown in Figure 5. Finally, using (5) one can find the overall QoE of each media playout system. The results are presented in Figure 6.
| (a) MOS of |
| (b) MOS of |
| (c) MOS of |
| (d) MOS of |
According to Figure 5, if only the QoE of an individual QoS metric is compared among different media playout systems, then it is very difficult to judge which playout scheme is the best. This is because there exists a trade-off among different QoS metrics. Therefore, it is not feasible to only use QoS values for performance comparisons among various playout systems. However, when the overall QoEs of different media playout systems are derived by using (5), as shown in Figure 6, performance comparisons among various playout systems become easy.
From Figure 5, we find that the linear slowdown AMP can only improve the QoE in underflow time ratio metric , while it degrades the QoE in normalized playout rate metric when compared with the nonadaptive playout system. And the overall QoE of the linear slowdown AMP, as shown in Figure 6, is nearly improved as compared to the nonadaptive playout system. As to the buffer variation AMP, the QoE performance in underflow time ratio metric is the best, while the QoE performance in loss rate metric becomes the worst, as shown in Figure 5. The overall QoE of the buffer variation AMP, as presented in Figure 6, becomes even worse than that of the nonadaptive playout system. However, in Figure 6 DPTA+APTA can significantly improve the overall QoE of a streaming service under various traffic load conditions. According to Figure 5, since DPTA algorithm can adapt the initial playout delay to network traffic conditions, DPTA+APTA achieves the best QoE in initial playout delay metric . Although the QoE of DPTA+APTA in normalized playout rate metric is the worst, DPTA+APTA significantly improves the QoEs in underflow time ratio and loss rate metrics. This is because APTA can adjust the playout rate based on the frame arrival process to stabilize the buffer fullness so that buffer underflow and overflow probabilities are significantly reduced. Finally, DPTA+APTA achieves the best overall QoE. Therefore, it is worthy for DPTA+APTA to effectively adapt the playout rate for improving the QoE of a video streaming service.
5. Conclusions
This study introduces a QoE-based evaluation method for AMP systems. The overall QoE of an AMP system can be computed according to the proposed integrated QoE function with multivariate QoS metrics. The obtained QoE can indicate the performance of an AMP system from the aspect of users’ perception. Thus, the knotty trade-off problem among various QoS metrics is resolved in the performance comparison of AMP systems. Numerical results show that some AMP mechanisms, such as linear slowdown and buffer variation schemes, do not perform better than the nonadaptive playout system from the viewpoint of users’ perception. But DPTA+APTA scheme can really improve the QoE of a video streaming service. In addition to the performance evaluation of AMP systems, we believe that the presented QoE-based evaluation approach can also be applied to the performance assessment of other multimedia service systems.
Acknowledgments
This work was supported by National Science Council of Republic of China under Grant NSC101-2221-E-182-004. The authors would also like to thank all reviewers for their valuable comments and suggestions.