Low-Complexity Saliency Detection Algorithm for Fast Perceptual Video Coding

Abstract

A low-complexity saliency detection algorithm for perceptual video coding is proposed; low-level encoding information is adopted as the characteristics of visual perception analysis. Firstly, this algorithm employs motion vector (MV) to extract temporal saliency region through fast MV noise filtering and translational MV checking procedure. Secondly, spatial saliency region is detected based on optimal prediction mode distributions in I-frame and P-frame. Then, it combines the spatiotemporal saliency detection results to define the video region of interest (VROI). The simulation results validate that the proposed algorithm can avoid a large amount of computation work in the visual perception characteristics analysis processing compared with other existing algorithms; it also has better performance in saliency detection for videos and can realize fast saliency detection. It can be used as a part of the video standard codec at medium-to-low bit-rates or combined with other algorithms in fast video coding.

1. Introduction

With the rapid developments of multimedia information processing and communication technology, video encoding has become the basic core technology of digital television, video conferencing, mobile media, 3D video coding, and so forth. During the past decades, in order to obtain video codec with low complexity, high quality, and high compression ratio, various technologies have been proposed for fast video coding [1].

Studies have shown that human visual system (HVS) is sensitive to the video scene perception and assigns different visual importance to different regions [2, 3]. Researchers have used advantages of visual attention in various multimedia processing applications such as image retargeting and video coding; the researching of saliency detection model for perceptual video coding is a hot topic. One of the key processing steps is to perform low-complexity calculations and obtain region of interest (ROI) in accordance with visual perception characteristics timely and effectively.

Up to now, saliency detection algorithms are widely used in extracting ROI in videos for various multimedia processing applications [4–8]. Moving zone detection technique for pixel precision is able to detect the moving foreground area, but its complexity in calculation makes it not applicable in real-time encoding. Liu et al., [6] proposed the moving zone detection algorithm, but the algorithm mainly employs motion vector information, making it even not applicable in effective detection on moving object in global motion zone. Yuming et al., in [7] a video saliency detection algorithm based on feature contrast is proposed, but the computational efficiency needs to be further improved.

Those saliency detection algorithms mentioned previously are facing a common problem: they are not only time-consuming but computation-consuming as well. Those algorithms do not pay more consideration to the effect of the real-time coding performance because of the additional computation for visual perception analysis. Complex saliency detection algorithm will increase the computational burden of video encoder, which is not conducive to the video coding standards in real-time multimedia communication application.

In this paper, a fast saliency detection algorithm based on low-level encoding information and HVS is proposed. In order to simplify the visual perception analysis progress, this algorithm correlates the encoding information in video bit-stream with the visual perception characteristics. The spatial and temporal saliency detection is carried out by means of MV. The prediction modes and other auxiliary coding information can save an amount of computing time in feature extraction for saliency detection. As almost no additional computation is increased to the video codec, the saliency detection computation complexity is lower compared with other existing algorithms. The saliency detection results are satisfied with the compared algorithms. So the proposed algorithm can reach the balance between the saliency detection accuracy and computational complexity.

This paper is organized as follows. Section 2 is an overview of the proposed framework and describes in detail each one of its subsystems. Section 3 gives the evaluation of the proposed algorithm, and in Section 4, some important conclusions are obtained and the further work is also introduced.

2. The Proposed Saliency Detection Algorithm

Motion is a highly salient feature which grabs one’s attention and keeps it locked on important features and objects. Interest in motion perception has a long history and it can be considered as a relatively well established discipline [9]. Motion perception is one of the most important visual processing mechanisms. The visual information related to temporal motion would generate stronger response in HVS. HVS always pays more attention to the objects with smooth movement (as shown in Figure 1). Spatial contrast is the most basic visual processing mechanism in HVS; it is the prerequisite for HVS perception of spatial shape such as texture and object (as shown in Figure 2).

Figure 1 

Tracking smooth movement targets (coastguard).

Figure 2 

Interest region of foreground goal (Suize).

The response intensity of temporal motion visual information caused by HVS is larger compared with that of spatial motion visual information caused by HVS [10]. In the proposed algorithm, temporal saliency detection is performed firstly, and then spatial saliency detection is adopted in order to optimize the visual perception characteristics analysis results.

2.1. Temporal Saliency Analysis and Detection

As we know, moving objects always have larger MV in video frame. It can be found in Figure 3 that the coding regions with larger MV happen to be the ROI (such as head, face, shoulders, and arms); the coding regions with smaller MV or zero MV are always in the static background which could only arouse lower attention of HVS. To sum up, as a relatively high consistency exists between MV and visual attention, MV can be regarded as the temporal characteristics of visual perception. Many existing algorithms are used in order to get the motion feature for the motion saliency detection. These algorithms are only effective for videos with a static background.

(a) Silent

(b) Salesman

(c) Paris

Figure 3 

MV distribution in static background video frames.

In an ideal case, foreground moving object can generate nonzero MV. As there are no relative movements in the background region, it should produce zero MV in static background. However, in reality, nonzero MV noise would be generated randomly due to external change of illumination and internal change of video encoding parameters (such as change of quantization steps, motion search scheme, and rate distortion optimization algorithm). As a result, corresponding MV detection mechanism should be proposed to get rid of the interference of MV noise.

Besides the MV noise interfering in temporal saliency detection, the translation MV interfering from the background should be considered. As shown in Figure 4, in Foreman sequence, the foreground object is the human’s head and shoulder. However, due to the rightward rotation of the camera, buildings on the right-hand side are moved into the scene, creating a globally distributed MV. In the same way, the horizontal displacement of camera causes obvious MV along the road side and the parked car in bus sequence. However, HVS is only interested in the moving bus on the road. Analogously, in Stefan sequence, the most interesting object is the tennis ball athlete which generated a lot of motion vector. But due to the moving of camera, there are amount of MV appeared on bleachers as well. Under these conditions with horizontal motion, the information of MV does not match the visual attention. Motion detection which is merely based on the size of MV can lead to errors in the temporal saliency judgment. As a result, corresponding MV detection mechanism needs to be formulated as well, in order to remove the interference caused by MV errors which are generated due to horizontal movement.

Foreman

Bus

Stefan

Figure 4 

MV distribution in translation background video frames.

In order to improve the temporal detection accuracy, the MV noise filtering and the attenuating translation MV interference error should be added. At the same time, the complexity of processing procedure should be controlled strictly. Otherwise it will influence the real-time performance of the saliency detection algorithm.

2.1.1. Filtering MV Noise

(1) Basic Principle of MV Noise Filtering. Compression coding uses the correlation between adjacent macro blocks to remove redundant information on spatial domain or temporal domain, in order to achieve a large amount of information with a small number of bits. In video coding framework based on blocks, the coded object is usually divided into several macro blocks or subblocks; then the macroblocks or subblocks that belong to an object should tend to have a similar motion vector or predictive coding mode and have similarity in structure [11]. Research shows that the moving objects in video sequence have the motion continuity and integrity. Motion continuity is reflected in there being a strong correlation between macro blocks in the current frame and the previous frame with corresponding position. Motion integrity reflects there existing great structural similarity between adjacent macro blocks in the same frame.

Therefore, based on the video sequence motion continuity and high temporal correlation, the MV of the macro block (MB) with the same position in the previous frame can provide very important prior information for the current MV of encoding MB.

In Figure 5, 𝑠^𝑡_(𝑥,𝑦) represents the current coded MB, 𝑡 represents the current encoded video frame, and (𝑥,𝑦) is the position coordinates. 𝑉_𝑠 is the MV of 𝑠^𝑡_(𝑥,𝑦). 𝑐^𝑡−1_(𝑥,𝑦) represents the MB in the previous frame with the same position coordinates as current coded MB and 𝑉_𝑐 is the MV of 𝑐^𝑡−1_(𝑥,𝑦).

Figure 5 

Schematic diagram of MBs with corresponding position in adjoining frames.

Through using large amounts of test sequences and statistics based on the H.264/AVC standard (JM18.7), it can be found that 𝑉_𝑠 and 𝑉_𝑐 have a high correlation.

Take Akiyo sequence and Foreman sequence as representations for gentle motion and active motion of the two kinds of video sequences; the quantization parameter (QP) is set as 28 and 32, respectively, through using full search prediction method, statistics the joint probability 𝑝(𝑉_𝑠∣𝑉_𝑐) of 𝑉_𝑠 and 𝑉_𝑐. The statistical results are shown in Table 1.

Table 1 

Probability of MV with the same position.

From Table 1 statistics data, it can be found that if 𝑉_𝑐=0, the probability of 𝑉_𝑠=0 is more than 60%. If 𝑉_𝑐≠0, the probability of 𝑉_𝑠≠0 and belong to the 𝑉_𝑐 (1±10%) is nearly 80% for gentle motion sequence and more than 65% for active motion sequence. If 𝑉_𝑐≠0, the probability of 𝑉_𝑠=0 is less than 20%.

The simulation results show that 𝑉_𝑐 can be taken as an important basis for determining 𝑉_𝑠 being MV noise or not. As there exists strong motion continuity and relativity of the moving object in a video sequence, in order to reduce the error judgment rate, in this paper, based on the average MV of reference region in previous frame, the basic principle of MV noise filtering is proposed as follows.

If 𝑉_𝑠 is generated in the current encoding MB 𝑠^𝑡_(𝑥,𝑦), there is a high probability that a MV with similar direction and size exists in the reference region 𝐶_rr of the corresponding position in the previous frame. If there is no MV in 𝐶_rr, 𝑉_𝑠 should be treated as MV noise and be filtered out. So how to determine the reference area 𝐶_rr is the key factor affecting the MV filtering results.

(2) Define Reference Region 𝐶_𝑟𝑟. As shown in Figure 6, 𝑐^𝑡−1_(𝑥,𝑦) is the MB which has the same position coordinates as MB 𝑠^𝑡_(𝑥,𝑦) in the previous frame. The rectangular area surrounded by dashed lines is defined as reference region 𝐶_rr.

Figure 6 

Schematic diagram of position relationship.

According to the direction of 𝑉_𝑠 (horizontal motion, vertical motion, and oblique motion), 𝐶_rr is determined as follows.

(i) 𝐶_𝑟𝑟 with Horizontal Motion. As shown in Figure 7, if one takes horizontal motion towards the right direction of 𝑉_𝑠 as an example, in the previous reference frame 𝐹_𝑡−1, find MB-1, which has the same position coordinate with the current encoding MB signed 𝑐(𝑥,𝑦).

Figure 7 

Schematic diagram of 𝐶rr with horizontal motion.

Firstly, take MB-1 as the starting point, then perform horizontal motion of 𝑖_𝑐 macro blocks in the opposite direction of 𝑣_𝑠, and get MB-2 signed 𝑐(𝑥,𝑦−𝑖_𝑐).

Secondly, centered at MB-2, make vertical extension of 𝑖_𝑐 macro blocks both upwards and downwards to obtain two vertical vertices, that is, MB-3 signed 𝑐(𝑥−𝑖_𝑐,𝑦−𝑖_𝑐) and MB-4 signed 𝑐(𝑥+𝑖_𝑐,𝑦−𝑖_𝑐).

Last, determine the rectangular reference region ① and 𝐶_rr is designated by four macro blocks which are MB-3, MB-4, MB-5, and MB-6.

If 𝑉_𝑠 is a horizontal motion towards the left direction, use the same method to determine the rectangular reference region ②; 𝐶_rr is designated and surrounded by four macro blocks which are MB-5, MB-6, MB-7, and MB-8.

In the above description, the position coordinates of MB-3 to MB-8 are given in Table 2.

Table 2 

Position coordinates of vertex macroblocks with horizontal motion.

In Table 2, 𝑖_𝑐 is defined as 𝑖_𝑐=𝑉_𝑠𝑥𝑤_𝑠+1. (1)|𝑉_𝑠𝑥| denotes the MV magnitude of the horizontal direction 𝑉_𝑠. 𝑤_𝑠 denotes the width of the current coding block. [⋅] represents the round numbers calculation.

(ii) 𝐶_𝑟𝑟  with Vertical Motion. If 𝑉_𝑠 make the vertical downward motion, the processing steps to determine reference region ③𝐶_rr are as follows.

Firstly, take MB-1 as the starting point, then perform vertical motion of 𝑗_𝑐 macro blocks in the opposite direction of 𝑉_𝑠 and get MB-2 signed 𝑐(𝑥−𝑗_𝑐,𝑦).

Secondly, centered at MB-2, make horizontal extension of 𝑗_𝑐 macro blocks both leftwards and rightwards to obtain two horizontal vertices: MB-3 signed 𝑐(𝑥−𝑗_𝑐,𝑦−𝑗_𝑐) and MB-4 signed 𝑐(𝑥−𝑗_𝑐,𝑦+𝑗_𝑐).

Last, determine the rectangular reference region ③; 𝐶_rr is designated and surrounded by four macro blocks which are MB-3, MB-4, MB-5, and MB-6 (as shown in Figure 8).

Figure 8 

Schematic diagram of 𝐶rr with vertical motion.

If 𝑉_𝑠 make vertical upward motion, use the same method to determine the rectangular reference region ④; 𝐶_rr is designated and surrounded by four macro blocks which are MB-5, MB-6, MB-7, and MB-8.

In the above description, the position coordinates of MB-3 to MB-8 are given in Table 3.

Table 3 

Position coordinates of vertex macroblock with vertical motion.

In Table 3, 𝑗_𝑐 is defined as 𝑗_𝑐=𝑉_𝑠𝑦ℎ_𝑠+1. (2)|𝑉_𝑠𝑦| denotes the MV magnitude of the vertical direction 𝑣_𝑠. ℎ_𝑠 denotes the height of the current coding block. [⋅] represent the round numbers calculation.

(iii) 𝐶_𝑟𝑟  with Oblique Motion. If 𝑉_𝑠 make the oblique motion, determine the reference regions of ⑤𝐶_rr, ⑥𝐶_rr, ⑦𝐶_rr, and ⑧𝐶_rr in the same way according to the different motion directions of 𝑉_𝑠. 𝐶_rr is given in Figure 9.

Figure 9 

Schematic diagram of 𝐶rr with oblique motion.

The position coordinates of MB-1 to MB-9 are given in Table 4.

Table 4 

Position coordinates of vertex macroblock with oblique motion.

In Table 4, 𝑖_𝑐 and 𝑗_𝑐 are defined as formulas (1) and (2).

In the proposed algorithm, reference region 𝐶_rr is not stationary, and the area and position of 𝐶_rr are changed adaptively according to the size and direction of 𝑉_𝑠.

(3) The MV Noise Filtering Computational Formula. Consider 𝑇₁𝑥,𝑦,MV=3,if𝑉_rr𝑉=02,elseif_𝑠≥𝑉_rr𝑇₂𝑥,𝑦,MVelse. (3)

In formula (3), 𝑉_rr is the averaged MV in 𝐶_rr; it is defined as 𝑉_rr=∑_∈𝐶rr𝑣_rrnum_𝐶rr. (4) Here, 𝑣_rr is the MV of MB in 𝐶_rr, num_𝐶rr is the summation times. (𝑥,𝑦) denotes the position coordinates of current encoding MB.

If |𝑉_rr|=0, consider 𝑉_𝑠 is caused by MV noise and should be filtered out. 𝑉_𝑠 is set as 0 and mark the current encoding block as 3, 𝑇₁(𝑥,𝑦,MV)=3.

If |𝑉_𝑠|≥|𝑉_rr|, there exist obvious motion characteristics in the current encoding MB compared with the MBs in 𝐶_rr, and the current encoding block belongs to dynamic foreground region which should be marked as 2, 𝑇₁(𝑥,𝑦,MV)=2.

Else, it means the current encoding MB has similar motion characteristics as its nearby MBs in 𝐶_rr, and the current encoding block’s temporal saliency characteristics are undetermined. So the translational MV checking should be carried out further in order to distinguish whether current encoding MB belongs to background region or foreground translation region.

2.1.2. Translational MV Checking

After MV noise filtering procedure, the translation MV interference attenuating step comes into consideration.

(1) Basic Principle of Translational MV Checking. In video coding based on the block matching, first step is to get the difference value between the best matching block and the original block, namely, the prediction error. If the prediction error value is smaller, after discrete cosine transform (DCT) transform the high frequency coefficients is less, then appearing probability of all zero quantized coefficients will be higher, and when number of coding bits is fewer, the higher the compression will be, which means the current block and the predicted block has more matching higher structural similarity, and the prediction coding effect is better.

In H.264/AVC standard, if one takes 4×4 subblock coding as example, the integer DCT can be described as 𝑌=𝐶_𝑓𝑋𝐶^𝑇_𝑓=[𝑋]⊗𝑎⊗𝐸111122−2−21−1−111−22−1121111−1−21−1−121−21−1₂𝑎𝑏2𝑎²𝑎𝑏2𝑎𝑏2𝑏²4𝑎𝑏2𝑏²4𝑎²𝑎𝑏2𝑎²𝑎𝑏2𝑎𝑏2𝑏²4𝑎𝑏2𝑏²4, (5) where 𝐶_𝑓𝑋𝐶^𝑇_𝑓 is integer core transform, 𝐸 is the constant scaling matrix, and ⊗ represents matrix multiplication.

For a residual value of prediction error for 4×4 subblock 𝑒(𝑚,𝑛), 0≤𝑚, 𝑛≤3, its transformation coefficient is 𝐸(𝑢,𝑣) computational formula is as follows: 𝐸()=𝑢,𝑣_{3,3𝑚,𝑛=0}𝑒()⋅(𝑢)𝑚,𝑛2.5𝐶√2()×cos2𝑚+1𝑢𝜋8⋅2.5𝐶(𝑣)√2()×cos2𝑛+1𝑣𝜋8. (6)

In formula (6), if 𝑢,𝑣=0, 𝐶(𝑢),𝐶(𝑣)=1/√2. If 𝑢,𝑣≠0, 𝐶(𝑢),𝐶(𝑣)=1. [⋅] represents the rounding computation.

After getting 𝐸(𝑢,𝑣), the computational formula of quantization coefficient 𝑍(𝑢,𝑣) is as follows: |𝑍(=|𝐸(()(𝑍((𝐸(𝑢,𝑣)|𝑢,𝑣)|×MF𝑢,𝑣,QP+𝑓≫𝑞bitsSign𝑢,𝑣))=Sign𝑢,𝑣))𝑞bits=15+floorQP6. (7)

Here, MF(𝑢,𝑣,QP) is the multiplier factor associated with QP. ≫ stands for binary right shift operation. 𝑓=2^𝑞bits/𝑛, 𝑛=3 or 6, corresponding intraframe coding block and interframe coding block, respectively.

If a quantized coefficient 𝑍(𝑢,𝑣) of an encoding block’s 𝐸(𝑢,𝑣) is equal to zero, it should be satisfied with the following conditions: |𝐸(<2𝑢,𝑣)|^𝑞bits−𝑓().MF𝑢,𝑣,QP (8)

The sum of absolute difference (SAD) of the 4×4 subblock can be obtained by adding prediction error absolute value of each region. Let SAD=^3,3_𝑚,𝑛=0𝑒𝑥,𝑦. (9)

So in the video coding standards based on the block-matching method, SAD is commonly used as the related function to measure the degree of correlation between the current encoding block and prediction block. The smaller value of SAD means there exiting stronger correlation between the two blocks, and they are more matchable. In this paper, the foreground translational region detection is on the basis of the of pixel region change detection theory [12].

(2) Translational MV Checking Formula. Let 𝑇₂𝑥,𝑦,MV=1,ifSAD_(𝑥,𝑦)≥SAD_𝑆𝑐,0,else. (10)

In formula (10), (𝑥,𝑦) represents the position coordinates of the encoding block. SAD_(𝑥,𝑦) is the sum of absolute difference of the current encoding block and its corresponding encoded block with the same position coordinates in previous frame. The value of SAD_(𝑥,𝑦) can be described as variation degree of encoding blocks in two adjacent video frames. SAD_(𝑥,𝑦) can be defined as follows: SAD_(𝑥,𝑦)=^𝑀𝑁_𝑖=1𝑗=1𝑠𝑖,𝑗−𝑐𝑖,𝑗. (11) Here, 𝑠(𝑖,𝑗) is the pixel value of the current encoding block. 𝑐(𝑖,𝑗) is the pixel value of the corresponding block in previous frame. 𝑀,𝑁 denote the partition dimensions of current encoding block, respectively.

If the value of SAD_(𝑥,𝑦) is high, it means that a great difference exists between the two adjacent frames. The current encoding block is considered in the foreground translational region under dynamic background condition and 𝑇₂(𝑥,𝑦,MV) should be marked as 1.

If the value of SAD_(𝑥,𝑦) is low, it means that a smaller difference exists between the two adjacent frames. And the current encoding block is considered in the background region and 𝑇₂(𝑥,𝑦,MV) should be marked as 0.

(3) Setting of Self-Adaptive Dynamic Threshold 𝑆𝐴𝐷_𝑆𝑐. As there exists diversified motion degree in video sequences, different encoding parameters, especially quantization steps, can affect the code distortion and cause change in the value of SAD_(𝑥,𝑦). How to measure SAD_(𝑥,𝑦) value becomes one of the important factors affecting the performance of the proposed algorithm. Obviously using the fixed threshold will bring judgment error. In order to reduce the detection error caused by these uncertainties mentioned above, the proposed algorithm performs translational MV interference detection with a self-adaptive dynamic threshold SAD_𝑆𝑐, which can be determined by using the averaged SAD_(𝑥,𝑦) value of all the encoding blocks in the background region in the previous frame. Let SAD_𝑆𝑐=∑_{(𝑥,𝑦)∈𝑆𝑐}SAD_(𝑥,𝑦)num_𝑆𝑐. (12) Here, 𝑆_𝑐 represents the background region in the previous frame. ∑_{(𝑥,𝑦)∈𝑆𝑐}SAD_(𝑥,𝑦) is the summation of all the SAD_(𝑥,𝑦) values for the encoding blocks enclosed in 𝑆_𝑐. num_𝑆𝑐 is the summation times.

(4) Temporal Saliency Flowchart. In summary, the computational formula of temporal saliency analysis and detection is as follows: 𝑇𝑥,𝑦,MV=3,if𝑉_rr𝑉=02,elseif_𝑠≥𝑉_rr1,elseifSAD_(𝑥,𝑦)≥SAD_𝑆𝑐0,else, (13) where 𝑇(𝑥,𝑦,MV)=3 means the current MV is MV noise;  𝑇(𝑥,𝑦,MV)=2 means the current encoding block is belongs to the foreground dynamic region;  𝑇(𝑥,𝑦,MV)=1 means the current encoding block is belongs to the foreground translational region; and  𝑇(𝑥,𝑦,MV)=0 means the current encoding block is in the background region.

It should be pointed out that after filtering out the MV noise, 𝑇(𝑥,𝑦,MV)=3, the current 𝑉_𝑠 would be set to zero; the current encoding block belongs to background region, which should be marked with 0.

The flow chart of temporal saliency detection is shown in Figure 10.

Figure 10 

Flow chart of temporal saliency detection.

According to the calculation procedure mentioned above, the current encoding frame can be sorted into temporal visual characteristic regions with different significance, based on the low-level encoding information 𝑉_𝑠 of current encoding block and its motion vector relativity with adjacent blocks in 𝐶_rr (shown in Figure 11).

Video frame

MV extraction and distribution

Detection results

Figure 11 

Schematic diagram of temporal saliency detection results based on MV.

As the calculation of SAD_(𝑥,𝑦) should be performed in interframe prediction mode decision and motion estimation, no additional calculation cost will be caused with the adoption of this method, so it is quite applicable in occasions with limited calculation resources.

2.2. Spatial Saliency Analysis and Detection

Because HVS is also sensitive to the change of spatial domain, in order to improve the visual perception of the analysis results, it needs to detect spatial saliency; analysis of the correlation between prediction mode and spatial visual features should also be performed. In the proposed algorithm, we take H.264/AVC standard as an example and discuss the correlation between prediction mode and visual spatial attention.

All the prediction modes of H.264/AVC coder are shown in Figure 12.

Figure 12 

Prediction modes in H.264/AVC.

It has been verified in previous studies that the optimal prediction mode decision has the following rules [13].

In I-frame encoding of H.264/AVC standard, the smooth regions are suitable for using intra 16×16, while regions with rich texture always select Intra 4×4.

In P-frame encoding, the optimal prediction mode selection depends on the matching degree of encoded MB in forecasting, and the prediction mode selection results can describe the encoded MB’s content richness and are consistent with the human visual selective attention.

As HVS is relatively nonsensitive to smooth background region, in H.264/AVC standard, the smooth regions usually choose the Intra 16×16 in I frame encoding or use macro-block prediction mode Inter 16 (skip, 16×16, 16×8, 8×16) in P frame encoding.

HVS usually assigns higher visual importance to figures in foreground regions with abundant texture features and moving objects; therefore, those regions mentioned always select Intra 4×4 in I frame encoding or use subblock prediction mode Inter 8 (8×8, 8×4, 4×8, 4×4) in P frame encoding (shown in Figure 13).

Claire

M & D

Akiyo

Silent

Figure 13 

Relationship between prediction modes with visual attention. ((a), (b)) I frame encoding, ((c), (d)) P frame encoding.

Although it is small probability event that using Intra mode in P-frame coding, once Intra mode is selected, means there appears new information or the encoding content is varied greatly in current frame compared with previous frame. It can be found that, in Figure 13(d), there are 6 MBs (with red dots mark) which select Intra 4×4 as the optimal prediction mode; because the woman raised right arm suddenly, the moving arm is just the ROI with higher human visual attention.

There is high consistency between prediction mode decision results and visual attention. Therefore, in the proposed algorithm, the prediction mode is regarded as spatial characteristic of visual perception analysis.

According to the analysis above, current encoding frame can be sorted into spatial visual characteristic regions with different significance according to optimal prediction mode decision results. The spatial saliency detection computational formula is as follows: 𝑆=𝑥,𝑦,Mode2,Mode_𝑃∈{Intra}1,Mode_𝑃∈{Inter8}orMode_𝐼∈Intra4×40,Mode_𝑃∈{Inter16}orMode_𝐼∈Intra16×16. (14)

Mode_𝑃, Mode_𝐼 represent the optimal prediction modes selected by the current MB in P-frame and I-frame coding, respectively.

If Mode_𝑃∈{Intra}, means in P-frame coding, then Intra 4×4 or Intra 16×16 is selected as the optimal prediction mode, the spatial saliency is high, the current encoding block belongs to the human visual sensitive region, and it can be expressed as 𝑆(𝑥,𝑦,Mode)=2.

If Mode_𝑃∈{Inter8} or Mode_𝐼∈Intra4×4, which means the current encoding block takes the subblock prediction mode (8×8, 8×4, 4×8, 4×4) as the optimal prediction mode in P-frame coding or uses Intra4×4 in I-frame coding, then the current encoding block has abundance of texture feature or changes in spatial domain, the current block belongs to attention region, the spatial saliency is high, and then it can be expressed as 𝑆(𝑥,𝑦,Mode)=1.

If Mode_𝑃∈{Inter16} or Mode_𝐼∈Intra16×16, which means the current encoding block is smooth and belong to the visual nonsensitive region, then the spatial changes are slight, the current block has low spatial visual characteristics significance, it belongs to nonsignificant region, and it can be signed with 𝑆(𝑥,𝑦,Mode)=0.

2.3. Combination of the Spatiotemporal Saliency Detection

In this paper, according to the spatiotemporal saliency detection results, we define video region of interest (VROI). Let VROI𝑥,𝑦,MV,Mode=𝑇𝑥,𝑦,MV‖𝑆𝑥,𝑦,Mode. (15)

The computational formula can be expressed as =𝑇‖𝑆𝑇‖𝑆𝑇‖𝑆VROI𝑥,𝑦,MV,Mode5,𝑆𝑥,𝑦,Mode=24,𝑇𝑥,𝑦,MV=2‖𝑆𝑥,𝑦,Mode=13,𝑇𝑥,𝑦,MV=1‖𝑆𝑥,𝑦,Mode=12,𝑥,𝑦,MV=2or𝑇𝑥,𝑦,MV=1𝑥,𝑦,Mode=01,𝑥,𝑦,MV=0or𝑇𝑥,𝑦,MV=3𝑥,𝑦,Mode=10,𝑥,𝑦,MV=0or𝑇𝑥,𝑦,MV=3𝑥,𝑦,Mode=0. (16)

In this algorithm, according to the calculation number of 𝑇(𝑥,𝑦,MV) and 𝑆(𝑥,𝑦,Mode), the priority level of VROI is divided into 6 grades, from high to low being 5~0. For example, if the current MB uses Intra mode in P-frame coding (𝑆(𝑥,𝑦,Mode)=2), this means that the MB is in the visual sensitive region and has the highest visual attention; it can be signed with VRIO(𝑥,𝑦,MV,Mode)=5, and so on.

The proposed algorithm framework is as depicted in Figure 14.

Figure 14 

Diagram of proposed saliency detection algorithm framework.

3. The Algorithm Performance Evaluation

3.1. Test Platform

In this paper, three existing algorithms are used to do the comparison experiment [4, 6, 7]. The experimental environment is set as Table 5.

Table 5 

Experimental environment.

We use 10 typical test sequences in multiple formats which include various types (such as 176×144, 352×288, and 416×240) of video with different scenes, motion, and flatness, separately, such as videos in daytime and nighttime, sports videos, news television, broadcast, and video surveillance.

3.2. Saliency Detection Results

According to the visual perception characteristics analysis and the saliency detection procedure mentioned previously, the VROI marking results are shown in Figure 15.

Figure 15 

The saliency detection results of the proposed algorithm.

In the output saliency detection results, the MB luminance values are proportional to the priority level of VRIO(𝑥,𝑦,MV,Mode). The region with higher visual sensitive, the corresponding MB luminance value is higher. In Figure 15, MVD is the motion vector diagram, and PMD is the prediction mode diagram. The detection results have good consistency with human visual system.

3.3. Algorithm Complexity Analysis

The computation time and the similar measure method are adopted to evaluate the performance of the algorithm [13]. In the similar measure method, Kullback-Leibler (KL) distance is used to measure the similarity between the saliency distributions at human saccade locations and random locations as ()=∑KL𝐻,𝑅^𝑘ℎ_𝑘logℎ_𝑘/𝑟_𝑘+∑_𝑘𝑟_𝑘log𝑟_𝑘/ℎ_𝑘2. (17)

Here 𝐻 and 𝑅 are saliency distributions at human saccade locations and random locations with probability density functions ℎ_𝑘 and 𝑟_𝑘, respectively.

The saliency detection algorithm with the higher KL distance can discriminate human saccade locations from the random locations more easily, and this means better performance in saliency detection for videos [14].

Statistical data in Table 6 show that the proposed algorithm can enhance the timeliness of calculation and the performance in video saliency detection markedly. Compared with [4, 6, 7], the calculation time for VROI detection can be saved up to 10.69%, 14.66%, and 5.29%; at the same time, the KL distance is increased by 0.50, 1.02, and 0.24, respectively. Especially for Bus, Stefan, Foreman, and Paris sequences, which contain a large number of global motion vectors or rich texture of background, the proposed algorithm can analyze visual perception characteristics and extraction VROI fast and accurately.

Table 6 

Algorithms performance comparison results.

In Table 6, mathematical symbol “−” denotes decrease and “+” denotes increase. Δ𝐶 Time (%), ΔKL𝐷 are defined as follows formula in (18) and (19), respectively: (%)=Δ𝐶TimeComputingTime_compared−ComputingTime_proposedComputingTime_compared×100%,ΔKL𝐷=KLDistance_proposed−KLDistance_compared. (18)(19)

In Table 6, the average computational time is the shortest, and the average KL distance of the proposed algorithm is the largest. This means the proposed algorithm can control the computational complexity strictly and discriminate human saccade locations from random locations more quickly and accurately than the other ones. The experiment results demonstrate that the performance of the proposed algorithm is the best among these compared ones in video saliency detection.

4. Conclusion

In this paper, the interdependency between video encoding information and HVS characteristics is studied; it proposes a video saliency detection algorithm based on visual perception characteristics analysis and low-layer encoding information which can get from the bit-stream directly. The simulation results show that the proposed algorithm has better performance than other existing ones. It can filter out the motion vector noise, weaken the interference of translational motion vector and get rid of visual redundancy, and it can be used in the detection of visual perception characteristics analysis and saliency detection fast and effectively. The complication of the proposed algorithm is low, and its detection results are more consistent with HVS compared with other existing algorithms. It can be used conveniently in many Internet-based multimedia applications such as video retrieval based on ROI and video quality assessment. It can also be applied to video coding standards, such as HEVC and H.264/AVC.

In the future, the various multimedia applications of the proposed video saliency detection algorithm combined with fast video coding technologies can realize fast video coding based on HVS for the latest video coding standard HEVC, and saliency detection technique can be taken as part of the video standard codec at medium-to-low bit-rates.

Acknowledgment

The research work is supported by the National Key Technology R&D Program of China with Grant no. 2011BAC12B03, the National Natural Young Science Foundation of China with Grant no. 61100131, and Beijing City Board of Education Project with Grant no. KM201110005007.