Video Shot Boundary Recognition Based on Adaptive Locality Preserving Projections

Abstract

A novel video shot boundary recognition method is proposed, which includes two stages of video feature extraction and shot boundary recognition. Firstly, we use adaptive locality preserving projections (ALPP) to extract video feature. Unlike locality preserving projections, we define the discriminating similarity with mode prior probabilities and adaptive neighborhood selection strategy which make ALPP more suitable to preserve the local structure and label information of the original data. Secondly, we use an optimized multiple kernel support vector machine to classify video frames into boundary and nonboundary frames, in which the weights of different types of kernels are optimized with an ant colony optimization method. Experimental results show the effectiveness of our method.

1. Introduction

Video shot boundary recognition is a fundamental process towards video summarization and analysis. There are many boundary recognition methods already presented [1, 2]. Common method to recognize shot boundary is comparing the difference of two adjacent frames with a threshold. In paper [3], the abrupt shot boundary is detected based on an adaptive threshold and gradual transition boundary is detected with a set of standard templates. Warhade et al. [4] detected shot boundary with cross-correlation coefficient, stationary wavelet transform, and combination of local. Thakar and Hadia [5] proposed a new gradual shot detection method in which the threshold can be adaptively determined based on the totaly information change of video frames. Huo et al. [6] used a statistical model according to the video frame differences to determine the adaptive threshold. In paper [7], the threshold is automatically determined according to the magnitude of color differences quantification. Warhade et al. [8] first extracted structure features from each video frame by using dual-tree complex wavelet transform and then decided the shot boundary based on the spatial domain similarity. In order to reduce the computation, Gao and Ma [9] used color histogram and mutual information to measure the difference between frames, and then the corner distribution of frames is utilized to exclude most of the false boundaries.

The main disadvantage of these methods is susceptible to the effect of thresholds, which can make a mistake for some complicated long gradual shots. To resolve this problem, people see the video shot recognition as a categorization task. In paper [10], a fuzzy logic method is used to detect shot boundary. This method contains two processing modes, where one is dedicated to detection of abrupt shot and the other for detection of gradual shot. In paper [11], the video feature including HSV (hue, saturation, value), edge orientation, and texture feature is obtained, and then the Kohonen self-organized network is used to recognize shot boundary. Huang et al. [12] classified video frames with a radial basis function neural network. Mohanta et al. [13] used a multilayer perception network to classify video frames based on local features matrix. To improve the recognition performance, Li et al. [14] first removed some frames from the original video which were clearly not shot boundaries, then used a novel SIFT key point matching algorithm to detect shot boundary. Zhao et al. [15] used context feature vector and Tabu-SVM to recognize shot boundary. In paper [16], the proposed approach first detected general shot boundary with Fisher criterion and then classified the cut and gradual shot with SVM. In order to improve the effect of SVM, Zhao et al. [17] optimized the parameters of SVM with particle swarm method. In addition, Lankinen and Kamarainen [18] detected shot boundary using a visual bag-of-words approach. Donate and Liu [19] extracted salient features from a video sequence and tracked them over time to estimate shot boundaries. Li and Chen [20] recognized shot boundary with macroblock type information which can save a lot of computation cost.

In this paper, we present a novel method to improve the shot boundary recognition accuracy. Firstly, based on the analysis of LPP, we present an adaptive LPP to extract more useful and discriminating features. Secondly, we recognize shot boundary with an optimized multiple kernel support vector machine.

The rest of this paper is organized as follows. Section 2 abstracts the theoretical fundamentals about LPP. In Section 3, we extract the video feature with improved LPP. Section 4 uses an optimized multiple kernel SVM to recognize shot boundary. Some experiments are used to evaluate the presented method in Section 5 and all the paper is concluded in Section 6.

2. Theoretical Fundamentals

2.1. Locality Preserving Projection

Locality preserving projection (LPP) is a dimensionality reduction method which can be explained by the graph theory [21]. Assume there is𝑚-dimensional data point set𝑋=[𝑥₁,𝑥₂,…,𝑥_𝑛]; we try to find a project matrix𝐴to project these data point into a low-dimensional subspace, and the projection can be expressed as𝑦_𝑖=𝐴^𝑇𝑥_𝑖.

The objective function of LPP is as follows: min_𝑖𝑗𝑦_𝑖−𝑦_𝑗²𝑆_𝑖𝑗, (1) where the weight matrix𝑆can be defined as follows: 𝑆_𝑖𝑗=1if𝑥_𝑖isamong𝐾nearestneighborsof𝑥_𝑗or𝑥_𝑗isamong𝐾nearestneighborsof𝑥_𝑖0else. (2) Then, the objective function of LPP can be converted into the following minimization problem: min_𝐴tr𝐴^𝑇𝑋𝐿𝑋^𝑇𝐴s.t.𝐴^𝑇𝑋𝐷𝑋^𝑇𝐴=1, (3) where𝐷_𝑖𝑗=∑_𝑗𝑊_𝑖𝑗is a diagonal matrix and𝐿=𝐷−𝑆is a Laplacian matrix.

Lastly, the project matrix𝐴can be obtained by solving a generalized eigenvalue problem as follows: 𝑋𝐿𝑋^𝑇𝐴=𝜆𝑋𝐷𝑋^𝑇𝐴. (4)

Let the column vectors𝑎₁,𝑎₂,…,𝑎_𝑑be the solutions of (4), ordered according to their eigenvalues,𝜆₁,𝜆₂,…,𝜆_𝑑; we can define the following transformation form: 𝑥_𝑖⟶𝑦_𝑖=𝐴^𝑇𝑥_𝑖,𝐴=𝑎₁,𝑎₂,…,𝑎_𝑑. (5)

2.2. Weight Definition

In the LPP, the weight between two points is defined to be a simple either 1 or 0 or heat kernel, which cannot reflect the class information. Given𝑋=[𝑥₁,𝑥₂,…,𝑥_𝑛], let𝐶_𝑖and𝑁(𝑥_𝑖)be the label and𝑘the nearest neighbors of the point𝑥_𝑖, Li et al. [22] presented an orthogonal discriminating projection (ODP), in which the weight𝑆_𝑖𝑗between two points is defined as follows: 𝑆_𝑖𝑗=−𝑑exp²𝑥_𝑖,𝑥_𝑗𝛽if𝑥_𝑖∈𝑁𝑥_𝑗,𝑥_𝑗∈𝑁𝑥_𝑖,𝐶_𝑖=𝐶_𝑗−𝑑exp²𝑥_𝑖,𝑥_𝑗𝛽−𝑑1−exp²𝑥_𝑖,𝑥_𝑗𝛽if𝑥_𝑖∈𝑁𝑥_𝑗,𝑥_𝑗∈𝑁𝑥_𝑖,𝐶_𝑖=𝐶_𝑗0otherwise, (6) where𝑑(𝑥_𝑖,𝑥_𝑗)denote the geodesic distance between points𝑥_𝑖and𝑥_𝑗, and𝛽is a parameter which is used as a regular.

Figure 1 shows that the typical plot of𝑆_𝑖𝑗is a function of𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽, where𝑆₁=exp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽),  𝑆₂=exp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽)(1−exp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽)),   𝑆₃=0.

Figure 1 

Type plot of𝑆𝑖𝑗as a function of𝑑2(𝑥𝑖,𝑥𝑗)/𝛽.

From Figure 1 we find that𝑆_𝑖𝑗is not a monotonically decreasing function of𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽, which is due to the fact that𝑆₂is a nonmonotonic function. When0<𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽<0.71,𝑆_𝑖𝑗is monotonically increasing. In the actual applications,𝑆_𝑖𝑗should decrease with the increase of𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽. Zhang et al. [23] proposed a modified ODP (MODP) with correlation coefficient‖(𝑥_𝑖−𝑥_𝑖)(𝑥_𝑗−𝑥_𝑗)‖/‖(𝑥_𝑖−𝑥_𝑖)‖⋅‖(𝑥_𝑗−𝑥_𝑗)‖between𝑥_𝑖and𝑥_𝑗.

Obviously, the above definitions about𝑆_𝑖𝑗consider only the space structure and not the manifold structure. Meanwhile, the presented𝑘nearest neighbors do not reflect the real information of manifold structure.

3. Video Feature Reduction

In this section, we extract more discriminating video features from the original color, shape, and texture feature. Firstly, we propose an improved LPP. We adaptively select the𝑘nearest neighbors of each point and introduce the model information into the new weight similarity which can make the weight be a monotonically decreasing function of distance. The major merit of ALPP is preserving the local structure and label information of original data. Then we use ALPP to extract more discriminating video features for shot boundary recognition.

3.1. Mode Detection

In order to preserve the mode information of data points, we use median-shift method relied on computing the median of local neighborhoods instead of the mean to detect mode. Considering that the median of a set is a point in the set, the method is more robust than mean-shift method. Most importantly, the median-shift method is not a nonparametric method that does not require a prior knowledge of the number of clusters nor does it place any limitations on the shape of the clusters. The process of mode detection is as follows [24].

Suppose we are given a set𝑆={𝑠₁,…,𝑠_𝑛},  𝑠_𝑖∈𝑅^𝑑,  𝑑>1; define the Tukey depth of a point to be DEPTH𝑠_𝑖=min𝑣^𝑇𝑠_𝑖∶𝑣∈𝑅^𝑑,‖𝑣‖=1, (7) where the Tukey depth of a point is the minimum of its depth along any projection vector𝑣.

Firstly, the median of𝑆is an element with maximal depth MEDIAN(𝑆)=max_{𝑠𝑖∈𝑆𝑣𝑘∈𝑉}DEPTH𝑠^𝑇_𝑖𝑣_𝑘. (8)

Then we seek the mode with the median-shift algorithms. For each point we wish to ascend in the direction of the positive gradient of the underlying probability density function. We define the median-shift for point𝑐in set𝑃as 𝑐^′=MEDIAN𝑝∈𝑃∣𝑝−𝑐≤𝑅, (9) where𝑟is a bandwidth parameter. Since𝑐^′uses necessarily a point in the dataset, there is no need for multiple iterations in this step. After one iteration all points are linked and we can only go through the list of discovered medians to find a mode. The results of this step are a set of modes representing clusters.

Next, we proceed by iteratively working on the reduced set of modes, replacing the median calculation by weighted median calculation until convergence, where weights are the number of points mapped to the given mode. The weights are taken into account during the calculation of the depth of each point in the next iteration by modifying the definitions as follows: RANK𝑠_𝑖=_{𝑠𝑗<𝑠𝑖}Weight𝑠_𝑗 (10) and defining𝑊=∑_𝑠𝑗Weight(𝑠_𝑗)as the total weights in the neighborhood of𝑠_𝑖, then DEPTH𝑠_𝑖=minRANK𝑠_𝑖,𝑊−RANK𝑠_𝑖. (11)

Finally, in case of data clustering, and not only mode detection, we map each data point to its closest mode. Let𝑀_𝑖be the model of the point𝑥_𝑖, and let𝑀be the set of𝑀_𝑖mode; we can obtain 𝑀=𝑀₁,𝑀₂,…,𝑀_𝑀, (12) where𝑀_𝑚={𝑥_𝑗∣𝑥_𝑗∈𝑀∧model(𝑥_𝑗)=𝑚}.

3.2. Adaptive Neighborhood Selection

Considering the unchangeable𝑘neighborhood not reflecting the mode information of manifold structure, we apply an adaptive strategy to select the neighborhood of each point.

Firstly, we define the manifold adjusted length of line segment 𝐿𝑥_𝑖,𝑥_𝑗=𝜌^{𝑑(𝑥𝑖,𝑥𝑗)}𝑥−1_𝑖and𝑥_𝑗areneighboringpointsinfiniteelse, (13) where𝑑(𝑥_𝑖,𝑥_𝑗)is the Euclidean distance between𝑥_𝑖and𝑥_𝑗and𝜌is a flexing factor. Obviously, this formulation can be utilized to describe the global consistency. In addition, the length of line segment between two points can be elongated or shortened by adjusting the flexing factor𝜌[25].

Then, let data points be the nodes of graph𝐺=(𝑉;𝐸)and let𝑝∈𝑉^𝑙be a path of length𝑙=|𝑝|−1connecting the nodes𝑝₁and𝑝_|𝑝|in which(𝑝_𝑘,𝑝_𝑘+1)∈𝐸,  1≤𝑘≤|𝑝|. Let𝑃_𝑖𝑗denote the set of all paths connecting nodes𝑥_𝑖and𝑥_𝑗; the manifold distance metric between two points is defined as follows: 𝐷𝑥_𝑖,𝑥_𝑗=min_{𝑝∈𝑃𝑖}^|𝑝|−1_𝑘=1𝐿𝑝_𝑘,𝑝_𝑘+1, (14) where𝐿(𝑎,𝑏)denotes the manifold adjusted length of line segment.

Next, the average manifold distance of point𝑥_𝑖is defined as follows: MS𝑥_𝑖=1𝑁_{𝑚𝑀𝑖=𝑀𝑗}𝐷𝑥_𝑖,𝑥_𝑗, (15) where𝑁_𝑚is the total number which meets the conditions𝑀_𝑖=𝑀_𝑗.

Lastly, the adaptive neighborhood of point𝑥_𝑖is constructed as 𝑁^′𝑥_𝑖=𝑥∣𝐷𝑥_𝑖,𝑥_𝑗<𝑀𝑆𝑥_𝑖∧𝑀_𝑖=𝑀_𝑗 (16) which shows that the neighborhood of point𝑥_𝑖is adaptively built with the points where the distance is shorter than the average distance.

The major merit of the adaptive neighborhood selection method can be summarized as follows.(1)The manifold distance metric can measure the geodesic distance along the manifold, which can elongate the distance among data points in different regions of high density and simultaneously shorten that in the same region of high density.(2)The neighborhood of each point is different from others, which is decided by the local density of the origin space. When the local density of𝑥_𝑖is lower, the neighborhood is higher and vice versa.

3.3. Improve Weight Definition

In order to resolve the problem which is described in Section 2.2, we improve the weight definition between two points𝑥_𝑖and𝑥_𝑗as follows: 𝑆_𝑖𝑗=−𝑑exp²𝑥_𝑖,𝑥_𝑗𝛽₁if𝑥_𝑖∈𝑁^′𝑥_𝑗,𝑥_𝑖∈𝑁^′𝑥_𝑗,𝐶_𝑖=𝐶_𝑗−𝑑exp²𝑥_𝑖,𝑥_𝑗𝛽₁−𝑑exp²𝑥_𝑖,𝑥_𝑗𝛽₂if𝑥_𝑖∈𝑁^′𝑥_𝑗,𝑥_𝑖∈𝑁^′𝑥_𝑗,𝐶_𝑖=𝐶_𝑗0otherwise, (17) where𝑑(𝑥_𝑖,𝑥_𝑗)denotes the distance between points𝑥_𝑖and𝑥_𝑗,   𝛽₁and𝛽₂are the regular parameters, and𝐶_𝑖is the label of the point𝑥_𝑖for definition of𝑁^′(𝑥_𝑖)please refer to (16).

Figure 2 shows that the typical plot of𝑆_𝑖𝑗is a function of𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽,   𝛽=𝛽₁=𝛽₂, where𝑆₁=exp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽₁),  𝑆₂=exp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽₁)exp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽₂), and𝑆₃=0.

Figure 2 

Type plot of𝑆𝑖𝑗as a function of𝑑2(𝑥𝑖,𝑥𝑗)/𝛽.

Similar to paper [26], letexp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽₁)be the local weight, and letexp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽₂)be the intermode discriminating weight. The new weight definition can be viewed as the local weight and discriminating weight. It means that the discriminating similarity reflects both the local neighborhood structure of model and label information of the data set.

The properties and the corresponding advantages of the improved weight definition can be summarized as follows.(1)The improved weight definition make use of the label information and model information to preserve the manifold information, which is very important for classification.(2)Since the value ofexp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽₂)ranges from 0 to 1, no matter how far the two points are, the inter-mode similarity can be limited in certain ranges.(3)With the decrease of the geodesic distance,exp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽₂)is a decrease, which means that two near points from different modes have a smaller similarity.(4)Note that𝑆₁and𝑆₂always decrease when𝑥_𝑖and𝑥_𝑗are far apart and they increase when𝑥_𝑖and𝑥_𝑗are close. Thus,𝑆_𝑖𝑗is a monotonically decreasing function of𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽.(5)The use of mode prior probabilities makes the newly designed discriminating similarity𝑆_𝑖𝑗more suitable for perverting the local structure information of original data.

3.4. Video Feature Reduction

In order to enhance the discriminating information for shot boundary recognition, we hope to combine the label information and model information to improve the discriminating ability and preserve the local neighborhood structure of the original data.

Due to introducing the similarity matrix𝑆, we define the local scatter matrix as follows [23]: 𝐽_𝐿(𝐴)=^𝑛𝑛_𝑖=1𝑗=1𝑆_𝑖𝑗𝑦_𝑖−𝑦_𝑗𝑦_𝑖−𝑦_𝑗^𝑇=^𝑛𝑛_𝑖=1𝑗=1𝑆_𝑖𝑗𝐴^𝑇𝑥_𝑖−𝐴^𝑇𝑥_𝑗𝐴^𝑇𝑥_𝑖−𝐴^𝑇𝑥_𝑗^𝑇=𝐴^𝑇𝑆_𝐿𝐴, (18) where𝑆_𝐿=∑^𝑛_𝑖=1∑^𝑛_𝑗=1𝑆_𝑖𝑗(𝑥_𝑖−𝑥_𝑗)(𝑥_𝑖−𝑥_𝑗)^𝑇=𝑋𝐿𝑋^𝑇,  𝐿=𝑀−𝑆,  𝑀_𝑖𝑖=∑_𝑗𝑆_𝑖𝑗.

Then we define the nonlocal scatter matrix as follows: 𝐽_𝑁(𝐴)=^𝑛𝑛_𝑖=1𝑗=11−𝑆_𝑖𝑗𝑦_𝑖−𝑦_𝑗𝑦_𝑖−𝑦_𝑗^𝑇=^𝑛𝑛_𝑖=1𝑗=11−𝑆_𝑖𝑗𝐴^𝑇𝑥_𝑖−𝐴^𝑇𝑥_𝑗𝐴^𝑇𝑥_𝑖−𝐴^𝑇𝑥_𝑗^𝑇=𝐴^𝑇𝑆^𝑇−𝑆^𝑁𝐴=𝐴^𝑇𝑆_𝑁𝐴, (19) where𝑆_𝑇=∑^𝑛_𝑖=1∑^𝑛_𝑗=1(𝑥_𝑖−𝑥_𝑗)(𝑥_𝑖−𝑥_𝑗)^𝑇,  𝑆_𝑁=𝑆_𝑇−𝑆_𝐿.

Lastly, the objective function of the improved LPP can be expressed as follows: argmax_𝐴𝐽(𝐴)=argmax𝐴^𝑇(1−𝛼)𝑆_𝑇−𝛼𝑆_𝐿𝐴s.t.𝐴^𝑇𝐴=𝐼, (20) where𝛼is an adjustable factor.

So we can find that𝐴consists of the eigenvectors associated with𝑑top eigenvalues of the following eigen-equation: (1−𝛼)𝑆_𝑇−𝛼𝑆_𝐿𝑎=𝜆𝑎. (21)

The algorithmic procedure of video feature extraction is stated below.(1)Extract original video feature including the color, shape and texture feature.(2)Perform PCA projection. In order to make the matrix𝑋𝑋^𝑇become nonsingular, we project the dataset into a PCA subspace with a transformation matrix.(3)Define the similarity matrix. For each point𝑥_𝑖, compute the similarity𝑊_𝑖𝑗=exp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽₁), if𝑥_𝑗is the adaptive neighbors of𝑥_𝑖and𝑐_𝑖=𝑐_𝑗, and compute the similarity𝑊_𝑖𝑗=exp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽₁)exp(−𝑑²(𝑥_𝑖,𝑥_𝑗)/𝛽₂), if𝑥_𝑗is the adaptive neighbors of𝑥_𝑖and𝑐_𝑖≠𝑐_𝑗.(4)Compute the diagonal matrix𝑀and Laplacian matrix𝐿and then compute the top𝑑eigenvalues and its corresponding eigenvectors based on (20).(5)Perform the ALPP transformation. Let𝐴be an optimal projection matrix; we can project the new data into low dimensionality with 𝑌=𝐴^𝑇𝑋. (22)

4. Shot Boundary Recognition

The process of shot boundary detection includes two steps. Firstly, we extract the video features using ALPP method. Secondly, we detect shot boundary using an optimized MKSVM.

4.1. Multiple Kernel SVM

Support vector machines are a family of pattern classification algorithms which is based on the idea of structural risk minimization rather than empirical risk minimization [27]. However, it is often unclear what the most suitable kernel for the task at hand is. Recently, the multiple kernel learning theory has been used for training different kernels by jointly optimizing both the coefficients of the classifiers and the weights of the kernels which have a more excellent effectiveness for object recognition than SVM [28]. In this paper, we combine several possible kernels to improve the precision of shot boundary recognition.

Let𝑑=[𝑑₁,𝑑₂,…𝑑_𝑀]^𝑇be a vector of weights for the mixture of kernels. A multiple kernel is the combination of the𝑀basis kernels𝐾_𝑚𝐾𝑥_𝑖,𝑥_𝑗=^𝑀_𝑖=1𝑑_𝑚𝐾_𝑚𝑥_𝑖,𝑥_𝑗s.t.𝑑_𝑚≥0_𝑀𝑖=1𝑑_𝑚=1. (23)

Assume there are a data set𝐷={𝑥_𝑖,𝑧_𝑖}^𝑚_𝑖=1of labeled examples, where𝑥_𝑖∈𝑋is the input vector and𝑧_𝑖∈{−1,1}.

According to paper [29], the primal form of multiple kernel support vector machine (MKSVM) is thus formulated as the following optimization problem: min_{𝑑,𝑏,𝜉}𝐽(𝑑)=12^𝑀_𝑚=11𝑑_𝑚𝑓_𝑚²+𝐶^𝑙_𝑖=1𝜉_𝑖s.t.𝑦_𝑖^𝑛_𝑖=1𝑓_𝑚𝑥_𝑖+𝑦_𝑖𝑏≥1−𝜉_𝑖𝜉_𝑖≥0,∀𝑖_𝑚𝑖=1𝑑_𝑚=1,𝑑_𝑚≥0,∀𝑚. (24)

Similar to the SVM, with the constraint on𝑑_𝑚, the above minimization problem can thus be transformed into the following dual problem: max_𝛼^𝑛_𝑖=1𝛼_𝑖−12^𝑛_𝑖,𝑗=1𝛼_𝑖𝛼_𝑗𝑦_𝑖𝑦_𝑗^𝑀_𝑖=1𝑑_𝑚𝑘𝑥_𝑖,𝑥_𝑗s.t.^𝑛_𝑖=1𝑦_𝑖𝛼_𝑖=0,∀𝑖=1,…,𝑛0≤𝛼_𝑖≤𝐶,^𝑀_𝑖=1𝑑_𝑚=1,𝑑_𝑚≥0. (25)

For the test input𝑥, the decision function of MKSVM can be computed as 𝐹_MKL(𝑥)=sign^𝑛𝑀_𝑖=1𝑚=1𝑑_𝑚𝐾_𝑚𝑥_𝑖,𝑥𝑎_𝑖+𝑏. (26)

4.2. Ant Colony Optimization Method

Ant colony optimization method (ACO) is an optimizing method inspired by the foraging behavior of ant colonies [30]. When ants walk between their nest and a food source, they mark the paths with special kind of chemical termed pheromone, and the shorter paths can attract more and more pheromone [31]. In the method, an ant determines its transfer direction according to the amount of pheromone in each path. Firstly, every ant constructs an edge from a start vertex to an end vertex. Then when all ants reach the end vertex, the edges are marked with a pheromone quantity. Thus the colony can converge to the shortest path [32]. In this paper, we apply the ant colony optimization method to solve the weights optimized problem in MKSVM.

Let𝑃^𝑘_𝑖𝑗(𝑡)be the proximity which an ant𝑘transfer from element𝑖to𝑗at iteration𝑡.𝑃^𝑘_𝑖𝑗(𝑡)can be defined as 𝑃^𝑘_𝑖𝑗(𝑡)=𝜏^𝛼_𝑖𝑗(𝑖−1^𝛽_{𝑖𝑗𝑧∈𝑉𝑘(𝑥)})𝜂𝜏^𝛼_𝑖𝑗(𝑖−1)𝜂^𝛽_𝑖𝑗if𝑧∈𝑉_𝑘(𝑥)0else, (27) where𝑉_𝑘(𝑥)is the element which has not been unvisited.𝜏_𝑖𝑗is the pheromone quantity of path(𝑖,𝑗),𝜂_𝑖𝑗is a heuristic measure of moving element𝑖to element𝑗,𝛼, and𝛽are two parameters that control the relative importance of the information heuristics and exception heuristics factor, respectively.

The amount of𝑝all𝑚ants pheromone trail on a path deposited step by step. After time𝑚, the pheromone quantity𝜏_𝑖𝑗(𝑡)associated with an edge joining element𝑖and𝑗is updated according to the following formula: 𝜏_𝑖𝑗(𝑡+𝑚)=1−𝜌𝜏_𝑖𝑗(𝑡−1)+^𝑛_𝑘=1Δ𝜏^𝑘_𝑖𝑗(𝑡), (28) where𝜌is a pheromone evaporation loss coefficient andΔ𝜏^𝑘_𝑖𝑗(𝑡)is the pheromone quantity deposited at iteration𝑖by ant𝑘on an edge joining element𝑖and𝑗. TheΔ𝜏^𝑘_𝑖𝑗(𝑡)is usually defined as Δ𝜏^𝑘_𝑖𝑗(𝑖)=𝑄𝐿_𝑘if𝑤_𝑖𝑗∈𝑇_𝑘(𝐼)0else, (29) where𝑄is a constant and𝐿_𝑘is the cost function of𝑘th ant.

For the process of ant colony optimization method please refer to paper [33].

4.3. Shot Boundary Recognition

The process of shot boundary recognition includes the following stages. Firstly, we extract the original feature. Then we reduce the video feature with improved LPP. Lastly, we classify the frame into boundary frame with the optimized MKSVM classifier based on ACO method.

The algorithm of video boundary recognition is described as follows.(1)Extract original video feature which includes colors, shape, and texture feature.(2)Reduce video feature with improved LPP.(3)Define input vector[(𝑦_𝑖−𝑘−𝑦_𝑖),…,(𝑦_𝑖+𝑘−𝑦_𝑖)], where𝑦_𝑖is the video feature vector and𝑖is𝑖th frame of video.(4)Label the boundary frames(+1)and nonboundary frame(−1)and build training sample set.(5)Initialize the parameters of ACO. Let time𝑡=0, interactive times𝐼_𝑡=0. Set the maximum iterative times𝐼_𝑡(max)and let initial pheromone𝜏_𝑥𝑦=𝑐andΔ𝜏_𝑥𝑦(0)=0.(6)Update𝐼_𝑡by𝐼_𝑡=𝐼_𝑡+1and choose the ant element𝑗as its transform direction according to function (27).(7)Update the taboo table pointer. Move the ant to the selected new element and add the element into ant taboo table.(8)If all elements of the set have been fully traversed, go to step 7 or else go to the next step.(9)Recalculate the pheromone of each path, if𝐼_𝑡<𝐼_𝑡(max), go to step 6, or else save the weights𝑑=[𝑑₁,𝑑₂,…𝑑_𝑀].(10)Classify the video frame into boundary frame based on the following rule: 𝐹_MKL(𝑥)=sign^𝑛𝑀_𝑖=1𝑚=1𝑑_𝑚𝐾_𝑚𝑥_𝑖,𝑥𝑎_𝑖+𝑏. (30)

5. Experiments and Analysis

In this section we present some experiments to validate the proposed approach. Firstly, we investigate the performance of the proposed ALPP method for video feature exaction experiment. Then we recognize shot boundary with the proposed method. The video database includes movie, news, sports, documentary, and MTV. The reason of selecting this type video is that news videos have many long abrupt shots, MTV has fast changes of scenes, sport have fast camera movements and zooming in, movie includes many gradual shots. Some shot samples are shown in Figure 3. For evaluation, we use the common figures of merit of the algorithm standard precision and recall [34] #Precision=Boundariescorrectlydetected#(),#TotalboundariesdetectedRecall=Boundariescorrectlydetected#.Totalgroundtruthboundaries (31)

(a) Abrupt shot

(b) Dissolve shot

(c) Fade out shot

Figure 3 

Some shot samples.

5.1. Video Feature Exaction Experiment

In order to testify the effectiveness of the adaptive LPP (ALPP), we extract video feature with ODP, MODP, and ALPP method. For convenience of comparison, we use the same method in paper [35] to detect shot boundary. In the experiment, original video feature is built on color feature, shape feature, and texture feature. Then ODP, MODP, and ALPP method, are used to extract video feature with𝛼=0.5and𝛾=0.6. In the ODP and MODP methods we adopt𝑘nearest neighbor criterion to define the adjacency matrix, in which the𝑘set to 10. The results of performance comparison by using ODP, MODP, and ALPP are shown in Table 1.

Table 1 

Comparison results between ODP, MODP, and ALPP.

From Table 1, we can find that ALPP obtains comparable recognition performance to LPP and ODP with the same shot boundary detect method. In ALPP method, the improved weight definition combines the label information and model information with adaptive nearest neighbor select strategy, which is very important to reflect the data information truthfully. The experiment shows that ALPP has more useful and discriminating ability to extract video feature than others.

5.2. Shot Boundary Recognition Experiment

In order to investigate the performance of the proposed method, we recognize different video shots, especially for gradual shots. The system performance is compared with the multilayer perception network method (MPN method) [36]. We use the original video feature and set the parameters𝛼and𝛾to be the same as before. In the proposed method, we use polynomial kernel, radial basis kernel and linear kernel to build MKSVM and set𝑑=3,  𝜎²=0.005,  𝛼=1,  𝛽=5,  𝜌=0.5, and𝑄=100. The experimental results are summarized in Figure 4.

Figure 4 

Comparison of two recognition methods.

From Figure 4, we found that the above methods can detect not only abrupt cuts but also gradual shots very well, but the proposed method achieves more desired performance than the MPN method for shot boundary recognition. The average precision and recall of the proposed method is up to 94.1% and 91.7%, which is higher by 3.5% and 3.1% than the MPN method, respectively. These results demonstrate that the proposed method is a good tool for shot boundary recognition by using the optimized MKSVM.

5.3. Discussion

Two experiments for different type video have been systematically performed, and so now we can conclude the following.(1)We improve the effect of shot boundary detection in two stages. In the feature extract stage, we use ALPP to extract more useful and discriminating video feature. In the shot boundary detection stage, we use optimized MKSVM obtaining more remarkably boundary detection accuracy.(2)For feature extraction, the proposed ALPP performed better than LPP and ODP. This is because the former makes use of the label information and model information with adaptive nearest neighbor select strategy. At the same time, the improved weight definition can guarantee that two near points from different modes have a smaller similarity.(3)Compared with the MPN method, the proposed method can yield better performance on shot boundary recognition. It owes much to the optimized MKSVM, in which the parameters are optimized by the ant colony method. It should be noted that there are some false detection results in the above methods, which may be due to the existence of irregular object movement and the small content change between consecutive frames.

6. Conclusion

In this paper, we present a new video shot boundary recognition method, which focuses on two key problems: extracting more useful and discriminating feature and improving the accuracy of shot boundary classifier. The major contributions of the paper are to propose an optimized locality preserving present method with model detection and optimized neighbor selection strategy. Meanwhile, an optimized shot boundary classifier based on MKSVM is designed with the ant colony optimization method. Experiments demonstrate that the proposed method is outstanding. The future work is to optimize the other parameters of MKSVM to achieve more desired result.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

This project is supported by the Postdoctoral Science Foundation of Central South University, Scientific Research Fund of Hunan University of Finance and Economics under Grant (no. K201205), the Construct Program of the Key Discipline in Hunan Province, Hunan Province Education and Science Issue “Performance Evaluation for College Teacher Based on Adaptive Learning” (no. XJK013CGD083), and the Research Foundation of Science & Technology Office of Hunan Province under Grant (no. 2012GK3064).