Video Genre Classification Using Weighted Kernel Logistic Regression

Abstract

Due to the widening semantic gap of videos, computational tools to classify these videos into different genre are highly needed to narrow it. Classifying videos accurately demands good representation of video data and an efficient and effective model to carry out the classification task. Kernel Logistic Regression (KLR), kernel version of logistic regression (LR), proves its efficiency as a classifier, which can naturally provide probabilities and extend to multiclass classification problems. In this paper, Weighted Kernel Logistic Regression (WKLR) algorithm is implemented for video genre classification to obtain significant accuracy, and it shows accurate and faster good results.

1. Introduction

Recently, video usage gains increasing importance, especially with the advance of recent internet technology and digital media; people have access to huge amount of data through internet and television. It is difficult for people to find videos of interest among these tremendous amounts of data and use them when there is a need, and it is not feasible to watch all the videos searching for the one of interest. Exploiting content-based of video footage is continuously needed for many applications, for example, for retrieving video sequence, creating automatic video summarization, or detecting specific action or activities in a video surveillance [1]. Many works are done dealing with video classification problem by categorizing videos in certain categories or genre, bridging the wide semantic gap between computed low-level features and high-level concepts and helping people to find their videos of interest within narrow domain. To get good understanding of video content, many different techniques have been developed and different video features have been identified for better video representation. Many techniques are used for video classification such as Bayesian, Hidden Markov Model (HMM), Gaussian Mixture Model (GMM), Neural-Network (NN), and Support Vector Machine (SVM). SVM is considered a state-of-the-art algorithm for classifying binary data through its implementation of kernels. Oh et al. developed general framework to perform the fundamental tasks for video data mining which are temporal segmentation of video sequences and feature extraction and how to capture the location of motions occurring in a segment, how to cluster those segmented pieces, and how to find whether a segment has normal or abnormal events [2]. Chen et al. proposed a multimedia data mining framework for extraction of soccer goal events in soccer videos, by using combined multimodal analysis and decision tree [3]. Li et al. also proposed a general framework for indexing and summarizing sports broadcast programs, with a high-level model of sports broadcast video using the concept of an event, defined according to domain specific knowledge for different types of sports [4]. Zhong and Chang present a general framework and effective algorithms to detect the syntactic structures that are at a level higher than shots [5]. Fan et al. propose framework to achieve more effective semantic video classification by integrating the unlabeled samples with a limited number of labeled samples [6].

Petrushin describes an approach to efficient search for rare and frequent video events and builds a robust GMM classifier to identify real time incoming events for indoor surveillance [7]. Hervieu et al. proposed a trajectory-based HMM framework for video content understanding to solve three tasks: unsupervised clustering of events, recognition of events corresponding to learned classes of dynamic video contents, and detection of unexpected events [1]. Adam et al. present a novel algorithm based on multiple local monitors which collect low-level statistics for detection of certain types of unusual events [8]. Shirahama et al. developed a method for retrieving events of interest in a video archive using “rough set theory” to extract multiple classification rules, each of which correctly identifies a subset of shots of the event [9].

Maalouf et al. [10, 11] show that Kernel Logistic Regression (KLR), kernel version of logistic regression (LR), proves its efficiency as a classifier, which can naturally provide probabilities and extend to multiclass classification problems. KLR is used for many reasons; the first is that LR and KLR are well studied, and they do not make assumption about the distribution of the independent variables, and last reason they can be extended to handle the multiclass classification problems. Thus with the right algorithm, the computation time can be reduced less than that for the other methods, such as Support Vector Machine (SVM), which require solving constrained quadratic optimization problem.

This paper is going to implement Weighted Kernel Logistic Regression (WKLR) algorithm for video genre classification, because it has direct probabilistic output and it could be extended directly to handle multiclass classification, not like SVM. The rest of the paper is organized as follows: Section 2 explains Logistic Regression and Kernel Logistic Regression model, Section 3 describes the Video Feature Extraction, Section 4 presents the proposed method, Section 5 experimental results are depicted, and Section 6 concludes the paper.

2. Logistic Regression and Kernel Logistic Regression Model

2.1. Logistic Regression

In logistic regression, a single outcome variable, 𝑦_𝑖(𝑖=1,…,𝑛), is coded 1 with probability 𝑝_𝑖 and 0 with probability 1−𝑝_𝑖. Then 𝑝_𝑖 varies as a function of some explanatory variables, such as 𝑥_𝑖 mathematically it is expressed as the following formula Maalouf et al. and King and Zeng [10, 12]: 𝐸𝑦_𝑖∣𝑥_𝑖,𝛽=𝑝_𝑖=𝑒^𝑥𝑖𝛽1+𝑒^{𝑥𝑖𝛽′}, (1) where 𝛽 is a vector of parameters with the assumption that 𝑋_𝑖0=1 so that the intercept 𝛽₀ is a constant term. So that the assumption that the intercept is included in the vector 𝛽 will continue from now on. The logistic (logit) transformation is the logarithm of the odds of the positive response and is defined as 𝑛_𝑖𝑝=ln^𝑖1−𝑝_𝑖=𝑥_𝑖𝛽. (2) The logit function can be expressed in matrix form as 𝑛=𝑋𝛽. (3) The regularized log likelihood is defined as ln𝐿𝛽=^𝑁_𝑖=1𝑦_𝑖ln𝑝_𝑖+1−𝑦_𝑖ln1−𝑝_𝑖−𝜆2𝛽², (4) where the regularization (penalty) term (𝜆/2)‖𝛽‖² is added to obtain better generalization.

For binary outputs, the loss function or the deviance DEV is the negative log likelihood and, given by the formula DEV𝛽=−2ln𝐿𝛽. (5)

2.2. Kernel Logistic Regression

For Kernel logistic regression and following Maalouf et al., vector 𝛽 can be expressed as a linear combination of the input vectors as the following [10]: 𝛽=𝑋^𝑇𝛼=^𝑁_𝑖=1𝛼_𝑖𝑋_𝑖, (6) where the vector 𝛼 is known as the dual variable with dimensions, 𝑁×1. Now the logit vector 𝑛 can be rewritten as 𝑛=𝑋𝑋^𝑇𝛼=𝐾𝛼, (7) where 𝐾=𝑋𝑋^𝑇 is symmetric positive semidefinite Gram matrix with 𝑁×𝑁 dimensions.

Considering the logit link function again shown previously, 𝑛_𝑖=𝑥_𝑖𝛽=𝛽₀+𝑥_𝑖1𝛽₁,…,𝑥_𝑖𝑑𝛽_𝑑, (8) where the vector 𝑥_𝑖 is given by 𝑥_𝑖=[1,𝑥_𝑖1,…,𝑥_𝑖𝑑], with 𝑖=1,…,𝑁. this linear function represents the simplest form of an identity mapping polynomial basis function ∅ of the feature space such that ∅𝑥_𝑖=∅𝑥_𝑖1,…,𝑥_𝑖𝑑=𝑥_𝑖. (9) So the logit link function can be rewritten as 𝑛_𝑖=∅𝑥_𝑖𝛽. (10) Generally the function ∅(⋅), maps the data from the lower dimension space to higher dimension space, so that ∅∶𝑥∈𝑅^𝑑⟶∅(𝑥)∈𝐹⊆𝑅̂. (11) The purpose of choosing the mapping ∅ is to convert nonlinear relation between the response variable and the independent variables into linear relation, Maalouf and Trafalis [11].

Logit link function in case of KLR then could be expressed as 𝑛_𝑖=^𝑁_𝑗=1𝛼_𝑗∅𝑥_𝑖,∅𝑥_𝑗=^𝑁_𝑗=1𝛼_𝑗𝑘𝑥_𝑖,𝑥_𝑗=𝑘_𝑖𝛼, (12) where 𝑘_𝑖 is the 𝑖th row in the kernel matrix 𝑘(𝑥_𝑖,𝑥_𝑗)=𝐾. Mercer’s sufficient and necessary conditions must be satisfied in kernel transformation function, which state that the kernel function must be expressed as inner product and must be positive and semidefinite, Maalouf and Trafalis [11]. Now 𝑛_𝑖=𝑘_𝑖𝑝𝛼=ln^𝑖1−𝑝_𝑖. (13) This implies that 𝑝_𝑖=𝑒^𝑛𝑖1+𝑒^𝑛𝑖=𝑒^𝑘𝑖𝛼1+𝑒^𝑘𝑖𝛼=11+𝑒^{−𝑘𝑖𝛼}. (14) So the regularization likelihood can be rewritten with respect to 𝛼 as ln𝐿(𝛼)=^𝑁_𝑖=1𝑦_𝑖ln𝑝_𝑖+1−𝑦_𝑖ln1−𝑝_𝑖−𝜆2𝛼^𝑇𝐾𝛼. (15) With deviance DEV(𝛼)=−2ln𝐿(𝛼), (16) iteratively reweighted least squares (IRLS) method is one of the most popular techniques used to find the MLE of 𝛽 in LR models, which uses Newton-Raphson algorithm to solve the score equations. KLR models can also be fitted using IRLS. Each iteration finds the weighted least squares (WLS) estimates for a given set of weights, which are used to construct a new set of weights Oh et al. [2]. The gradient and Hessian are obtained by differentiating ln 𝐿(𝛼) with respect to 𝛼. In matrix form, the gradient is ∇_𝛼ln𝐿(𝛼)=𝐾^𝑇𝑦−𝑝−𝜆𝐾𝛼=0, (17) where 𝑝 is the probability vector whose elements are given in (14). And the Hessian with respect to 𝛼 is ∇²_𝛼ln𝐿(𝛼)=−𝐾^𝑇𝑉𝐾−𝜆𝐾, (18) where 𝑉 is a diagonal matrix with diagonal elements 𝑝_𝑖(1−𝑝_𝑖) for 𝑖=1,…,𝑁. The Newton-Raphson update with respect to 𝛼 on the (𝑐+1)-th iteration is 𝛼^(𝑐+1)=𝛼^(𝑐)+𝐾^𝑇𝑉𝐾+𝜆𝐾⁻¹𝐾^𝑇𝑦−𝑝−𝜆𝐾𝛼^(𝑐). (19) Since 𝛼^(𝑐)=(𝐾^𝑇𝑉𝐾+𝜆𝐾)⁻¹(𝐾^𝑇𝑉𝐾+𝜆𝐾)𝛼^(𝑐), then (10) can be rewritten as 𝛼^(𝑐+1)=𝐾^𝑇𝑉𝐾+𝜆𝐾⁻¹𝐾^𝑇𝑉𝑧^(𝑐), (20) where 𝑧^(𝑐)=𝐾𝛼^(𝑐)+𝑉⁻¹(𝑦−𝑝) is the adjusted dependent variable or adjusted response.

2.3. WKLR Algorithm

Consider 𝐾^𝑇𝑉𝐾+𝜆𝐾𝛼^(𝑐+1)=𝐾^𝑇𝑉𝑧^(𝑐) (21) is a system linear equations with kernel matrix 𝐾, vector of adjusted responses 𝑧, and weighted matrix 𝑉, and the weights and response vector both depend on 𝛼^(𝑐), which is the current estimate of the parameter vector, so specifying an initial estimate 𝛼⁽⁰⁾ for 𝛼 can be solved iteratively, using the Conjugate Gradient (CG) method, which is equivalent to minimizing the quadratic problem 12𝛼^(𝑐+1)𝐾^𝑇𝑉𝐾+𝜆𝐾𝛼^(𝑐+1)−𝛼^(𝑐+1)𝐾^𝑇𝑉𝑧^(𝑐). (22) In order to avoid the long computations that the CG may suffer from, a limit can be placed on the number of CG iterations, thus creating an approximate or truncated Newton direction, Maalouf and Trafalis [11].

Algorithm 1 describes Iterative Weighted Kernel Logistic Regression Maximum Likelihood Estimate (IWKLR MLE) using IRLS.

Data: 𝐾,𝑦,𝛼(0) Result: 𝛼 Begin   𝐶 = 0    while DEV(𝑐)−DEV(𝑐+1)DEV(𝑐+1)>𝜀1 and 𝐶 ≤ MAX ILRS Iteration Do     for 𝑖←1  𝑡𝑜  𝑛  do      𝑝𝑖=11+𝑒−𝑥𝑖𝛼;            // compute probabilities      𝑣𝑖=𝑝𝑖(1−𝑝𝑖);           // compute werights      𝑧𝑖=𝐾𝑖𝛼(𝑐)+(𝑦𝑖−𝑝𝑖)𝑝𝑖(1−𝑝𝑖);         // compute adjusted response     𝑉 = diag(𝑣1,…,𝑣𝑁);     𝐾𝑇𝑉𝐾+𝜆𝐾𝛼(𝑐+1)=𝐾𝑇𝑉𝑧(𝑐);      // compute 𝛼     𝐶=𝐶+1 End

Algorithm 1 

3. Video Feature Extraction

Extracting appropriate features is vital for acceptable design of any pattern classifier. In video classification studies, the features used can be categorized as one of three which are text features, audio features, and visual features; some studies divide the features into two main categories: text features and nontext features, and the latter is divided into low-level and semantic video features. Text features contain text extracted from video and user-generated text feature. Nontext features are features extracted from images, audio, and motion; such features could be referred to as low-level features which could be defined as features extracted from video clips and audio track without reference to any external knowledge [13, 14]. Some of researches use features that correspond to cinematic principle for visual features which are composed of color, motion and average shot length. One difficulty in using low-level feature especially visual features is the huge amount of data, and the most used solution for this is using a key frame to represent the shot or by using dimensionality reduction techniques such as Principal Component Analysis (PCA) and wavelet transforms application [13, 15, 16]. Discrete transform is widely used for feature extraction and data redundancy reduction. Proportional advantages of deterministic transforms make them an interesting type of feature extraction approaches. One of the important discrete transform, is Discrete Cosine Transform (DCT), and special properties of the DCT make it a powerful transform in video processing applications [17–20]. DCT has ability for data decorrelation. There are fast algorithms for DCT realization. When applying DCT to video, some coefficients are selected and others discarded for dimension reduction of data. DCT coefficient selection is an important part of feature extraction process. DCT feature extraction is composed of two steps: the first one DCT is applied to the entire key frame to obtain the DCT coefficients, then in the second step some of coefficients are selected to construct feature vectors. The size of the DCT coefficient matrix is the same as the input frame. DCT by itself does not decrease data dimension, so it compresses most signal information in small percent of coefficients. DCT coefficients for 𝑀×𝑁 frame are calculated as follows [18]: 𝑓()=1𝑢,𝑣√∝(𝑢)∝(𝑣)×𝑀𝑁_{𝑀−1𝑥=0𝑁−1𝑦=0}𝑓()𝑥,𝑦×cos2𝑥+1𝑢𝜋2𝑀×cos2𝑦+1𝑣𝜋,2𝑁𝑢=0,1,…𝑀,𝑣=0,1,…𝑁, (23) where ∝(𝜔) is defined by ∝(𝜔)=1√2𝜔=0. (24)𝑓(𝑥,𝑦) is the frame intensity function, and 𝐹(𝑢,𝑣) is 2D matrix of DCT coefficients. As mentioned that DCT coefficient matrix is the same as the frame size and to obtain feasible and compact low-dimensional representation of the features, PCA could be used [21]. Given N d-dimensional feature vectors [22,…,N], the mean vector calculated as 1𝜏=𝑁^𝑁_𝑖=1𝑥_𝑖. (25) And covariance matrix is calculated as 1𝑐=𝑁^𝑁_𝑖=1𝑥_𝑖−𝜏𝑥_𝑖−𝜏^𝑇; (26) then the Principal Component assumes the first 𝑃 significant eigenvectors of 𝑐, that is, {𝑣_𝑖,𝑖=1,2,…,𝑃}. By constructing the eigenmatrix 𝑈=[𝑣₁,𝑣₂,…,𝑣_𝑃] of 𝑑𝑋𝑃 dimension, an arbitrary 𝑑-dimensional original feature vector 𝑥 can be represented as a new low 𝑃-dimensional vector 𝑦=𝑈^𝑇(𝑥−𝜏), and 𝑃≪𝑁, and 𝑃≪𝑑 [21]. In our proposed method we deal with shot or scene concept and take a key frame to represent the whole shot or scene. There are different algorithms implemented for extracting key frame; most of these algorithms depend on the motion part to extract the key frames. After extracting the key frames, we used DCT to obtain the DCT coefficients. PCA is applied on the DCT coefficients to select the most significant DCT coefficients.

4. Proposed Method

In this paper, first shot detection is used to extract the key frames from the input videos; shot detection is used because it is more sophisticated method for video summary [19, 22]. Secondly we construct 𝑋 and 𝑌 variables; variable represents the features of video shot represented by the DCT_PCA data, and 𝑌 variable determines that video shot is set manually. Then feature should be scaled to values between 0 and 1 using the following equation: 𝑓(𝑥)=11+𝑒^−𝑥. (27) The advantage of this scaling is to avoid features in greater numeric range dominating those in smaller numeric range. Another advantage is to avoid numerical difficulties during the calculation when large feature value might cause numerical problem. Finally WKLR is implemented on the prepared data for classification aiming to achieve significant accuracy and making WKLR to be an effective method for video classification.

4.1. Performance Measures

The predictions quality evaluated using five measures: Matthews correlation coefficient (MCC), 𝑄_total, Predicted Positive Value (PPV), sensitivity, and specificity. FP: false positive, FN: false negative, TP: true positive, and TN: true negative. Matthews correlation coefficient can be in the range of −1 to 1, where 1 is a perfect correlation and −1 is the perfect anticorrelation. A value of 0 indicates no correlation. Its value can be calculated as follows: =()MCCTP∗TN−FP∗FN()∗()∗()∗().TP+FNTN+FPTP+FPTN+FN (28)

𝑄_total is the percentage of correctly classified residues, also called the prediction accuracy. It is given as follows: 𝑄_total=(TP+TN).TP+TN+FN+FP (29)

PPV is the predicted positive value, also called the precision or 𝑄_pred. It is given as follows: 𝑄_pred=TPTP+FP×100. (30)

Sensitivity is also called recall or 𝑄_obs and is the fraction of the total positive examples that are correctly predicted, and it is calculated as follows: 𝑄_obs=TPTP+FN×100. (31)

Specificity is the fraction of total negative examples that are correctly predicted; its value can be calculated as follows: 𝑆pecificity=TNTN+FP×100. (32)

5. Experimental Results

The steps mentioned previously are tested on group of videos downloaded from youTube and Youku websites, containing the three categories, News, Sport, and Movies, and the selected frames of all those videos are almost about 10000 frames to test the accuracy of the proposed method. In the training phase, the data distributed randomly, the algorithm of WKLR MLE was implemented with 10-fold cross-validation, and max number of iterations is set to 30 for truncated Newton method.

Figure 1, Tables 1 and 2 show the accuracy and classification performance of each category against the other two categories

Table 1 

Number of positive and negative occurrences for each genre.

Table 2 

The accuracy and MCC for each genre.

(a) News ROC curve 

(b) Sport ROC curve

(c) Movie ROC curve

Figure 1 

ROC curve for news, sport, and movie, respectively.

As shown in Table 2, using KLR with same Gaussian kernel shows that the proposed approach could achieve considerable improvement in performance on small to medium size data sets; the average accuracy of our proposed method is 88%, compared to 86% accuracy of the work done in [23] in which the last paper proposed a new technique for improving unconstrained face recognition based on leveraging weakly labeled web videos. KLR has also advantages including that KLR has direct extension to multiclass classification and it can yield probabilistic output without intervention.

6. Conclusion

This paper proposed Weighted Kernel Logistic Regression (WKLR) in order to enhance the accuracy and processing time of video classification, which shows easy to implement and good acceptable results regarding to performance. One of the benefits of using WKLR is that it uses algorithms which are less complex for unconstrained optimization methods compared to algorithms used for constrained optimization methods such as SVM. An other benefit is that WKLR takes the advantage of speed of the IRLS and the power of the kernel methods. Future studies may compare this method using different kernels and test the algorithm on more datasets. They may also try to improve the speed of the algorithm. Methods such as the trust region Newton might also be utilized because of their stability and rigorous mathematical derivation.