[Retracted] News Video Classification Model Based on ResNet-2 and Transfer Learning

Gao, Yiping

doi:https://doi.org/10.1155/2021/5865200

Security and Communication Networks

On this page

Abstract Introduction Literature Review Experimental Results and Analysis Conclusions Data Availability Conflicts of Interest References Copyright Related Articles

Research Article Retraction

!

This article has been Retracted. To view the article details, please click the ‘Retraction’ tab above.

Special Issue

Massive Machine-Type Communications for Internet of Things

View this Special Issue

Research Article | Open Access

Volume 2021 | Article ID 5865200 | https://doi.org/10.1155/2021/5865200

[Retracted] News Video Classification Model Based on ResNet-2 and Transfer Learning

Yiping Gao¹

Academic Editor: Jian Su

Received02 Nov 2021

Revised14 Nov 2021

Accepted20 Nov 2021

Published16 Dec 2021

Abstract

A large amount of useful information is included in the news video, and how to classify the news video information has become an important research topic in the field of multimedia technology. News videos are enormously informative, and employing manual classification methods is too time-consuming and vulnerable to subjective judgment. Therefore, developing an automated news video analysis and retrieval method becomes one of the most important research contents in the current multimedia information system. Therefore, this paper proposes a news video classification model based on ResNet-2 and transfer learning. First, a model-based transfer method was adopted to transfer the commonality knowledge of the pretrained model of the Inception-ResNet-v2 network on ImageNet, and a news video classification model was constructed. Then, a momentum update rule is introduced on the basis of the Adam algorithm, and an improved gradient descent method is proposed in order to obtain an optimal solution of the local minima of the function in the learning process. The experimental results show that the improved Adam algorithm can iteratively update the network weights through the adaptive learning rate to reach the fastest convergence. Compared with other convolutional neural network models, the modified Inception-ResNet-v2 network model achieves 91.47% classification accuracy for common news video datasets.

1. Introduction

Today, video media plays an increasingly prominent role in enriching people’s lives, education, and entertainment. Video is a kind of media with rich content, which can provide more vivid information than words, sounds, and images [1–5]. News is a kind of video, which is an important way for people to understand the society and is closely related to people’s life. Now, there are a lot of news programs, and the amount of information is also very large. Therefore, it becomes an important demand that people can easily find relevant content of their own interest in a large number of news programs.

Content-based retrieval refers to retrieval according to the semantic features or audio-visual features of media objects [6–8]. Semantic features refer to the content information of video segments, while audio-visual features refer to some physical features that can be directly obtained from sounds and images, such as colors, textures, and shapes in images, motions of objects and lenses in videos, and tonal loudness and timbre in sounds [9–12]. This is a very practical technology with a wide range of applications. Now, content-based video retrieval has achieved some results. However, the research of content-based video retrieval for news is not much.

Deep learning abandons the complex operation process of traditional algorithms, and convolution neural network (CNN) [13, 14] has achieved great success in image recognition and image segmentation at first. With the continuous breakthroughs in typical network structures, deep neural networks such as recurrent neural network (RNN) [15], deep belief network (DBN) [16], and generative adversarial networks (GAN) [17] have emerged, which can better enhance the feature extraction ability of models by supervised learning [18, 19]. Therefore, based on the theory and technology of content-based video retrieval, this paper focuses on the related technology and implementation of news video retrieval based on deep learning.

2. Literature Review

Traditional video classification and recognition methods generally use artificially designed features to model, extract the size, shape, color, texture, and other information features of video key frames, and fuse one or more features to build a classifier to realize automatic video classification and recognition. For example, Arivazhagan et al. [20] proposed to apply completely local binary pattern (CLBP) as texture feature to image recognition. In this method, color and texture features are fused, and the nearest neighbor classifier is used to accomplish the classification task, and the factors of illumination intensity change are considered. Zhou et al. [21] extracted three features of color, shape, and size, selected K-nearest neighbor (KNN) classifier to classify and recognize various images, and achieved good recognition accuracy, with the recognition accuracy as high as 90%. However, all these methods require manual design of image features. Although they are excellent in accuracy and robustness, manual design of feature extractors usually requires a lot of work. Convolutional neural network omits the process of feature extraction by manual design and fuses attention mechanism to extract geometric transformation information in images, so as to improve the accuracy of image recognition and the stability of the network.

At present, the research on news video recognition using deep convolutional neural network is still limited mainly because there is no common news video dataset with large enough data amount and high enough quality, so it is difficult to train an excellent classification and recognition model. Therefore, in this paper, Inception-ResNet-v2, which is trained in ImageNet large dataset [22–24] and has better performance than Inception-ResNet-v1, is used as the pretraining model, and the model-based transfer learning method [25] is used for the experiment. The main innovations of this paper are as follows: (1) the transfer learning technology is applied to the news video classification model based on the concept-ResNet-v2 network, which effectively improves the overfitting and makes the model have better generalization ability; (2) an improved Adam gradient descent method is proposed to improve the convergence rate of the model.

3. Classification Model Based on Inception-ResNet-v2 Network and Transfer Learning

3.1. Deep Neural Network

Convolution neural network is the most widely used deep learning model in the field of computer vision. Its earliest theoretical model is the neurocognitive machine proposed by Japanese scholar Fukushima [26]. The neural cognitive machine has a good recognition ability even when the target object is slightly distorted. On the basis of neurocognitive machine, a multilayer feedforward neural network model LeNet-5 appeared and was successfully applied in handwritten character recognition. The model mainly includes input layer, convolution layer, pooling layer, full connection layer, and output layer, which laid the foundation for the later convolution neural network structure. In 2012, the AlexNet model won the ILSVRC competition, which made convolutional neural network become a research hotspot, and then, more excellent convolutional neural networks were proposed. A typical convolutional neural network model architecture is shown in Figure 1.

Convolution neural network extracts features by convolution operation on local “receptive field,” and it is mainly used in image processing. CNN is a kind of feedforward neural network with deep structure. Firstly, the image is input at the input layer and then calculated by convolution layer, pooling layer, and nonlinear activation function, and the semantic information of high-level abstraction is gradually extracted from the image. This is the “feedforward operation” of convolutional neural network. Finally, for the fully connected layer, all the features extracted from the previous network are connected for prediction, and the difference between the detected value and the true labeled value of the network is calculated. The loss is propagated back to the first convolution layer from the fully connected layer by the gradient descent method so that all the parameters of the network are updated, and the whole network model converges after several rounds of training.

The formula for calculating a single deconvolution layer is as follows:

In this layer, an image composed of feature images of color channels is used as the input. Each channel of the image can be expressed as the linear sum of potential feature maps and convolution kernel.

The deconvolution layer makes the potential feature graph sparse by introducing regularization terms. The total loss function of deconvolution layer is composed as follows:where is sparse norm and is constant.

The implementation process of deconvolution is shown in Figure 2.

3.2. Transfer Learning

In the training of deep neural network, a large enough dataset with high quality is an important basis for the accuracy and high reliability of the training model. However, in practical application, for the research of video classification and recognition, because the common experimental dataset is small, the classification model trained by the Inception-ResNet-v2 network has the problems of low accuracy and poor generalization ability.

Therefore, this paper adopts the idea of the transfer learning method and regards the Inception-ResNet-v2 network model after being pretrained by ImageNet large training set as a general image feature extractor. By transferring the extracted general image underlying feature knowledge to the news classification task, as an initialization parameter of the network, a small amount of video data can also learn and train a high-performance news classification model. The comparison between traditional machine learning process and transfer learning process is shown in Figure 3.

Firstly, the definition of transfer learning is analyzed. Given a source domain and a learning task , if there is a target domain and a learning task , the goal of transfer learning is to use the useful information learned in and to help the target prediction function in , where or . According to learning methods, transfer learning can be divided into four categories: sample-based transfer learning method, feature-based transfer learning method, model-based transfer learning method, and relationship-based transfer learning method [27]. The model-based transfer learning method refers to the method of finding the shared parameter information between the source domain and the target domain to realize the transfer, so this paper adopts the model-based transfer learning method.

3.3. Inception-ResNet-v2 Classification Model

At present, the classical convolutional neural network models are LeNet, AlexNet, VGGNet, GoogLeNet, ResNet, DenseNet, and ResNeXt. On the basis of the above structures, Szegedy et al. proposed the Inception-v2 structure and the Inception-v3 structure. Batch normalization is added to the Inception-v2 model, which makes the output of each layer obey the distribution of mean value of 0 and variance of 1. Inception-v3 integrates all the advantages of Inception-v2. Compared with v2, Inception-v3 uses asymmetric convolution to reduce the number of parameters and the amount of computation. On the contrary, label smoothing regularization is adopted to prevent overfitting.

In 2016, the Google team released Inception-ResNet-v2 CNN, which scored the best result in the benchmark test of LSVRC image classification. It is inspired by the residuals network (ResNet) and is a variation of the Inception-v3 model. Residual connections can make shortcuts exist in the model, thus simplifying the concept module and completing the deeper neural network training. The network structure of Inception-ResNet-v2 is shown in Figure 4. Compared with the current common deep convolutional neural network models such as GoogLeNet and ResNet, this paper uses Inception-ResNet-v2 network as the basic framework, as shown in Figure 5.

The network structure of Inception-ResNet-v2 consists of a combination of Inception-v4 and ResNet. The three Inception-ResNet blocks (Inception-ResNet-A, Inception-ResNet-B, and Inception-ResNet-C) add direct connections to diversify the channels. Compared with Inception-v4, it has less parameters and faster convergence. At the same time, it also has a certain reduction in the performance requirements of the machine and can set higher parameters in the same experimental environment. The convolutional core of Inception-ResNet-v2 has more varied channels than Inception-ResNet-v1. For CNN, the commonly used optimization algorithms are gradient descent, etc. The network depth increases gradually, and the training error decreases first and then increases.

The construction ideas of the proposed classification model structure are as follows:(1)Using the pretraining model Inception-ResNet-v2 on ImageNet large-scale image dataset and combining with the model-based transfer learning method, the underlying features of the image learned by the pretraining model convolution module are migrated to the classified task as the initialization parameters of the network(2)Training the classification model with the extracted feature map as input and replacing the output of the last full connection layer of the pretraining network with the category number of the news video dataset in this paper(3)Completing the classification and recognition task on the news video dataset established in this paper

According to the above ideas, the classification model based on the transfer learning and the Inception-ResNet-v2 pretraining network is shown in Figure 6.

This paper uses the model-based transfer learning method to build the classification model structure, which not only saves training time and reduces the requirements of experimental hardware configuration but also solves the overshoot problem caused by the small sample training process so that the generalization ability of the model is better.

3.4. Model Optimization Based on the Gradient Descent Method

Gradient descent method is the most commonly used objective function optimization algorithm in the field of deep learning [28, 29], and its purpose is to find the local minimum of the function. The gradient descent method generally conforms to the law that the function value is closer to the target value, the corresponding gradient decreases, and the descent is slower. The gradient descent method is an algorithm for neural network to obtain the optimal solution in the learning process. Commonly used are batch gradient descent (BGD), stochastic gradient descent (SGD), miniBatch gradient descent (MBGD), and Adam.

Adam, as one of the adaptive gradient algorithms, combines the ideas of MBGD algorithm and SGD algorithm and calculates the mean and variance of gradients to dynamically adjust the learning rate. This algorithm is not sensitive to gradient scaling and diagonal rescaling, so it is very suitable for dealing with sparse data and nonstatic targets. It is one of the algorithms with the best gradient descent performance at present. Adam algorithm calculation formula is as follows:where is momentum coefficient, and the default value is 0.9, is a constant and defaults to 0.999, is the learning rate, and are the weight values of step t and step t + 1, respectively, and is 10⁻⁸.

In this paper, a momentum updating rule is introduced based on Adam, and the deviation correction term of the momentum vector is synthesized, so as to dynamically adjust the momentum deviation. The update process of dynamic Adam algorithm is as follows:

We can get from formula (4) thatwhere the default value of is 0.99.

4. Experimental Results and Analysis

4.1. Experimental Environment and Dataset

Windows10 64-bit operating system was used in the experiment, and the processor configuration was Inter(R) Xeon(R) Silver 4110 [email protected] GHz. The establishment, training, and testing of convolutional neural network are programmed by python language, the open source artificial neural network library Keras is called to create the model, and NVIDIA GeForce GTX 2080 is used to accelerate the training. A total of 362 news video samples were collected through the Internet and cooperative shooting, and training and testing datasets which can meet the needs of deep neural network training were made to prepare for the subsequent classification model training. Part of the data is shown in Figure 7.

(a)

(b)

(c)

(d)

4.2. Evaluation Index

Since there are few types of news videos, this paper uses Top-1 accuracy (Acctop-1) as the evaluation index:where N represents the total number of videos and represents the number of videos correctly classified.

4.3. Selection of Learning Rate and Batch Quantity

The learning rate controls the step size of gradient descent, and different learning rates have great influence on the convergence and classification accuracy of the model. In order to optimize the experimental results, the relationship between learning rate and classification results is analyzed. Experiment with different learning rate models is carried out under default parameter settings. Figure 8 shows the change of the loss value with iteration times during the training process of the model. The amount of data needed to train the Inception-ResNet-v2 network model is generally relatively large. In the experimental process, batch training is generally adopted, that is, a batch of sample data is read in at one time. The selection of batch size is related to the memory size of GPU. If the batch size is too small, the parallel computing capacity of GPU cannot be fully utilized; if it is too large, it may exceed the computing capacity of GPU, resulting in overflow of video memory. Figure 9 shows the influence of different batch quantities on the model optimization process when the initial learning rate is 0.01.

It can be seen from Figure 8 that, on the whole, a higher learning rate can achieve faster convergence. The model of 0.01 converges slowly, and the learning rate of 0.2 can achieve rapid convergence, but the final loss value is higher. This is because too high a learning rate may miss the optimal solution and reduce the classification accuracy. However, when the learning rate is 0.01, better results can be obtained, so the initial learning rate of subsequent experiments is set to 0.01.

It can be seen from Figure 9 that the larger the batch size, the better the performance of the loss value. The model with batch size of 64 converges fastest and loses the least. However, due to the limited configuration, video memory overflow will occur when the batch size is 64, so the batch size parameter selected in this experiment is 32.

4.4. Comparison of Optimization Algorithms

For the experimental schemes numbered 1–4, five gradient descent optimization algorithms, BGD, MBGD, SGD, Adam, and dynamic Adam, are used to optimize the parameters of this model. The experimental results are shown in Table 1.

It can be seen from Table 1 that the dynamic Adam optimization algorithm has higher accuracy than the other four algorithms, which shows that the dynamic Adam optimization algorithm has faster convergence speed, higher efficiency of network parameter optimization, and better learning effect.

4.5. Comparison of Network Models

For the experimental schemes numbered 1–10, ten pretrained convolution neural network models, AlexNet, VGG16, VGG19, Inception-v3, Inception-v4, ResNet-50, ResNet-101, ResNet-152, Inception-ResNet-v1, and Inception-ResNet-v2 were used for the experiment. The experimental results are shown in Table 2, where the gradient descent optimization algorithm is unified as dynamic Adam. The transfer training strategy is the whole layer of the training and pretraining network models, and the training iteration steps are 25000.

As shown in Table 2, the training result of ResNet-50 network model has the lowest accuracy, while Inception-ResNet-V2 has the highest, and the accuracy is gradually improved with the increase of the depth of the network model, which indicates that the depth of the convolutional neural network has a certain influence on the result of news video classification and recognition. The deeper network structure has stronger ability to extract video information and higher classification accuracy.

5. Conclusions

In this paper, a news video classification model based on the Inception-ResNet-v2 network and transfer learning is implemented. In this model, ImageNet network is used as the pretraining model, and the model-based transfer learning method is adopted to construct the Inception-ResNet-v2 network model structure, and the dynamic Adam algorithm is adopted as the model optimization method. The experimental results show that the Inception-ResNet-v2 network has a stronger ability to extract news video information, which is more conducive to the realization of classification tasks. The improved Adam algorithm can achieve the fastest convergence by iteratively updating the network weights with the adaptive learning rate. Compared with other convolutional neural network models, the classification accuracy of the proposed Inception-ResNet-v2 network model for common news video datasets is 91.47%.

Data Availability

The experimental data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that they have no conflicts of interest to report regarding the present study.

References

G. Gao, W. Zhang, Y. Wen, Z. Wang, and W. Zhu, “Towards cost-efficient video transcoding in media cloud: insights learned from user viewing patterns,” IEEE Transactions on Multimedia, vol. 17, no. 8, pp. 1286–1296, 2015.
View at: Publisher Site | Google Scholar
S. Norazean, M. A. Mazli, and G. Faizul, “Students' perceptions on using different listening assessment methods: audio-only and video media[J],” English Language Teaching, vol. 10, no. 8, pp. 93–97, 2017.
View at: Google Scholar
K. K. Loh, B. Z. H. Tan, and S. W. H. Lim, “Media multitasking predicts video-recorded lecture learning performance through mind wandering tendencies,” Computers in Human Behavior, vol. 63, pp. 943–947, 2016.
View at: Publisher Site | Google Scholar
J. Adams, G. Christian, and T. Tarshis, “Managing media: reflections on media and video game use from a therapeutic perspective,” Journal of the American Academy of Child & Adolescent Psychiatry, vol. 54, no. 5, pp. 341-342, 2015.
View at: Publisher Site | Google Scholar
I. A. K. S. Puspita Dewi and N. W. Arini, “The positive impact of teams games tournament learning model assisted with video media on students' mathematics learning outcomes,” Journal of Education Technology, vol. 4, no. 3, pp. 367–371, 2020.
View at: Publisher Site | Google Scholar
Y. Hao, T. Mu, and R. Hong, “Stochastic multiview hashing for large-scale near-duplicate video retrieval[J],” IEEE Transactions on Multimedia, vol. 19, no. 99, pp. 1–14, 2016.
View at: Google Scholar
R. Fernandez-Beltran and F. Pla, “Latent topics-based relevance feedback for video retrieval,” Pattern Recognition, vol. 51, pp. 72–84, 2016.
View at: Publisher Site | Google Scholar
Y. Zhu, X. Huang, Q. Huang, and Q. Tian, “Large-scale video copy retrieval with temporal-concentration SIFT,” Neurocomputing, vol. 187, no. 4, pp. 83–91, 2016.
View at: Publisher Site | Google Scholar
R. Harakawa, T. Ogawa, and M. Haseyama, “[Paper] accurate and efficient extraction of hierarchical structure ofWeb communities forWeb video retrieval,” ITE Transactions on Media Technology and Applications, vol. 4, no. 1, pp. 49–59, 2016.
View at: Publisher Site | Google Scholar
L. Gu, J. Liu, and A. Qu, “Performance evaluation and scheme selection of shot boundary detection and keyframe extraction in content-based video retrieval,” International Journal of Digital Crime and Forensics, vol. 9, no. 4, pp. 15–29, 2017.
View at: Publisher Site | Google Scholar
J. Jung, I. Yoon, S. Lee, and J. Paik, “Normalized metadata generation for human retrieval using multiple video surveillance cameras,” Sensors, vol. 16, no. 7, p. 963, 2016.
View at: Publisher Site | Google Scholar
R. M. Bommisetty, A. Khare, and M. Khare, “Content-based video retrieval using integration of curvelet transform and simple linear iterative clustering[J],” International Journal of Image and Graphics, vol. 132, no. 16, pp. 6–9, 2021.
View at: Google Scholar
S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137–1149, 2017.
View at: Publisher Site | Google Scholar
T. Roska and L. O. Chua, “The CNN universal machine: an analogic array computer[J],” IEEE Transactions on Circuits & Systems II Analog & Digital Signal Processing, vol. 40, no. 3, pp. 163–173, 2015.
View at: Google Scholar
Z. Zeng, T. Huang, and W. X. Zheng, “Multistability of recurrent neural networks with time-varying delays and the piecewise linear activation function[J],” Neurocomputing, vol. 21, no. 8, pp. 1371–1377, 2016.
View at: Google Scholar
H. Zhang, Y. Li, Z. Lv, A. K. Sangaiah, and T. Huang, “A real-time and ubiquitous network attack detection based on deep belief network and support vector machine,” IEEE/CAA Journal of Automatica Sinica, vol. 7, no. 3, pp. 790–799, 2020.
View at: Publisher Site | Google Scholar
J. M. Wolterink, T. Leiner, M. A. Viergever, and I. Isgum, “Generative adversarial networks for noise reduction in low-dose CT,” IEEE Transactions on Medical Imaging, vol. 36, no. 12, pp. 2536–2545, 2017.
View at: Publisher Site | Google Scholar
G. E. Dahl, D. Yu, and L. Deng, “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition[J],” IEEE Transactions on Audio Speech and Language Processing, vol. 20, no. 1, pp. 30–42, 2011.
View at: Google Scholar
J. Feng, Y. Lei, and L. Jing, “Deep neural networks: a promising tool for fault characteristic mining and intelligent diagnosis of rotating machinery with massive data[J],” Mechanical Systems and Signal Processing, vol. 72-73, pp. 303–315, 2016.
View at: Google Scholar
S. Arivazhagan, R. N. Shebiah, R. Harini, and S. Swetha, “Human action recognition from RGB-D data using complete local binary pattern,” Cognitive Systems Research, vol. 58, pp. 94–104, 2019.
View at: Publisher Site | Google Scholar
N. R. Zhou, X. X. Liu, and Y. L. Chen, “Quantum K-Nearest-Neighbor image classification algorithm based on K-L transform[J],” International Journal of Theoretical Physics, vol. 60, no. 4, pp. 1–16, 2021.
View at: Publisher Site | Google Scholar
M. Perez, S. Avila, D. Moreira et al., “Video pornography detection through deep learning techniques and motion information,” Neurocomputing, vol. 230, no. 5, pp. 279–293, 2017.
View at: Publisher Site | Google Scholar
Y. L. Zheng and L. Zhang, “Plant leaf image recognition method based on convolutional neural network based on Transfer Learning [J],” Journal of agricultural machinery, vol. 49, no. S1, pp. 354–359, 2018.
View at: Google Scholar
Y. Xiao, X. Huang, and K. Liu, “Model transferability from ImageNet to lithography hotspot detection,” Journal of Electronic Testing, vol. 37, no. 1, pp. 141–149, 2021.
View at: Publisher Site | Google Scholar
Y. Ma, G. Luo, X. Zeng, and A. Chen, “Transfer learning for cross-company software defect prediction,” Information and Software Technology, vol. 54, no. 3, pp. 248–256, 2012.
View at: Publisher Site | Google Scholar
V. Jayaram, M. Alamgir, Y. Altun, B. Scholkopf, and M. Grosse-Wentrup, “Transfer Learning in Brain-Computer Interfaces Abstract\uFFFDThe performance of brain-computer interfaces (BCIs) improves with the amount of avail,” IEEE Computational Intelligence Magazine, vol. 11, no. 1, pp. 20–31, 2016.
View at: Publisher Site | Google Scholar
F. Montalbo, “A computer-aided diagnosis of brain tumors using a fine-tuned YOLO-based model with transfer learning[J],” KSII Transactions on Internet and Information Systems, vol. 14, no. 12, pp. 4816–4834, 2021.
View at: Google Scholar
D. Needell, N. Srebro, and R. Ward, “Stochastic gradient descent and the randomized Kaczmarz algorithm[J],” Mathematical Programming, vol. 155, no. 12, pp. 549–573, 2013.
View at: Google Scholar
F. Ramos and L. Ott, “Hilbert maps: scalable continuous occupancy mapping with stochastic gradient descent[J],” The International Journal of Robotics Research, vol. 35, no. 14, pp. 1717–1730, 2015.
View at: Google Scholar

Copyright

Copyright © 2021 Yiping Gao. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

1331

Downloads

577

Citations