摘要
特种视频(本文特指暴力视频)的智能分类技术有助于实现网络信息内容安全的智能监控。针对现有特种视频多模态特征融合时未考虑语义一致性等问题,本文提出了一种基于音视频多模态特征融合与多任务学习的特种视频识别方法。首先,提取特种视频的表观信息和运动信息随时空变化的视觉语义特征及音频信息语义特征;然后,构建具有语义保持的共享特征子空间,以实现音视频多种模态特征的融合;最后,提出基于音视频特征的语义一致性度量和特种视频分类的多任务学习特种视频分类理论框架,设计了对应的损失函数,实现了端到端的特种视频智能识别。实验结果表明,本文提出的算法在Violent Flow和MediaEval VSD 2015两个数据集上平均精度分别为97.97%和39.76%,优于已有研究。结果证明了该算法的有效性,有助于提升特种视频监控的智能化水平。
Classification of special videos is significant for intelligent surveillance of internet content.Existing algorithms that fuse multimodal features forclassification of special videoscannot measure multimodal audio-visual semantic correspondence.An algorithm for recognizing special videos based on multimodal audio-visual feature fusion was proposed herein over the framework of multitask learning.First,audio semantic features and spatial-temporal visual semantic cues,including appearance and motion,were extracted.A latent subspace to fuse audio and visual features whilst preserving their semantic information was learned and developed through jointly learning audio-visual semantic correspondence and special video classification.Subsequently,a multitask learning loss function was presented viacombination of the correspondence loss,obtained based on the measured audio-visual semantic information,and the cross-entropy loss of special video classification.Finally,an end-to-end intelligent system for special video recognition was implemented.Experimental results demonstrate that the accuracy of the proposed algorithm is 97.97%with respect to the Violent Flow dataset,and the average accuracy is 39.76%with respect to the Media Eval VSD 2015 dataset,where by the algorithm outperforms the other existing methods.These results show that the proposed algorithm is effective for improving the intelligence of network content surveillance.
作者
吴晓雨
顾超男
王生进
WU Xiao-yu;GU Chao-nan;WANG Sheng-jin(School of Information and Communication,University of China,Beijing 100024,China;Department of Electronic Engineer,Tsiinghua University,Beijing 100084,China)
出处
《光学精密工程》
EI
CAS
CSCD
北大核心
2020年第5期1177-1186,共10页
Optics and Precision Engineering
基金
国家自然科学基金资助项目(No.61801441)
北京信息科学与技术国家研究中心跨媒体智能专项资助(No.BNR2019TD01022)
“北京市高精尖”学科建设项目(中国传媒大学互联网信息学科)
中国传媒大学中央高校基本科研业务费专项资金资助项目(No.CUC2019B066,No.CUC18A002-2)。
关键词
特种视频识别
特征提取
多模态特征融合
语义一致性度量
多任务学习
special video recognition
feature extraction
multimodal feature fusion
semantic correspondence measurement
multitask learning
