摘要
视频片段检索旨在利用用户给出的自然语言查询语句,在一个长视频中找到最符合语句描述的目标视频片段.视频中包含丰富的视觉、文本、语音信息,如何理解视频中提供的信息,以及查询语句提供的文本信息,并进行跨模态信息的对齐与交互,是视频片段检索任务的核心问题.系统梳理了当前视频片段检索领域中的相关工作,将它们分为两大类:基于排序的方法和基于定位的方法.其中,基于排序的方法又可细分为预设候选片段的方法和有指导地生成候选片段的方法,而基于定位的方法则可分为一次定位的方法和迭代定位的方法.同时对该领域的数据集和评价指标进行了介绍,并对一些模型在多个常用数据集上的性能进行了总结与整理.此外,介绍了该任务的延伸工作,如大规模视频片段检索工作等.最后,对视频片段检索未来的发展方向进行了展望.
Given a natural language sentence as the query,the task of video moment retrieval aims to localize the most relevant video moment in a long untrimmed video.Based on the rich visual,text,and audio information contained in the video,how to fully understand the visual information provided in the video and utilize the text information provided by the query sentence to enhance the generalization and robustness of model,and how to align and interact cross-modal information are crucial challenges of the video moment retrieval.This study systematically sorts out the work in the field of video moment retrieval,and divides them into ranking-based methods and localization-based methods.Thereinto,the ranking-based methods can be further divided into the methods of presetting candidate clips,and the methods of generating candidate clips with guidance;the localization-based methods can be divided into one-time localization methods and iterative localization ones.The datasets and evaluation metrics of this fieldf are also summarized and the latest advances are reviewed.Finally,the related extension task is introduced,e.g.,moment localization from video corpus,and the survey is concluded with a discussion on promising trends.
作者
王妍
詹雨薇
罗昕
刘萌
许信顺
WANG Yan;ZHAN Yu-Wei;LUO Xin;LIU Meng;XU Xin-Shun(School of Software,Shandong University,Jinan 250101,China;School of Computer Science and Technology,Shandong Jianzhu University,Jinan 250101,China)
出处
《软件学报》
EI
CSCD
北大核心
2023年第2期985-1006,共22页
Journal of Software
基金
国家自然科学基金(61991411,61872428,62006142,62172256)
山东省重点研发项目(2019JZZY010127)
山东省自然科学基金(ZR2019ZD06,ZR2020QF036)。
关键词
视频片段检索
自然语言时序定位视频片段
视频理解
深度学习
人工智能
video moment retrieval
temporal activity localization via language
video understanding
deep learning
artificial intelligence
