摘要
现有的一阶段方法分别提取视觉特征映射和文本特征,并进行多模态推理来预测被引用对象的边界框.这些方法存在以下两个缺点:首先,预先训练的视觉特征提取器在视觉特征中引入了与文本无关的视觉信号,阻碍了多模态交互;其次,现有模型的推理过程缺乏对语言建模的可视化指导.基于上述缺点,现有的一阶段方法的推理能力是有限的.提出了一种提取文本相关视觉特征映射的低阶交互和一种整合视觉特征的高阶交互来指导语言建模,并进一步对视觉特征进行多步推理.在此基础上,提出了一种新的网络结构,称为双路径多级交互网络.在5种常用的视觉定位数据集上进行了实验,结果表明该方法具有较好的性能和实时性.
This study explores the multimodal understanding and reasoning for one-stage visual grounding.Existing one-stage methods extract visual feature maps and textual features separately,and then,multimodal reasoning is performed to predict the bounding box of the referred object.These methods suffer from the following two weaknesses:Firstly,the pre-trained visual feature extractors introduce textunrelated visual signals into the visual features that hinder multimodal interaction.Secondly,the reasoning process followed in these two methods lacks visual guidance for language modeling.It is clear from these shortcomings that the reasoning ability of existing one-stage methods is limited.We propose a lowlevel interaction to extract text-related visual feature maps,and a high-level interaction to incorporate visual features in guiding the language modeling and further performing multistep reasoning on visual features.Based on the proposed interactions,we present a novel network architecture called the dual-path multilevel interaction network(DPMIN).Furthermore,experiments on five commonly used visual grounding datasets are conducted.The results demonstrate the superior performance of the proposed method and its real-time applicability.
作者
王月
叶加博
林欣
WANG Yue;YE Jiabo;LIN Xin(School of Computer Science and Technology,East China Normal University,Shanghai 200062,China)
出处
《华东师范大学学报（自然科学版）》
CAS
CSCD
北大核心
2024年第2期65-75,共11页
Journal of East China Normal University(Natural Science)
关键词
视觉定位
多模态推理
引用表达
visual grounding
multimodal understanding
referring expressions
