摘要
图像描述生成有着重要的理论意义与应用价值,在计算机视觉与自然语言处理领域皆受到广泛关注.基于注意力机制的图像描述生成方法,在同一时刻融合当前词和视觉信息以生成目标词,忽略了视觉连贯性及上下文信息,导致生成描述与参考描述存在差异.针对这一问题,提出一种基于视觉关联与上下文双注意力机制的图像描述生成方法(visual relevance and context dual attention, VRCDA).视觉关联注意力在传统视觉注意力中增加前一时刻注意力向量以保证视觉连贯性,上下文注意力从全局上下文中获取更完整的语义信息,以充分利用上下文信息,进而指导生成最终的图像描述文本.在MSCOCO和Flickr30k两个标准数据集上进行了实验验证,结果表明所提出的VRCDA方法能够有效地生成图像语义描述,相比于主流的图像描述生成方法,在各项评价指标上均取得了较高的提升.
Image captioning is of great theoretical significance and application value, which has attracted wide attention in computer vision and natural language processing. The existing attention mechanism-based image captioning methods integrate the current word and visual cues at the same moment to generate the target word, but they neglect the visual relevance and contextual information, which results in a difference between the generated caption and the ground truth. To address this problem, this paper presents the visual relevance and context dual attention(VRCDA) method. The visual relevance attention incorporates the attention vector of the previous moment into the traditional visual attention to ensure visual relevance, and the context attention is used to obtain much complete semantic information from the global context for better use of the context. In this way, the final image caption is generated via visual relevance and context information. The experiments on the MSCOCO and Flickr30k benchmark datasets demonstrate that VRCDA can effectively describe the image semantics, and compared with several state-of-the-art methods of image captioning, VRCDA can yield superior performance in all evaluation metrics.
作者
刘茂福
施琦
聂礼强
LIU Mao-Fu;SHI Qi;NIE Li-Qiang(School of Computer Science and Technology,Wuhan University of Science and Technology,Wuhan 430065,China;School of Computer Science and Technology,Shandong University,Qingdao 266237,China)
出处
《软件学报》
EI
CSCD
北大核心
2022年第9期3210-3222,共13页
Journal of Software
