摘要
多模态数据的指数级增长使得传统数据库在存储和检索方面遇到挑战,而多模态哈希通过融合多模态特征并映射成二进制哈希码,能够有效地降低数据库的存储开销并提高其检索效率.虽然目前已经有许多针对多模态哈希的工作取得了较好的效果,但是仍然存在着3个重要问题:(1)已有方法偏向于考虑所有样本都是模态完整的,然而在实际检索场景中,样本缺失部分模态的情况依然存在;(2)大多数方法都是基于浅层学习模型,这不可避免地限制了模型的学习能力,从而影响最终的检索效果;(3)针对模型学习能力弱的问题已提出了基于深度学习框架的方法,但是它们在提取各个模态的特征后直接采用了向量拼接等粗粒度特征融合方法,未能有效地捕获深层语义信息,从而弱化了哈希码的表示能力并影响最终的检索效果.针对以上问题,提出了PMH-F^(3)模型.该模型针对样本缺失部分模态的情况,实现了部分多模态哈希.同时,基于深层网络架构,利用Transformer编码器,以自注意力方式捕获深层语义信息,并实现细粒度的多模态特征融合.基于MIRFlickr和MSCOCO数据集进行了充分实验并取得了最优的检索效果.实验结果表明:所提出的PMH-F^(3)模型能够有效地实现部分多模态哈希,并可应用于大规模多模态数据检索.
Due to the exponential growth of multimodal data,traditional databases are confronted with challenges in terms of storage and retrieval.Multimodal hashing is able to effectively reduce the storage cost of databases and improve retrieval efficiency by fusing multimodal features and mapping them into binary hash codes.Although many works on multimodal hashing perform well,there are also three important problems to be solved:(1)Existing methods tend to consider that all samples are modality-complete,while in practical retrieval scenarios,it is also common for samples to miss partial modalities;(2)Most methods are based on shallow learning models,which inevitably limits models’learning ability and affects the final retrieval performance;(3)Some methods based on deep learning framework have been proposed to address the issue of weak learning ability,but they directly use coarse-grained feature fusion methods,such as concatenation,after extracting features from different modalities,which fails to effectively capture deep semantic information,thereby weakening the representation ability of hash codes and affecting the final retrieval performance.In response to the above problems,the PMH-F^(3) model is proposed.This model implements partial multimodal hashing for the case of samples missing partial modalities.The model is based on deep network architecture,and the Transformer encoder is used to capture deep semantics in self-attention manner,achieving fine-grained multimodal feature fusion.Sufficient experiments are conducted on MIR Flickr and MS COCO datasets and the best retrieval performance is achieved.The results of experiments show that PMH-F^(3) model can effectively implement partial multimodal hashing and can be applied to large-scale multimodal data retrieval.
作者
殷崭祚
李博涵
王萌
黄瑞龙
吴文隆
王昊奋
YIN Zhan-Zuo;LI Bo-Han;WANG Meng;HUANG Rui-Long;WU Wen-Long;WANG Hao-Fen(College of Computer Science and Technology,Nanjing University of Aeronautics and Astronautics,Nanjing 211106,China;Collaborative Innovation Center of Novel Software Technology and Industrialization,Nanjing 211106,China;National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology(Northwestern Polytechnical University),Xi’an 710119,China;College of Design and Innovation,Tongji University,Shanghai 200092,China)
出处
《软件学报》
EI
CSCD
北大核心
2024年第3期1074-1089,共16页
Journal of Software
基金
国家重点研发计划(2020YFB1708100)
“十四五”民用航天技术预先研究项目(D020101)
国家自然科学基金(62172351)
高安全系统的软件开发与验证技术工业和信息化部重点实验室资助项目(NJ2018014)
河北省软件工程重点实验室项目。
关键词
部分多模态哈希
多模态数据检索
细粒度特征融合
partial multimodal hashing
multimodal data retrieval
fine-grained feature fusion
