摘要
针对现有壁画深度学习修复方法,存在上下文信息关注不足及结果欠佳的问题,提出了一种多层次特征融合与超图卷积的生成对抗修复模型。首先,利用金字塔特征分层对壁画进行多尺度层次特征提取,并采用混合空洞卷积单元扩大多层特征提取感受野,以克服单尺度卷积操作对于壁画特征提取能力不足的问题。然后,提出多分支短链融合层及门控机制融合多分支特征方法,将相邻分支间的特征信息进行融合,使融合后的壁画特征图中既有同分支的特征,又有相邻分支的特征,以提高特征信息的利用率;并引入门控机制对特征进行选择融合,以减少细节信息的丢失。接着,将融合特征通过卷积长短期记忆网络(ConvLSTM)特征注意力方法,增强对壁画上下文信息的关注。最后,设计超图卷积壁画长程特征增强模块,通过在编码器和解码器的跳跃连接之间建立超图卷积层,利用超图卷积捕获编码器的空间特征信息,并将其迁移到解码器中,有助于解码器更好地生成壁画图像,以加强特征的长程依赖关系,并与SN-PatchGAN判别器对抗博弈从而完成修复。通过敦煌壁画数字化修复实验,结果表明:所提方法客观评价优于对比算法,对于破损壁画修复结果更加清晰自然。
Aiming at the problems of low feature utilization and insufficient attention to context information in existing deep learning image res-toration methods,a generative adversarial mural restoration model based on multi-level feature fusion and hypergraph convolution was proposed.Firstly,a pyramid feature extraction layer structure was designed,and the multi-scale feature extraction was carried out by using the pyramid fea-ture layer.The mixed dilated convolution unit was used to expand the receptive field of multi-layer feature extraction,which overcomes the prob-lem of insufficient feature extraction ability of single-scale convolution operation.Then,a multi-branch short chain fusion layer and a gating method were proposed to fuse the multi branch features,and the feature information between the adjacent branches was fused,so that the fused mural feature map has both the same branch features and the adjacent branch features,improving the utilization rate of feature information,and introducing the gating mechanism to select and fuse features to reduce the loss of detail information.Next,the fused features were passed through the ConvLSTM feature attention to enhance the attention to the mural context information.Finally,a hypergraph convolution mural long-range feature enhancement module was designed.By establishing a hypergraph convolution layer between the skip connection of the encoder and the decoder,the spatial feature information of the encoder was captured by hypergraph convolution and transferred to the decoder,which helpes the decoder to generate mural images and strengthenes the long-range dependence of features.Afterwards,the mural restoration was completed in a game against the SN-PatchGAN discriminator.Through the restoration experiments of digital Dunhuang murals,the results showed that the pro-posed method is superior to the comparative algorithms,and the restoration results of damaged murals are clearer and more natural.
作者
陈永
陶美风
赵梦雪
CHEN Yong;TAO Meifeng;ZHAO Mengxue(School of Elec.and Info.Eng.,Lanzhou Jiaotong Univ.,Lanzhou 730070,China;Gansu Provincial Eng.Research Center for Artificial Intelligence and Graphics&Image Processing,Lanzhou 730070,China)
出处
《工程科学与技术》
EI
CAS
CSCD
北大核心
2024年第3期208-218,共11页
Advanced Engineering Sciences
基金
教育部人文社会科学研究青年基金项目(19YJC760012)
兰州交通大学基础拔尖人才项目(2022JC36)
兰州交通大学重点研发项目(ZDYF2304)。
