摘要
目标检测模型的轻量化研究虽已产生诸多代表性成果,但现有方法在模型高比例剪枝时会出现检测精度断崖式衰减。在探索主流目标检测网络剪枝性能衰减的根源时发现剪枝后梯度的波动是影响模型性能的关键。为此构建了基于强关联平滑约束的剪枝框架(Pruning Framework based on Strong Correlation Smoothing Constraint,SCSC)。首先将历史梯度及当前梯度定义为自蒸馏理论中的教师及学生,通过学生模仿教师的方式使学生梯度最大程度接近教师梯度,实现梯度平滑;其次依据梯度平滑结果提出基于强关联约束的剪枝方案,将历史梯度与当前梯度组成强关联组,通过强化历史梯度对当前梯度更新的贡献增强模型权重参数稀疏度。在PASCAL VOC2007数据集进行测试,SCSC对比主流剪枝方法取得了2个百分点的平均精度提升;在KITTI数据集中,SCSC剪枝率为80%时,相较于原网络识别精度衰减仅为3个百分点。
Although the research on lightweight object detection models has produced many representative results,these models still suffer from a cliff-like decay of detection accuracy when they are pruned at a high ratio.Some researchers find that the fluctuation of the gradient after pruning is the key factor affecting the model performance when exploring the root cause of the pruning performance degradation of the mainstream object detection networks.Therefore,a pruning framework based on gradient selfdistillation smoothing is constructed and called as SCSC,a pruning framework with strongly correlation smoothing constraints.First,the historical gradient and the current gradient are defined as the teacher and the student in the self-distillation theory,and the student gradient approaches the teacher gradient as much as possible by imitating the teacher,achieving gradient smoothing.Second,based on the gradient smoothing result,a pruning scheme based on strong correlation constraints is proposed.This scheme forms a strong correlation group with the historical gradient and the current gradient,and enhances the sparsity of the model weight parameters by strengthening the contribution of the historical gradient to the current gradient update.Through the experiments on the PASCAL VOC2007 dataset,SCSC achieves a 2 percentages improvement in average precision compared with mainstream pruning methods;on the KITTI dataset,when the SCSC pruning rate is 80%,the recognition accuracy decay is only decreased 3 percentages from that of the original network.
作者
康彬
李卓
邱坤
窦海娥
王磊
郑宝玉
KANG Bin;LI Zhuo;QIU Kun;DOU Haie;WANG Lei;ZHENG Baoyu(School of Internet of Things,Nanjing University of Posts and Telecommunications,Nanjing 210003,China;School of Applied Technology College,Nanjing University of Posts and Telecommunications,Nanjing 210042,China;School of Communications and Information Engineering,Nanjing University of Posts and Telecommunications,Nanjing 210003,China)
出处
《南京邮电大学学报（自然科学版）》
北大核心
2024年第3期72-79,共8页
Journal of Nanjing University of Posts and Telecommunications：Natural Science Edition
基金
国家自然科学基金(62171232,62071255,62371253,62001248)
江苏省重点研发计划(BE2023087)
江苏省高校重点项目(20KJA510009)资助项目。
关键词
卷积神经网络
知识蒸馏
模型剪枝
convolutional neural networks
knowledge distillation
model pruning
