摘要
神经网络压缩技术的出现缓解了深度神经网络模型在资源受限设备中的应用难题,如移动端或嵌入式设备.但神经网络压缩技术在压缩处理的自动化、稀疏度与硬件部署之间的矛盾、避免压缩后模型重训练等方面存在困难.本文在回顾经典神经网络模型和现有神经网络压缩工具的基础上,总结参数剪枝、参数量化、低秩分解和知识蒸馏四类压缩方法的代表性压缩算法的优缺点,概述压缩方法的评测指标和常用数据集,并分析各种压缩方法在不同任务和硬件资源约束中的性能表现,展望神经网络压缩技术具有前景的研究方向.
The development of neural network compression relieves the difficulty of deep neural networks running on resource-restricted devices,such as mobile or embedded devices.However,neural network compression encounters challenges in automation of compression,conflict of the sparsity and hardware deployment,avoidance of retraining compressed networks and other issues.This paper firstly reviews classic neural network models and current compression toolkits.Secondly,this paper summarizes advantages and weaknesses of representative compression methods of parameter pruning,quantization,low-rank factorization and distillation.This paper lists evaluating indicators and common datasets for the performance evaluation and then analyzes compression performance in different tasks and resource constraints.Finally,promising development trends are stated in this paper as references for promoting the neural network compression technique.
作者
尹文枫
梁玲燕
彭慧民
曹其春
赵健
董刚
赵雅倩
赵坤
YIN Wen-Feng;LIANG Ling-Yan;PENG Hui-Min;CAO Qi-Chun;ZHAO Jian;DONG Gang;ZHAO Ya-Qian;ZHAO Kun(Inspur Electronic Information Industry Co.Ltd.,Jinan 250101,China;Guangdong Inspur Big Data Research Co.Ltd.,Guangzhou 510632,China)
出处
《计算机系统应用》
2020年第9期16-25,共10页
Computer Systems & Applications
关键词
神经网络压缩
参数剪枝
参数量化
低秩分解
知识蒸馏
neural network compression
parameter pruning
parameter quantizatipn
low-rank factorization
knowledge distillation
