深度卷积神经网络的柔性剪枝策略

doi:10.11959/j.issn.1000-0801.2022004

电信科学 ›› 2022, Vol. 38 ›› Issue (1): 83-94.doi: 10.11959/j.issn.1000-0801.2022004

深度卷积神经网络的柔性剪枝策略

陈靓¹^,², 钱亚冠¹^,², 何志强¹^,², 关晓惠³, 王滨⁴, 王星⁴

¹ 浙江科技学院理学院/大数据学院，浙江杭州 310023
² 海康威视-浙江科技学院边缘智能安全联合实验室，浙江杭州 310023
³ 浙江水利水电学院信息工程与艺术设计学院，浙江杭州310023
⁴ 浙江大学电气工程学院，浙江杭州 310063

修回日期:2021-12-13 出版日期:2022-01-20 发布日期:2022-01-01
作者简介:陈靓（1995- ），男，浙江科技学院硕士生，主要研究方向为网络剪枝
钱亚冠（1976- ），男，博士，浙江科技学院副教授，主要研究方向为深度学习、人工智能安全、大数据处理
何志强（1996- ），男，浙江科技学院硕士生，主要研究方向为网络剪枝
关晓惠（1977- ），女，博士，浙江水利水电学院副教授，主要研究方向为数字图像处理与模式识别
王滨（1978- ），男，博士，浙江大学研究员，主要研究方向为人工智能安全、物联网安全、密码学
王星（1985- ），男，博士，浙江大学在站博士后，主要研究方向为机器学习与物联网安全
基金资助:
国家重点研发计划项目(2018YFB2100400);国家自然科学基金资助项目(61902082)

A flexible pruning on deep convolutional neural networks

Liang CHEN¹^,², Yaguan QIAN¹^,², Zhiqiang HE¹^,², Xiaohui GUAN³, Bin WANG⁴, Xing WANG⁴

¹ School of Science/School of Big-data Science, Zhejiang University of Science and Technology, Hangzhou 310023, China
² Hikvision-Zhejiang University of Science and Technology Edge Intelligence Security Lab, Hangzhou 310023, China
³ College of Information Engineering ＆ Art Design, Zhejiang University of Water Resources and Electric Power, Hangzhou 310023, China
⁴ College of Electrical Engineering, Zhejiang University, Hangzhou 310063, China

Revised:2021-12-13 Online:2022-01-20 Published:2022-01-01
Supported by:
The National Key Research and Development Program of China(2018YFB2100400);The National Natural Science Foundation of China(61902082)

摘要/Abstract

摘要：

摘要：尽管深度卷积神经网络在多种应用中取得了极大的成功，但其结构的冗余性导致模型过大的存储容量和过高的计算代价，难以部署到资源受限的边缘设备中。网络剪枝是消除网络冗余的一种有效途径，为了找到在有限资源下最佳的神经网络模型架构，提出了一种高效的柔性剪枝策略。一方面，通过计算通道贡献量，兼顾通道缩放系数的分布情况；另一方面，通过对剪枝结果的合理估计及预先模拟，提高剪枝过程的效率。基于VGG16与ResNet56在CIFAR-10的实验结果表明，柔性剪枝策略分别降低了71.3%和54.3%的浮点运算量，而准确率仅分别下降0.15个百分点和0.20个百分点。

关键词: 卷积神经网络, 网络剪枝, 缩放系数, 通道贡献量

Abstract:

Despite the successful application of deep convolutional neural networks, due to the redundancy of its structure, the large memory requirements and the high computing cost lead it hard to be well deployed to the edge devices with limited resources.Network pruning is an effective way to eliminate network redundancy.An efficient flexible pruning strategy was proposed in the purpose of the best architecture under the limited resources.The contribution of channels was calculated considering the distribution of channel scaling factors.Estimating the pruning result and simulating in advance increase efficiency.Experimental results based on VGG16 and ResNet56 on CIFAR-10 show that the flexible pruning reduces FLOPs by 71.3% and 54.3%, respectively, while accuracy by only 0.15 percentage points and 0.20 percentage points compared to the benchmark model.

Key words: convolutional neural network, network pruning, scaling factor, channel contribution

中图分类号:

TP183

陈靓, 钱亚冠, 何志强, 关晓惠, 王滨, 王星. 深度卷积神经网络的柔性剪枝策略[J]. 电信科学, 2022, 38(1): 83-94.

Liang CHEN, Yaguan QIAN, Zhiqiang HE, Xiaohui GUAN, Bin WANG, Xing WANG. A flexible pruning on deep convolutional neural networks[J]. Telecommunications Science, 2022, 38(1): 83-94.

图/表 10

图1

图2

图3

图4

表1

表2

表3

图5

图6

表4

参考文献 33

[1]	KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017,60(6): 84-90.
[2]	HE K M , ZHANG X Y , REN S Q ,et al. Deep residual learning for image recognition[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016: 770-778.
[3]	唐博恒, 柴鑫刚 . 基于云边协同的计算机视觉推理机制[J]. 电信科学, 2021,37(5): 72-81.
	TANG B H , CHAI X G . Cloud-edge collaboration based computer vision inference mechanism[J]. Telecommunications Science, 2021,37(5): 72-81.
[4]	WANG Y , BIAN Z P , HOU J H ,et al. Convolutional neural networks with dynamic regularization[EB]. 2019.
[5]	COURBARIAUX M , BENGIO Y , DAVID J P . Binary Connect:training deep neural networks with binary weights during propagations[J]. CoRR, 2015.
[6]	HAN S , POOL J , TRAN J ,et al. Learning both weights and connections for efficient neural networks[J]. CoRR, 2015.
[7]	WEN W J , YANG F , SU Y F ,et al. Learning low-rank structured sparsity in recurrent neural networks[C]// Proceedings of 2020 IEEE International Symposium on Circuits and Systems (ISCAS). Piscataway:IEEE Press, 2020: 1-4.
[8]	HE Y H , ZHANG X Y , SUN J . Channel pruning for accelerating very deep neural networks[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 1398-1406.
[9]	LI H , KADAV A , DURDANOVIC I ,et al. Pruning filters for efficient ConvNets[EB]. 2016.
[10]	HU H Y , PENG R , TAI Y W ,et al. Network trimming:a data-driven neuron pruning approach towards efficient deep architectures[EB]. 2016.
[11]	LUO J H , WU J X , LIN W Y . ThiNet:a filter level pruning method for deep neural network compression[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 5068-5076.
[12]	LIU Z , LI J G , SHEN Z Q ,et al. Learning efficient convolutional networks through network slimming[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 2755-2763.
[13]	YE J B , LU X , LIN Z ,et al. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers[EB]. 2018.
[14]	IOFFE S , SZEGEDY C . Batch normalization:accelerating deep network training by reducing internal covariate shift[J]. CoRR, 2015.
[15]	HE Y , DONG X Y , KANG G L ,et al. Asymptotic soft filter pruning for deep convolutional neural networks[J]. IEEE Transactions on Cybernetics, 2020,50(8): 3594-3604.
[16]	LIN M B , JI R R , ZHANG Y X ,et al. Channel pruning via automatic structure search[C]// Proceedings of Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. California:International Joint Conferences on Artificial Intelligence Organization, 2020.
[17]	LECUN Y , DENKER J S , Solla S A . Optimal brain damage[C]// Proceedings of the Advances in Neural Information Processing Systems. Berlin:Springer, 1989: 598-605.
[18]	LIU Z , SUN M J , ZHOU T H ,et al. Rethinking the value of network pruning[EB]. 2018.
[19]	YE Y , YOU G M , FWU J K ,et al. Channel pruning via optimal thresholding[M]// Communications in Computer and Information Science. Cham: Springer International Publishing, 2020: 508-516.
[20]	ZHUANG T , ZHANG Z X , HUANG Y H ,et al. Neuron-level structured pruning using polarization regularizer[C]// Advances in Neural Information Processing Systems. 2020: 1-13.
[21]	RONG J T , YU X Y , ZHANG M Y ,et al. Soft Taylor pruning for accelerating deep convolutional neural networks[C]// Proceedings of IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society. Piscataway:IEEE Press, 2020: 5343-5349.
[22]	CAI L H , AN Z L , YANG C G ,et al. Softer pruning,incremental regularization[C]// Proceedings of 2020 25th International Conference on Pattern Recognition (ICPR). Piscataway:IEEE Press, 2021: 224-230.
[23]	XU X Z , CHEN Q M , XIE L ,et al. Batch-normalization-based soft filter pruning for deep convolutional neural networks[C]// Proceedings of 2020 16th International Conference on Control,Automation,Robotics and Vision (ICARCV). Piscataway:IEEE Press, 2020: 951-956.
[24]	LIU Z C , MU H Y , ZHANG X Y ,et al. MetaPruning:meta learning for automatic neural network channel pruning[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2019: 3295-3304.
[25]	DING Y X , ZHAO W G , WANG Z P ,et al. Automaticlly learning featurs of android apps using CNN[C]// Proceedings of 2018 International Conference on Machine Learning and Cybernetics (ICMLC). Piscataway:IEEE Press, 2018: 331-336.
[26]	LIN M B , JI R R , ZHANG Y X ,et al. Channel pruning via automatic structure search[C]// Proceedings of Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. California:International Joint Conferences on Artificial Intelligence Organization, 2020: 673-679.
[27]	SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition[J]. CoRR, 2014.
[28]	KRIZHEVSKY A , HINTON G . Learning multiple layers of features from tiny images[J]. Handbook of Systemic Autoimmune Diseases. 2009,1(4)
[29]	LUO J H , WU J X . Neural network pruning with residual-connections and limited-data[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 1455-1464.
[30]	ZHOU Y F , ZHANG Y , WANG Y F ,et al. Accelerate CNN via recursive Bayesian pruning[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2019: 3305-3314.
[31]	PENG H Y , WU J X , CHEN S F ,et al. Collaborative channel pruning for deep networks[C]// Proceedings of the International Conference on Machine Learning. New York:ACM Press, 2019: 5113-5122.
[32]	YANG H R , WEN W , LI H ,et al. DeepHoyer:learning sparser neural network with differentiable scale-invariant sparsity measures[EB]. 2019.
[33]	HE Y , DING Y H , LIU P ,et al. Learning filter pruning criteria for deep convolutional neural networks acceleration[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 2006-2015.

数据集	剪枝方法	准确率	准确率下降	浮点运算量/10⁶	浮点运算量减少
CIFAR-10	基线	93.88%	-	314	-
	NS^[12]	93.62%	0.26%	155	51%
	OT^[19]	93.87%	0.01%	163	48%
	PR^[20]	93.92%	-0.04%	144	54%
	AOFP^[25]	94.03%	-0.15%	186	41%
	本文	94.04%	-0.16%	140	55%
CIFAR-100	基线	73.83%	-	314	-
	NS^[12]	74.20%	-0.37%	195	38%
	OT^[19]	73.99%	0.21%	197	37%
	PR^[20]	74.25%	-0.42%	179	43%
	本文	74.33%	-0.50%	174	45%

剪枝方法	剪枝率	网络架构	总通道数	浮点运算量/10⁶	参数量/10⁶	准确率（Scratch-E）	准确率（Scratch-B）
基线	0	64-64-128-128-256-256-256-512-512-512-512-512-512	4 224	314	20.04	93.88%	93.88%
NS^[12]	70%	30-62-123-126-247-238-199-101-27-10-9-6-89	1 268	155	1.68	93.44%	93.91%
OT^[19]	73%	26-59-114-120-206-172-128-98-56-38-27-32-57	1 133	113	1.16	93.28%	93.86%
本文	74%	32-64-115-128-228-198-118-65-25-12-19-22-74	1 100	140	1.26	93.47%	94.04%
本文	76%	31-64-107-128-214-180-103-56-21-10-16-19-59	1 017	116	1.05	93.30%	93.91%
RBP^[30]	86%	50-63-123-108-104-57-23-14-9-8-6-7-11	583	9.0	0.39	92.43%	93.20%
本文	80%	26-62-91-121-181-152-86-41-13-7-10-11-61	862	90	0.78	92.92%	93.73%
本文	86%	19-47-65-92-126-103-52-23-6-3-4-5-47	592	47	0.37	91.92%	93.37%

数据集	剪枝方法	准确率	准确率下降	浮点运算量/10⁶	浮点运算量减少
CIFAR-10	基线	93.80%	—	127	—
	NS^[12]	93.27%	0.53%	66	48%
	PR^[20]	93.83%	-0.03%	67	47%
	SFP^[21]	93.35%	0.45%	59	53%
	BN-SFP^[23]	93.29%	0.51%	59	53%
	ASFP^[15]	93.12%	0.68%	59	53%
	ABCPruner^[26]	93.23%	0.57%	58	54%
	DCP^[31]	93.81%	-0.01%	67	47%
	DeepHoyer^[32]	93.54%	0.26%	67	47%
	LFPC^[33]	93.72%	0.08%	67	47%
	本文	93.85%	-0.05%	66	48%
	本文	93.60%	0.20%	58	54%
CIFAR-100	基线	72.49%	—	127	—
	NS^[12]	71.40%	1.09%	96	24%
	PR^[20]	72.43%	0.06%	95	25%
	本文	72.51%	-0.02%	95	25%

数据集	模型	剪枝方法	准确率	准确率提升	浮点运算量/10⁶	总通道数
CIFAR-10	VGG16	基线	93.88%	—	314	4 224
		本文（重构）	94.20%	0.32%	313	2 359
	ResNet56	基线	93.80%	—	127	2 032
		本文（重构）	94.04%	0.24%	126	2 181
CIFAR-100	VGG16	基线	73.83%	—	314	4 224
		本文（重构）	74.20%	0.37%	310	3 115
	ResNet56	基线	72.49%	—	127	2 032
		本文（重构）	72.72%	0.23%	114	2 141

深度卷积神经网络的柔性剪枝策略

A flexible pruning on deep convolutional neural networks

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 33

相关文章 7

Metrics

推荐阅读 0

[1]	刘子豪, 贾小军, 张素兰, 徐志玲, 张俊. 一种融合MeanShift聚类分析和卷积神经网络的Vibe++背景分割方法[J]. 电信科学, 2021, 37(3): 133-145.
[2]	曹靖城, 张继东, 史国杰. 一种使用边缘增强技术提高相似图片检索召回率的方法[J]. 电信科学, 2021, 37(1): 76-84.
[3]	贾川民, 赵政辉, 王苫社, 马思伟. 基于神经网络的图像视频编码[J]. 电信科学, 2019, 35(5): 32-42.
[4]	苏赋,吕沁,罗仁泽. 基于深度学习的图像分类研究综述[J]. 电信科学, 2019, 35(11): 58-74.
[5]	李强,康子路. 基于深度时空特征卷积—池化的视频人群计数方法[J]. 电信科学, 2018, 34(6): 72-79.
[6]	徐宏伟,严迪群,阳帆,王让定,金超,向立. 基于卷积神经网络的电子变调语音检测算法[J]. 电信科学, 2018, 34(2): 46-57.
[7]	邢惠钧,昌硕. 基于移动小车的行人监控系统[J]. 电信科学, 2017, 33(2): 120-127.