深度卷积神经网络的柔性剪枝策略

doi:10.11959/j.issn.1000-0801.2022004

Abstract

Abstract:

Despite the successful application of deep convolutional neural networks, due to the redundancy of its structure, the large memory requirements and the high computing cost lead it hard to be well deployed to the edge devices with limited resources.Network pruning is an effective way to eliminate network redundancy.An efficient flexible pruning strategy was proposed in the purpose of the best architecture under the limited resources.The contribution of channels was calculated considering the distribution of channel scaling factors.Estimating the pruning result and simulating in advance increase efficiency.Experimental results based on VGG16 and ResNet56 on CIFAR-10 show that the flexible pruning reduces FLOPs by 71.3% and 54.3%, respectively, while accuracy by only 0.15 percentage points and 0.20 percentage points compared to the benchmark model.

Key words: convolutional neural network, network pruning, scaling factor, channel contribution

CLC Number:

TP183

Liang CHEN, Yaguan QIAN, Zhiqiang HE, Xiaohui GUAN, Bin WANG, Xing WANG. A flexible pruning on deep convolutional neural networks[J]. Telecommunications Science, 2022, 38(1): 83-94.

Figures/Tables 10

References 33

[1]	KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017,60(6): 84-90.
[2]	HE K M , ZHANG X Y , REN S Q ,et al. Deep residual learning for image recognition[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2016: 770-778.
[3]	唐博恒, 柴鑫刚 . 基于云边协同的计算机视觉推理机制[J]. 电信科学, 2021,37(5): 72-81.
	TANG B H , CHAI X G . Cloud-edge collaboration based computer vision inference mechanism[J]. Telecommunications Science, 2021,37(5): 72-81.
[4]	WANG Y , BIAN Z P , HOU J H ,et al. Convolutional neural networks with dynamic regularization[EB]. 2019.
[5]	COURBARIAUX M , BENGIO Y , DAVID J P . Binary Connect:training deep neural networks with binary weights during propagations[J]. CoRR, 2015.
[6]	HAN S , POOL J , TRAN J ,et al. Learning both weights and connections for efficient neural networks[J]. CoRR, 2015.
[7]	WEN W J , YANG F , SU Y F ,et al. Learning low-rank structured sparsity in recurrent neural networks[C]// Proceedings of 2020 IEEE International Symposium on Circuits and Systems (ISCAS). Piscataway:IEEE Press, 2020: 1-4.
[8]	HE Y H , ZHANG X Y , SUN J . Channel pruning for accelerating very deep neural networks[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 1398-1406.
[9]	LI H , KADAV A , DURDANOVIC I ,et al. Pruning filters for efficient ConvNets[EB]. 2016.
[10]	HU H Y , PENG R , TAI Y W ,et al. Network trimming:a data-driven neuron pruning approach towards efficient deep architectures[EB]. 2016.
[11]	LUO J H , WU J X , LIN W Y . ThiNet:a filter level pruning method for deep neural network compression[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 5068-5076.
[12]	LIU Z , LI J G , SHEN Z Q ,et al. Learning efficient convolutional networks through network slimming[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2017: 2755-2763.
[13]	YE J B , LU X , LIN Z ,et al. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers[EB]. 2018.
[14]	IOFFE S , SZEGEDY C . Batch normalization:accelerating deep network training by reducing internal covariate shift[J]. CoRR, 2015.
[15]	HE Y , DONG X Y , KANG G L ,et al. Asymptotic soft filter pruning for deep convolutional neural networks[J]. IEEE Transactions on Cybernetics, 2020,50(8): 3594-3604.
[16]	LIN M B , JI R R , ZHANG Y X ,et al. Channel pruning via automatic structure search[C]// Proceedings of Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. California:International Joint Conferences on Artificial Intelligence Organization, 2020.
[17]	LECUN Y , DENKER J S , Solla S A . Optimal brain damage[C]// Proceedings of the Advances in Neural Information Processing Systems. Berlin:Springer, 1989: 598-605.
[18]	LIU Z , SUN M J , ZHOU T H ,et al. Rethinking the value of network pruning[EB]. 2018.
[19]	YE Y , YOU G M , FWU J K ,et al. Channel pruning via optimal thresholding[M]// Communications in Computer and Information Science. Cham: Springer International Publishing, 2020: 508-516.
[20]	ZHUANG T , ZHANG Z X , HUANG Y H ,et al. Neuron-level structured pruning using polarization regularizer[C]// Advances in Neural Information Processing Systems. 2020: 1-13.
[21]	RONG J T , YU X Y , ZHANG M Y ,et al. Soft Taylor pruning for accelerating deep convolutional neural networks[C]// Proceedings of IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society. Piscataway:IEEE Press, 2020: 5343-5349.
[22]	CAI L H , AN Z L , YANG C G ,et al. Softer pruning,incremental regularization[C]// Proceedings of 2020 25th International Conference on Pattern Recognition (ICPR). Piscataway:IEEE Press, 2021: 224-230.
[23]	XU X Z , CHEN Q M , XIE L ,et al. Batch-normalization-based soft filter pruning for deep convolutional neural networks[C]// Proceedings of 2020 16th International Conference on Control,Automation,Robotics and Vision (ICARCV). Piscataway:IEEE Press, 2020: 951-956.
[24]	LIU Z C , MU H Y , ZHANG X Y ,et al. MetaPruning:meta learning for automatic neural network channel pruning[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2019: 3295-3304.
[25]	DING Y X , ZHAO W G , WANG Z P ,et al. Automaticlly learning featurs of android apps using CNN[C]// Proceedings of 2018 International Conference on Machine Learning and Cybernetics (ICMLC). Piscataway:IEEE Press, 2018: 331-336.
[26]	LIN M B , JI R R , ZHANG Y X ,et al. Channel pruning via automatic structure search[C]// Proceedings of Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. California:International Joint Conferences on Artificial Intelligence Organization, 2020: 673-679.
[27]	SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition[J]. CoRR, 2014.
[28]	KRIZHEVSKY A , HINTON G . Learning multiple layers of features from tiny images[J]. Handbook of Systemic Autoimmune Diseases. 2009,1(4)
[29]	LUO J H , WU J X . Neural network pruning with residual-connections and limited-data[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 1455-1464.
[30]	ZHOU Y F , ZHANG Y , WANG Y F ,et al. Accelerate CNN via recursive Bayesian pruning[C]// Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway:IEEE Press, 2019: 3305-3314.
[31]	PENG H Y , WU J X , CHEN S F ,et al. Collaborative channel pruning for deep networks[C]// Proceedings of the International Conference on Machine Learning. New York:ACM Press, 2019: 5113-5122.
[32]	YANG H R , WEN W , LI H ,et al. DeepHoyer:learning sparser neural network with differentiable scale-invariant sparsity measures[EB]. 2019.
[33]	HE Y , DING Y H , LIU P ,et al. Learning filter pruning criteria for deep convolutional neural networks acceleration[C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway:IEEE Press, 2020: 2006-2015.

Metrics

Recommended 0

No Suggested Reading articles found!

数据集	剪枝方法	准确率	准确率下降	浮点运算量/10⁶	浮点运算量减少
CIFAR-10	基线	93.88%	-	314	-
	NS^[12]	93.62%	0.26%	155	51%
	OT^[19]	93.87%	0.01%	163	48%
	PR^[20]	93.92%	-0.04%	144	54%
	AOFP^[25]	94.03%	-0.15%	186	41%
	本文	94.04%	-0.16%	140	55%
CIFAR-100	基线	73.83%	-	314	-
	NS^[12]	74.20%	-0.37%	195	38%
	OT^[19]	73.99%	0.21%	197	37%
	PR^[20]	74.25%	-0.42%	179	43%
	本文	74.33%	-0.50%	174	45%

剪枝方法	剪枝率	网络架构	总通道数	浮点运算量/10⁶	参数量/10⁶	准确率（Scratch-E）	准确率（Scratch-B）
基线	0	64-64-128-128-256-256-256-512-512-512-512-512-512	4 224	314	20.04	93.88%	93.88%
NS^[12]	70%	30-62-123-126-247-238-199-101-27-10-9-6-89	1 268	155	1.68	93.44%	93.91%
OT^[19]	73%	26-59-114-120-206-172-128-98-56-38-27-32-57	1 133	113	1.16	93.28%	93.86%
本文	74%	32-64-115-128-228-198-118-65-25-12-19-22-74	1 100	140	1.26	93.47%	94.04%
本文	76%	31-64-107-128-214-180-103-56-21-10-16-19-59	1 017	116	1.05	93.30%	93.91%
RBP^[30]	86%	50-63-123-108-104-57-23-14-9-8-6-7-11	583	9.0	0.39	92.43%	93.20%
本文	80%	26-62-91-121-181-152-86-41-13-7-10-11-61	862	90	0.78	92.92%	93.73%
本文	86%	19-47-65-92-126-103-52-23-6-3-4-5-47	592	47	0.37	91.92%	93.37%

数据集	剪枝方法	准确率	准确率下降	浮点运算量/10⁶	浮点运算量减少
CIFAR-10	基线	93.80%	—	127	—
	NS^[12]	93.27%	0.53%	66	48%
	PR^[20]	93.83%	-0.03%	67	47%
	SFP^[21]	93.35%	0.45%	59	53%
	BN-SFP^[23]	93.29%	0.51%	59	53%
	ASFP^[15]	93.12%	0.68%	59	53%
	ABCPruner^[26]	93.23%	0.57%	58	54%
	DCP^[31]	93.81%	-0.01%	67	47%
	DeepHoyer^[32]	93.54%	0.26%	67	47%
	LFPC^[33]	93.72%	0.08%	67	47%
	本文	93.85%	-0.05%	66	48%
	本文	93.60%	0.20%	58	54%
CIFAR-100	基线	72.49%	—	127	—
	NS^[12]	71.40%	1.09%	96	24%
	PR^[20]	72.43%	0.06%	95	25%
	本文	72.51%	-0.02%	95	25%

数据集	模型	剪枝方法	准确率	准确率提升	浮点运算量/10⁶	总通道数
CIFAR-10	VGG16	基线	93.88%	—	314	4 224
		本文（重构）	94.20%	0.32%	313	2 359
	ResNet56	基线	93.80%	—	127	2 032
		本文（重构）	94.04%	0.24%	126	2 181
CIFAR-100	VGG16	基线	73.83%	—	314	4 224
		本文（重构）	74.20%	0.37%	310	3 115
	ResNet56	基线	72.49%	—	127	2 032
		本文（重构）	72.72%	0.23%	114	2 141

A flexible pruning on deep convolutional neural networks

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 33

Related Articles 5

Metrics

Recommended 0

[1]	Zihao LIU, Xiaojun JIA, Sulan ZHANG, Zhiling XU, Jun ZHANG. Vibe++ background segmentation method combining MeanShift clustering analysis and convolutional neural network [J]. Telecommunications Science, 2021, 37(3): 133-145.
[2]	Jingcheng CAO, Jidong ZHANG, Guojie SHI. A method for improving recall rate of similar image retrieval by using edge enhancement technology [J]. Telecommunications Science, 2021, 37(1): 76-84.
[3]	Qiang LI,Zilu KANG. Video crowd counting method based on conv-pooling deep spatial and temporal features [J]. Telecommunications Science, 2018, 34(6): 72-79.
[4]	Hongwei XU,Diqun YAN,Fan YANG,Rangding WANG,Chao JIN,Li XIANG. Detection algorithm of electronic disguised voice based on convolutional neural network [J]. Telecommunications Science, 2018, 34(2): 46-57.
[5]	Huijun XING,Shuo CHANG. Pedestrian surveillance system based on mobile vehicle [J]. Telecommunications Science, 2017, 33(2): 120-127.