Telecommunications Science ›› 2020, Vol. 36 ›› Issue (4): 115-124.doi: 10.11959/j.issn.1000-0801.2020119
• Research and Development • Previous Articles Next Articles
Rui MIN
Revised:
2020-03-26
Online:
2020-04-20
Published:
2020-04-24
CLC Number:
Rui MIN. A survey of efficient deep neural network[J]. Telecommunications Science, 2020, 36(4): 115-124.
[1] | KRIZHEVSKY A , SUTSKEVER I , HINTON G E . Imagenet classification with deep convolutional neural networks[C]// Proceedings of the Advances in Neural Information Processing Systems. Lake Tahoe:MIT Press, 2012. |
[2] | HU J , SHEN L , SUN G . Squeeze-and-excitation networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018. |
[3] | SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition[C]// Proceedings of International Conference on Learning Representations. San Diego:Elsevier, 2015. |
[4] | XIE S , GIRSHICK R , DOLL R P ,et al. Aggregated residual transformations for deep neural networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2017. |
[5] | LECUN Y , DENKER J S , SOLLA S A . Optimal brain damage[C]// Proceedings of the Advances in Neural Information Processing Systems. Piscataway:IEEE Press, 1990. |
[6] | HOWARD A G , ZHU M , CHEN B ,et al. Mobilenets:efficient convolutional neural networks for mobile vision applications[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2017 |
[7] | SANDLER M , HOWARD A , ZHU M ,et al. Mobilenetv2:inverted residuals and linear bottlenecks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018. |
[8] | ZHANG X , ZHOU X , LIN M ,et al. Shufflenet:an extremely efficient convolutional neural network for mobile devices[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2018. |
[9] | MCCULLOCH W S , PITTS W . A logical calculus of the ideas immanent in nervous activity[J]. The Bulletin of Mathematical Biophysics, 1943,5(4): 115-133. |
[10] | ROSENBLATT F . The perceptron:a probabilistic model for information storage and organization in the brain[J]. Psychological Review, 1958,65(6):386. |
[11] | MARVIN M , SEYMOUR P . Perceptrons[M]. Cambridge: MIT PressPress, 1969. |
[12] | HOPFIELD J J . Neural networks and physical systems with emergent collective computational abilities[J]. Proceedings of the National Academy of Sciences, 1982,79(8): 2554-2558. |
[13] | HINTON G E , SEJNOWSKI T J . Learning and relearning in Boltzmann machines[J]. Parallel Distributed Processing:Explorations in the Microstructure of Cognition, 1986,1(282-317):2. |
[14] | MCCLELLAND J L , RUMELHART D E , GROUP P R . Parallel distributed processing[M]. Cambridge: MIT Press Press, 1987. |
[15] | LECUN Y , BOTTOU L , BENGIO Y ,et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998,86(11): 2278-324. |
[16] | HINTON G E , SALAKHUTDINOV R R . Reducing the dimensionality of data with neural networks[J]. Science, 2006,313(5786): 504-507. |
[17] | TAIGMAN Y , YANG M , RANZATO M A ,et al. Deepface:closing the gap to human-level performance in face verification[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2014. |
[18] | SUN Y , WANG X , TANG X . Deep learning face representation from predicting 10,000 classes[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2014. |
[19] | SCHROFF F , KALENICHENKO D , PHILBIN J . Facenet:A unified embedding for face recognition and clustering[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2015. |
[20] | DENG J , GUO J , XUE N ,et al. Arcface:additive angular margin loss for deep face recognition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2019. |
[21] | WEN Y , ZHANG K , LI Z ,et al. A discriminative feature learning approach for deep face recognition[C]// Proceedings of the European Conference on Computer Vision. Glasgow:Springer, 2016. |
[22] | GIRSHICK R , DONAHUE J , DARRELL T ,et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2014. |
[23] | GIRSHICK R , . Fast R-CNN[C]// Proceedings of the IEEE International Conference on Computer Vision. Piscataway:IEEE Press, 2015. |
[24] | REN S , HE K , GIRSHICK R ,et al. Faster R-CNN:towards real-time object detection with region proposal networks[C]// Proceedings of the Advances in Neural Information Processing Systems. Cambridge:MIT Press, 2015. |
[25] | HE K , GKIOXARI G , DOLL R P ,et al. Mask R-CNN[C]// Proceedings of the IEEE International Conference on Computer Vision. Piscataway:IEEE Press, 2017. |
[26] | WANG L , XIONG Y , WANG Z ,et al. Temporal segment networks:towards good practices for deep action recognition[C]// Proceedings of the European Conference on Computer Vision. Amsterdam:Springer, 2016. |
[27] | MIKOLOV T , YIH W-T , ZWEIG G . Linguistic regularities in continuous space word representations[C]// Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies. Piscataway:IEEE Press, 2013. |
[28] | KALCHBRENNER N , GREFENSTETTE E , BLUNSOM P . A convolutional neural network for modelling sentences[J]. arXiv:14042188, 2014 |
[29] | HASSIBI B , STORK D G , WOLFF G J . Optimal brain surgeon and general network pruning[C]// Proceedings of the IEEE International Conference on Neural Networks. Piscataway:IEEE Press, 1993. |
[30] | SRINIVAS S , BABU R V . Data-free parameter pruning for deep neural networks[C]// Proceedings of the British Machine Vision Conference. Swansea:BMVA Press, 2015 |
[31] | HAN S , POOL J , TRAN J ,et al. Learning both weights and connections for efficient neural network[C]// Proceedings of the Advances in Neural Information Processing Systems. Cambridge:MIT Press , 2015. |
[32] | HAN S , LIU X , MAO H ,et al. EIE:efficient inference engine on compressed deep neural network[C]// Proceedings of the 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA). Piscataway:IEEE Press, 2016. |
[33] | PARK J , LI S , WEN W ,et al. Faster cnns with direct sparse convolutions and guided pruning[C]// International Conference on Learning Representations. San Diego:Elsevier, 2016, |
[34] | LIU B , WANG M , FOROOSH H ,et al. Sparse convolutional neural networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2015. |
[35] | ANWAR S , HWANG K , SUNG W . Structured pruning of deep convolutional neural networks[J]. ACM Journal on Emerging Technologies in Computing Systems (JETC), 2017,13(3):32. |
[36] | LEBEDEV V , LEMPITSKY V . Fast convnets using group-wise brain damage[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2016. |
[37] | HU H , PENG R , TAI Y-W ,et al. Network trimming:a data-driven neuron pruning approach towards efficient deep architectures[C]// Proceedings of International Conference on Learning Representations. San Diego:Elsevier, 2016. |
[38] | MOLCHANOV P , TYREE S , KARRAS T ,et al. Pruning convolutional neural networks for resource efficient transfer learning[C]// Proceedings of International Conference on Learning Representations. San Diego:Elsevier, 2017. |
[39] | KIM Y-D , PARK E , YOO S ,et al. Compression of deep convolutional neural networks for fast and low power mobile applications[C]// International Conference on Learning Representations. San Diego:Elsevier, 2016. |
[40] | ALEXANDER N , PODOPRIKHIN D , OSOKIN A ,et al. Tensorizing neural networks[C]// Proceedings of the Advances in Neural Information Processing Systems. Montreal:MIT Press , 2015. |
[41] | DENTON E L , ZAREMBA W , BRUNA J ,et al. Exploiting linear structure within convolutional networks for efficient evaluation[C]// Proceedings of the Advances in Neural Information Processing Systems. Montreal:MIT Press, 2014. |
[42] | JADERBERG M , VEDALDI A , ZISSERMAN A . Speeding up convolutional neural networks with low rank expansions[C]// Proceedings of the British Machine Vision Conference. Nottingham:BMVA Press, 2014, |
[43] | CHEN W , WILSON J , TYREE S ,et al. Compressing neural networks with the hashing trick[C]// Proceedings of the International Conference on Machine Learning.[S.l.:s.n]. 2015. |
[44] | COURBARIAUX M , BENGIO Y , DAVID J-P . Training deep neural networks with low precision multiplications[C]// International Conference on Learning Representations. San Diego:Elsevier, 2015. |
[45] | DETTMERS T , . 8-bit approximations for parallelism in deep learning[C]// 4th International Conference on Learning Representations. San Diego:Elsevier, 2016. |
[46] | RASTEGARI M , ORDONEZ V , REDMON J ,et al. Xnor-net:Imagenet classification using binary convolutional neural networks[C]// Proceedings of the European Conference on Computer Vision. Amsterdam:Springer, 2016. |
[47] | LI F , ZHANG B , LIU B . Ternary weight networks[J]. arXiv preprint arXiv:160504711, 2016, |
[48] | HINTON G , VINYALS O , DEAN J . Distilling the knowledge in a neural network[C]// Proceedings of the Advances in Neural Information Processing Systems. Montreal:MIT Press, 2015. |
[49] | C,BUCILUǎ C , CARUANA R , NICULESCU-MIZIL A , . Model compression[C]// Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York:ACM Press, 2006. |
[50] | BA J , CARUANA R . Do deep nets really need to bedeep?[C]// Proceedings of the Advances in Neural Information Processing Systems. Montreal:MIT Press , 2014. |
[51] | ROMERO A , BALLAS N , KAHOU S E ,et al. Fitnets:hints for thin deep nets[C]// International Conference on Learning Representations. San Diego:Elsevier, 2015. |
[52] | ZAGORUYKO S , KOMODAKIS N . Paying more attention to attention:Improving the performance of convolutional neural networks via attention transfer[C]// /International Conference on Learning Representations. San Diego:Elsevier, 2016. |
[53] | LI Q , JIN S , YAN J . Mimicking very efficient network for object detection[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2017. |
[54] | LUO P , ZHU Z , LIU Z ,et al. Face model compression by distilling knowledge from neurons[C]// Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence.[S.l.:s.n. ], 2016. |
[55] | LIU Y , CAO J , LI B ,et al. Knowledge distillation via instance relationship graph[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2019. |
[56] | PENG B , JIN X , LIU J ,et al. Correlation congruence for knowledge distillation[C]// Proceedings of the IEEE International Conference on Computer Vision. Seoul:Elsevier, 2019. |
[57] | LAN X , ZHU X , GONG S . Knowledge distillation by on-the-fly native ensemble[C]// Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montreal:MIT Press, 2018. |
[58] | BAKER B , GUPTA O , NAIK N ,et al. Designing neural network architectures using reinforcement learning[J]. arXiv:1611.02167, 2016 |
[59] | ZOPH B , LE Q V . Neural architecture search with reinforcement learning[C]// Proceedings of International Conference on Learning Representations. Toulon:Elsevier, 2017. |
[60] | REAL E , MOORE S , SELLE A ,et al. Large-scale evolution of image classifiers[C]// Proceedings of the 34th International Conference on Machine Learning.[S.l.:s.n]. 2017: 2902-2911. |
[61] | LIU H , SIMONYAN K , YANG Y . Darts:differentiable architecture search[C]// Proceedings of International Conference on Learning Representations. New Orleans:Elsevier, 2019. |
[1] | Min LU, Zehao QIN, Zhihui CHEN, Min ZHANG, Guangxue YUE. 1D-Concatenate based channel estimation DNN model optimization method [J]. Telecommunications Science, 2023, 39(4): 71-86. |
[2] | Panpan LI, Zhengxia XIE, Guangxue YUE, Xin LIU. Research progress and trends of deep learning based wireless communication receiving method [J]. Telecommunications Science, 2022, 38(2): 1-17. |
[3] | Shujun SUN, Shengliang PENG, Yudong YAO, Xi YANG. A survey of deep learning based modulation recognition [J]. Telecommunications Science, 2021, 37(5): 82-90. |
[4] | Yajie LI,Yongli ZHAO,Shoudong LIU,Jie ZHANG. Overview of research on fiber nonlinear equalization algorithm based on artificial intelligence [J]. Telecommunications Science, 2020, 36(3): 61-70. |
[5] | Hansheng LIU,Hongyu TANG,Mingxia BO,Jianfeng NIU,Tianbo LI,Lingxiao LI. A multi-source threat intelligence confidence value evaluation method based on machine learning [J]. Telecommunications Science, 2020, 36(1): 119-126. |
[6] | Haikun WANG,Jia PAN,Cong LIU. Research development and forecast of automatic speech recognition technologies [J]. Telecommunications Science, 2018, 34(2): 1-11. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||
|