神经结构搜索进展概述

doi:10.11959/j.issn.1000-0801.2019097

电信科学 ›› 2019, Vol. 35 ›› Issue (5): 43-50.doi: 10.11959/j.issn.1000-0801.2019097

• 专题：智能通信技术及应用 • 上一篇下一篇

神经结构搜索进展概述

何明捷^1,²,张杰¹,山世光^1,²

¹ 中国科学院计算技术研究所中国科学院智能信息处理重点实验室，北京 100190
² 中国科学院大学，北京 100049

修回日期:2019-05-06 出版日期:2019-05-20 发布日期:2019-05-21
作者简介:何明捷（1990- ），男，中国科学院计算技术研究所工程师，中国科学院大学博士生，主要研究方向为计算机视觉与机器学习。|张杰（1988- ），男，中国科学院计算技术研究所助理研究员，主要研究方向为计算机视觉与机器学习。|山世光（1975- ），男，中国科学院计算技术研究所研究员、博士生导师，中国科学院智能信息处理重点实验室常务副主任。国家自然科学基金委员会“优秀青年科学基金项目”入选者，第三批国家“万人计划”入选者，科技部“创新人才推进计划”中青年科技创新领军人才，CCF 青年科学家奖获得者。主要研究方向为计算机视觉和机器学习。
基金资助:
国家自然科学基金资助项目(61806188)

A survey of neural architecture search

Mingjie HE^1,²,Jie ZHANG¹,Shiguang SHAN^1,²

¹ Key Lab of Intelligent Information Processing,Institute of Computing Technology Chinese Academy of Sciences,Beijing 100190,China
² University of Chinese Academy of Sciences,Beijing 100049,China

Revised:2019-05-06 Online:2019-05-20 Published:2019-05-21
Supported by:
The National Natural Science Foundation of China(61806188)

摘要/Abstract

摘要：

近年来，深度学习技术在大量的计算视觉任务上取得了巨大的成功，深度神经结构是一个决定性能的关键要素，全自动的神经结构搜索方法的研究近年来受到了越来越多的关注。全自动的神经结构搜索方法是指针对特定的任务，通过算法自动地学习出适用的深度神经结构。各类神经结构搜索方法在探索高性能、高效率的神经结构方面已经展示出了巨大的潜力。从性能评估方法、搜索空间、结构搜索策略 3 个维度对神经结构搜索方法进行了分类概述：重点介绍了4种降低计算开销的性能评估方法，2类典型的神经结构搜索空间以及基于离散空间和基于连续空间的2种搜索策略。基于连续空间的NAS算法正逐渐成为NAS算法的重要发展方向。

关键词: 神经结构搜索, 深度学习, 强化学习

Abstract:

Recently,deep learning has achieved impressive success on various computer vision tasks.The neural architecture is usually a key factor which directly determines the performance of the deep learning algorithm.The automated neural architecture search methods have attracted more and more attentions in recent years.The neural architecture search is the automated process of seeking the optimal neural architecture for specific tasks.Currently,the neural architecture search methods have shown great potential in exploring high-performance and high-efficiency neural architectures.In this paper,a survey in this research field and categorize existing methods based on their performance estimation methods,search spaces and architecture search strategies were presented.Specifically,there were four performance estimation methods for computation cost reduction,two typical neural architecture search spaces and two types of search strategies based on discrete and continuous spaces respectively.Neural architecture search methods based on continuous space are becoming the trend of researches on neural architecture search.

Key words: neural architecture search, deep learning, reinforcement learning

中图分类号:

TP183

何明捷,张杰,山世光. 神经结构搜索进展概述[J]. 电信科学, 2019, 35(5): 43-50.

Mingjie HE,Jie ZHANG,Shiguang SHAN. A survey of neural architecture search[J]. Telecommunications Science, 2019, 35(5): 43-50.

图/表 5

图1

图2

图3

图4

表1

参考文献 29

[1]	DENG J , DONG W , SOCHER R ,et al. ImageNet:a large-scale hierarchical image database[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR),June 20-25,2009,Miami,Florida,USA. Piscataway:IEEE Press, 2009.
[2]	KRIZHEVSKY A , SUTSKEVER I , HINTON G E . Imagenet classification with deep convolutional neural networks[C]// Advances in Neural Information Processing Systems (NIPS),Dec 3-8,2012,Lake Tahoe,Nevada,USA. New York:Curran Associates Inc, 2012.
[3]	SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition[J]. Computer Science, 2004.
[4]	SZEGEDY C , LIU W , JIA Y ,et al. Going deeper with convolutions[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR),June 7-12,2015,Boston,MA,USA. Piscataway:IEEE Press, 2015.
[5]	HE K , ZHANG X , REN S ,et al. Deep residual learning for image recognition[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR),June 27-30,2016,Las Vegas,NV,USA. Piscataway:IEEE Press, 2016.
[6]	HE K , ZHANG X , REN S ,et al. Identity mappings in deep residual networks[C]// European Conference on Computer Vision (ECCV),Oct 11-14,2016,Amsterdam,Netherlands. Cham:Springer, 2016.
[7]	HUANG G , LIU Z , VAN DER MAATEN L ,et al. Densely connected convolutional networks[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR),July 21-26,2017,Honolulu,HI,USA. Piscataway:IEEE Press, 2017.
[8]	XIE S , GIRSHICK R , DOLL R P ,et al. Aggregated residual transformations for deep neural networks[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR),July 21-26,2017,Honolulu,HI,USA. Piscataway:IEEE Press, 2017.
[9]	CHEN Y , LI J , XIAO H ,et al. Dual path networks[C]// Advances in Neural Information Processing Systems (NIPS),Dec 4-9,2017,Long Beach,CA,USA. New York:Curran Associates Inc, 2017.
[10]	HU J , SHEN L , SUN G . Squeeze-and-excitation networks[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR),June 18-22,2018,Salt Lake,UT,USA. Piscataway:IEEE Press, 2018.
[11]	ELSKEN T , METZEN J H , HUTTER F . Neural architecture search:A survey[J]. arXiv:1808.05377, 2018.
[12]	LIU C , ZOPH B , NEUMANN M ,et al. Progressive neural architecture search[C]// European Conference on Computer Vision (ECCV),Sep 8-14,2018,Munich,Germany. Cham:Springer, 2018.
[13]	LUO R , TIAN F , QIN T ,et al. Neural architecture optimization[C]// Advances in Neural Information Processing Systems (NIPS),Dec 3-8,2018,Montreal,Canada. New York:Curran Associates,Inc, 2018.
[14]	PHAM H , GUAN M Y , ZOPH B ,et al. Efficient neural architecture search via parameter sharing[C]// International Conference on Machine Learning (ICML),July 10-15,2018,Stockholmsm?ssan,Stockholm,Sweden.[S.l.:sn]. 2018.
[15]	ZHONG Z , YAN J , WU W ,et al. Practical block-wise neural network architecture generation[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR),June 18-22,2018,Salt Lake City,UT,USA. Piscataway:IEEE Press, 2018.
[16]	ZOPH B , VASUDEVAN V , SHLENS J ,et al. Learning transferable architectures for scalable image recognition[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR),June 18-22,2018,Salt Lake City,UT,USA. Piscataway:IEEE Press, 2018.
[17]	TAN M , CHEN B , PANG R ,et al. Mnasnet:platform-aware neural architecture search for mobile[J]. arXiv:1807.11626, 2018.
[18]	REAL E , AGGARWAL A , HUANG Y ,et al. Regularized evolution for image classifier architecture search[C]// Thirty-Third AAAI Conference on Artificial Intelligence,Jan 27-Feb 1,2019,Honolulu,Hawaii,USA.[S.l.:sn]. 2019.
[19]	CAI H , YANG J , ZHANG W ,et al. Path-level network transformation for efficient architecture search[C]// International Conference on Machine Learning (ICML),July 10-15,2018,Stockholmsm?ssan,Stockholm,Sweden.[S.l.:sn]. 2018.
[20]	CHRABASZCZ P , LOSHCHILOV I , HUTTER F . A downsampled variant of imagenet as an alternative to the cifar datasets[J]. arXiv:1707.08819, 2017.
[21]	RAWAL A , MIIKKULAINEN R . From nodes to networks:evolving recurrent neural networks[J]. arXiv:1803.04439, 2018.
[22]	KRIZHEVSKY A , HINTON G . Learning multiple layers of features from tiny images[R]. 2009.
[23]	SANDLER M , HOWARD A , ZHU M ,et al. Mobilenetv2:inverted residuals and linear bottlenecks[C]// IEEE Conference on Computer Vision and Pattern Recognition (CVPR),June 18-22,2018,Salt Lake City,UT,USA. Piscataway:IEEE Press, 2018.
[24]	ZOPH B , LE Q V . Neural architecture search with reinforcement learning[C]// International Conference on Learning Representations (ICLR),April 24-26,2017,Toulon,France.[S.l.:sn]. 2017.
[25]	BAKER B , GUPTA O , NAIK N ,et al. Designing neural network architectures using reinforcement learning[C]// International Conference on Learning Representations (ICLR),April 24-26,2017,Toulon,France.[S.l.:sn]. 2017.
[26]	CAI H , CHEN T , ZHANG W ,et al. Reinforcement learning for architecture search by network transformation[J]. arXiv:1707.04873, 2017.
[27]	LIU H , SIMONYAN K , VINYALS O ,et al. Hierarchical representations for efficient architecture search[C]// International Conference on Learning Representations (ICLR),April 30-May 3,2018,Vancouver,BC,Canada.[S.l.:s. n], 2018.
[28]	REAL E , MOORE S , SELLE A ,et al. Large-scale evolution of image classifiers[C]// International Conference on Machine Learning (ICML),Aug 6-11,2017,Sydney,NSW,Australia.[S.l.:sn]. 2017.
[29]	XIE L , YUILLE A . Genetic CNN[C]// IEEE International Conference on Computer Vision (ICCV),Oct 22-29,2017,Venice,Italy. Piscataway:IEEE Press, 2017.

方法名称	优化目标	搜索空间	搜索策略	参数数量/MB	CIFAR-10 Top1 Error	GPU数量	GPU运行天数
NASNet^[16]	精度	基于cell结构	离散，强化学习	27.6	2.97%	500	4
PNASNet^[12]	精度	基于cell结构	离散，强化学习	3.2	3.63%^[14]	100^[14]	1.5^[14]
MNASNet^[17]*	精度，速度	基于chain结构	离散，强化学习	4.4	-	-	-
ENAS^[14]	精度	基于cell结构	离散，强化学习	4.6	3.54%	1	0.45
BlockQNN^[15]	精度	基于cell结构	离散，强化学习	39.8	3.54%	32	3
AmoebaNet^[18]	精度	基于cell结构	离散，进化算法	3.2	3.34%	450	7
NAONet^[13]	精度	基于cell结构	连续优化	28.6	2.98%	200	1

神经结构搜索进展概述

A survey of neural architecture search

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 5

参考文献 29

相关文章 15

Metrics

推荐阅读 0

[1]	胡珈玮, 刘晓谦, 唐昕柯, 董宇涵. 基于DQN的UUV辅助水下无线光通信轨迹规划系统[J]. 电信科学, 2023, 39(5): 42-47.
[2]	卢敏, 胡娟, 张先超, 丁伟健, 乐光学. 基于用户多特征融合的个性化推荐模型[J]. 电信科学, 2023, 39(5): 101-115.
[3]	王雪荣, 唐政治, 李银川, 齐美玉, 朱建波, 张亮. 基于优化决策树的时延敏感流智能感知调度[J]. 电信科学, 2023, 39(4): 120-132.
[4]	廖熙雯, 冷甦鹏, 明昱君, 李天扬. 基于数字孪生的城市交通流智能预测与导引策略[J]. 电信科学, 2023, 39(3): 70-79.
[5]	康宇, 刘雅琼, 赵彤雨, 寿国础. AI算法在车联网通信与计算中的应用综述[J]. 电信科学, 2023, 39(1): 1-19.
[6]	诸葛斌, 尹正虎, 斯文学, 颜蕾, 董黎刚, 蒋献. 基于学生知识追踪的多指标习题推荐算法[J]. 电信科学, 2022, 38(9): 129-143.
[7]	邓丹昊, 王朝炜, 江帆, 王卫东. 无人机辅助无蜂窝大规模MIMO中的空地协同调度[J]. 电信科学, 2022, 38(8): 37-44.
[8]	周杰, Esono Mikue Bernardo Esono, 王学英, 周惠婷, 罗宏. 基于SLM-PTS算法融合的NC-OFDM峰均比优化[J]. 电信科学, 2022, 38(7): 63-74.
[9]	李昕, 孙君. 基于价值差异学习的多小区mMTC接入算法[J]. 电信科学, 2022, 38(6): 82-90.
[10]	李攀攀, 谢正霞, 乐光学, 刘鑫. 基于深度学习的无线通信接收方法研究进展与趋势[J]. 电信科学, 2022, 38(2): 1-17.
[11]	申情, 郭文宾, 楼俊钢, 余强国. 考虑多层次潜在特征的个性化推荐模型[J]. 电信科学, 2022, 38(2): 71-83.
[12]	金楠, 王瑞琴, 陆悦聪. 基于艾宾浩斯遗忘曲线和注意力机制的推荐算法[J]. 电信科学, 2022, 38(10): 89-97.
[13]	陈志宏, 王明晓. 计算机视觉在智慧安防中的应用[J]. 电信科学, 2021, 37(8): 142-147.
[14]	章坚武, 王路鑫, 孙玲芬, 章谦骅, 单杭冠. 人工智能在5G系统中的应用综述[J]. 电信科学, 2021, 37(5): 14-31.
[15]	唐博恒, 柴鑫刚. 基于云边协同的计算机视觉推理机制[J]. 电信科学, 2021, 37(5): 72-81.