基于混合maxout单元的卷积神经网络性能优化

doi:10.11959/j.issn.1000-436x.2017145

摘要/Abstract

摘要：

针对深度卷积神经网络中 maxout 单元非最大特征无法传递、特征图像子空间池化表达能力不足的局限性，提出混合maxout (mixout,mixed maxout)单元。首先，计算相同输入在不同卷积变换下所形成的特征图像子空间的指数概率分布；其次，根据概率分布计算特征图像子空间的期望；最后，利用伯努利分布对子空间的最大值与期望值加权，均衡单元模型。分别构建基于 mixout 单元的简单模型和网中网模型进行实验，结果表明 mixout单元模型性能较好。

关键词: 深度学习, 卷积神经网络, maxout单元, 激活函数

Abstract:

The maxout units have the problem of not delivering non-max features， resulting in the insufficient of pooling operation over a subspace that is composed of several linear feature mappings,when they are applied in deep convolutional neural networks.The mixed maxout (mixout) units were proposed to deal with this constrain.Firstly,the exponential probability of the feature mappings getting from different linear transformations was computed.Then,the averaging of a subspace of different feature mappings by the exponential probability was computed.Finally,the output was randomly sampled from the max feature and the mean value by the Bernoulli distribution,leading to the better utilizing of model averaging ability of dropout.The simple models and network in network models was built to evaluate the performance of mixout units.The results show that mixout units based models have better performance.

Key words: deep learning, convolutional neural network, maxout units, activation function

中图分类号:

TP391.3

赵慧珍,刘付显,李龙跃,罗畅. 基于混合maxout单元的卷积神经网络性能优化[J]. 通信学报, 2017, 38(7): 105-114.

Hui-zhen ZHAO,Fu-xian LIU,Long-yue LI,Chang LUO. Improving deep convolutional neural networks with mixed maxout units[J]. Journal on Communications, 2017, 38(7): 105-114.

图/表 12

图1

图2

图3

图4

图5

图6

图7

图8

表1

表2

图9

图10

参考文献 27

[1]	周昌令, 栾兴龙, 肖建国 . 基于深度学习的域名查询行为向量空间嵌入[J]. 通信学报, 2016,37(3): 165-174.
	ZHOU C L , LUAN X L , XIAO J G . Vector space embedding of DNS query behaviors by deep learning[J]. Journal on Communications, 2016,37(3): 165-174.
[2]	杨钊, 陶大鹏, 张树业 ,等. 大数据下的基于深度神经网的相似汉字识别[J]. 通信学报, 2014,35(9): 184-189.
	YANG Z , TAO D P , ZHANG S Y ,et al. Similar handwritten Chinese character recognition based on deep neural networks with big data[J]. Journal on Communications, 2014,35(9): 184-189.
[3]	SPRINGENBERG J T , RIEDMILLER M . Improving deep neural networks with probabilistic maxout units[J]. arXiv preprint arXiv:1312.6116, 2013.
[4]	HINTON G E , SRIVASTAVA N , KRIZHEVSKY A ,et al. Improving neural networks by preventing co-adaptation of feature detectors[J]. arXiv preprint arXiv:1207.0580, 2012.
[5]	KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks[C]// The 26th Annual Conference on Neural Information Processing Systems. 2012: 1097-1105.
[6]	WANG S I , MANNING C D . Fast dropout training[C]// The 30th International Conference on Machine Learning. 2013: 118-126.
[7]	BA J , FREY B . Adaptive dropout for training deep neural networks[C]// The Advances in Neural Information Processing Systems. 2013: 3084-3092.
[8]	TOMPSON J , GOROSHIN R , JAIN A ,et al. Efficient object localization using convolutional networks[C]// The IEEE Conference on Computer Vision and Pattern Recognition. 2015: 648-656.
[9]	WAN L , ZEILER M , ZHANG S ,et al. Regularization of neural networks using dropconnect[C]// The 30th International Conference on Machine Learning. 2013: 1058-1066.
[10]	GOODFELLOW I J , WARDE-FARLEY D , MIRZA M ,et al. Maxout networks[C]// The 30th International Conference on Machine Learning. 2013: 1319-1327.
[11]	ZEILER M D , FERGUS R . Stochastic pooling for regularization of deep convolutional neural networks[J]. arXiv Preprint arXiv:1301.3557, 2013.
[12]	NAIR V , HINTON G E . Rectified linear units improve restricted Boltzmann machines[C]// The 27th International Conference on Machine Learning. 2010: 807-814.
[13]	MAAS A L , HANNUN A Y , NG A Y . Rectifier nonlinearities improve neural network acoustic models[C]// The 30th International Conference on Machine Learning. 2013,30(1).
[14]	HE K , ZHANG X , REN S ,et al. Delving deep into rectifiers:surpassing human-level performance on ImageNet classification[C]// The IEEE International Conference on Computer Vision. 2015: 1026-1034.
[15]	WANG S . General constructive representations for continuous piecewise-linear functions[J]. IEEE Transactions on Circuits and Systems I:Regular Papers, 2004,51(9): 1889-1896.
[16]	LI J , WANG X , XU B . Understanding the dropout strategy and analyzing its effectiveness on LVCSR[C]// 38th IEEE International Conference on Acoustics,Speech,and Signal Processing. 2013: 7614-7618.
[17]	YU D , WANG H , CHEN P ,et al. Mixed pooling for convolutional neural networks[C]// International Conference on Rough Sets and Knowledge Technology. 2014: 364-375.
[18]	LIN M , CHEN Q , YAN S . Network in network[J]. Journal of Artificial Intelligence Research， 2011，41（2）： 297-327.
[19]	KRIZHEVSKY A , HINTON G E . Learning multiple layers of features from tiny images[R]. Computer Science Department,University of Toronto,Tech. 2009.
[20]	HE K M , ZHANG X Y , REN S Q ,et al. Deep residual learning for image recognition[J]. arxiv Prepraint arXiv:1512.03385, 2015.
[21]	NETZER Y , WANG T , COATES A ,et al. Reading digits in natural images with unsupervised feature learning[R]. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[22]	CHANG J R , CHEN Y S . Batch-normalized maxout network in network[J]. arXiv Preprint arXiv:1511.02583, 2015.
[23]	JIN X , XU C , FENG J ,et al. Deep learning with s-shaped rectified linear activation units[J]. arXiv Preprint arXiv:1512.07030, 2015.
[24]	LIAO Z , CARNEIRO G . On the importance of normalisation layers in deep learning with piecewise linear activation units[C]// The 2016 IEEE Winter Conference on Applications of Computer Vision. 2016: 1-8.
[25]	XU B , WANG N , CHEN T ,et al. Empirical evaluation of rectified activations in convolutional network[J]. arXiv Preprint arXiv:1505.00853, 2015.
[26]	SRIVASTAVA N , HINTON G , KRIZHEVSKY A ,et al. Dropout:a simple way to prevent neural networks from overfitting[J]. The Journal of Machine Learning Research, 2014,15(1): 1929-1958.
[27]	CLEVERT D A , UNTERTHINER T , HOCHREITER S . Fast and accurate deep network learning by exponential linear units (ELUS)[J]. arXiv preprint arXiv:1511.07289, 2015.

M-NIN网络结构	层次	mixout单元数	核尺寸	子空间维数	补充像素	滑动像素
	卷积层	192	5×5	0	2	1
M-NIN区块1	MMLP -1	160	1×1	5	0	1
	MMLP-2	96	1×1	5	0	1
	池化层	最大池化	3×3	0	2	2
	卷积层	192	5×5	0	2	1
M-NIN区块2	MMLP -1	192	1×1	5	0	1
	MMLP -2	192	1×1	5	0	1
	池化层	最大池化	3×3	0	2	2
	卷积层	192	3×3	0	2	1
M-NIN区块3	MMLP -1	192	1×1	5	0	1
	MMLP -2	160	1×1	5	0	1
	池化层	平均池化	8×8	0	2	1
softmax层				分类10/100

模型	CIFAR-10	CIFAR-100	SVHN
ReLU函数	12.45%	42.90%	2.78%
LReLU函数	11.20%	42.04%	2.69%
PReLU函数	11.79%	41.63%	2.51%
maxout单元	11.68%	38.57%	2.47%
probout单元	11.35%	38.14%	2.39%
mixout单元	11.02%	37.39%	2.31%