深度学习中的内存管理问题研究综述

doi:10.11959/j.issn.2096-0271.2020033

Abstract

Abstract:

In recent years,deep learning has achieved great success in many fields.As the deep neural network develops towards a deeper and wider direction,the training and inference of a deep neural network face huge memory pressure.The limited memory space of accelerating devices has become an important factor restricting the rapid development of deep neural network.How to achieve efficient memory management in deep learning has become a key point in the development of deep learning.Therefore,the basic characteristics of deep neural network were introduced firstly and memory bottleneck in deep learning training was analyzed.Some representative research works were classified,and their advantages and disadvantages were analyzed.Finally,some important direction and tendency of memory management in deep learning were suggested.

Key words: memory management, deep learning, memory swapping, recomputation, memory sharing, compression

CLC Number:

TP183

Weiliang MA, Xuan PENG, Qian XIONG, Xuanhua SHI, Hai JIN. Memory management in deep learning:a survey[J]. Big Data Research, 2020, 6(4): 56-68.

Figures/Tables 2

解决方案	内存管理技术	优点	缺点
vDNN^[18]	内存交换	基于分层特征的内存交换策略能够大幅降低GPU内存占用	通信和计算不能完全重叠，带来较大的性能开销；适用范围有限
cDMA^[28]	内存交换、压缩	在vDNN方案的基础上实现了对交换数据的压缩，能够有效降低性能损失	可应用压缩策略的数据类型有限，内存占用降低的效果一般
Layrub^[21]	内存共享、内存交换	针对不同类型的神经网络应用不同的内存管理策略，几乎适用于所有的神经网络	基于分层策略的内存交换策略会带来一定的性能损失
GradientCheckpoint^[24]	内存共享、重计算	对于n层线性神经网络，该方案的空间复杂度仅为 $O \sqrt{n}$	通过计算来换取内存，带来额外的性能开销，降低了训练性能；仅适用于线性神经网络
SuperNeurons^[22]	内存共享、内存交换、重计算	基于对计算图的静态分析，将3种策略应用于神经网络的训练中，可有效降低训练过程中的内存占用	静态分析不能很好地评估不同策略带来的性能损失，此外基于分层的粗粒度的内存管理策略无法实现最优的内存管理方案
Capuchin^[23]	内存交换、重计算	基于张量的细粒度的内存管理策略能够对不同的张量应用不同的内存管理策略，充分发挥不同策略的优势，在保证性能的同时，可有效降低内存占用	重计算策略带来的性能开销不可忽略，在算力较差的加速设备中性能损失更加明显

References 30

[1]	陈游旻, 李飞, 舒继武 . 大数据环境下的存储系统构建：挑战、方法和趋势[J]. 大数据, 2019,5(4): 27-40.
	CHEN Y M , LI F , SHU J W . Building storage systems in big data era:challenges,methods and trends[J]. Big Data Research, 2019,5(4): 27-40.
[2]	HE K M , ZHANG X Y , REN S Q ,et al. Deep residual learning for image recognition[C]// The 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2016: 770-778.
[3]	IOFFE S , SZEGEDY C . Batch normalization:accelerating deep network training by reducing internal covariate shift[C]// The 32nd International Conference on Machine Learning. New York:ACM Press, 2015: 448-456.
[4]	KRIZHEVSKY A , SUTSKEVER I , HINTON G . ImageNet classification with deep convolutional neural networks[C]// The 26th Annual Conference on Neural Information Processing Systems. Cambridge:MIT Press, 2012: 1106-1114.
[5]	HOCHREITER S , SCHMIDHUBER J . Long short-term memory[J]. Neural Computation, 1997,9(8): 1735-1780.
[6]	SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for largescale image recognition[C]// The 3rd International Conference on Learning Representations.[S.l.:s.n]. 2014
[7]	SZEGEDY C , LIU W , JIA Y Q ,et al. Going deeper with convolutions[C]// The 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE Press, 2015: 1-9.
[8]	ZAGORUYKO S , KOMODAKIS N . Wide residual networks[J]. Computer Science, 2016,arXiv:1605.07146.
[9]	DEVLIN J , CHANG M-W , LEE K ,et al. BERT:pretraining of deep bidirectional transformers for language understanding[J]. Computer Science, 2018,arXiv:1810.04805.
[10]	王孝远, 廖小飞, 刘海坤 ,等. 面向大数据的异构内存系统[J]. 大数据, 2018,4(4): 15-34.
	WANG X Y , LIAO X F , LIU H K ,et al. Big data oriented hybrid memory systems[J]. Big Data Research, 2018,4(4): 15-34.
[11]	李鑫, 陈璇, 黄志球 . 面向大数据应用的混合内存架构特征分析[J]. 大数据, 2018,4(3): 61-80.
	LI X , CHEN X , HUANG Z Q . Analysis on hybrid memory architecture for big data application[J]. Big Data Research, 2018,4(3): 61-80.
[12]	LECUN Y , BOTTOU L , BENGIO Y ,et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998,86: 2278-2324.
[13]	ABADI M , BARHAM P , CHEN J M ,et al. TensorFlow:a system for large-scale machine learning[C]// The 12th USENIX Symposium on Operating Systems Design and Implementation. Berkeley:USENIX Association, 2016: 265-283.
[14]	BERGSTRA J , BREULEUX O , FREDERIC B ,et al. Theano:a CPU and GPU math compiler in Python[C]// The 9th Python for Scientific Computing Conference.[S.l.:s.n]. 2010: 1-7.
[15]	PASZKE A , GROSS S , MASSA F ,et al. PyTorch:an imperative style,highperformance deep learning library[C]// The 2019 Annual Conference on Neural Information Processing Systems. Cambridge:MIT Press, 2019: 8024-8035.
[16]	CHEN T Q , LI M , LI Y T ,et al. MXNet:a flexible and efficient machine learning library for heterogeneous distributed systems[J]. Computer Science, 2015,arXiv:1512.01274.
[17]	DEAN J , CORRADO G , MONGA R ,et al. Large scale distributed deep networks[C]// The 26th Annual Conference on Neural Information Processing Systems. Cambridge:MIT Press, 2012: 1232-1240.
[18]	RHU M , GIMELSHEIN N , CLEMONS J ,et al. vDNN:virtualized deep neural networks for scalable,memoryefficient neural network design[C]// The 49th Annual IEEE/ACM International Symposium on Microarchitecture. Piscataway:IEEE Press, 2016: 1-13.
[19]	CHEN M , SUN M M , YANG J ,et al. Training deeper models by GPU memory optimization on TensorFlow[C]// Advances in Neural Information Processing Systems 30.[S.l.:s.n]. 2017.
[20]	CHEN X M , CHEN D Z , HAN Y H ,et al. moDNN:memory optimal deep neural network training on graphics processing units[J]. IEEE Transactions on Parallel and Distributed Systems, 2019,30(3): 646-661.
[21]	JIN H , LIU B , JIANG W B ,et al. Layercentric memory reuse and data migration for extreme-scale deep learning on many-core architectures[J]. ACM Transactions on Architecture and Code Optimization, 2018,15(3): 1-26.
[22]	WANG L N , YE J M , ZHAO Y Y ,et al. Superneurons:dynamic GPU memory management for training deep neural networks[C]// The 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. New York:ACM Press, 2018: 41-53.
[23]	PENG X , SHI X H , DAI H L ,et al. Capuchin:tensor-based GPU memory management for deep learning[C]// The 24th International Conference on Architectural Support for Programming Languages and Operating Systems. New York:ACM Press, 2020: 891-905.
[24]	CHEN T Q , XU B , ZHANG C Y ,et al. Training deep nets with sublinear memory cost[J]. Computer Science, 2016,arXiv:1604.06174.
[25]	GRUSLYS A , MUNOS R , DANIHELKA I ,et al. Memory-efficient backpropagation through time[C]// The 2016 Annual Conference on Neural Information Processing Systems. Cambridge:MIT Press, 2016: 4125-4133.
[26]	KUSUMOTO M , INOUE T , WATANABE G ,et al. A graph theoretic framework of recomputation algorithms for memoryefficient backpropagation[C]// The 2019 Annual Conference on Neural Information Processing Systems. Cambridge:MIT Press, 2019: 1161-1170.
[27]	JAIN P , JAIN A , NRUSIMHA A ,et al. Checkmate:breaking the memory wall with optimal tensor rematerialization[C]// Machine Learning and Systems 2020.[S.l.:s.n]. 2020: 497-511.
[28]	RHU M,O’CONNOR M , CHATTERJEE N ,et al. Compressing DMA engine:leveraging activation sparsity for training deep neural networks[C]// The IEEE 24th International Symposium on High Performance Computer Architecture. Piscataway:IEEE Press, 2018: 78-91.
[29]	JAIN A , PHANISHAYEE A , MARS J ,et al. Gist:efficient data encoding for deep neural network training[C]// The 45th ACM/IEEE Annual International Symposium on Computer Architecture. Piscataway:IEEE Press, 2018: 776-789.
[30]	HAN S , MAO H Z , DALLY W . Deep compression:compressing deep neural network with pruning,trained quantization and Huffman coding[C]// The 4th International Conference on Learning Representations.[S.l.:s.n]. 2016.

Metrics

Recommended 0

No Suggested Reading articles found!

Memory management in deep learning:a survey

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 2

References 30

Related Articles 15

Metrics

Recommended 0

[1]	Yimin DENG, Xulong ZHANG, Shijing SI, Jianzong WANG, Jing XIAO. Human avatars synthesis technologies: a survey [J]. Big Data Research, 2023, 9(3): 114-139.
[2]	Yayun HE, Junqing PENG, Jianzong WANG, Jing XIAO. Rhythm dancer: 3D dance generation by keymotion transition graph and pose-interpolation network [J]. Big Data Research, 2023, 9(1): 23-37.
[3]	Yumeng CUI, Jingya WANG, Shangyi YAN, Zhizhong TAO. Automatic key information extraction of police records based on deep learning [J]. Big Data Research, 2022, 8(6): 127-142.
[4]	Zhitao ZHU, Shijing SI, Jianzong WANG, Jing XIAO. Survey on federated recommendation systems [J]. Big Data Research, 2022, 8(4): 105-132.
[5]	Jie WANG, Songyan ZHANG, Jiye LIANG. A semi-supervised deep learning algorithm combining consistency regularization and manifold regularization [J]. Big Data Research, 2022, 8(3): 103-114.
[6]	Kangting XU, Wei Song. A Chinese text sentiment analysis method combining language knowledge and deep learning [J]. Big Data Research, 2022, 8(3): 115-127.
[7]	Zhitao ZHAO, Lijun ZHAO, Zheng ZHANG, Ping TANG. Integration of remote sensing intelligent processing algorithm using container cloud technology [J]. Big Data Research, 2022, 8(2): 58-74.
[8]	Kai ZHANG, Yang CHE. Method of accelerating deep learning with optimized distributed cache in containers [J]. Big Data Research, 2021, 7(5): 150-163.
[9]	Jingxi WEN, Hufei YU, Jiang XIN, Yan TANG. Analysis of gender differences in the brain based on deep learning [J]. Big Data Research, 2021, 7(4): 130-140.
[10]	Xin PENG, Chi CHEN, Yun LIN. Context-based intelligent recommendation for code reuse [J]. Big Data Research, 2021, 7(1): 37-47.
[11]	Lihui WANG, Yongbin QIN. State of the art and future perspectives of the applications of deep learning in the medical image analysis [J]. Big Data Research, 2020, 6(6): 83-104.
[12]	Hufei YU, Jingxi WEN, Jiang XIN, Yan TANG. Study on domain adaptation of medical data based on generative adversarial network [J]. Big Data Research, 2020, 6(5): 45-54.
[13]	Ting SONG, Zhanwei CHEN, Haifeng YANG. Aspect sentiment analysis based on a hierarchical attention network [J]. Big Data Research, 2020, 6(5): 82-91.
[14]	Fan YU. Research on the next-generation deep learning framework [J]. Big Data Research, 2020, 6(4): 69-80.
[15]	Luchen LIU, Jianhao SHEN, Ming ZHANG, Zichang WANG, Haoran LI, Zequn LIU. Deep learning based patient representation learning framework of heterogeneous temporal events data [J]. Big Data Research, 2019, 5(1): 25-38.