增强型深度确定策略梯度算法

doi:10.11959/j.issn.1000-436x.2018238

Journal on Communications ›› 2018, Vol. 39 ›› Issue (11): 106-115.doi: 10.11959/j.issn.1000-436x.2018238

• Papers • Previous Articles Next Articles

Enhanced deep deterministic policy gradient algorithm

Jianping CHEN^1,^2,^3,⁴,Chao HE^1,^2,³,Quan LIU⁵,Hongjie WU^1,^2,^3,⁴,Fuyuan HU^1,^2,^3,⁴,Qiming FU^1,^2,^3,⁴()

¹ Institute of Electronics and Information Engineering,Suzhou University of Science and Technology,Suzhou 215009,China
² Jiangsu Province Key Laboratory of Intelligent Building Energy Efficiency,Suzhou University of Science and Technology,Suzhou 215009,China
³ Suzhou Key Laboratory of Mobile Networking and Applied Technologies,Suzhou University of Science and Technology,Suzhou 215009,China
⁴ Virtual Reality Key Laboratory of Intelligent Interaction and Application Technology of Suzhou,Suzhou University of Science and Technology,Suzhou 215009,China
⁵ School of Computer Science and Technology,Soochow University,Suzhou 215006,China

Revised:2018-08-01 Online:2018-11-01 Published:2018-12-10
Supported by:
The National Natural Science Foundation of China(61502329);The National Natural Science Foundation of China(61772357);The National Natural Science Foundation of China(61750110519);The National Natural Science Foundation of China(61772355);The National Natural Science Foundation of China(61702055);The National Natural Science Foundation of China(61672371);The National Natural Science Foundation of China(61602334);The National Natural Science Foundation of China(61502323);The Natural Science Foundation of Jiangsu Province(BK20140283);The Key Research and Development Program of Jiangsu Province(BE2017663);High School Natural Foundation of Jiangsu Province(13KJB520020);Suzhou Industrial Application of Basic Research Program Part(SYG201422)

Abstract

Abstract:

With the problem of slow convergence for deep deterministic policy gradient algorithm,an enhanced deep deterministic policy gradient algorithm was proposed.Based on the deep deterministic policy gradient algorithm,two sample pools were constructed,and the time difference error was introduced.The priority samples were added when the experience was played back.When the samples were trained,the samples were selected from two sample pools respectively.At the same time,the bisimulation metric was introduced to ensure the diversity of the selected samples and improve the convergence rate of the algorithm.The E-DDPG algorithm was used to pendulum problem.The experimental results show that the E-DDPG algorithm can effectively improve the convergence performance of the continuous action space problems and have better stability.

Key words: deep reinforcement learning, sample ranking, bisimulation metric, temporal difference error

CLC Number:

TP391

Jianping CHEN,Chao HE,Quan LIU,Hongjie WU,Fuyuan HU,Qiming FU. Enhanced deep deterministic policy gradient algorithm[J]. Journal on Communications, 2018, 39(11): 106-115.

Figures/Tables 10

References 18

[1]	SUTTON R S , BARTO G A . Reinforcement learning:an introduction[M]. Cambridge: MIT pressPress, 1998.
[2]	朱斐, 刘全, 傅启明 ,等. 一种用于连续动作空间的最小二乘行动者-评论家方法[J]. 计算机研究与发展, 2014,51(3): 548-558.
	ZHU F , LIU Q , FU Q M . A least square actor-critic approach for continuous action space[J]. Journal of Computer Research and Development, 2014,51(3): 548-558.
[3]	孙志军, 薛磊, 许阳明 ,等. 深度学习研究综述[J]. 计算机应用研究, 2012,29(8): 2806-2810.
	SUN Z J , XUE L , XU Y M ,et al. Overview of deep learning[J]. Application Research of Computers, 2012,29(8): 2806-2810.
[4]	LECUN Y , BENGIO Y , HINTON G . Deep learning[J]. Nature, 2015,521(7553): 436-444.
[5]	HINTON G E , OSINDERO S , TEH Y W . A fast learning algorithm for deep belief nets[J]. Neural Computation, 2006,18(7): 1527-1554.
[6]	SILVER D , HUANG A , MADDISON C J ,et al. Mastering the game of Go with deep neural networks and tree search[J]. Nature, 2016,529(7587): 484-489.
[7]	SILVER D , SCHRITTWIESER J , SIMONYAN K ,et al. Mastering the game of go without human knowledge[J]. Nature, 2017,550(7676): 354-359.
[8]	MNIH V , KAVUKCUOFLU K , SILVER D ,et al. Playing atari with deep reinforcement learning[C]// Workshops at the 26th Neural Information Processing Systems. 2013.
[9]	MNIH V , KAVUKCUOFLU K , SILVER D ,et al. Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540): 529-533.
[10]	WATKINS C J C H . Learning from delayed rewards[J]. Robotics and Autonomous Systems, 1989,15(4): 233-235.
[11]	VAN H V , GUEZ A , SILVER D . Deep reinforcement learning with double q-learning[C]// The AAAI Conference on Artificial Intelligence. 2016.
[12]	HASSELT H V , . Double Q-learning[C]// The Advances in Neural Information Processing Systems. 2010.
[13]	SCHAUL T , QUAN J , ANTONOGLOU I ,et al. Prioritized experience replay[C]// The 4th International Conference on Learning Representations. 2016: 322-355.
[14]	SUTTON R S , MCALLESTER D , SINGH S ,et al. Policy gradient methods for reinforcement learning with function approximation[J]. Advances in Neural Information Processing Systems, 2000,12: 1057-1063.
[15]	LILLICRAP T P , HUNT J J , PRITZEL A ,et al. Continuous control with deep reinforcement learning[C]// The 4th International Conference on Learning Representations. 2015.
[16]	SILVER D , LEVER G , HEESS N ,et al. Deterministic policy gradient algorithms[C]// The International Conference on Machine Learning. 2014.
[17]	GIVAN R , DEAN T , GREIG M . Equivalence notions and model minimization in Markov decision processes[J]. Artificial Intelligence, 2003,147(1-2): 163-223.
[18]	FERNS N , PANANGADEN P , PRECUP D . Metrics for finite markov decision processes[C]// The 20th Conference on Uncertainty in Artificial Intelligence. 2004.

Metrics

Recommended 0

No Suggested Reading articles found!

Enhanced deep deterministic policy gradient algorithm

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 18

Related Articles 15

Metrics

Recommended 0

[1]	Biao JIN, Yikang LI, Zhiqiang YAO, Yulin CHEN, Jinbo XIONG. GenFedRL: a general federated reinforcement learning framework for deep reinforcement learning agents [J]. Journal on Communications, 2023, 44(6): 183-197.
[2]	Yuancheng LI, Yongtai QIN. Deep reinforcement learning based algorithm for real-time QoS optimization of software-defined security middle platform [J]. Journal on Communications, 2023, 44(5): 181-192.
[3]	Guoliang XU, Feng TAN, Yongyi RAN, Feng CHEN. Joint beam hopping and coverage control optimization algorithm for multibeam satellite system [J]. Journal on Communications, 2023, 44(4): 78-86.
[4]	Zongxuan SHA, Ru HUO, Chuang SUN, Shuo WANG, Tao HUANG. Forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning [J]. Journal on Communications, 2022, 43(8): 30-40.
[5]	Yu ZHANG, Min CHENG. Joint optimization of edge computing and caching in NDN [J]. Journal on Communications, 2022, 43(8): 164-175.
[6]	Xianchao ZHANG, Yao ZHAO, Haijun YE, Rui FAN. Intelligent transmit power control algorithm for the multi-user interference of wireless network [J]. Journal on Communications, 2022, 43(2): 15-21.
[7]	Xin SU, Leilei MENG, Yiqing ZHOU, Wu CELIMUGE. Maritime mobile edge computing offloading method based on deep reinforcement learning [J]. Journal on Communications, 2022, 43(10): 133-145.
[8]	Li’na DU, Li ZHUO, Shuo YANG, Jiafeng LI, Jing ZHANG. Survey on reinforcement learning based adaptive bit rate algorithm for mobile video streaming services [J]. Journal on Communications, 2021, 42(9): 205-217.
[9]	Wenchen HE, Shaoyong GUO, Xuesong QIU, Liandong CHEN, Suxiang ZHANG. Node selection method in federated learning based on deep reinforcement learning [J]. Journal on Communications, 2021, 42(6): 62-71.
[10]	Guolin SUN,Ruijie OU,Guisong LIU. Deep reinforcement learning-based resource reservation algorithm for emergency Internet-of-things slice [J]. Journal on Communications, 2020, 41(9): 8-20.
[11]	Yingchang LIANG,Junjie TAN,Dusit Niyato. Overview on intelligent wireless communication technology [J]. Journal on Communications, 2020, 41(7): 1-17.
[12]	Tingting LIU,Yi’nan LUO,Chenyang YANG. Distributed interference coordination based on multi-agent deep reinforcement learning [J]. Journal on Communications, 2020, 41(7): 38-48.
[13]	Pei ZHANG,Shuaijun LIU,Zhiguo MA,Xiaohui WANG,Junde SONG. Improved satellite resource allocation algorithm based on DRL and MOP [J]. Journal on Communications, 2020, 41(6): 51-60.
[14]	Quan LIU,Yubin JIANG,Zhihui HU. Advantage estimator based on importance sampling [J]. Journal on Communications, 2019, 40(5): 108-116.
[15]	Xiaomin LIAO,Shaohu YAN,Jia SHI,Zhenyu TAN,Zhongling ZHAO,Zan LI. Deep reinforcement learning based resource allocation algorithm in cellular networks [J]. Journal on Communications, 2019, 40(2): 11-18.