通信学报 ›› 2018, Vol. 39 ›› Issue (11): 106-115.doi: 10.11959/j.issn.1000-436x.2018238

• 学术论文 • 上一篇    下一篇

增强型深度确定策略梯度算法

陈建平1,2,3,4,何超1,2,3,刘全5,吴宏杰1,2,3,4,胡伏原1,2,3,4,傅启明1,2,3,4()   

  1. 1 苏州科技大学电子与信息工程学院,江苏 苏州 215009
    2 苏州科技大学江苏省建筑智慧节能重点实验室,江苏 苏州 215009
    3 苏州科技大学苏州市移动网络技术与应用重点实验室,江苏 苏州 215009
    4 苏州科技大学苏州市虚拟现实智能交互及应用技术重点实验室,江苏 苏州 215009
    5 苏州大学计算机科学与技术学院,江苏 苏州 215006
  • 修回日期:2018-08-01 出版日期:2018-11-01 发布日期:2018-12-10
  • 作者简介:陈建平(1963?),男,江苏南京人,博士,苏州科技大学教授,主要研究方向为大数据分析与应用、建筑节能、智能信息处理。|何超(1993?),男,江苏徐州人,苏州科技大学硕士生,主要研究方向为强化学习、深度学习、建筑节能。|刘全(1969?),男,内蒙古牙克石人,博士,苏州大学教授、博士生导师,主要研究方向为智能信息处理、自动推理与机器学习。|吴宏杰(1977?),男,江苏苏州人,博士,苏州科技大学副教授,主要研究方向为深度学习、模式识别、生物信息。|胡伏原(1978?),男,湖南岳阳人,博士,苏州科技大学教授,主要研究方向为模式识别与机器学习。|傅启明(1985?),男,江苏淮安人,博士,苏州科技大学讲师,主要研究方向为强化学习、深度学习及建筑节能。
  • 基金资助:
    国家自然科学基金资助项目(61502329);国家自然科学基金资助项目(61772357);国家自然科学基金资助项目(61750110519);国家自然科学基金资助项目(61772355);国家自然科学基金资助项目(61702055);国家自然科学基金资助项目(61672371);国家自然科学基金资助项目(61602334);国家自然科学基金资助项目(61502323);江苏省自然科学基金资助项目(BK20140283);江苏省重点研发计划基金资助项目(BE2017663);江苏省高校自然科学研究基金资助项目(13KJB520020);苏州市应用基础研究计划工业部分基金资助项目(SYG201422)

Enhanced deep deterministic policy gradient algorithm

Jianping CHEN1,2,3,4,Chao HE1,2,3,Quan LIU5,Hongjie WU1,2,3,4,Fuyuan HU1,2,3,4,Qiming FU1,2,3,4()   

  1. 1 Institute of Electronics and Information Engineering,Suzhou University of Science and Technology,Suzhou 215009,China
    2 Jiangsu Province Key Laboratory of Intelligent Building Energy Efficiency,Suzhou University of Science and Technology,Suzhou 215009,China
    3 Suzhou Key Laboratory of Mobile Networking and Applied Technologies,Suzhou University of Science and Technology,Suzhou 215009,China
    4 Virtual Reality Key Laboratory of Intelligent Interaction and Application Technology of Suzhou,Suzhou University of Science and Technology,Suzhou 215009,China
    5 School of Computer Science and Technology,Soochow University,Suzhou 215006,China
  • Revised:2018-08-01 Online:2018-11-01 Published:2018-12-10
  • Supported by:
    The National Natural Science Foundation of China(61502329);The National Natural Science Foundation of China(61772357);The National Natural Science Foundation of China(61750110519);The National Natural Science Foundation of China(61772355);The National Natural Science Foundation of China(61702055);The National Natural Science Foundation of China(61672371);The National Natural Science Foundation of China(61602334);The National Natural Science Foundation of China(61502323);The Natural Science Foundation of Jiangsu Province(BK20140283);The Key Research and Development Program of Jiangsu Province(BE2017663);High School Natural Foundation of Jiangsu Province(13KJB520020);Suzhou Industrial Application of Basic Research Program Part(SYG201422)

摘要:

针对深度确定策略梯度算法收敛速率较慢的问题,提出了一种增强型深度确定策略梯度(E-DDPG)算法。该算法在深度确定策略梯度算法的基础上,重新构建两个新的样本池——多样性样本池和高误差样本池。在算法执行过程中,训练样本分别从多样性样本池和高误差样本池按比例选取,以兼顾样本多样性以及样本价值信息,提高样本的利用效率和算法的收敛性能。此外,进一步从理论上证明了利用自模拟度量方法对样本进行相似性度量的合理性,建立值函数与样本相似性之间的关系。将E-DDPG算法以及DDPG算法用于经典的Pendulum问题和MountainCar问题,实验结果表明,E-DDPG具有更好的收敛稳定性,同时具有更快的收敛速率。

关键词: 深度强化学习, 样本排序, 自模拟度量, 时间差分误差

Abstract:

With the problem of slow convergence for deep deterministic policy gradient algorithm,an enhanced deep deterministic policy gradient algorithm was proposed.Based on the deep deterministic policy gradient algorithm,two sample pools were constructed,and the time difference error was introduced.The priority samples were added when the experience was played back.When the samples were trained,the samples were selected from two sample pools respectively.At the same time,the bisimulation metric was introduced to ensure the diversity of the selected samples and improve the convergence rate of the algorithm.The E-DDPG algorithm was used to pendulum problem.The experimental results show that the E-DDPG algorithm can effectively improve the convergence performance of the continuous action space problems and have better stability.

Key words: deep reinforcement learning, sample ranking, bisimulation metric, temporal difference error

中图分类号: 

No Suggested Reading articles found!