基于鸽群的鲁棒强化学习算法

doi:10.11959/j.issn.2096-109x.2022064

网络与信息安全学报 ›› 2022, Vol. 8 ›› Issue (5): 66-74.doi: 10.11959/j.issn.2096-109x.2022064

• 专题：大数据与人工智能安全 • 上一篇下一篇

基于鸽群的鲁棒强化学习算法

张明英¹, 华冰², 张宇光¹, 李海东¹, 郑墨泓³

¹ 中国电子技术标准化研究院，北京 100007
² 南京航空航天大学航天学院，江苏南京 211106
³ 中国电子科技集团公司第七研究所，广东广州 510000

修回日期:2022-07-15 出版日期:2022-10-15 发布日期:2022-10-01
作者简介:张明英（1985- ），男，广西北海人，中国电子技术标准化研究院高级工程师，主要研究方向为人工智能、知识图谱、大数据
华冰（1978- ），女，江苏南京人，南京航空航天大学副研究员，主要研究方向为飞行器导航、智能数据处理
张宇光（1991- ），男，内蒙古包头人，中国电子技术标准化研究院工程师，主要研究方向为数据安全、人工智能安全、个人信息保护、计算机视觉、视觉生成
李海东（1992- ），男，湖北孝感人，中国电子技术标准化研究院工程师，主要研究方向为人工智能安全、大数据安全、个人信息保护
郑墨泓（1995- ），女，广东潮阳人，中国电子科技集团公司第七研究所助理工程师，主要研究方向为航天器姿态控制、无人机组网
基金资助:
科技创新2030重大项目(2020AAA0107804)

Robust reinforcement learning algorithm based on pigeon-inspired optimization

Mingying ZHANG¹, Bing HUA², Yuguang ZHANG¹, Haidong LI¹, Mohong ZHENG³

¹ China Electronics Standardization Institute, Beijing 100007, China
² College of Astronautics, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China
³ The 7th Research Institute of China Electronics Technology Group Corporation, Guangzhou 510000, China

Revised:2022-07-15 Online:2022-10-15 Published:2022-10-01
Supported by:
Science and Technology Innovation 2030 Major Project(2020AAA0107804)

摘要/Abstract

摘要：

强化学习是一种人工智能算法，具有计算逻辑清晰、模型易扩展的优点，可以在较少甚至没有先验信息的前提下，通过和环境交互并最大化值函数，调优策略性能，有效地降低物理模型引起的复杂性。基于策略梯度的强化学习算法目前已成功应用于图像智能识别、机器人控制、自动驾驶路径规划等领域。然而强化学习高度依赖采样的特性决定了其训练过程需要大量样本来收敛，且决策的准确性易受到与仿真环境中不匹配的轻微干扰造成严重影响。特别是当强化学习应用于控制领域时，由于无法保证算法的收敛性，难以对其稳定性进行证明，为此，需要对强化学习进行改进。考虑到群体智能算法可通过群体协作解决复杂问题，具有自组织性及稳定性强的特征，利用其对强化学习进行优化求解是一个提高强化学习模型稳定性的有效途径。结合群体智能中的鸽群算法，对基于策略梯度的强化学习进行改进：针对求解策略梯度时存在迭代求解可能无法收敛的问题，提出了基于鸽群的强化学习算法，以最大化未来奖励为目的求解策略梯度，将鸽群算法中的适应性函数和强化学习结合估计策略的优劣，避免求解陷入死循环，提高了强化学习算法的稳定性。在具有非线性关系的两轮倒立摆机器人控制系统上进行仿真验证，实验结果表明，基于鸽群的强化学习算法能够提高系统的鲁棒性，降低计算量，减少算法对样本数据库的依赖。

关键词: 鸽群算法, 强化学习, 策略梯度, 鲁棒性

Abstract:

Reinforcement learning(RL) is an artificial intelligence algorithm with the advantages of clear calculation logic and easy expansion of the model.Through interacting with the environment and maximizing value functions on the premise of obtaining little or no prior information, RL can optimize the performance of strategies and effectively reduce the complexity caused by physical models .The RL algorithm based on strategy gradient has been successfully applied in many fields such as intelligent image recognition, robot control and path planning for automatic driving.However, the highly sampling-dependent characteristics of RL determine that the training process needs a large number of samples to converge, and the accuracy of decision making is easily affected by slight interference that does not match with the simulation environment.Especially when RL is applied to the control field, it is difficult to prove the stability of the algorithm because the convergence of the algorithm cannot be guaranteed.Considering that swarm intelligence algorithm can solve complex problems through group cooperation and has the characteristics of self-organization and strong stability, it is an effective way to be used for improving the stability of RL model.The pigeon-inspired optimization algorithm in swarm intelligence was combined to improve RL based on strategy gradient.A RL algorithm based on pigeon-inspired optimization was proposed to solve the strategy gradient in order to maximize long-term future rewards.Adaptive function of pigeon-inspired optimization algorithm and RL were combined to estimate the advantages and disadvantages of strategies, avoid solving into an infinite loop, and improve the stability of the algorithm.A nonlinear two-wheel inverted pendulum robot control system was selected for simulation verification.The simulation results show that the RL algorithm based on pigeon-inspired optimization can improve the robustness of the system, reduce the computational cost, and reduce the algorithm’s dependence on the sample database.

Key words: pigeon-inspired optimization algorithm, strengthen learning, policy gradient, robustness

中图分类号:

TP393

张明英, 华冰, 张宇光, 李海东, 郑墨泓. 基于鸽群的鲁棒强化学习算法[J]. 网络与信息安全学报, 2022, 8(5): 66-74.

Mingying ZHANG, Bing HUA, Yuguang ZHANG, Haidong LI, Mohong ZHENG. Robust reinforcement learning algorithm based on pigeon-inspired optimization[J]. Chinese Journal of Network and Information Security, 2022, 8(5): 66-74.

图/表 11

图1

图2

图3

表1

表2

图4

图5

图6

图7

图8

表3

参考文献 20

[1]	PETERS J , SCHAAL S . Policy gradient methods for robotics[C]// 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems. 2006: 2219-2225.
[2]	BAUM Y , AMICO M , HOWELL S ,et al. Experimental deep reinforcement learning for error-robust gate-set design on a superconducting quantum computer[J]. PRX Quantum, 2021,2(4): 040324.
[3]	HUA J , ZENG L , LI G ,et al. Learning for a robot:deep reinforcement learning,imitation learning,transfer learning[J]. Sensors, 2021,21(4): 1278.
[4]	SIVAK V V , EICKBUSCH A , LIU H ,et al. Model-free quantum control with reinforcement learning[J]. Physical Review X, 2022,12(1): 011059.
[5]	AGARWAL N , HAZAN E , MAJUMDAR A ,et al. A regret minimization approach to iterative learning control[C]// International Conference on Machine Learning (PMLR). 2021: 100-109.
[6]	KRIZHEVSKY A , SUTSKEVER I , HINTON G E . Imagenet classification with deep convolutional neural networks[J]. Advances in Neural Information Processing Systems. 2012: 1097-1105.
[7]	YARATS D , FERGUS R , LAZARIC A ,et al. Reinforcement learning with prototypical representations[C]// International Conference on Machine Learning (PMLR). 2021: 11920-11931.
[8]	DAHL G E , YU D , DENG L ,et al. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition[J]. IEEE Transactions on Audio,Speech,and Language Processing, 2011: 30-42.
[9]	HAOXIANG W , SMYS S . Overview of configuring adaptive activation functions for deep neural networks—a comparative study[J]. Journal of Ubiquitous Computing and Communication Technologies (UCCT), 2021,3(1): 10-22.
[10]	MISHRA A , LATORRE J A , Pool J ,et al. Accelerating sparse deep neural networks[J]. arXiv preprint arXiv:2104.08378, 2021.
[11]	SILVER D , HUANG A , MADDISON C J ,et al. Mastering the game of go with deep neural networks and tree search[J]. Nature, 2016,529:484.
[12]	VINYALS O , BABUSCHKIN I , CZARNECKI W M ,et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning[J]. Nature, 2019,575: 350-354.
[13]	HEESS N , WAYNE G , SILVER D ,et al. Learning continuous control policies by stochastic value gradients[C]// Advances in Neural Information Processing Systems. 2015:28.
[14]	CHEN Z , CHEN B , XIE S ,et al. Efficiently training on-policy actor-critic networks in robotic deep reinforcement learning with demonstration-like sampled exploration[C]// 2021 3rd International Symposium on Robotics ＆ Intelligent Manufacturing Technology (ISRIMT). 2021: 292-298.
[15]	WANG C , LING Y . Actor-critic tracking with precise scale estimation and advantage function[J]. Journal of Physics Conference Series, 2021,1827(1): 012064.
[16]	ZHANG S , DUAN H . Gaussian pigeon-inspired optimization approach to orbital spacecraft formation reconfiguration[J]. Chinese Journal of Aeronautics, 2015,28(1): 200-205.
[17]	ZHANG B , DUAN H . Three-dimensional path planning for uninhabited combat aerial vehicle based on predator-prey pigeon-inspired optimization in dynamic environment[J]. IEEE/ACM Transactions on Computational Biology ＆ Bioinformatics, 2017,14(1): 97-107.
[18]	周雨鹏 . 基于鸽群算法的函数优化问题求解[D]. 长春:东北师范大学, 2016.
	ZHOU Y P . Function optimization problem solving based on pigeon swarm algorithm[D]. Changchun:Northeast Normal University, 2016.
[19]	顾清华, 孟倩倩 . 优化复杂函数的粒子群-鸽群混合优化算法[J]. 计算机工程与应用, 2019,55(22): 46-52.
	GU Q H , MENG Q Q . Hybrid particle swarm optimization and pigeon—inspired optimization algorithm for solving complex functions[J]. Computer Engineering and Applications, 2019,55(22): 46-52.
[20]	胡耀龙, 冯强, 海星朔 ,等. 基于自适应学习策略的改进鸽群优化算法[J]. 北京航空航天大学学报, 2020,46(12): 2348-2356.
	HU Y L , FENG Q , HAI X S ,et al. Improved pigeon-inspired optimization algorithm based on adaptive learning strategy[J]. Journal of Beijing University of Aeronautics and Astronautics, 2020,46(12): 2348-2356.

算例	控制算法	数据库
1	基于鸽群的鲁棒强化学习算法	与任务匹配
2	基于鸽群的鲁棒强化学习算法	与任务不匹配
3	基于策略梯度的强化学习算法	与任务匹配
4	基于策略梯度的强化学习算法	与任务不匹配

物理量	参数	物理量	参数
m_B	2.5 kg	m_W	0.636 kg
l	0.026 m	J	5.175×10^-4kg.m²

算例	控制算法	数据库是否匹配	控制结果	完成1次训练的时间/s	是否存在无法判断策略梯度的情况
1	基于鸽群的鲁棒强化学习算法	是	完成控制任务	3.4	否
2	基于鸽群的鲁棒强化学习算法	否	完成控制任务	3.4	否
3	基于策略梯度的强化学习算法	是	完成控制任务	4.1	是
4	基于策略梯度的强化学习算法	否	未完成	4.1	是

基于鸽群的鲁棒强化学习算法

Robust reinforcement learning algorithm based on pigeon-inspired optimization

在线阅读

pdf下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 20

相关文章 8

Metrics

推荐阅读 0

[1]	肖天, 江智昊, 唐鹏, 黄征, 郭捷, 邱卫东. 基于深度强化学习的高性能导向性模糊测试方案[J]. 网络与信息安全学报, 2023, 9(2): 132-142.
[2]	林点, 潘理, 易平. 面向图像识别的卷积神经网络鲁棒性研究进展[J]. 网络与信息安全学报, 2022, 8(3): 111-122.
[3]	石灏苒, 吉立新, 刘树新, 王庚润. 基于半局部结构的异常连边识别算法[J]. 网络与信息安全学报, 2022, 8(1): 63-72.
[4]	王鹏程, 郑海斌, 邹健飞, 庞玲, 李虎, 陈晋音. 面向商用活体检测平台的鲁棒性评估[J]. 网络与信息安全学报, 2022, 8(1): 180-189.
[5]	王正龙, 张保稳. 生成对抗网络研究综述[J]. 网络与信息安全学报, 2021, 7(4): 68-85.
[6]	陈晋音, 张敦杰, 黄国瀚, 林翔, 鲍亮. 面向图神经网络的对抗攻击与防御综述[J]. 网络与信息安全学报, 2021, 7(3): 1-28.
[7]	吴奇,陈鸿昶. 低故障恢复开销的软件定义网络控制器布局算法[J]. 网络与信息安全学报, 2020, 6(6): 97-104.
[8]	徐堂炜,张海璐,刘楚环,肖亮,朱珍民. 基于强化学习的低时延车联网群密钥分配管理技术[J]. 网络与信息安全学报, 2020, 6(5): 119-125.