基于深度强化学习的微服务多维动态防御策略研究

doi:10.11959/j.issn.1000-436x.2023077

通信学报 ›› 2023, Vol. 44 ›› Issue (4): 50-63.doi: 10.11959/j.issn.1000-436x.2023077

基于深度强化学习的微服务多维动态防御策略研究

周大成, 陈鸿昶, 何威振, 程国振, 扈红超

信息工程大学信息技术研究所，河南郑州 450002

修回日期:2023-02-23 出版日期:2023-04-25 发布日期:2023-04-01
作者简介:周大成（1995- ），男，河南息县人，信息工程大学博士生，主要研究方向为网络空间安全、云计算等
陈鸿昶（1964- ），男，河南新密人，博士，信息工程大学教授、博士生导师，主要研究方向为网络空间安全、数据分析等
何威振（1996- ），男，安徽亳州人，信息工程大学博士生，主要研究方向为网络空间安全、云计算等
程国振（1986- ），男，山东菏泽人，博士，信息工程大学副教授、硕士生导师，主要研究方向为网络空间安全、软件定义网络等
扈红超（1982- ），男，河南商丘人，博士，信息工程大学教授、博士生导师，主要研究方向为网络空间安全、拟态防御等
基金资助:
国家自然科学基金资助项目(62072467);国家重点研发计划基金资助项目(2021YFB1006200);国家重点研发计划基金资助项目(2021YFB1006201)

Research on multidimensional dynamic defense strategy for microservice based on deep reinforcement learning

Dacheng ZHOU, Hongchang CHEN, Weizhen HE, Guozhen CHENG, Hongchao HU

Institute of Information Technology, Information Engineering University, Zhengzhou 450002, China

Revised:2023-02-23 Online:2023-04-25 Published:2023-04-01
Supported by:
The National Natural Science Foundation of China(62072467);The National Key Research and Develop-ment Program of China(2021YFB1006200);The National Key Research and Develop-ment Program of China(2021YFB1006201)

摘要/Abstract

摘要：

针对云原生中安全防御策略在动态请求流量下难以兼顾服务质量的问题，提出基于深度强化学习的微服务多维动态防御策略，简称D2RA策略，在流量动态变化时给出兼顾安全防御和服务质量的动态配置方案。首先，基于微服务多副本部署和微服务调用链的特点，建立微服务系统状态图来刻画微服务的请求流量、系统配置与安全性、服务质量、资源开销之间的关系；其次，设计D2RA框架并提出基于深度Q网络的动态策略优化算法，为微服务提供动态请求流量下最优系统配置快速更新方案。仿真实验结果表明，D2RA在动态请求流量下可有效进行资源分配，相对于对比方法在防御有效性和服务质量方面分别取得19.07%和42.31%的优化。

关键词: 微服务, 云原生, 动态防御, 强化学习, 深度Q网络

Abstract:

Aiming at the problem that it is hard for security defense strategies in cloud native to guarantee the quality of service under dynamic requests, a multidimensional dynamic defense strategy for microservice based on deep reinforcement learning, named D2RA strategy, was proposed to provide dynamic configuration schemes that ensure security defense performance and quality of service for microservices under dynamical requests.Firstly, based on the characteristics of multiple replicas and invocation chains of microservices, a microservices state graph was established to depict the maps between requests, system configuration and security performance, quality of service, and resource overhead of microservices.Secondly, the D2RA framework was designed and a dynamic strategy optimization algorithm based on deep Q-network was proposed for microservices to provide fast and optimal system configurations update scheme under dynamic requests.The simulation results show that D2RA effectively allocate resources under dynamic requests, and achieve 19.07% more defense effectiveness and 42.31% higher quality of service as compared to the existing methods.

Key words: microservice, cloud native, dynamic defense, reinforcement learning, deep Q-network

中图分类号:

TP309.1

周大成, 陈鸿昶, 何威振, 程国振, 扈红超. 基于深度强化学习的微服务多维动态防御策略研究[J]. 通信学报, 2023, 44(4): 50-63.

Dacheng ZHOU, Hongchang CHEN, Weizhen HE, Guozhen CHENG, Hongchao HU. Research on multidimensional dynamic defense strategy for microservice based on deep reinforcement learning[J]. Journal on Communications, 2023, 44(4): 50-63.

图/表 12

图1

图2

表1

图3

图4

表2

仿真参数设置"

仿真参数	取值
单次攻击成功概率 $p_{i}^{succ}$	[0.3,0.5]的随机值
攻击所需次数?	[5,15]的随机整数
攻击时间间隔 $T^{att} / s$	0.5
攻击风险调节系数ξ	0.8
各个微服务副本的服务率μ	0.2
微服务响应时间服务等级目标 $ψ_{SLO} / s$	0.5
微服务安全性目标权重 $ω_{1}$	0.4
微服务服务质量目标权重 $ω_{2}$	0.3
微服务资源开销目标权重 $ω_{3}$	0.3
收益折扣因子γ	0.9
采样批量batch	32
学习率	0.000 1
目标网络更新步长C	200
初始探索概率 $ε_{\max}$	1
最终探索概率 $ε_{\min}$	0.1
ε下降所需步数M	30 000
经验回放池大小	50 000

表2

图5

图6

图7

图8

图9

图10

参考文献 24

[1]	GANNON D , BARGA R , SUNDARESAN N . Cloud-native applications[J]. IEEE Cloud Computing, 2017,4(5): 16-21.
[2]	TH?NES J . Microservices[J]. IEEE Software, 2015,32(1): 116.
[3]	LUO S T , XU H L , LU C Z ,et al. Characterizing microservice dependency and performance:Alibaba trace analysis[C]// Proceedings of the ACM Symposium on Cloud Computing. New York:ACM Press, 2021: 412-426.
[4]	NIFE F N , KOTULSKI Z . Application-aware firewall mechanism for software defined networks[J]. Journal of Network and Systems Management, 2020,28(3): 605-626.
[5]	BáNáTI A , KAIL E , KARóCZKAI K ,et al. Authentication and authorization orchestrator for microservice-based software architectures[C]// Proceedings of 2018 41st International Convention on Information and Communication Technology,Electronics and Microelectronics. Piscataway:IEEE Press, 2018: 1180-1184.
[6]	BARDAS A G , SUNDARAMURTHY S C , OU X ,et al. MTD CBITS:moving target defense for cloud-based IT systems[C]// Proceedings of 22nd European Symposium on Research in Computer Security. Berlin:Springer, 2017: 167-186.
[7]	TORKURA K A , SUKMANA M I H , KAYEM A V D M ,et al. A cyber risk based moving target defense mechanism for microservice architectures[C]// Proceedings of 2018 IEEE International Conference on Parallel ＆ Distributed Processing with Applications,Ubiquitous Computing ＆ Communications,Big Data ＆ Cloud Computing,Social Computing ＆ Networking,Sustainable Computing ＆ Communications. Piscataway:IEEE Press, 2018: 932-939.
[8]	JIN H , LI Z , ZOU D Q ,et al. DSEOM:a framework for dynamic security evaluation and optimization of MTD in container-based cloud[J]. IEEE Transactions on Dependable and Secure Computing, 2019,18(3): 1125-1136.
[9]	张帅, 郭云飞, 孙鹏浩 ,等. 云原生下基于深度强化学习的移动目标防御策略优化方案[J]. 电子与信息学报, 2023,45(2): 608-616.
	ZHANG S , GUO Y F , SUN P H ,et al. Optimization scheme of moving target defense strategy based on deep reinforcement learning in cloud native environment[J]. Journal of Electronics ＆ Information Technology, 2023,45(2): 608-616.
[10]	DUC T L , LEIVA R G , CASARI P ,et al. Machine learning methods for reliable resource provisioning in edge-cloud computing:a survey[J]. ACM Computing Surveys, 2020,52(5): 1-39.
[11]	ZHOU D C , CHEN H C , SHANG K ,et al. Cushion:a proactive resource provisioning method to mitigate SLO violations for containerized microservices[J]. IET Communications, 2022,16: 2105-2122.
[12]	YADAV T , RAO A M . Technical aspects of cyber kill chain[C]// Proceedings of International Symposium on Security in Computing and Communication. Berlin:Springer, 2015: 438-452.
[13]	NOUREDDINE M A , FAWAZ A , SANDERS W H ,et al. A game-theoretic approach to respond to attacker lateral movement[C]// Proceedings of 7th International Conference on Decision and Game Theory for Security. New York:ACM Press, 2016: 294-313.
[14]	ALMOHRI H M J , WATSON L T , EVANS D . Misery digraphs:delaying intrusion attacks in obscure clouds[J]. IEEE Transactions on Information Forensics and Security, 2018,13(6): 1361-1375.
[15]	张福, 程度, 胡俊 . ATT＆CK框架实践指南[M]. 北京: 电子工业出版社, 2022.
	ZHANG F , CHENG D , HU J . ATT＆CK framework practice guide[M]. Beijing: Publish House of Electronics Industry, 2022.
[16]	GAN Y , ZHANG Y Q , CHENG D L ,et al. An open-source benchmark suite for microservices and their hardware-software implications for cloud ＆ edge systems[C]// Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems. New York:ACM Press, 2019: 3-18.
[17]	CHO J H , SHARMA D P , ALAVIZADEH H ,et al. Toward proactive,adaptive defense:a survey on moving target defense[J]. IEEE Communications Surveys ＆ Tutorials, 2020,22(1): 709-745.
[18]	CAI G L , WANG B S , HU W ,et al. Moving target defense:state of the art and characteristics[J]. Frontiers of Information Technology ＆ Electronic Engineering, 2016,17(11): 1122-1153.
[19]	ZHANG X , SEN S , KURNIAWAN D ,et al. E2E:embracing user heterogeneity to improve quality of experience on the web[C]// Proceedings of the ACM Special Interest Group on Data Communication. New York:ACM Press, 2019: 289-302.
[20]	GIAS A U , CASALE G , WOODSIDE M . ATOM:model-driven autoscaling for microservices[C]// Proceedings of 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). Piscataway:IEEE Press, 2019: 1994-2004.
[21]	LI Z , JIN H , ZOU D Q ,et al. Exploring new opportunities to defeat low-rate DDoS attack in container-based cloud environment[J]. IEEE Transactions on Parallel and Distributed Systems, 2019,31(3): 695-706.
[22]	BHASI V M , GUNASEKARAN J R , THINAKARAN P ,et al. Kraken:adaptive container provisioning for deploying dynamic DAGs in serverless platforms[C]// Proceedings of the ACM Symposium on Cloud Computing. New York:ACM Press, 2021: 153-167.
[23]	MNIH V , KAVUKCUOGLU K , SILVER D ,et al. Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540): 529-533.
[24]	ABELS A , ROIJERS D , LENAERTS T ,et al. Dynamic weights in multi-objective deep reinforcement learning[C]// Proceedings of International Conference on Machine Learning. Saarland:DBLP, 2019: 11-20.

动作	含义
i⁺	选定微服务的索引增加1
i^-	选定微服务的索引减少1
δ N⁺	选定微服务的副本增加一个
δ N^-	选定微服务的副本减少一个
δ T⁺	选定微服务的副本清洗周期增加1 s
δ T^-	选定微服务的副本清洗周期减少1 s
0	不采取任何动作

基于深度强化学习的微服务多维动态防御策略研究

Research on multidimensional dynamic defense strategy for microservice based on deep reinforcement learning

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 24

相关文章 15

Metrics

推荐阅读 0

[1]	马玲, 樊漆亮, 许婷, 郭冠琛, 张圣林, 孙永谦, 张玉志. 基于强化学习的在线离线混部云环境下的调度框架[J]. 通信学报, 2023, 44(6): 90-102.
[2]	金彪, 李逸康, 姚志强, 陈瑜霖, 熊金波. GenFedRL：面向深度强化学习智能体的通用联邦强化学习框架[J]. 通信学报, 2023, 44(6): 183-197.
[3]	李元诚, 秦永泰. 基于深度强化学习的软件定义安全中台QoS实时优化算法[J]. 通信学报, 2023, 44(5): 181-192.
[4]	许国良, 谭峰, 冉泳屹, 陈丰. 面向多波束卫星系统的波束跳变与覆盖控制联合优化算法[J]. 通信学报, 2023, 44(4): 78-86.
[5]	夏景明, 刘玉风, 谈玲. 基于蜂窝网络的多无人机能量消耗最优化算法研究[J]. 通信学报, 2023, 44(2): 185-197.
[6]	许文俊, 吴思雷, 王凤玉, 林兰, 李国军, 张治. 基于多智能体强化学习的大规模灾后用户分布式覆盖优化[J]. 通信学报, 2022, 43(8): 1-16.
[7]	沙宗轩, 霍如, 孙闯, 汪硕, 黄韬. 基于深度强化学习的转发效能感知流量调度算法[J]. 通信学报, 2022, 43(8): 30-40.
[8]	马帅, 李兵, 盛海鸿, 谷荣妍, 周辉, 王洪梅, 王悦, 李世银. 基于深度强化学习的可见光定位通信一体化功率分配研究[J]. 通信学报, 2022, 43(8): 121-130.
[9]	张宇, 程旻. NDN中边缘计算与缓存的联合优化[J]. 通信学报, 2022, 43(8): 164-175.
[10]	左珮良, 侯少龙, 郭超, 蒋华, 王文博. 基于强化学习的多层卫星网络边缘安全决策方法[J]. 通信学报, 2022, 43(6): 189-199.
[11]	张先超, 赵耀, 叶海军, 樊锐. 无线网络多用户干扰下智能发射功率控制算法[J]. 通信学报, 2022, 43(2): 15-21.
[12]	李传煌, 陈泱婷, 唐晶晶, 楼佳丽, 谢仁华, 方春涛, 王伟明, 陈超. QL-STCT：一种SDN链路故障智能路由收敛方法[J]. 通信学报, 2022, 43(2): 131-142.
[13]	陈赓, 齐书虎, 沈斐, 曾庆田. 面向B5G多业务场景基于D3QN的双时间尺度网络切片算法[J]. 通信学报, 2022, 43(11): 213-224.
[14]	陈晋音, 胡书隆, 邢长友, 张国敏. 面向智能渗透攻击的欺骗防御方法[J]. 通信学报, 2022, 43(10): 106-120.
[15]	苏新, 孟蕾蕾, 周一青, CELIMUGE Wu. 基于深度强化学习的海洋移动边缘计算卸载方法[J]. 通信学报, 2022, 43(10): 133-145.