基于深度强化学习的微服务多维动态防御策略研究

doi:10.11959/j.issn.1000-436x.2023077

Abstract

Abstract:

Aiming at the problem that it is hard for security defense strategies in cloud native to guarantee the quality of service under dynamic requests, a multidimensional dynamic defense strategy for microservice based on deep reinforcement learning, named D2RA strategy, was proposed to provide dynamic configuration schemes that ensure security defense performance and quality of service for microservices under dynamical requests.Firstly, based on the characteristics of multiple replicas and invocation chains of microservices, a microservices state graph was established to depict the maps between requests, system configuration and security performance, quality of service, and resource overhead of microservices.Secondly, the D2RA framework was designed and a dynamic strategy optimization algorithm based on deep Q-network was proposed for microservices to provide fast and optimal system configurations update scheme under dynamic requests.The simulation results show that D2RA effectively allocate resources under dynamic requests, and achieve 19.07% more defense effectiveness and 42.31% higher quality of service as compared to the existing methods.

Key words: microservice, cloud native, dynamic defense, reinforcement learning, deep Q-network

CLC Number:

TP309.1

Dacheng ZHOU, Hongchang CHEN, Weizhen HE, Guozhen CHENG, Hongchao HU. Research on multidimensional dynamic defense strategy for microservice based on deep reinforcement learning[J]. Journal on Communications, 2023, 44(4): 50-63.

Figures/Tables 12

仿真参数	取值
单次攻击成功概率 $p_{i}^{succ}$	[0.3,0.5]的随机值
攻击所需次数?	[5,15]的随机整数
攻击时间间隔 $T^{att} / s$	0.5
攻击风险调节系数ξ	0.8
各个微服务副本的服务率μ	0.2
微服务响应时间服务等级目标 $ψ_{SLO} / s$	0.5
微服务安全性目标权重 $ω_{1}$	0.4
微服务服务质量目标权重 $ω_{2}$	0.3
微服务资源开销目标权重 $ω_{3}$	0.3
收益折扣因子γ	0.9
采样批量batch	32
学习率	0.000 1
目标网络更新步长C	200
初始探索概率 $ε_{\max}$	1
最终探索概率 $ε_{\min}$	0.1
ε下降所需步数M	30 000
经验回放池大小	50 000

References 24

[1]	GANNON D , BARGA R , SUNDARESAN N . Cloud-native applications[J]. IEEE Cloud Computing, 2017,4(5): 16-21.
[2]	TH?NES J . Microservices[J]. IEEE Software, 2015,32(1): 116.
[3]	LUO S T , XU H L , LU C Z ,et al. Characterizing microservice dependency and performance:Alibaba trace analysis[C]// Proceedings of the ACM Symposium on Cloud Computing. New York:ACM Press, 2021: 412-426.
[4]	NIFE F N , KOTULSKI Z . Application-aware firewall mechanism for software defined networks[J]. Journal of Network and Systems Management, 2020,28(3): 605-626.
[5]	BáNáTI A , KAIL E , KARóCZKAI K ,et al. Authentication and authorization orchestrator for microservice-based software architectures[C]// Proceedings of 2018 41st International Convention on Information and Communication Technology,Electronics and Microelectronics. Piscataway:IEEE Press, 2018: 1180-1184.
[6]	BARDAS A G , SUNDARAMURTHY S C , OU X ,et al. MTD CBITS:moving target defense for cloud-based IT systems[C]// Proceedings of 22nd European Symposium on Research in Computer Security. Berlin:Springer, 2017: 167-186.
[7]	TORKURA K A , SUKMANA M I H , KAYEM A V D M ,et al. A cyber risk based moving target defense mechanism for microservice architectures[C]// Proceedings of 2018 IEEE International Conference on Parallel ＆ Distributed Processing with Applications,Ubiquitous Computing ＆ Communications,Big Data ＆ Cloud Computing,Social Computing ＆ Networking,Sustainable Computing ＆ Communications. Piscataway:IEEE Press, 2018: 932-939.
[8]	JIN H , LI Z , ZOU D Q ,et al. DSEOM:a framework for dynamic security evaluation and optimization of MTD in container-based cloud[J]. IEEE Transactions on Dependable and Secure Computing, 2019,18(3): 1125-1136.
[9]	张帅, 郭云飞, 孙鹏浩 ,等. 云原生下基于深度强化学习的移动目标防御策略优化方案[J]. 电子与信息学报, 2023,45(2): 608-616.
	ZHANG S , GUO Y F , SUN P H ,et al. Optimization scheme of moving target defense strategy based on deep reinforcement learning in cloud native environment[J]. Journal of Electronics ＆ Information Technology, 2023,45(2): 608-616.
[10]	DUC T L , LEIVA R G , CASARI P ,et al. Machine learning methods for reliable resource provisioning in edge-cloud computing:a survey[J]. ACM Computing Surveys, 2020,52(5): 1-39.
[11]	ZHOU D C , CHEN H C , SHANG K ,et al. Cushion:a proactive resource provisioning method to mitigate SLO violations for containerized microservices[J]. IET Communications, 2022,16: 2105-2122.
[12]	YADAV T , RAO A M . Technical aspects of cyber kill chain[C]// Proceedings of International Symposium on Security in Computing and Communication. Berlin:Springer, 2015: 438-452.
[13]	NOUREDDINE M A , FAWAZ A , SANDERS W H ,et al. A game-theoretic approach to respond to attacker lateral movement[C]// Proceedings of 7th International Conference on Decision and Game Theory for Security. New York:ACM Press, 2016: 294-313.
[14]	ALMOHRI H M J , WATSON L T , EVANS D . Misery digraphs:delaying intrusion attacks in obscure clouds[J]. IEEE Transactions on Information Forensics and Security, 2018,13(6): 1361-1375.
[15]	张福, 程度, 胡俊 . ATT＆CK框架实践指南[M]. 北京: 电子工业出版社, 2022.
	ZHANG F , CHENG D , HU J . ATT＆CK framework practice guide[M]. Beijing: Publish House of Electronics Industry, 2022.
[16]	GAN Y , ZHANG Y Q , CHENG D L ,et al. An open-source benchmark suite for microservices and their hardware-software implications for cloud ＆ edge systems[C]// Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems. New York:ACM Press, 2019: 3-18.
[17]	CHO J H , SHARMA D P , ALAVIZADEH H ,et al. Toward proactive,adaptive defense:a survey on moving target defense[J]. IEEE Communications Surveys ＆ Tutorials, 2020,22(1): 709-745.
[18]	CAI G L , WANG B S , HU W ,et al. Moving target defense:state of the art and characteristics[J]. Frontiers of Information Technology ＆ Electronic Engineering, 2016,17(11): 1122-1153.
[19]	ZHANG X , SEN S , KURNIAWAN D ,et al. E2E:embracing user heterogeneity to improve quality of experience on the web[C]// Proceedings of the ACM Special Interest Group on Data Communication. New York:ACM Press, 2019: 289-302.
[20]	GIAS A U , CASALE G , WOODSIDE M . ATOM:model-driven autoscaling for microservices[C]// Proceedings of 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). Piscataway:IEEE Press, 2019: 1994-2004.
[21]	LI Z , JIN H , ZOU D Q ,et al. Exploring new opportunities to defeat low-rate DDoS attack in container-based cloud environment[J]. IEEE Transactions on Parallel and Distributed Systems, 2019,31(3): 695-706.
[22]	BHASI V M , GUNASEKARAN J R , THINAKARAN P ,et al. Kraken:adaptive container provisioning for deploying dynamic DAGs in serverless platforms[C]// Proceedings of the ACM Symposium on Cloud Computing. New York:ACM Press, 2021: 153-167.
[23]	MNIH V , KAVUKCUOGLU K , SILVER D ,et al. Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540): 529-533.
[24]	ABELS A , ROIJERS D , LENAERTS T ,et al. Dynamic weights in multi-objective deep reinforcement learning[C]// Proceedings of International Conference on Machine Learning. Saarland:DBLP, 2019: 11-20.

Metrics

Recommended 0

No Suggested Reading articles found!

动作	含义
i⁺	选定微服务的索引增加1
i^-	选定微服务的索引减少1
δ N⁺	选定微服务的副本增加一个
δ N^-	选定微服务的副本减少一个
δ T⁺	选定微服务的副本清洗周期增加1 s
δ T^-	选定微服务的副本清洗周期减少1 s
0	不采取任何动作

Research on multidimensional dynamic defense strategy for microservice based on deep reinforcement learning

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 24

Related Articles 15

Metrics

Recommended 0

[1]	Ling MA, Qiliang FAN, Ting XU, Guanchen GUO, Shenglin ZHANG, Yongqian SUN, Yuzhi ZHANG. Scheduling framework based on reinforcement learning in online-offline colocated cloud environment [J]. Journal on Communications, 2023, 44(6): 90-102.
[2]	Biao JIN, Yikang LI, Zhiqiang YAO, Yulin CHEN, Jinbo XIONG. GenFedRL: a general federated reinforcement learning framework for deep reinforcement learning agents [J]. Journal on Communications, 2023, 44(6): 183-197.
[3]	Yuancheng LI, Yongtai QIN. Deep reinforcement learning based algorithm for real-time QoS optimization of software-defined security middle platform [J]. Journal on Communications, 2023, 44(5): 181-192.
[4]	Guoliang XU, Feng TAN, Yongyi RAN, Feng CHEN. Joint beam hopping and coverage control optimization algorithm for multibeam satellite system [J]. Journal on Communications, 2023, 44(4): 78-86.
[5]	Wenjun XU, Silei WU, Fengyu WANG, Lan LIN, Guojun LI, Zhi ZHANG. Large-scale post-disaster user distributed coverage optimization based on multi-agent reinforcement learning [J]. Journal on Communications, 2022, 43(8): 1-16.
[6]	Zongxuan SHA, Ru HUO, Chuang SUN, Shuo WANG, Tao HUANG. Forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning [J]. Journal on Communications, 2022, 43(8): 30-40.
[7]	Shuai MA, Bing LI, Haihong SHENG, Rongyan GU, Hui ZHOU, Hongmei WANG, Yue WANG, Shiyin LI. Research on power allocation of integrated VLPC based on deep reinforcement learning [J]. Journal on Communications, 2022, 43(8): 121-130.
[8]	Yu ZHANG, Min CHENG. Joint optimization of edge computing and caching in NDN [J]. Journal on Communications, 2022, 43(8): 164-175.
[9]	Peiliang ZUO, Shaolong HOU, Chao GUO, Hua JIANG, Wenbo WANG. Security decision method for the edge of multi-layer satellite network based on reinforcement learning [J]. Journal on Communications, 2022, 43(6): 189-199.
[10]	Xianchao ZHANG, Yao ZHAO, Haijun YE, Rui FAN. Intelligent transmit power control algorithm for the multi-user interference of wireless network [J]. Journal on Communications, 2022, 43(2): 15-21.
[11]	Chuanhuang LI, Yangting CHEN, Jingjing TANG, Jiali LOU, Renhua XIE, Chuntao FANG, Weiming WANG, Chao CHEN. QL-STCT: an intelligent routing convergence method for SDN link failure [J]. Journal on Communications, 2022, 43(2): 131-142.
[12]	Jinyin CHEN, Shulong HU, Changyou XING, Guomin ZHANG. Deception defense method against intelligent penetration attack [J]. Journal on Communications, 2022, 43(10): 106-120.
[13]	Xin SU, Leilei MENG, Yiqing ZHOU, Wu CELIMUGE. Maritime mobile edge computing offloading method based on deep reinforcement learning [J]. Journal on Communications, 2022, 43(10): 133-145.
[14]	Li’na DU, Li ZHUO, Shuo YANG, Jiafeng LI, Jing ZHANG. Survey on reinforcement learning based adaptive bit rate algorithm for mobile video streaming services [J]. Journal on Communications, 2021, 42(9): 205-217.
[15]	Jiujiu CHEN, Chunyan FENG, Caili GUO, Yang YANG, Qizheng SUN, Meiyi ZHU. Video semantics-driven resource allocation algorithm in Internet of vehicles [J]. Journal on Communications, 2021, 42(7): 1-11.