基于深度强化学习的软件定义安全中台QoS实时优化算法

doi:10.11959/j.issn.1000-436x.2023090

Abstract

Abstract:

To overcome the problem that the real-time optimization of the quality of service (QoS) in software-defined security scenarios was hindered by the mismatch between security protection measures and business scenarios, which led to difficulties in application and performance degradation., a novel algorithm based on deep reinforcement learning for optimizing QoS in software defined security middle platforms (SDSmp) in real-time was proposed.Firstly, the fragmented security requirements and infrastructure were integrated into the SDSmp cloud model.Then by leveraging the power of deep reinforcement learning and cloud computing technology, the real-time matching and dynamic adaptation capabilities of the security middle platform were enhanced.Finally, a real-time scheduling strategy for security middle platform resources that meet QoS goals was generated.Experimental results demonstrate that compared to existing real-time methods, the proposed algorithm not only ensures load balancing but also improves job success rate by 18.7% for high QoS and reduces the average response time by 34.2%, and it is highly robust and better suited for real-time environments than existing methods.

Key words: software defined security, deep reinforcement learning, security middle platform, quality of service

CLC Number:

TP393

Yuancheng LI, Yongtai QIN. Deep reinforcement learning based algorithm for real-time QoS optimization of software-defined security middle platform[J]. Journal on Communications, 2023, 44(5): 181-192.

Figures/Tables 16

References 25

[1]	LIU Y B , LU X Y , JIAN Y ,et al. SDSA:a framework of a software-defined security architecture[J]. China Communications, 2016,13(2): 178-188.
[2]	ALHAJ A N , DUTTA N . Analysis of security attacks in SDN network:a comprehensive survey[C]// Proceedings of Contemporary Issues in Communication,Cloud and Big Data Analytics. Berlin:Springer, 2022: 27-37.
[3]	QIU R X , QIN Y T , LI Y C ,et al. A software-defined security middle platform architecture[C]// Proceedings of the 5th International Conference on Computer Science and Software Engineering. New York:ACM Press, 2022: 647-651.
[4]	WEI Y , PAN L , LIU S J ,et al. DRL-scheduling:an intelligent QoS-aware job scheduling framework for applications in clouds[J]. IEEE Access, 2018,6: 55112-55125.
[5]	ZHANG G , QIU X F , CHANG W . Scheduling of security resources in software defined security architecture[C]// Proceedings of 2017 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC). Piscataway:IEEE Press, 2018: 494-503.
[6]	MOUSSAID N E , TOUMANARI A , AZHARI M E . Security analysis as software-defined security for SDN environment[C]// Proceedings of the Fourth International Conference on Software Defined Systems. Piscataway:IEEE Press, 2017: 87-92.
[7]	MOHAMED A , HAMDAN M , KHAN S ,et al. Software-defined networks for resource allocation in cloud computing:a survey[J]. Computer Networks, 2021,195:108151.
[8]	QI C , WU J X , HU H C ,et al. Dynamic-scheduling mechanism of controllers based on security policy in software-defined network[J]. Electronics Letters, 2016,52(23): 1918-1920.
[9]	LUO S , BEN S M . Orchestration of software-defined security services[C]// Proceedings of 2016 IEEE International Conference on Communications Workshops. Piscataway:IEEE Press, 2016: 436-441.
[10]	兰巨龙, 张学帅, 胡宇翔 ,等. 基于深度强化学习的软件定义网络QoS优化[J]. 通信学报, 2019,40(12): 60-67.
	LAN J L , ZHANG X S , HU Y X ,et al. Software-defined networking QoS optimization based on deep reinforcement learning[J]. Journal on Communications, 2019,40(12): 60-67.
[11]	吴平, 常朝稳, 左志斌 ,等. 基于地址重载的SDN分组转发验证[J]. 通信学报, 2022,43(3): 88-100.
	WU P , CHANG C W , ZUO Z B ,et al. Address overloading-based packet forwarding verification in SDN[J]. Journal on Communications, 2022,43(3): 88-100.
[12]	MORALES E F , ZARAGOZA J H . An introduction to reinforcement learning[J]. Machine Learning, 2011:doi.10.4018/978-1-60960-165-2.Ch004.
[13]	MNIH V , KAVUKCUOGLU K , SILVER D ,et al. Playing atari with deep reinforcement learning[J]. arXiv Preprint,arXiv:1312.5602, 2013.
[14]	MNIH V , KAVUKCUOGLU K , SILVER D ,et al. Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540): 529-533.
[15]	CHENG L , KALAPGAR A , JAIN A ,et al. Cost-aware real-time job scheduling for hybrid cloud using deep reinforcement learning[J]. Neural Computing and Applications, 2022,34(21): 18579-18593.
[16]	ABUNDO M , DI-VALERIO V , CARDELLINI V , et al . QoS-aware bidding strategies for VM spot instances:a reinforcement learning approach applied to periodic long running jobs[C]// Proceedings of 2015 IFIP/IEEE International Symposium on Integrated Network Management. Piscataway:IEEE Press, 2015: 53-61.
[17]	ARULKUMARAN K , DEISENROTH M P , BRUNDAGE M ,et al. Deep reinforcement learning:a brief survey[J]. IEEE Signal Processing Magazine, 2017,34(6): 26-38.
[18]	盛妍, 朱青, 张明杰 ,等. 基于数据中台的智能标签关键技术研究与应用[J]. 电子技术应用, 2022,48(3): 73-77.
	SHENG Y , ZHU Q , ZHANG M J ,et al. Research and application on key technology of intelligent tag based on data middle platform[J]. Application of Electronic Technique, 2022,48(3): 73-77.
[19]	CHENG F , HUANG Y F , TANPURE B ,et al. Cost-aware job scheduling for cloud instances using deep reinforcement learning[J]. Cluster Computing, 2022,25(1): 619-631.
[20]	HUANG Y F , CHENG L , XUE L T ,et al. Deep adversarial imitation reinforcement learning for QoS-aware cloud job scheduling[J]. IEEE Systems Journal, 2022,16(3): 4232-4242.
[21]	JAIN V , KUMAR B . QoS-aware task offloading in fog environment using multi-agent deep reinforcement learning[J]. Journal of Network and Systems Management, 2022,31(1): 1-32.
[22]	CHO C , SHIN S , JEON H ,et al. QoS-aware workload distribution in hierarchical edge clouds:a reinforcement learning approach[J]. IEEE Access, 2020,8: 193297-193313.
[23]	LILLICRAP T P , HUNT J J , PRITZEL A ,et al. Continuous control with deep reinforcement learning[J]. arXiv Preprint,arXiv:1509.02971, 2015.
[24]	NEJAD M M , MASHAYEKHY L , GROSU D . Truthful greedy mechanisms for dynamic virtual machine provisioning and allocation in clouds[J]. IEEE Transactions on Parallel and Distributed Systems, 2015,26(2): 594-603.
[25]	WANG L , GELENBE E . Adaptive dispatching of tasks in the cloud[J]. IEEE Transactions on Cloud Computing, 2015,6(1): 33-45.

Metrics

Recommended 0

No Suggested Reading articles found!

参数	含义
J^id	安全业务前台作业的ID
J ^at	安全业务前台作业到达时间
J^t	安全业务前台作业类型（计算或I/O密集型）
J^l	安全业务前台作业长度（所需的指令、服务）
J^q	安全业务前台作业QoS要求
J ^rt	安全业务前台作业响应时间
J ^et	安全业务前台作业执行时间
J ^wt	安全业务前台作业等待时间
V ^id	安全中台资源（VM）的ID
V^t	安全中台资源（VM）类型（计算或I/O密集型）
V^p	处理速度（每秒处理的指令、服务）
	安全中台资源（VM）的计算处理速度
	安全中台资源（VM）的读写处理速度
V ^it	安全中台资源（VM）的空闲时间
R	奖励（体现QoS、作业调度成功率、响应时间等）
Suc	作业调度成功率（作业是否调度成功满足QoS）

安全中台资源	计算型作业		I/O型作业
安全中台资源	平均值/MIPS	标准差/MIPS	平均值/MIPS	标准差/MIPS
计算密集型资源	1 000	100	500	50
I/O密集型资源	500	50	1 000	100

负载模式	到达率	平均值	标准差	对应现实场景
随机	[0,100%]	53.53%	29.51%	随机使用
低频	[20%,40%]	30.07%	6.36%	低频使用
高频	[60%,80%]	70.32%	5.57%	高频使用

负载模式	算法	平均响应时间/s	作业调度成功率	负载均衡率
随机	random	0.807	51.3%	75.4%
	round-robin	0.426	72.1%	76.1%
	earliest	0.412	74.4%	76.7%
	DQN	0.203	95.7%	65.5%
	suitable	0.255	82.7%	64.1%
	sensibleR	1.108	43.8%	75.7%
低频	random	0.237	99.5%	29.8%
	round-robin	0.163	99.9%	29.7%
	earliest	0.158	99.9%	27.4%
	DQN	0.115	99.9%	26.8%
	suitable	0.057	99.9%	23.8%
	sensibleR	0.254	98.4%	33.7%
高频	random	11.637	11.4%	98.4%
	round-robin	10.362	12.6%	97.7%
	earliest	3.527	13.8%	91.4%
	DQN	0.357	93.7%	76.2%
	suitable	0.658	70.3%	73.8%
	sensibleR	11.246	12.2%	98.1%

Deep reinforcement learning based algorithm for real-time QoS optimization of software-defined security middle platform

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 25

Related Articles 15

Metrics

Recommended 0

[1]	Biao JIN, Yikang LI, Zhiqiang YAO, Yulin CHEN, Jinbo XIONG. GenFedRL: a general federated reinforcement learning framework for deep reinforcement learning agents [J]. Journal on Communications, 2023, 44(6): 183-197.
[2]	Guoliang XU, Feng TAN, Yongyi RAN, Feng CHEN. Joint beam hopping and coverage control optimization algorithm for multibeam satellite system [J]. Journal on Communications, 2023, 44(4): 78-86.
[3]	Zongxuan SHA, Ru HUO, Chuang SUN, Shuo WANG, Tao HUANG. Forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning [J]. Journal on Communications, 2022, 43(8): 30-40.
[4]	Yu ZHANG, Min CHENG. Joint optimization of edge computing and caching in NDN [J]. Journal on Communications, 2022, 43(8): 164-175.
[5]	Jianxun LIU, Linghang DING, Guosheng KANG, Buqing CAO, Yong XIAO. Joint QoS prediction for Web services based on deep fusion of features [J]. Journal on Communications, 2022, 43(7): 215-226.
[6]	Damin ZHANG, Yi WANG, Chengcheng ZOU, Peiwen ZHAO, Linna ZHANG. Resource allocation strategies for improved mayfly algorithm in cognitive heterogeneous cellular network [J]. Journal on Communications, 2022, 43(6): 156-167.
[7]	Xianchao ZHANG, Yao ZHAO, Haijun YE, Rui FAN. Intelligent transmit power control algorithm for the multi-user interference of wireless network [J]. Journal on Communications, 2022, 43(2): 15-21.
[8]	Xin SU, Leilei MENG, Yiqing ZHOU, Wu CELIMUGE. Maritime mobile edge computing offloading method based on deep reinforcement learning [J]. Journal on Communications, 2022, 43(10): 133-145.
[9]	Li’na DU, Li ZHUO, Shuo YANG, Jiafeng LI, Jing ZHANG. Survey on reinforcement learning based adaptive bit rate algorithm for mobile video streaming services [J]. Journal on Communications, 2021, 42(9): 205-217.
[10]	Wenchen HE, Shaoyong GUO, Xuesong QIU, Liandong CHEN, Suxiang ZHANG. Node selection method in federated learning based on deep reinforcement learning [J]. Journal on Communications, 2021, 42(6): 62-71.
[11]	Guolin SUN,Ruijie OU,Guisong LIU. Deep reinforcement learning-based resource reservation algorithm for emergency Internet-of-things slice [J]. Journal on Communications, 2020, 41(9): 8-20.
[12]	Yingchang LIANG,Junjie TAN,Dusit Niyato. Overview on intelligent wireless communication technology [J]. Journal on Communications, 2020, 41(7): 1-17.
[13]	Tingting LIU,Yi’nan LUO,Chenyang YANG. Distributed interference coordination based on multi-agent deep reinforcement learning [J]. Journal on Communications, 2020, 41(7): 38-48.
[14]	Pei ZHANG,Shuaijun LIU,Zhiguo MA,Xiaohui WANG,Junde SONG. Improved satellite resource allocation algorithm based on DRL and MOP [J]. Journal on Communications, 2020, 41(6): 51-60.
[15]	Gang LI,Zhijun WU. Task scheduling algorithm for system-wide information management based on multiple QoS constraints [J]. Journal on Communications, 2019, 40(7): 27-37.