基于深度强化学习的软件定义安全中台QoS实时优化算法

doi:10.11959/j.issn.1000-436x.2023090

摘要/Abstract

摘要：

针对软件定义安全场景中的服务质量（QoS）实时优化方案因安全防护手段与业务场景不匹配而导致的适用困难和性能下降的问题，提出了基于深度强化学习的软件定义安全中台QoS实时优化算法。首先，将碎片化的安全需求与安全基础设施统一到软件定义安全中台云模型中；然后，通过深度强化学习结合云计算技术提高安全中台的实时匹配和动态适应能力；最后，生成满足QoS目标的安全中台资源实时调度策略。实验结果表明，与现有实时算法相比，所提算法不但保证负载均衡，还提高了 18.7%的作业调度成功率以提高服务质量，降低了34.2%的平均响应时间，具有很好的稳健性，更适用于实时环境。

关键词: 软件定义安全, 深度强化学习, 安全中台, 服务质量

Abstract:

To overcome the problem that the real-time optimization of the quality of service (QoS) in software-defined security scenarios was hindered by the mismatch between security protection measures and business scenarios, which led to difficulties in application and performance degradation., a novel algorithm based on deep reinforcement learning for optimizing QoS in software defined security middle platforms (SDSmp) in real-time was proposed.Firstly, the fragmented security requirements and infrastructure were integrated into the SDSmp cloud model.Then by leveraging the power of deep reinforcement learning and cloud computing technology, the real-time matching and dynamic adaptation capabilities of the security middle platform were enhanced.Finally, a real-time scheduling strategy for security middle platform resources that meet QoS goals was generated.Experimental results demonstrate that compared to existing real-time methods, the proposed algorithm not only ensures load balancing but also improves job success rate by 18.7% for high QoS and reduces the average response time by 34.2%, and it is highly robust and better suited for real-time environments than existing methods.

Key words: software defined security, deep reinforcement learning, security middle platform, quality of service

中图分类号:

TP393

李元诚, 秦永泰. 基于深度强化学习的软件定义安全中台QoS实时优化算法[J]. 通信学报, 2023, 44(5): 181-192.

Yuancheng LI, Yongtai QIN. Deep reinforcement learning based algorithm for real-time QoS optimization of software-defined security middle platform[J]. Journal on Communications, 2023, 44(5): 181-192.

图/表 16

图1

图2

表1

图3

表2

表3

表4

图4

图5

图6

图7

图8

图9

图10

图11

图12

参考文献 25

[1]	LIU Y B , LU X Y , JIAN Y ,et al. SDSA:a framework of a software-defined security architecture[J]. China Communications, 2016,13(2): 178-188.
[2]	ALHAJ A N , DUTTA N . Analysis of security attacks in SDN network:a comprehensive survey[C]// Proceedings of Contemporary Issues in Communication,Cloud and Big Data Analytics. Berlin:Springer, 2022: 27-37.
[3]	QIU R X , QIN Y T , LI Y C ,et al. A software-defined security middle platform architecture[C]// Proceedings of the 5th International Conference on Computer Science and Software Engineering. New York:ACM Press, 2022: 647-651.
[4]	WEI Y , PAN L , LIU S J ,et al. DRL-scheduling:an intelligent QoS-aware job scheduling framework for applications in clouds[J]. IEEE Access, 2018,6: 55112-55125.
[5]	ZHANG G , QIU X F , CHANG W . Scheduling of security resources in software defined security architecture[C]// Proceedings of 2017 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC). Piscataway:IEEE Press, 2018: 494-503.
[6]	MOUSSAID N E , TOUMANARI A , AZHARI M E . Security analysis as software-defined security for SDN environment[C]// Proceedings of the Fourth International Conference on Software Defined Systems. Piscataway:IEEE Press, 2017: 87-92.
[7]	MOHAMED A , HAMDAN M , KHAN S ,et al. Software-defined networks for resource allocation in cloud computing:a survey[J]. Computer Networks, 2021,195:108151.
[8]	QI C , WU J X , HU H C ,et al. Dynamic-scheduling mechanism of controllers based on security policy in software-defined network[J]. Electronics Letters, 2016,52(23): 1918-1920.
[9]	LUO S , BEN S M . Orchestration of software-defined security services[C]// Proceedings of 2016 IEEE International Conference on Communications Workshops. Piscataway:IEEE Press, 2016: 436-441.
[10]	兰巨龙, 张学帅, 胡宇翔 ,等. 基于深度强化学习的软件定义网络QoS优化[J]. 通信学报, 2019,40(12): 60-67.
	LAN J L , ZHANG X S , HU Y X ,et al. Software-defined networking QoS optimization based on deep reinforcement learning[J]. Journal on Communications, 2019,40(12): 60-67.
[11]	吴平, 常朝稳, 左志斌 ,等. 基于地址重载的SDN分组转发验证[J]. 通信学报, 2022,43(3): 88-100.
	WU P , CHANG C W , ZUO Z B ,et al. Address overloading-based packet forwarding verification in SDN[J]. Journal on Communications, 2022,43(3): 88-100.
[12]	MORALES E F , ZARAGOZA J H . An introduction to reinforcement learning[J]. Machine Learning, 2011:doi.10.4018/978-1-60960-165-2.Ch004.
[13]	MNIH V , KAVUKCUOGLU K , SILVER D ,et al. Playing atari with deep reinforcement learning[J]. arXiv Preprint,arXiv:1312.5602, 2013.
[14]	MNIH V , KAVUKCUOGLU K , SILVER D ,et al. Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540): 529-533.
[15]	CHENG L , KALAPGAR A , JAIN A ,et al. Cost-aware real-time job scheduling for hybrid cloud using deep reinforcement learning[J]. Neural Computing and Applications, 2022,34(21): 18579-18593.
[16]	ABUNDO M , DI-VALERIO V , CARDELLINI V , et al . QoS-aware bidding strategies for VM spot instances:a reinforcement learning approach applied to periodic long running jobs[C]// Proceedings of 2015 IFIP/IEEE International Symposium on Integrated Network Management. Piscataway:IEEE Press, 2015: 53-61.
[17]	ARULKUMARAN K , DEISENROTH M P , BRUNDAGE M ,et al. Deep reinforcement learning:a brief survey[J]. IEEE Signal Processing Magazine, 2017,34(6): 26-38.
[18]	盛妍, 朱青, 张明杰 ,等. 基于数据中台的智能标签关键技术研究与应用[J]. 电子技术应用, 2022,48(3): 73-77.
	SHENG Y , ZHU Q , ZHANG M J ,et al. Research and application on key technology of intelligent tag based on data middle platform[J]. Application of Electronic Technique, 2022,48(3): 73-77.
[19]	CHENG F , HUANG Y F , TANPURE B ,et al. Cost-aware job scheduling for cloud instances using deep reinforcement learning[J]. Cluster Computing, 2022,25(1): 619-631.
[20]	HUANG Y F , CHENG L , XUE L T ,et al. Deep adversarial imitation reinforcement learning for QoS-aware cloud job scheduling[J]. IEEE Systems Journal, 2022,16(3): 4232-4242.
[21]	JAIN V , KUMAR B . QoS-aware task offloading in fog environment using multi-agent deep reinforcement learning[J]. Journal of Network and Systems Management, 2022,31(1): 1-32.
[22]	CHO C , SHIN S , JEON H ,et al. QoS-aware workload distribution in hierarchical edge clouds:a reinforcement learning approach[J]. IEEE Access, 2020,8: 193297-193313.
[23]	LILLICRAP T P , HUNT J J , PRITZEL A ,et al. Continuous control with deep reinforcement learning[J]. arXiv Preprint,arXiv:1509.02971, 2015.
[24]	NEJAD M M , MASHAYEKHY L , GROSU D . Truthful greedy mechanisms for dynamic virtual machine provisioning and allocation in clouds[J]. IEEE Transactions on Parallel and Distributed Systems, 2015,26(2): 594-603.
[25]	WANG L , GELENBE E . Adaptive dispatching of tasks in the cloud[J]. IEEE Transactions on Cloud Computing, 2015,6(1): 33-45.

参数	含义
J^id	安全业务前台作业的ID
J ^at	安全业务前台作业到达时间
J^t	安全业务前台作业类型（计算或I/O密集型）
J^l	安全业务前台作业长度（所需的指令、服务）
J^q	安全业务前台作业QoS要求
J ^rt	安全业务前台作业响应时间
J ^et	安全业务前台作业执行时间
J ^wt	安全业务前台作业等待时间
V ^id	安全中台资源（VM）的ID
V^t	安全中台资源（VM）类型（计算或I/O密集型）
V^p	处理速度（每秒处理的指令、服务）
	安全中台资源（VM）的计算处理速度
	安全中台资源（VM）的读写处理速度
V ^it	安全中台资源（VM）的空闲时间
R	奖励（体现QoS、作业调度成功率、响应时间等）
Suc	作业调度成功率（作业是否调度成功满足QoS）

安全中台资源	计算型作业		I/O型作业
安全中台资源	平均值/MIPS	标准差/MIPS	平均值/MIPS	标准差/MIPS
计算密集型资源	1 000	100	500	50
I/O密集型资源	500	50	1 000	100

负载模式	到达率	平均值	标准差	对应现实场景
随机	[0,100%]	53.53%	29.51%	随机使用
低频	[20%,40%]	30.07%	6.36%	低频使用
高频	[60%,80%]	70.32%	5.57%	高频使用

负载模式	算法	平均响应时间/s	作业调度成功率	负载均衡率
随机	random	0.807	51.3%	75.4%
	round-robin	0.426	72.1%	76.1%
	earliest	0.412	74.4%	76.7%
	DQN	0.203	95.7%	65.5%
	suitable	0.255	82.7%	64.1%
	sensibleR	1.108	43.8%	75.7%
低频	random	0.237	99.5%	29.8%
	round-robin	0.163	99.9%	29.7%
	earliest	0.158	99.9%	27.4%
	DQN	0.115	99.9%	26.8%
	suitable	0.057	99.9%	23.8%
	sensibleR	0.254	98.4%	33.7%
高频	random	11.637	11.4%	98.4%
	round-robin	10.362	12.6%	97.7%
	earliest	3.527	13.8%	91.4%
	DQN	0.357	93.7%	76.2%
	suitable	0.658	70.3%	73.8%
	sensibleR	11.246	12.2%	98.1%