基于深度强化学习的SDN服务质量智能优化算法

doi:10.11959/j.issn.2096-3750.2023.00316

Abstract

Abstract:

Deep reinforcement learning has strong abilities of decision-making and generalization and often applies to the quality of service (QoS) optimization in software defined network (SDN).However, traditional deep reinforcement learning algorithms have problems such as slow convergence and instability.An algorithm of quality of service optimization algorithm of based on deep reinforcement learning (AQSDRL) was proposed to solve the QoS problem of SDN in the data center network (DCN) applications.AQSDRL introduces the softmax deep double deterministic policy gradient (SD3) algorithm for model training, and a SumTree-based prioritized empirical replay mechanism was used to optimize the SD3 algorithm.The samples with more significant temporal-difference error (TD-error) were extracted with higher probability to train the neural network, effectively improving the convergence speed and stability of the algorithm.The experimental results show that the proposed AQSDRL effectively reduces the network transmission delay and improves the load balancing performance of the network than the existing deep reinforcement learning algorithms.

Key words: deep reinforcement learning, SDN, QoS, DCN, SumTree

CLC Number:

TP393

Cenhuishan LIAO, Junyan CHEN, Guanping LIANG, Xiaolan XIE, Xiaoye LU. Quality of service optimization algorithm based on deep reinforcement learning in software defined network[J]. Chinese Journal on Internet of Things, 2023, 7(1): 73-82.

Figures/Tables 8

References 30

[1]	TANHA M , SAJJADI D , RUBY R ,et al. Traffic engineering enhancement by progressive migration to SDN[J]. IEEE Communications Letters, 2018,22(3): 438-441.
[2]	ONGARO F , CERQUEIRA E , FOSCHINI L ,et al. Enhancing the quality level support for real-time multimedia applications in software-defined networks[C]// Proceedings of 2015 International Conference on Computing,Networking and Communications (ICNC). Piscataway:IEEE Press, 2015: 505-509.
[3]	ALIZADEH M , EDSALL T , DHARMAPURIKAR S ,et al. CONGA:distributed congestion-aware load balancing for datacenters[C]// Proceedings of the 2014 ACM conference on SIGCOMM. New York:ACM Press, 2014: 503-514.
[4]	ZOU G B , LI T F , JIANG M ,et al. Deep TSQP:temporal-aware service QoS prediction via deep neural network and feature integration[J]. Knowledge-Based Systems, 2022,241:108062.
[5]	CHEN J Y , WANG Y , HUANG X F ,et al. ALBLP:adaptive load-balancing architecture based on link-state prediction in software-defined networking[J]. Wireless Communications and Mobile Computing,2022, 2022:8354150.
[6]	NUGRAHAB , MURTHYRN . Deep learning-based slow DDoS attack detection in SDN-based networks[C]// Proceedings of 2020 IEEE Conference on Network Function Virtualization and Software Defined Networks. Piscataway:IEEE Press, 2020: 51-56.
[7]	NOVAES M P , CARVALHO L F , LLORET J ,et al. Long short-term memory and fuzzy logic for anomaly detection and mitigation in software-defined network environment[J]. IEEE Access, 2020(8): 83765-83781.
[8]	BHATIA J , DAVE R , BHAYANI H ,et al. SDN-based real-time urban traffic analysis in VANET environment[J]. Computer Communications, 2020,149: 162-175.
[9]	TROIA S , ALVIZU R , MAIER G . Reinforcement learning for service function chain reconfiguration in NFV-SDN metro-core optical networks[J]. IEEE Access, 2019(7): 167944-167957.
[10]	LIN S C , AKYILDIZ I F , WANG P ,et al. QoS-aware adaptive routing in multi-layer hierarchical software defined networks:a reinforcement learning approach[C]// Proceedings of 2016 IEEE International Conference on Services Computing. Piscataway:IEEE Press, 2016: 25-33.
[11]	YOUNUS M U , KHAN M K , ANJUM M R ,et al. Optimizing the lifetime of software defined wireless sensor network via reinforcement learning[J]. IEEE Access, 2020(9): 259-272.
[12]	CASAS-VELASCO D M , RENDON O M C , DA FONSECA N L S . Intelligent routing based on reinforcement learning for software-defined networking[J]. IEEE Transactions on Network and Service Management, 2021,18(1): 870-881.
[13]	AL-JAWAD A , COM?A I S , SHAH P ,et al. An innovative reinforcement learning-based framework for quality of service provisioning over multimedia-based SDN environments[J]. IEEE Transactions on Broadcasting, 2021,67(4): 851-867.
[14]	XU Z Y , WU K , ZHANG W Y ,et al. PnP-DRL:a plug-and-play deep reinforcement learning approach for experience-driven networking[J]. IEEE Journal on Selected Areas in Communications, 2021,39(8): 2476-2486.
[15]	LIU W X , CAI J , CHEN Q C ,et al. DRL-R:deep reinforcement learning approach for intelligent routing in software-defined data-center networks[J]. Journal of Network and Computer Applications, 2021,177:102865.
[16]	HU Y X , LI Z Y , LAN J L ,et al. EARS:intelligence-driven experiential network architecture for automatic routing in software-defined networking[J]. China Communications, 2020,17(2): 149-162.
[17]	BOUZIDI E H , OUTTAGARTS A , LANGAR R ,et al. Deep Q-network and traffic prediction based routing optimization in software defined networks[J]. Journal of Network and Computer Applications, 2021(192):103181.
[18]	兰巨龙, 于倡和, 胡宇翔 ,等. 基于深度增强学习的软件定义网络路由优化机制[J]. 电子与信息学报, 2019,41(11): 2669-2674.
	LAN J L , YU C H , HU Y X ,et al. A SDN routing optimization mechanism based on deep reinforcement learning[J]. Journal of Electronics ＆ Information Technology, 2019,41(11): 2669-2674.
[19]	兰巨龙, 张学帅, 胡宇翔 ,等. 基于深度强化学习的软件定义网络QoS优化[J]. 通信学报, 2019,40(12): 60-67.
	LAN J L , ZHANG X S , HU Y X ,et al. Software-defined networking QoS optimization based on deep reinforcement learning[J]. Journal on Communications, 2019,40(12): 60-67.
[20]	MAI T L , YAO H P , ZHANG N ,et al. Transfer reinforcement learning aided distributed network slicing optimization in industrial IoT[J]. IEEE Transactions on Industrial Informatics, 2022,18(6): 4308-4316.
[21]	CHEN J Y , WANG Y , OU J T ,et al. ALBRL:automatic load-balancing architecture based on reinforcement learning in software-defined networking[J]. Wireless Communications and Mobile Computing,2022, 2022:3866143.
[22]	FUJIMOTO S , VAN HOOF H , MEGER D ,et al. Addressing function approximation error in actor-critic methods[EB]. 2018.
[23]	孙鹏浩, 兰巨龙, 申涓 ,等. 一种基于深度增强学习的智能路由技术[J]. 电子学报, 2020,48(11): 2170-2177.
	SUN P H , LAN J L , SHEN J ,et al. An intelligent routing technology based on deep reinforcement learning[J]. Acta Electronica Sinica, 2020,48(11): 2170-2177.
[24]	孙鹏浩, 兰巨龙, 申涓 ,等. 基于牵引控制的深度强化学习路由策略生成[J]. 计算机研究与发展, 2021,58(7): 1563-1572.
	SUN P H , LAN J L , SHEN J ,et al. Pinning control-based routing policy generation using deep reinforcement learning[J]. Journal of Computer Research and Development, 2021,58(7): 1563-1572.
[25]	PAN L , CAI Q P , HUANG L B . Softmax deep double deterministic policy gradients[J]. Advances in Neural Information Processing Systems, 2020(33): 11767-11777.
[26]	KHAN A A , ZAFRULLAH M , HUSSAIN M ,et al. Performance analysis of OSPF and hybrid networks[C]// Proceedings of 2017 International Symposium on Wireless Systems and Networks (ISWSN). Piscataway:IEEE Press, 2017: 1-4.
[27]	CHIESA M , KINDLER G , SCHAPIRA M . Traffic engineering with equal-cost-multipath:an algorithmic perspective[J]. IEEE/ACM Transactions on Networking, 2017,25(2): 779-792.
[28]	SUN P H , GUO Z H , LI J F ,et al. Enabling scalable routing in software-defined networks with deep reinforcement learning on critical nodes[J]. IEEE/ACM Transactions on Networking, 2022,30(2): 629-640.
[29]	SUN P H , LAN J L , LI J F ,et al. A scalable deep reinforcement learning approach for traffic engineering based on link control[J]. IEEE Communications Letters, 2021,25(1): 171-175.
[30]	National Science Foundation. National science foundation network[EB]. 2022.

Metrics

Recommended 0

No Suggested Reading articles found!

超参	数值
探索噪声（expl_noise）	0.3
采样大小（batch_size）	8
奖励折扣因子（discount）	0.99
网络参数更新步长（tau）	0.005
截断目标策略噪音（noise_clip）	0.5
策略网络学习率（actor_lr）	3×10^-4
价值网络学习率（critic_lr）	3×10^-4

Quality of service optimization algorithm based on deep reinforcement learning in software defined network

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 30

Related Articles 7

Metrics

Recommended 0

[1]	Zihui LUO, Chengling JIANG, Liang LIU, Xiaolong ZHENG, Huadong MA. Research on deep reinforcement learning based intelligent shop scheduling method [J]. Chinese Journal on Internet of Things, 2022, 6(1): 53-64.
[2]	Haibo MEI, Kun YANG, Xinyu FAN. Deep reinforcement learning to enhance the energy-efficient performance of UAV-enabled F-RAN [J]. Chinese Journal on Internet of Things, 2021, 5(2): 48-59.
[3]	Hao YUAN, Deke GUO, Guoming TANG, Lailong LUO. Online energy-aware task dispatching with QoS guarantee in edge computing [J]. Chinese Journal on Internet of Things, 2021, 5(2): 71-77.
[4]	Yinglei TENG,Xin LI,Jian WANG,Bogen CAI,Mei SONG. SDN enabled space-terrestrial integrated network architecture of railway system [J]. Chinese Journal on Internet of Things, 2020, 4(3): 30-41.
[5]	Yi ZHOU,Xiaoyong MA,Fuxiao GAO,Wei LI,Nan CHENG,Ning LU. Autonomous deployment and energy efficiency optimization strategy of UAV based on deep reinforcement learning [J]. Chinese Journal on Internet of Things, 2019, 3(2): 47-55.
[6]	Ruijin DING,Feifei GAO,Ling XING. Intelligent routing strategy in the Internet of things based on deep reinforcement learning [J]. Chinese Journal on Internet of Things, 2019, 3(2): 56-63.
[7]	Guanhua QIAO,Supeng LENG,Hao LIU,Kaisheng HUANG,Fan WU. Task collaborative offloading scheme in vehicle multi-access edge computing network [J]. Chinese Journal on Internet of Things, 2019, 3(1): 51-59.