基于深度强化学习的转发效能感知流量调度算法

doi:10.11959/j.issn.1000-436x.2022148

通信学报 ›› 2022, Vol. 43 ›› Issue (8): 30-40.doi: 10.11959/j.issn.1000-436x.2022148

基于深度强化学习的转发效能感知流量调度算法

沙宗轩¹, 霍如¹^,², 孙闯³, 汪硕²^,⁴, 黄韬²^,⁴

¹ 北京工业大学信息学部，北京 100124
² 网络通信与安全紫金山实验室, 江苏南京 211111
³ 清华大学自动化系，北京 100084
⁴ 北京邮电大学网络与交换国家重点实验室，北京 100876

修回日期:2022-07-19 出版日期:2022-08-25 发布日期:2022-08-01
作者简介:沙宗轩（1990- ），男，回族，安徽蚌埠人，北京工业大学博士生，主要研究方向为未来网络、网络人工智能深度学习、强化学习等
霍如（1988- ），女，黑龙江哈尔滨人，博士，北京工业大学讲师，主要研究方向为未来网络、工业互联网、边缘计算、网络资源管理、区块链等
孙闯（1989- ），男，黑龙江哈尔滨人，博士，清华大学在站博士后，主要研究方向为传感器测量与测试技术、初始仪器精密标定方法和惯性导航技术等
汪硕（1991- ），男，河南灵宝人，博士，北京邮电大学讲师，主要研究方向为数据中心网络、软件定义网络、网络流量调度等
黄韬（1980- ），男，重庆人，博士，北京邮电大学教授，主要研究方向为未来网络体系架构、软件定义网络、网络虚拟化等
基金资助:
2020年工业互联网创新发展工程基金资助项目（工业互联网标识资源搜索系统）;中国通信学会青年人才托举计划-托举培养协议基金资助项目(YESS20200287)

Forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning

Zongxuan SHA¹, Ru HUO¹^,², Chuang SUN³, Shuo WANG²^,⁴, Tao HUANG²^,⁴

¹ Information Department, Beijing University of Technology, Beijing 100124, China
² Purple Mountain Laboratories, Nanjing 211111, China
³ Department of Automation, Tsinghua University, Beijing 100084, China
⁴ State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China

Revised:2022-07-19 Online:2022-08-25 Published:2022-08-01
Supported by:
The MIIT of China 2020 (Identification Resources Search System for Industrial Internet of Things);Young Elite Scientist Sponsorship Program by CAST and China-CIC(YESS20200287)

摘要/Abstract

摘要：

软件定义网络（SDN）通过将控制平面与数据平面分离，可实现灵活的流量调度，更有效地利用网络资源。但是，流表项数量、设备负载率及连接主机数量增加等因素的共同作用会导致 SDN 交换机的转发效能降低，进而影响端到端的数据传输时延。为了解决上述问题，提出了基于深度强化学习的转发效能感知流量调度算法。首先，将交换机状态信息统一到感知模型中，通过神经网络建立交换机状态信息和转发效能的映射关系。然后，结合网络状态和流量信息，通过深度强化学习产生流量调度策略。最后，通过由最短路径和负载均衡算法产生的专家样本引导模型训练，不仅使模型学习到专家样本的知识以提升性能，同时提升模型训练效率。实验结果表明，与其他算法相比，所提算法不仅使端到端的平均传输时延降低了15.31%，而且保证了网络整体的负载均衡。

关键词: 软件定义网络, 深度强化学习, 流量调度, 转发效能感知, 负载均衡

Abstract:

The software defined network separates the control plane from the data plane to achieve flexible traffic scheduling, which can use network resources more efficiently.However, with the increase of the number of flow entries, load rate, the number of connected hosts, and other factors, the forwarding efficiency of the SDN switch will be reduced, which will affect the end-to-end transmission delay.To solve the above problems, the forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning was proposed.First, the switch state was integrated into the perception model, and the mapping relationship between switch state information and forwarding efficiency was established based on neural network.Then, combined with network state and traffic information, traffic scheduling policy was generated through deep reinforcement learning.Finally, the expert samples generated by the shortest path and load balance algorithms could guide the model training, which enabled the model to learn knowledge from expert samples to improve performance and accelerated the training process.The experimental results show that the proposed algorithm not only reduces the average end-to-end transmission delay by 15.31%, but also ensures the overall load balance of the network, compared with other algorithms.

Key words: software defined network, deep reinforcement learning, traffic scheduling, forwarding efficiency aware, load balance

中图分类号:

TN911.5

沙宗轩, 霍如, 孙闯, 汪硕, 黄韬. 基于深度强化学习的转发效能感知流量调度算法[J]. 通信学报, 2022, 43(8): 30-40.

Zongxuan SHA, Ru HUO, Chuang SUN, Shuo WANG, Tao HUANG. Forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning[J]. Journal on Communications, 2022, 43(8): 30-40.

图/表 6

图1

图2

表1

图3

图4

图5

参考文献 27

[1]	周桐庆, 蔡志平, 夏竟 ,等. 基于软件定义网络的流量工程[J]. 软件学报, 2016,27(2): 394-417.
	ZHOU T Q , CAI Z P , XIA J ,et al. Traffic engineering for software defined networks[J]. Journal of Software, 2016,27(2): 394-417.
[2]	MENDIOLA A , ASTORGA J , JACOB E ,et al. A survey on the contributions of software-defined networking to traffic engineering[J]. IEEE Communications Surveys ＆ Tutorials, 2017,19(2): 918-953.
[3]	杨思锦, 庄雷, 宋玉 ,等. 多模态网络中时间敏感网络模态的智能调度机制[J]. 通信学报, 2022,43(5): 82-91.
	YANG S J , ZHUANG L , SONG Y ,et al. Intelligent scheduling mechanism of time-sensitive network modal in polymorphic network[J]. Journal on Communications, 2022,43(5): 82-91.
[4]	SARASWAT S , AGARWAL V , GUPTA H P ,et al. Challenges and solutions in software defined networking:a survey[J]. Journal of Network and Computer Applications, 2019,141: 23-58.
[5]	黄韬, 刘江, 汪硕 ,等. 未来网络技术与发展趋势综述[J]. 通信学报, 2021,42(1): 130-150.
	HUANG T , LIU J , WANG S ,et al. Survey of the future network technology and trend[J]. Journal on Communications, 2021,42(1): 130-150.
[6]	LECUN Y , BENGIO Y , HINTON G . Deep learning[J]. Nature, 2015,521(7553): 436-444.
[7]	SUTTON R S , BARTO A G . Reinforcement learning:an introduction[M]. Massachusetts: MIT Press, 1998.
[8]	XIAO Y , LIU J , WU J W ,et al. Leveraging deep reinforcement learning for traffic engineering:a survey[J]. IEEE Communications Surveys ＆ Tutorials, 2021,23(4): 2064-2097.
[9]	XIE J F , YU F R , HUANG T ,et al. A survey of machine learning techniques applied to software defined networking (SDN):research issues and challenges[J]. IEEE Communications Surveys ＆ Tutorials, 2019,21(1): 393-430.
[10]	HARTMAN T , HASSIDIM A , KAPLAN H ,et al. How to split a flow?[C]// Proceedings of IEEE INFOCOM. Piscataway:IEEE Press, 2012: 828-836.
[11]	ZHANG J J , YE M H , GUO Z H ,et al. CFR-RL:traffic engineering with reinforcement learning in SDN[J]. IEEE Journal on Selected Areas in Communications, 2020,38(10): 2249-2259.
[12]	ZHANG J J , GUO Z H , YE M H ,et al. SmartEntry:mitigating routing update overhead with reinforcement learning for traffic engineering[C]// Proceedings of the Workshop on Network Meets AI ＆ ML. Piscataway:IEEE Press, 2020: 1-7.
[13]	HUANG X H , YUAN T T , QIAO G H ,et al. Deep reinforcement learning for multimedia traffic control in software defined networking[J]. IEEE Network, 2018,32(6): 35-41.
[14]	HUANG X H , ZENG M , XIE K . Intelligent traffic control for QoS optimization in hybrid SDNs[J]. Computer Networks, 2021,189: 107877.
[15]	ZHANG Q Y , WANG X W , LV J H ,et al. Intelligent content-aware traffic engineering for SDN:an AI-driven approach[J]. IEEE Network, 2020,34(3): 186-193.
[16]	XU Z Y , TANG J , MENG J S ,et al. Experience-driven networking:a deep reinforcement learning based approach[C]// Proceedings of IEEE Conference on Computer Communications. Piscataway:IEEE Press, 2018: 1871-1879.
[17]	KATO N , FADLULLAH Z M , MAO B M ,et al. The deep learning vision for heterogeneous network traffic control:proposal,challenges,and future perspective[J]. IEEE Wireless Communications, 2017,24(3): 146-153.
[18]	ZHANG X C , WANG Y L , GENG G G ,et al. Delay-optimized multicast tree packing in software-defined networks[J]. IEEE Transactions on Services Computing, 2021,PP(99): 1.
[19]	WU T , ZHOU P , WANG B H ,et al. Joint traffic control and multi-channel reassignment for core backbone network in SDN-IoT:a multi-agent deep reinforcement learning approach[J]. IEEE Transactions on Network Science and Engineering, 2021,8(1): 231-245.
[20]	SAHA N , MISRA S , BERA S . Q-flag:QoS-aware flow-rule aggregation in software-defined IoT networks[J]. IEEE Internet of Things Journal, 2022,9(7): 4899-4906.
[21]	FILALI A , KOBBANE A , ELMACHKOUR M ,et al. SDN controller assignment and load balancing with minimum quota of processing capacity[C]// Proceedings of IEEE International Conference on Communications (ICC). Piscataway:IEEE Press, 2018: 1-6.
[22]	SAVAS S S , TORNATORE M , DIKBIYIK F ,et al. RASCAR:recovery-aware switch-controller assignment and routing in SDN[J]. IEEE Transactions on Network and Service Management, 2018,15(4): 1222-1234.
[23]	WANG T , LIU F M , XU H . An efficient online algorithm for dynamic SDN controller assignment in data center networks[J]. IEEE/ACM Transactions on Networking, 2017,25(5): 2788-2801.
[24]	BERA S , MISRA S , SAHA N . Traffic-aware dynamic controller assignment in SDN[J]. IEEE Transactions on Communications, 2020,68(7): 4375-4382.
[25]	BOUZIDI E H , OUTTAGARTS A , LANGAR R ,et al. Dynamic clustering of software defined network switches and controller placement using deep reinforcement learning[J]. Computer Networks, 2022,207:108852.
[26]	LIN N , ZHAO Q , ZHAO L ,et al. A novel cost-effective controller placement scheme for software-defined vehicular networks[J]. IEEE Internet of Things Journal, 2021,8(18): 14080-14093.
[27]	MAITY I , MISRA S , MANDAL C . CORE:prediction-based control plane load reduction in software-defined IoT networks[J]. IEEE Transactions on Communications, 2021,69(3): 1835-1844.

属性	配置
操作系统	Ubuntu 16
CPU	Intel(R) Core(TM) i5-10400F
内存	16.0 GB
Python	Ver.3.7
Tensorflow	Ver.1.14
Mininet	Ver.2.3.0
Ryu	Ver.4.34
Iperf	Ver.2.0.9

基于深度强化学习的转发效能感知流量调度算法

Forwarding efficiency aware traffic scheduling algorithm based on deep reinforcement learning

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 6

参考文献 27

相关文章 15

Metrics

推荐阅读 0

[1]	金彪, 李逸康, 姚志强, 陈瑜霖, 熊金波. GenFedRL：面向深度强化学习智能体的通用联邦强化学习框架[J]. 通信学报, 2023, 44(6): 183-197.
[2]	李元诚, 秦永泰. 基于深度强化学习的软件定义安全中台QoS实时优化算法[J]. 通信学报, 2023, 44(5): 181-192.
[3]	许国良, 谭峰, 冉泳屹, 陈丰. 面向多波束卫星系统的波束跳变与覆盖控制联合优化算法[J]. 通信学报, 2023, 44(4): 78-86.
[4]	王东滨, 吴东哲, 智慧, 郭昆, 张勖, 时金桥, 张宇, 陆月明. 软件定义网络抗拒绝服务攻击的流表溢出防护[J]. 通信学报, 2023, 44(2): 1-11.
[5]	张宇, 程旻. NDN中边缘计算与缓存的联合优化[J]. 通信学报, 2022, 43(8): 164-175.
[6]	尹长川, 李妍珏, 朱海龙, 何欣欣, 韩文璇. HSTC：TSN中的混合流量调度机制[J]. 通信学报, 2022, 43(6): 119-132.
[7]	燕昺昊, 刘勤让, 沈剑良, 汤先拓, 梁栋. 软件定义网络中一种快速无循环路径迁移策略[J]. 通信学报, 2022, 43(5): 24-35.
[8]	吴平, 常朝稳, 左志斌, 马莹莹. 基于地址重载的SDN分组转发验证[J]. 通信学报, 2022, 43(3): 88-100.
[9]	张先超, 赵耀, 叶海军, 樊锐. 无线网络多用户干扰下智能发射功率控制算法[J]. 通信学报, 2022, 43(2): 15-21.
[10]	李传煌, 陈泱婷, 唐晶晶, 楼佳丽, 谢仁华, 方春涛, 王伟明, 陈超. QL-STCT：一种SDN链路故障智能路由收敛方法[J]. 通信学报, 2022, 43(2): 131-142.
[11]	裴金川, 胡宇翔, 田乐, 胡涛, 李子勇. 联合路由规划的时间敏感网络流量调度方法[J]. 通信学报, 2022, 43(12): 54-65.
[12]	苏新, 孟蕾蕾, 周一青, CELIMUGE Wu. 基于深度强化学习的海洋移动边缘计算卸载方法[J]. 通信学报, 2022, 43(10): 133-145.
[13]	杜丽娜, 卓力, 杨硕, 李嘉锋, 张菁. 基于强化学习的移动视频流业务码率自适应算法研究进展[J]. 通信学报, 2021, 42(9): 205-217.
[14]	杨明川, 薛冠昌, 李清毅. 基于邻居卫星负载状态的低轨卫星分布式路由算法[J]. 通信学报, 2021, 42(8): 43-51.
[15]	吴平, 常朝稳, 马莹莹. 基于端址重载的SDN包转发验证[J]. 通信学报, 2021, 42(7): 70-83.