基于深度强化学习的软件定义网络QoS优化

doi:10.11959/j.issn.1000-436x.2019227

通信学报 ›› 2019, Vol. 40 ›› Issue (12): 60-67.doi: 10.11959/j.issn.1000-436x.2019227

基于深度强化学习的软件定义网络QoS优化

兰巨龙,张学帅(),胡宇翔,孙鹏浩

国家数字交换系统工程技术研究中心，河南郑州 450001

修回日期:2019-10-28 出版日期:2019-12-25 发布日期:2020-01-16
作者简介:兰巨龙（1962- ），男，河北张家口人，博士，国家数字交换系统工程技术研究中心教授、博士生导师，主要研究方向为未来信息通信网络关键理论与技术|张学帅（1994- ），男，山东菏泽人，国家数字交换系统工程技术研究中心硕士生，主要研究方向为软件定义网络|胡宇翔（1982- ），男，河南周口人，博士，国家数字交换系统工程技术研究中心副教授、博士生导师，主要研究方向为未来网络关键技术、网络智慧化等|孙鹏浩（1992- ），男，山东青岛人，国家数字交换系统工程技术研究中心博士生，主要研究方向为软件定义网络、流量工程等
基金资助:
国家重点研发计划基金资助项目(2017YFB0803204);国家自然科学基金资助项目(61521003);国家自然科学基金资助项目(61702547);国家自然科学基金资助项目(61872382);广东省重点领域研发计划基金资助项目(2018B010113001)

Software-defined networking QoS optimization based on deep reinforcement learning

Julong LAN,Xueshuai ZHANG(),Yuxiang HU,Penghao SUN

National Digital Switching System Engineering ＆Research Center,Zhengzhou 450001,China

Revised:2019-10-28 Online:2019-12-25 Published:2020-01-16
Supported by:
The National Key Research and Development Program of China(2017YFB0803204);The National Natural Science Foundation of China(61521003);The National Natural Science Foundation of China(61702547);The National Natural Science Foundation of China(61872382);The Research and Development Program in Key Areas of Guangdong Province(2018B010113001)

摘要/Abstract

摘要：

为解决软件定义网络场景中，当前主流的基于启发式算法的QoS优化方案常因参数与网络场景不匹配出现性能下降的问题，提出了基于深度强化学习的软件定义网络QoS优化算法。首先将网络资源和状态信息统一到网络模型中，然后通过长短期记忆网络提升算法的流量感知能力，最后基于深度强化学习生成满足QoS目标的动态流量调度策略。实验结果表明，相对于现有算法，所提算法不但保证了端到端传输时延和分组丢失率，而且提高了22.7%的网络负载均衡程度，增加了8.2%的网络吞吐率。

关键词: 软件定义网络, 深度强化学习, 长短期记忆, 服务质量

Abstract:

To solve the problem that the QoS optimization schemes which based on heuristic algorithm degraded often due to the mismatch between parameters and network characteristics in software-defined networking scenarios,a software-defined networking QoS optimization algorithm based on deep reinforcement learning was proposed.Firstly,the network resources and state information were integrated into the network model,and then the flow perception capability was improved by the long short-term memory,and finally the dynamic flow scheduling strategy,which satisfied the specific QoS objectives,were generated in combination with deep reinforcement learning.The experimental results show that,compared with the existing algorithms,the proposed algorithm not only ensures the end-to-end delay and packet loss rate,but also improves the network load balancing by 22.7% and increases the throughput by 8.2%.

Key words: software-defined networking, deep reinforcement learning, long short-term memory, quality of service

中图分类号:

TP393

兰巨龙,张学帅,胡宇翔,孙鹏浩. 基于深度强化学习的软件定义网络QoS优化[J]. 通信学报, 2019, 40(12): 60-67.

Julong LAN,Xueshuai ZHANG,Yuxiang HU,Penghao SUN. Software-defined networking QoS optimization based on deep reinforcement learning[J]. Journal on Communications, 2019, 40(12): 60-67.

图/表 7

图1

图2

表1

图3

图4

图5

图6

参考文献 20

[1]	MCKEOWN N . Software-defined networking[J]. INFOCOM Keynote Talk, 2009,17(2): 30-32.
[2]	张朝昆, 崔勇, 唐翯祎 ,等. 软件定义网络（SDN）研究进展[J]. 软件学报, 2015,26(1): 62-81.
	ZHANG C K , CUI Y , TANG H Y ,et al. State-of-the-art survey on software-defined-networking (SDN)[J]. Journal of Software, 2015,26(1): 62-81.
[3]	董谦, 李俊, 马宇翔 ,等. 软件定义网络中基于分段路由的流量调度方法[J]. 通信学报, 2018,39(11): 23-35.
	DONG Q , LI J , MA Y X ,et al. Traffic scheduling method based on segment routing in software-defined networking[J]. Journal on Com-munications, 2018,39(11): 23-35.
[4]	HARTMAN T , HASSIDIM A , KAPLAN H ,et al. How to split a flow?[C]// 2012 Proceedings IEEE INFOCOM. IEEE, 2012: 828-836.
[5]	ZHANG S R , . Valiant load-balancing:building networks that can support all traffic matrices[M]// London:Algorithms for Next Generation Networks. Springer, 2010: 19-30.
[6]	SUTTON R S , BARTO A G . Reinforcement learning:an introduction[M]. MIT Press, 2018.
[7]	LECUN Y , BENGIO Y , HINTON G . Deep learning[J]. Nature, 2015,521(7553):436.
[8]	MAO H , NETRAVALI R , ALIZADEH M . Neural adaptive video streaming with pensieve[C]// The Conference of the ACM Special Interest Group on Data Communication. ACM, 2017: 197-210.
[9]	XIAO S , HE D , GONG Z . Deep-q:traffic-driven QoS inference using deep generative network[C]// The 2018 Workshop on Network Meets AI ＆ ML. ACM, 2018: 67-73.
[10]	CHEN L , CHEN K , BAI W ,et al. Scheduling mix-flows in commodity datacenters with karuna[C]// The 2016 ACM SIGCOMM Conference. ACM, 2016: 174-187.
[11]	ONGARO F , CERQUEIRA E , FOSCHINI L ,et al. Enhancing the quality level support for real-time multimedia applications in software-defined networks[C]// 2015 International Conference on Computing,Networking and Communications (ICNC). IEEE, 2015: 505-509.
[12]	ALIZADEH M , EDSALL T , DHARMAPURIKAR S ,et al. CONGA:distributed congestion-aware load balancing for datacenters[J]. ACM SIGCOMM Computer Communication Review, 2014,44(4): 503-514.
[13]	XU Z , TANG J , MENG J ,et al. Experience-driven networking:a deep reinforcement learning based approach[C]// IEEE INFOCOM 2018 IEEE Conference on Computer Communications. IEEE, 2018: 1871-1879.
[14]	XU Z , TANG J , YIN C ,et al. Experience-driven congestion control:when multi-path TCP meets deep reinforcement learning[J]. IEEE Journal on Selected Areas in Communications, 2019,37(6): 1325-1336.
[15]	MCKEOWN N , ANDERSON T , BALAKRISHNAN H ,et al. OpenFlow:enabling innovation in campus networks[J]. ACM SIGCOMM Computer Communication Review, 2008,38(2): 69-74.
[16]	BOSSHART P , DALY D , GIBB G ,et al. P4:programming protocol-independent packet processors[J]. ACM SIGCOMM Computer Communication Review, 2014,44(3): 87-95.
[17]	ZHOU W , LI L , LUO M ,et al. REST API design patterns for SDN northbound API[C]// 2014 28th International Conference on Advanced Information Networking and Applications Workshops. IEEE, 2014: 358-365.
[18]	CLEMM A , CHANDRAMOULI M , KRISHNAMURTHY S . DNA:an SDN framework for distributed network analytics[C]// 2015 IFIP/IEEE International Symposium on Integrated Network Management (IM). IEEE, 2015: 9-17.
[19]	LILLICRAP T P , HUNT J J , PRITZEL A ,et al. Continuous control with deep reinforcement learning[C]// Eighth International Conference on Learning Representations. ICLR, 2016: 187-200.
[20]	R?CKE H , . Optimal hierarchical decompositions for congestion minimization in networks[C]// The Fortieth Annual ACM Symposium on Theory of Computing. ACM, 2008: 255-264.

属性	配置
操作系统	Ubuntu 16.04
CPU	Intel^?Xeon E5
GPU	NVIDIA Tesla P100
内存	64 GB

基于深度强化学习的软件定义网络QoS优化

Software-defined networking QoS optimization based on deep reinforcement learning

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 20

相关文章 15

Metrics

推荐阅读 0

[1]	金彪, 李逸康, 姚志强, 陈瑜霖, 熊金波. GenFedRL：面向深度强化学习智能体的通用联邦强化学习框架[J]. 通信学报, 2023, 44(6): 183-197.
[2]	李元诚, 秦永泰. 基于深度强化学习的软件定义安全中台QoS实时优化算法[J]. 通信学报, 2023, 44(5): 181-192.
[3]	许国良, 谭峰, 冉泳屹, 陈丰. 面向多波束卫星系统的波束跳变与覆盖控制联合优化算法[J]. 通信学报, 2023, 44(4): 78-86.
[4]	王东滨, 吴东哲, 智慧, 郭昆, 张勖, 时金桥, 张宇, 陆月明. 软件定义网络抗拒绝服务攻击的流表溢出防护[J]. 通信学报, 2023, 44(2): 1-11.
[5]	沙宗轩, 霍如, 孙闯, 汪硕, 黄韬. 基于深度强化学习的转发效能感知流量调度算法[J]. 通信学报, 2022, 43(8): 30-40.
[6]	张宇, 程旻. NDN中边缘计算与缓存的联合优化[J]. 通信学报, 2022, 43(8): 164-175.
[7]	刘建勋, 丁领航, 康国胜, 曹步清, 肖勇. 基于特征深度融合的Web服务QoS联合预测[J]. 通信学报, 2022, 43(7): 215-226.
[8]	张达敏, 王义, 邹诚诚, 赵沛雯, 张琳娜. 认知异构蜂窝网络中改进蜉蝣算法的资源分配策略[J]. 通信学报, 2022, 43(6): 156-167.
[9]	燕昺昊, 刘勤让, 沈剑良, 汤先拓, 梁栋. 软件定义网络中一种快速无循环路径迁移策略[J]. 通信学报, 2022, 43(5): 24-35.
[10]	吴平, 常朝稳, 左志斌, 马莹莹. 基于地址重载的SDN分组转发验证[J]. 通信学报, 2022, 43(3): 88-100.
[11]	张先超, 赵耀, 叶海军, 樊锐. 无线网络多用户干扰下智能发射功率控制算法[J]. 通信学报, 2022, 43(2): 15-21.
[12]	李传煌, 陈泱婷, 唐晶晶, 楼佳丽, 谢仁华, 方春涛, 王伟明, 陈超. QL-STCT：一种SDN链路故障智能路由收敛方法[J]. 通信学报, 2022, 43(2): 131-142.
[13]	苏新, 孟蕾蕾, 周一青, CELIMUGE Wu. 基于深度强化学习的海洋移动边缘计算卸载方法[J]. 通信学报, 2022, 43(10): 133-145.
[14]	王洪雁, 袁海. 基于骨骼及表观特征融合的动作识别方法[J]. 通信学报, 2022, 43(1): 138-148.
[15]	杜丽娜, 卓力, 杨硕, 李嘉锋, 张菁. 基于强化学习的移动视频流业务码率自适应算法研究进展[J]. 通信学报, 2021, 42(9): 205-217.