基于强化学习的服务链映射算法

doi:10.11959/j.issn.1000-436x.2018002

通信学报 ›› 2018, Vol. 39 ›› Issue (1): 90-100.doi: 10.11959/j.issn.1000-436x.2018002

基于强化学习的服务链映射算法

魏亮,黄韬,张娇,王泽南,刘江,刘韵洁

北京邮电大学网络与交换技术国家重点实验室，北京 100876

修回日期:2017-11-29 出版日期:2018-01-01 发布日期:2018-02-07
作者简介:魏亮（1981-），男，江苏扬州人，北京邮电大学博士生，主要研究方向为未来网络、软件定义网络、网络功能虚拟化等。|黄韬（1980-），男，重庆人，博士，北京邮电大学教授，主要研究方向为路由与交换、软件定义网络、内容分发网络等。|张娇（1986-），女，河北保定人，北京邮电大学副教授，主要研究方向为数据中心网络、网络功能虚拟化、软件定义网络、未来网络体系架构等。|王泽南（1994-），男，浙江湖州人，北京邮电大学博士生，主要研究方向为网络功能虚拟化、网络智能等。|刘江（1983-），男，河南郑州人，博士，北京邮电大学副教授，主要研究方向为网络体系架构、网络虚拟化、软件定义网络、信息中心网络等。|刘韵洁（1943-），男，山东烟台人，中国工程院院士，北京邮电大学教授，主要研究方向为未来网络技术、网络体系架构、网络融合与演进等。
基金资助:
国家高技术研究发展计划（“863”计划）基金资助项目(2015AA016101);国家自然科学基金资助项目(61501042);北京科技新星基金资助项目(Z151100000315078)

Service chain mapping algorithm based on reinforcement learning

Liang WEI,Tao HUANG,Jiao ZHANG,Zenan WANG,Jiang LIU,Yunjie LIU

State Key Laboratory of Networking and Switching Technology,Beijing University of Posts and Telecommunications,Beijing 100876,China

Revised:2017-11-29 Online:2018-01-01 Published:2018-02-07
Supported by:
The National High Technology Research and Development Program of China (863 Program)(2015AA016101);The National Natural Science Foundation of China(61501042);Beijing New-Star Plan of Science and Technology(Z151100000315078)

摘要/Abstract

摘要：

提出基于人工智能技术的多智能体服务链资源调度架构，设计一种基于强化学习的服务链映射算法。通过Q-learning的机制，根据系统状态、执行部署动作后的奖惩反馈来决定服务链中各虚拟网元的部署位置。实验结果表明，与经典算法相比，该算法有效降低了业务的平均传输延时，提升了系统的负载均衡情况。

关键词: 网络功能虚拟化, 人工智能, 服务链, 强化学习

Abstract:

A service chain resource scheduling architecture of multi-agent based on artificial intelligence technology was proposed.Meanwhile,a service chain mapping algorithm based on reinforcement learning was designed.Through the Q-learning mechanism,the location of each virtual network element in the service chain was determined according to the system status and the reward and punishment feedback after the deployment.The experimental results show that compared with the classical algorithms,the algorithm effectively reduces the average transmission delay of the service and improves the load balance of the system.

Key words: network function virtualization, artificial intelligence, service chain, reinforcement learning

中图分类号:

TP302

魏亮,黄韬,张娇,王泽南,刘江,刘韵洁. 基于强化学习的服务链映射算法[J]. 通信学报, 2018, 39(1): 90-100.

Liang WEI,Tao HUANG,Jiao ZHANG,Zenan WANG,Jiang LIU,Yunjie LIU. Service chain mapping algorithm based on reinforcement learning[J]. Journal on Communications, 2018, 39(1): 90-100.

图/表 7

表1

变量及意义"

变量	意义
G = (V,E)	网络拓扑
v _i∈V	其中一个节点
e _i∈E	其中一条链路
$C_{v_{i}}$	服务器节点总的vCPU数量
$\tilde{C_{v_{i}}}$	服务器节点已经使用的vCPU数量
$D (v_{i}, v_{j})$	节点v_i和节点v_j之间的最短路径的链路延时
$F = (f_{1}, f_{2}, f_{3}, \dots)$	系统支持的VNF的类型
$S_{i} = (s_{i}_{, 1}, s_{i}_{, 2}, s_{i}_{, 3}, \dots)$	服务链请求，由不同的网络功能按次序组成
f(s_i,j)	网络功能s_i,j所需要的VNF类型
\|S_i\|	服务链的的长度
v(s_i,j)	s_i,j部署的服务器节点位置

表1

图1

图2

图3

图4

图5

图6

参考文献 24

[1]	HAN B , GOPALAKRISHNAN V , JI L ,et al. Network function virtualization:challenges and opportunities for innovations[J]. IEEE Communications Magazine, 2015,53(2): 90-97.
[2]	MIJUMBI R , SERRAT J , GORRICHO J L ,et al. Network function virtualization:state-of-the-art and research challenges[J]. IEEE Communications Surveys ＆ Tutorials, 2016,18(1): 236-262.
[3]	RIERA J F , ESCALONA E , BATALLé J ,et al. Virtual network function scheduling:concept and challenges[C]// 2014 International Conference on Smart Communications in Network Technologies (SaCoNeT). 2014: 1-5.
[4]	MIJUMBI R , SERRAT J , GORRICHO J L ,et al. Design and evaluation of algorithms for mapping and scheduling of virtual network functions[C]// 2015 1st IEEE Conference on Network Softwarization (NetSoft). 2015: 1-9.
[5]	KREUTZ D , RAMOS F M V , VERíSSIMO P E ,et al. Software-defined networking:a comprehensive survey[J]. Proceedings of the IEEE, 2015,103(1): 14-76.
[6]	NUNES B A A , MENDONCA M , NGUYEN X N ,et al. A survey of software-defined networking:past,present,and future of programmable networks[J]. IEEE Communications Surveys ＆ Tutorials, 2014,16(3): 1617-1634.
[7]	LI Y , CHEN M . Software-defined network function virtualization:a survey[J]. IEEE Access, 2015,3: 2542-2553.
[8]	BHAMARE D , JAIN R , SAMAKA M ,et al. A survey on service function chaining[J]. Journal of Network and Computer Applications, 2016,75: 138-155.
[9]	KUO T W , LIOU B H , LIN K C J ,et al. Deploying chains of virtual network functions:on the relation between link and server usage[C]// IEEE INFOCOM 2016-the 35th Annual IEEE International Conference on Computer Communications. 2016: 1-9.
[10]	MECHTRI M , GHRIBI C , ZEGHLACHE D . A scalable algorithm for the placement of service function chains[J]. IEEE Transactions on Network and Service Management, 2016,13(3): 533-546.
[11]	WANG L , LU Z , WEN X ,et al. Joint optimization of service function chaining and resource allocation in network function virtualization[J]. IEEE Access, 2016,4: 8084-8094.
[12]	YE Z , CAO X , WANG J ,et al. Joint topology design and mapping of service function chains for efficient,scalable,and reliable network functions virtualization[J]. IEEE Network, 2016,30(3): 81-87.
[13]	REDDY V S , BAUMGARTNER A , BAUSCHERT T . Robust embedding of VNF/service chains with delay bounds[C]// 2016 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN). 2016: 93-99.
[14]	MEHRAGHDAM S , KELLER M , KARL H . Specifying and placing chains of virtual network functions[C]// 2014 IEEE 3rd International Conference on Cloud Networking (CloudNet). 2014: 7-13.
[15]	ZHANG Q , XIAO Y , LIU F ,et al. Joint optimization of chain placement and request scheduling for network function virtualization[C]// 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS). 2017: 731-741.
[16]	HIRWE A , KATAOKA K . LightChain:a lightweight optimisation of VNF placement for service chaining in NFV[C]// 2016 IEEE NetSoft Conference and Workshops (NetSoft). 2016: 33-37.
[17]	ZHANG B , ZHANG P , ZHAO Y . Co-scaler:cooperative scaling of software-defined NFV service function chain[C]// 2016 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN). 2016: 33-38.
[18]	SAHHAF S , TAVERNIER W , COLLE D ,et al. Network service chaining with efficient network function mapping based on service decompositions[C]// 2015 1st IEEE Conference on Network Softwarization (NetSoft). 2015: 1-5.
[19]	MOENS H , TURCK F D . Customizable function chains:managing service chain variability in hybrid NFV networks[J]. IEEE Transactions on Network and Service Management, 2016,13(4): 711-724.
[20]	FEI X , LIU F , XU H ,et al. Towards load-balanced VNF assignment in geo-distributed NFV Infrastructure[C]// IEEE/ACM 25th International Symposium on Quality of Service (IWQoS). 2017: 1-10.
[21]	DIETRICH D , ABUJODA A , RIZK A ,et al. Multi-provider service chain embedding with nestor[J]. IEEE Transactions on Network and Service Management, 2017,14(1): 91-105.
[22]	HABIB A , KHAN M I . Reinforcement learning based autonomic virtual machine management in clouds[C]// 2016 5th International Conference on Informatics,Electronics and Vision (ICIEV). 2016: 1083-1088.
[23]	GROLéAT T , POUYLLAU H . Distributed inter-domain SLA negotiation using Reinforcement Learning[C]// 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011)and Workshops. 2011: 33-40.
[24]	LIN S C , AKYILDIZ I F , WANG P ,et al. QoS-aware adaptive routing in multi-layer hierarchical software defined networks:a reinforcement learning approach[C]// 2016 IEEE International Conference on Services Computing (SCC). 2016: 25-33.

基于强化学习的服务链映射算法

Service chain mapping algorithm based on reinforcement learning

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 24

相关文章 15

Metrics

推荐阅读 0

[1]	马玲, 樊漆亮, 许婷, 郭冠琛, 张圣林, 孙永谦, 张玉志. 基于强化学习的在线离线混部云环境下的调度框架[J]. 通信学报, 2023, 44(6): 90-102.
[2]	金彪, 李逸康, 姚志强, 陈瑜霖, 熊金波. GenFedRL：面向深度强化学习智能体的通用联邦强化学习框架[J]. 通信学报, 2023, 44(6): 183-197.
[3]	李元诚, 秦永泰. 基于深度强化学习的软件定义安全中台QoS实时优化算法[J]. 通信学报, 2023, 44(5): 181-192.
[4]	周大成, 陈鸿昶, 何威振, 程国振, 扈红超. 基于深度强化学习的微服务多维动态防御策略研究[J]. 通信学报, 2023, 44(4): 50-63.
[5]	许国良, 谭峰, 冉泳屹, 陈丰. 面向多波束卫星系统的波束跳变与覆盖控制联合优化算法[J]. 通信学报, 2023, 44(4): 78-86.
[6]	江沸菠, 彭于波, 董莉. 面向6G的深度图像语义通信模型[J]. 通信学报, 2023, 44(3): 198-208.
[7]	陈浩, 杨芫, 徐明伟, 裴丹, 尤艺霖. 支持多模态网络的可扩展异构服务功能链并行编排部署系统[J]. 通信学报, 2022, 43(9): 1-11.
[8]	许文俊, 吴思雷, 王凤玉, 林兰, 李国军, 张治. 基于多智能体强化学习的大规模灾后用户分布式覆盖优化[J]. 通信学报, 2022, 43(8): 1-16.
[9]	沙宗轩, 霍如, 孙闯, 汪硕, 黄韬. 基于深度强化学习的转发效能感知流量调度算法[J]. 通信学报, 2022, 43(8): 30-40.
[10]	徐泽汐, 庄雷, 张坤丽, 桂明宇. 基于知识图谱的服务功能链在线部署算法[J]. 通信学报, 2022, 43(8): 41-51.
[11]	马帅, 李兵, 盛海鸿, 谷荣妍, 周辉, 王洪梅, 王悦, 李世银. 基于深度强化学习的可见光定位通信一体化功率分配研究[J]. 通信学报, 2022, 43(8): 121-130.
[12]	张宇, 程旻. NDN中边缘计算与缓存的联合优化[J]. 通信学报, 2022, 43(8): 164-175.
[13]	左珮良, 侯少龙, 郭超, 蒋华, 王文博. 基于强化学习的多层卫星网络边缘安全决策方法[J]. 通信学报, 2022, 43(6): 189-199.
[14]	王敬宇, 庄子睿. 知识定义多模态网络按需服务体系研究[J]. 通信学报, 2022, 43(4): 71-82.
[15]	杨力, 潘成胜, 孔相广, 黄琦龙, 戚耀文. 5G融合卫星网络研究综述[J]. 通信学报, 2022, 43(4): 202-215.