基于强化学习的在线离线混部云环境下的调度框架

doi:10.11959/j.issn.1000-436x.2023119

通信学报 ›› 2023, Vol. 44 ›› Issue (6): 90-102.doi: 10.11959/j.issn.1000-436x.2023119

基于强化学习的在线离线混部云环境下的调度框架

马玲¹, 樊漆亮¹, 许婷¹, 郭冠琛², 张圣林¹, 孙永谦¹, 张玉志¹

¹ 南开大学软件学院，天津 300350
² 北京大学计算机学院，北京 100871

修回日期:2023-06-13 出版日期:2023-06-25 发布日期:2023-06-01
作者简介:马玲（1985- ），女，吉林洮南人，博士，南开大学副教授，主要研究方向为人工智能
樊漆亮（1999- ），男，江西南昌人，南开大学硕士生，主要研究方向为时间序列异常检测
许婷（1991- ），女，河南正阳人，南开大学博士生，主要研究方向为异常检测、故障定位、根因分析和故障预测等
郭冠琛（2000- ），女，山东淄博人，北京大学硕士生，主要研究方向为异常检测、故障定位、根因分析和故障预测等
张圣林（1989- ），男，山东滨州人，博士，南开大学副教授，主要研究方向为异常检测、故障定位、根因分析和故障预测等
孙永谦（1988- ），男，河北石家庄人，博士，南开大学助理教授，主要研究方向为异常检测、故障定位、根因分析和故障预测等
张玉志（1964- ），男，河北邢台人，博士，南开大学讲席教授、博士生导师，主要研究方向为人工智能和软件工程
基金资助:
国家自然科学基金资助项目(62272249);国家自然科学基金资助项目(61901234)

Scheduling framework based on reinforcement learning in online-offline colocated cloud environment

Ling MA¹, Qiliang FAN¹, Ting XU¹, Guanchen GUO², Shenglin ZHANG¹, Yongqian SUN¹, Yuzhi ZHANG¹

¹ College of Software, NanKai University, Tianjin 300350, China
² School of Computer Science, Peking University, Beijing 100871, China

Revised:2023-06-13 Online:2023-06-25 Published:2023-06-01
Supported by:
The National Natural Science Foundation of China(62272249);The National Natural Science Foundation of China(61901234)

摘要/Abstract

摘要：

目前针对云计算平台的强化学习调度算法考虑的场景较单一，或者忽略了任务的资源约束并简单地将所有机器看作同一类型，存在资源利用率较低及调度效率不高等不足。为了解决云环境中的在线离线混部调度问题，提出 JobFusion 框架。首先，通过集成带连通性约束的层次要素算法，在基于虚拟化技术的云计算平台中构建高效的资源划分方案；其次，为了解决扩展性问题，使用图卷积神经网络对具有任意层次约束关系及任意数量的任务进行嵌入，以捕获工作流的关键路径等信息；最后，集成了表现优异的强化学习模型对任务实施调度。实验结果表明，相较对比方法，JobFusion提高了39.86%的资源利用率，且最多降低了64.36%的平均任务完成时间。

关键词: 强化学习, 图嵌入, 层次聚类, 云计算, 虚拟化

Abstract:

Some reinforcement learning-based scheduling algorithms for cloud computing platforms barely considered one scenario or ignored the resource constraints of jobs and treated all machines as the same type, which caused low resource utilization or insufficient scheduling efficiency.To address the scheduling problems in online-offline colocated cloud environment, a framework named JobFusion was proposed.Firstly, an efficient resource partitioning scheme was built in the cloud computing platform supporting virtualization technology by integrating the hierarchical clustering method with connectivity constraints.Secondly, a graph convolutional neural network was utilized to embed the attributes of elastic dimension with various constraints and the jobs with various numbers, to capture the critical path information of workflow.Finally, existing high-performance reinforcement learning methods were integrated for scheduling jobs.According to the results of evaluation experiments, JobFusion improves the resource utilization by 39.86% and reduces the average job completion time by up to 64.36% compared with baselines.

Key words: reinforcement learning, graph embedding, hierarchical cluster, cloud computing, virtualization

中图分类号:

TP18

马玲, 樊漆亮, 许婷, 郭冠琛, 张圣林, 孙永谦, 张玉志. 基于强化学习的在线离线混部云环境下的调度框架[J]. 通信学报, 2023, 44(6): 90-102.

Ling MA, Qiliang FAN, Ting XU, Guanchen GUO, Shenglin ZHANG, Yongqian SUN, Yuzhi ZHANG. Scheduling framework based on reinforcement learning in online-offline colocated cloud environment[J]. Journal on Communications, 2023, 44(6): 90-102.

图/表 17

图1

图2

图3

图4

图5

图6

表1

表2

图7

图8

图9

图10

图11

图12

图13

图14

图15

参考文献 15

[1]	SRIVASTAVA P , KHAN R . A review paper on cloud computing[J]. International Journal of Advanced Research in Computer Science and Software Engineering, 2018,8(6): 17.
[2]	MAO H Z , SCHWARZKOPF M , VENKATAKRISHNAN S B ,et al. Learning scheduling algorithms for data processing clusters[C]// Proceedings of the ACM Special Interest Group on Data Communication. New York:ACM Press, 2019: 270-288.
[3]	MNIH V , BADIA A P , MIRZA M ,et al. Asynchronous methods for deep reinforcement learning[C]// Proceedings of the 33rd International Conference on International Conference on Machine Learnin. New York:ACM Press, 2016: 1928-1937.
[4]	YU H F , IRISSAPPANE A A , WANG H ,et al. FaaSRank:learning to schedule functions in serverless platforms[C]// Proceedings of 2021 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS). Piscataway:IEEE Press, 2022: 31-40.
[5]	ZHANG Y , ZHU H , TANG D ,et al. Dynamic job shop scheduling based on deep reinforcement learning for multi-agent manufacturing systems[J]. Robotics and Computer-Integrated Manufacturing, 2022:doi.org/10.1016/j.rcim.2022.102412.
[6]	SONG W , CHEN X Y , LI Q Q ,et al. Flexible job-shop scheduling via graph neural network and deep reinforcement learning[J]. IEEE Transactions on Industrial Informatics, 2023,19(2): 1600-1610.
[7]	ZHOU G , WEN R , TIAN W ,et al. Deep reinforcement learning-based algorithms selectors for the resource scheduling in hierarchical cloud computing[J]. Journal of Network and Computer Applications, 2022,208: 103520
[8]	SHENG J , HU Y , ZHOU W ,et al. Learning to schedule multi-NUMA virtual machines via reinforcement learning[J]. Pattern Recognition, 2022,121: 108254.
[9]	LI W G , LI J L , WU W G . A dynamic scheduling algorithm with time varying resource constraints in colocation data centers[C]// Proceedings of the 13th International Conference on Information and Communication Systems (ICICS). Piscataway:IEEE Press, 2022: 115-120.
[10]	ZHANG X L , LI L Q , WANG Y ,et al. Zeus:improving resource efficiency via workload colocation for massive kubernetes clusters[J]. IEEE Access, 2021,9: 105192-105204.
[11]	PI A D , ZHOU X B , XU C Z . Holmes:SMT interference diagnosis and CPU scheduling for job co-location[C]// Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing. New York:ACM Press, 2022: 110-121.
[12]	SUNNY M , SEN S J , SABU S E ,et al. Deploy-Web hosting using docker container[C]// Advances in Computing and Network Communications. Singapore:Springer, 2021: 335-345.
[13]	JAYAPRAKASH S , NAGARAJAN M D , PRADO R P D ,et al. A systematic review of energy management strategies for resource allocation in the cloud:clustering,optimization and machine learning[J]. Energies, 2021:doi.org/10.3390/en14175322.
[14]	CHHAJED D , LOWE T J . Building intuition:insights from basic operations management models and principles[M]. New York: Springer, 2008.
[15]	XU H W , CHEN W X , ZHAO N W ,et al. Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications[C]// Proceedings of the 2018 World Wide Web Conference. New York:ACM Press, 2018: 187-196.

硬件	规格参数
CPU	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20 GHz×2
DRAM	128 GB

物理机资源类型	CPU资源/核	内存资源/GB	机器数量/台
类型1	32	64	6 000
类型2	92	288	3 000

基于强化学习的在线离线混部云环境下的调度框架

Scheduling framework based on reinforcement learning in online-offline colocated cloud environment

在线阅读

PDF下载

可视化

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献 15

相关文章 15

Metrics

推荐阅读 0

[1]	金彪, 李逸康, 姚志强, 陈瑜霖, 熊金波. GenFedRL：面向深度强化学习智能体的通用联邦强化学习框架[J]. 通信学报, 2023, 44(6): 183-197.
[2]	李元诚, 秦永泰. 基于深度强化学习的软件定义安全中台QoS实时优化算法[J]. 通信学报, 2023, 44(5): 181-192.
[3]	周大成, 陈鸿昶, 何威振, 程国振, 扈红超. 基于深度强化学习的微服务多维动态防御策略研究[J]. 通信学报, 2023, 44(4): 50-63.
[4]	许国良, 谭峰, 冉泳屹, 陈丰. 面向多波束卫星系统的波束跳变与覆盖控制联合优化算法[J]. 通信学报, 2023, 44(4): 78-86.
[5]	陈浩, 杨芫, 徐明伟, 裴丹, 尤艺霖. 支持多模态网络的可扩展异构服务功能链并行编排部署系统[J]. 通信学报, 2022, 43(9): 1-11.
[6]	许文俊, 吴思雷, 王凤玉, 林兰, 李国军, 张治. 基于多智能体强化学习的大规模灾后用户分布式覆盖优化[J]. 通信学报, 2022, 43(8): 1-16.
[7]	沙宗轩, 霍如, 孙闯, 汪硕, 黄韬. 基于深度强化学习的转发效能感知流量调度算法[J]. 通信学报, 2022, 43(8): 30-40.
[8]	徐泽汐, 庄雷, 张坤丽, 桂明宇. 基于知识图谱的服务功能链在线部署算法[J]. 通信学报, 2022, 43(8): 41-51.
[9]	马帅, 李兵, 盛海鸿, 谷荣妍, 周辉, 王洪梅, 王悦, 李世银. 基于深度强化学习的可见光定位通信一体化功率分配研究[J]. 通信学报, 2022, 43(8): 121-130.
[10]	张宇, 程旻. NDN中边缘计算与缓存的联合优化[J]. 通信学报, 2022, 43(8): 164-175.
[11]	孙雁飞, 尹嘉峥, 亓晋, 胡筱旋, 陈梦婷, 董振江. 基于动态图嵌入的车联网拓扑控制[J]. 通信学报, 2022, 43(6): 133-142.
[12]	左珮良, 侯少龙, 郭超, 蒋华, 王文博. 基于强化学习的多层卫星网络边缘安全决策方法[J]. 通信学报, 2022, 43(6): 189-199.
[13]	杨力, 潘成胜, 孔相广, 黄琦龙, 戚耀文. 5G融合卫星网络研究综述[J]. 通信学报, 2022, 43(4): 202-215.
[14]	张先超, 赵耀, 叶海军, 樊锐. 无线网络多用户干扰下智能发射功率控制算法[J]. 通信学报, 2022, 43(2): 15-21.
[15]	李传煌, 陈泱婷, 唐晶晶, 楼佳丽, 谢仁华, 方春涛, 王伟明, 陈超. QL-STCT：一种SDN链路故障智能路由收敛方法[J]. 通信学报, 2022, 43(2): 131-142.