大数据 ›› 2020, Vol. 6 ›› Issue (1): 81-98.doi: 10.11959/j.issn.2096-0271.2020007

• 应用 • 上一篇    下一篇

云环境下大规模分布式计算数据感知的调度系统

刘汪根1,郑淮城1,荣国平2   

  1. 1 星环信息科技(上海)有限公司,上海 200233
    2 南京大学软件学院,江苏 南京 210093
  • 出版日期:2020-01-15 发布日期:2020-02-21
  • 作者简介:刘汪根(1985- ),男,星环信息科技(上海)有限公司研发总监、总架构师,主要研究方向为新一代的大数据架构、分布式数据库技术和容器云等|郑淮城(1987- ),男,星环信息科技(上海)有限公司软件工程师,星环原生云操作系统研发负责人,主要研究方向为复杂业务场景的底层容器云技术工程化|荣国平(1977- ),男,博士,南京大学软件学院副研究员,主要研究方向为DevOps、微服务架构、虚拟化技术等

A scheduler system for large-scale distributed data computing in cloud

Wanggen LIU1,Huaicheng ZHENG1,Guoping RONG2   

  1. 1 Transwarp Technology (Shanghai) Co.,Ltd.,Shanghai 200233,China
    2 Software Institute of Nanjing University,Nanjing 210093,China
  • Online:2020-01-15 Published:2020-02-21

摘要:

介绍了新的调度系统,包括资源调度、应用编排、配置标签中心、云网络和云存储服务等子系统。系统通过数据拓扑感知能力保证了计算和数据的局部性,节约网络I/O开销;通过优化点对点大数据量读取的资源调度,解决网络风暴造成的影响;通过网络和磁盘隔离技术以及可抢占的方式来保证服务等级协议。

关键词: 云计算, 调度系统, 大数据, AI平台, 数据局部性, 分布式计算, 抢占

Abstract:

A novel scheduler system including resource scheduling,application scheduling,configuration and label management center,cloud network and cloud storage services was introduced.The locality of computation and data was ensured by the ability of data topology awareness,and the I/O cost was saved.The impact of network storm was solved by optimizing the resource scheduling of point to point large data reading.The service level protocol was guaranteed by network and disk isolation technology and preemptive way.

Key words: cloud computing, scheduling system, big data, artificial intelligence platform, data locality, distributed computing, preemption

中图分类号: 

No Suggested Reading articles found!