基于PaaS技术的大数据云化平台实践

doi:10.11959/j.issn.1000-0801.2018043

Abstract

Abstract:

As a platform of data storage,processing,and service,big data platform needs to support internal and external various big data applications.A solution applied PaaS technology to the big data platform was proposed.Details of the key technical solutions such as the architecture of PaaS cloud computing platform,hierarchical scheduling scheme,multi-tenant management and global scheduling of Hadoop tasks were described.It was proved that the platform could remarkably improve big data application development efficiency and help application deployment fast.In the meantime,good elastic scaling capability can effectively improve big data application system resource utilization.

Key words: big data, PaaS, Hadoop, YARN scheduler, cloud service

CLC Number:

TP311

Haiyong XU,Yan HUANG. Applications of a big data platform based on PaaS[J]. Telecommunications Science, 2018, 34(1): 148-157.

Figures/Tables 13

References 11

[1]	Gartner. Platform as a service(PaaS)[EB/OL].（2014-04-28）[2017-11-28]. .
[2]	齐磊, 张海峰, 张天骁 ,等. 基于容器技术的 PaaS 云平台方案[J]. 电信科学, 2017,33(4): 177-182.
	QI L , ZHANG H F , ZHANGT X ,et al. PaaS cloud platform based on container technology[J]. Telecommunications Science, 2017,33(4): 177-182.
[3]	VERMA A , PEDROSA L , KORUPOLU M ,et al. Large-scale cluster management at Google with Borg[C]// 10th European Conference on Computer Systems,April 21-24,2015,Bordeaux,France. New York:ACM Press, 2015:18.
[4]	张基, 李大中, 张呈宇 ,等. DC/OS 关键技术与应用场景[J]. 电信科学, 2016,32(12): 13-19.
	ZHANG J , LI D Z , ZHANG C Y ,et al. Key technologies and application scenario of DC/OS[J]. Telecommunications Science, 2016,32(12): 13-19.
[5]	BERNSTEIN D . Containers and cloud:from LXC to Docker to Kubernetes[J]. IEEE Cloud Computing, 2015,1(3): 81-84.
[6]	BALALAIE A , HEYDARNOORI A , JAMSHIDI P . Microservices Architecture enables DevOps:migration to a cloud-native architecture[J]. IEEE Software , 2016,33(3): 42-52.
[7]	王峰, 雷葆华 . Hadoop分布式文件系统的模型分析[J]. 电信科学, 2010,26(12): 95-99.
	WANG F , LEI B H . Modeling and analysis of Hadoop distributed file system[J]. Telecommunications Science, 2010,26(12): 95-99.
[8]	KONAR M , . Apache Hadoop YARN:yet another resource negotiator[C]// ACM Symposium on Cloud Computing,Oct 1-3,2013,Santa Clara,CA,USA. New York:ACM Press, 2013:5.
[9]	LU X , LIANG F , WANG B ,et al. DataMPI:Extending MPI to Hadoop-like big data computing[C]// International Parallel and Distributed Processing Symposium,May 19-23,2014,Phoenix,Arizona,USA. Piscataway:IEEE Press, 2014: 829-838.
[10]	郑文武, 李先绪, 黄执勤 . 云计算中的并行计算技术分析[J]. 电信科学, 2012,27(12): 31-38.
	ZHENG W W , LI X X , HUANG Z Q . Technolical analysis of parallel computing in cloud computing[J]. Telecommunications Science, 2012,27(12): 31-38.
[11]	李小宁, 李磊, 金连文 ,等. 基于OpenStack构建私有云计算平台[J]. 电信科学, 2012,27(9): 1-8.
	LI X N , LI L , JIN L W ,et al. Constructing a private cloud computing platform based on OpenStack[J]. Telecommunications Science, 2012,27(9): 1-8.

Metrics

Recommended 0

No Suggested Reading articles found!

领域	开源调度器	全局调度器
多租户调度	在同构集群上，只能选择容量调度器或公平调度器两者之一，且FusionInsight集群当前不支持公平调度器。容量调度器只支持百分比方式配置，而公平调度器只支持绝对值方式	支持异构集群和多资源池；
		在单一策略框架中同时支持百分比调度和绝对值方式配置资源需求；
		支持预留，以保证直接访问资源
数据位置感知调度	从节点到作业的调度策略导致降低数据本地话命中率，潜在影响应用的执行性能	从作业到节点的调度策略，可具有更精确的数据位置感知，数据本地化调度的作业命中率比较高
基于机器负载的均衡调度	不支持	在调度时考虑机器的负载和资源分配情况，做到均衡调度
租户内多用户公平调度	不支持	租户内用户的公平调度，支持关键字default、others
作业等待原因	不支持	作业等待原因信息可显示为什么作业需等待

对比项	传统开发模式	大数据云化开发模式
硬件资源准备时间	1～3个月	小时级别
开发工具准备时间	1～3周	小时级别
数据准备时间	1～3个月	小时级别
应用部署时间	小时级别	分钟级别
资源利用率	资源独占，利用率低	资源池化共享，资源利用率高

[1]	Lu HAN, Weiyu CHEN, Fei ZHANG, Jianfeng HE, Huaizhen SU. Big data classification method of non relational distributed submission information under differentiated requirements [J]. Telecommunications Science, 2023, 39(6): 114-121.
[2]	Xue HAN. Damage toughness assessment method of power backbone communication network based on power big data [J]. Telecommunications Science, 2023, 39(5): 136-143.
[3]	Yudi SUN. A prediction model of massive 5G network users’ revisit behavior based on telecom big data [J]. Telecommunications Science, 2023, 39(2): 157-162.
[4]	Aihua LI, Xiaobo WU, Chao CHEN, Bin WEI, Yuanyuan SHI. Big data intelligent analysis technology for 5G network [J]. Telecommunications Science, 2022, 38(8): 129-139.
[5]	Gengsheng HUANG, Yuhong HUANG, Manxue GUO, Jianping ZHENG, Xin GE. Architecture of a new infrastructure for data service in DT era [J]. Telecommunications Science, 2022, 38(7): 138-145.
[6]	Panpan LI, Zhengxia XIE, Guangxue YUE, Xin LIU. Research progress and trends of deep learning based wireless communication receiving method [J]. Telecommunications Science, 2022, 38(2): 1-17.
[7]	Haibo ZHAO, Zhijun XIANG, Linsong XIAO. A big data framework for short-term power load forecasting using heterogenous data [J]. Telecommunications Science, 2022, 38(12): 103-111.
[8]	Zhihong CHEN, Yuan YAO. Research on integration of government information system based on cloud computing [J]. Telecommunications Science, 2021, 37(9): 118-128.
[9]	Zhiyong LIU, Zhongjiang HE, Yilong RUAN, Junfeng SHAN, Chao ZHANG. Big data security features and operation practices [J]. Telecommunications Science, 2021, 37(5): 160-169.
[10]	Xiaojun LIU,Juan WU,Xiaoqing XU. Big data architecture analysis and data security fusion technology [J]. Telecommunications Science, 2020, 36(7): 146-155.
[11]	Yujiao LIU,Kunhuang SONG,Xiang WANG. Analysis of economic prosperity index based on big data of electricity [J]. Telecommunications Science, 2020, 36(6): 166-171.
[12]	Fuzhou SUN,Jin QIAN,Jing YANG,Ling QIAN. Open source development in the cloud services era [J]. Telecommunications Science, 2020, 36(11): 156-164.
[13]	Weihua CHENG,Xiaorong HE. Quality analysis method of Web browsing based on big data [J]. Telecommunications Science, 2020, 36(11): 174-181.
[14]	Xiaomin TAN,Ai FANG,Bing LIANG,Haojie YANG. Locating causes of abnormality of EPG experience based on decision tree [J]. Telecommunications Science, 2019, 35(8): 158-164.
[15]	Zelin LU,Qiangzhi LI. Evolutionary logic and operational mechanism of E-commerce platform [J]. Telecommunications Science, 2019, 35(7): 152-158.

Applications of a big data platform based on PaaS

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 11

Related Articles 15

Metrics

Recommended 0