基于Kudu的大数据平台实时业务处理能力提升方案

doi:10.11959/j.issn.1000-0801.2019158

Abstract

Abstract:

Aiming at the difficulty of real-time business processing capability of Hadoop big data platform,the theory of the most advanced Kudu column storage was studied as an effective complement of HDFS block storage,and the use of primary key index and memory acceleration provided by Kudu and Spark was illustrated to effectively solve the big data platform cannot support the technical implementation methods of business pain points such as real-time warehousing,incremental update and SQL-related query.The experimental results prove the effect of the method on improving the real-time business processing capability of the big data platform.

Key words: Kudu, big data, column storage, primary key index, memory acceleration, real-time access, incremental update, SQL join query

CLC Number:

TP311.13

Feiyang GU, Ying KONG. Scheme of enhancing real-time business processing capabilities based on Kudu for the big data platform[J]. Telecommunications Science, 2019, 35(10): 151-156.

Figures/Tables 7

References 4

[1]	汤劲松, 蔡韵, 王晓亮 ,等. 电信运营商大数据基础能力建设与实践[J]. 电信科学, 2018,34(1): 158-165.
	TANG J S , CAI Y , WANG X L ,et al. Construction and practice of big data fundamental ability for telecom operators[J]. Telecommunications Science, 2018,34(1): 158-165.
[2]	王瑞通, 李炜春 . 大数据基础存储系统技术研究[J]. 计算机技术与发展, 2017(8): 66-72.
	WANG R T , LI W C . Research of big data storage system technology[J]. The Computer Technology and Development, 2017(8): 66-72.
[3]	陈涛, 鲁萌, 陈彦名 . 运营商大数据技术应用研究[J]. 电信科学, 2017,33(1): 130-134.
	CHEN T , LU M , CHEN Y M . Research on operators’ big data technologies and applications[J]. Telecommunications Science, 2017,33(1): 130-134.
[4]	鲍远松 . 基于 Kudu 的结构化数据存储分析方案设计[J]. 信息技术与标准化, 2017(10): 60-63.
	BAO Y S . Structured data storage analysis scheme design based on Kudu[J]. Information Technology and Standardization, 2017(10): 60-63.

Metrics

Recommended 0

No Suggested Reading articles found!

表名	导入数据量/条	时间/s
wls_carrier	11 284	2.43
trs_dwdmwave	28 083	5.92

表名	5 min		15 min
表名	更新数据量	时间/s	更新数据量	时间/s
inc_addr_segm_eqp	33	6	42	8
inc_asn_link_route	5 032	6	35 981	35
inc_cbl_pair	176	7	1 606	13
inc_c_servicecustomer	174	6	2 088	6
cbl_jnt_box	4	6	32	7

实时业务应用	Kudu应用前	Kudu应用后
流式数据实时入库	块存储	列存储
	HDFS整个数据块写入实时性较差，分钟级别	Tablet逐条写入实时性高、秒级
流式数据增量更新	HDFS不支持更新	列存储支持更新
	不支持增量更新，只能插入	支持增量更新和插入
	全量更新时间为分钟至小时级别	增量更新时间为秒级
实时数据复杂SQL查询	不支持内存缓存	支持内存缓存
	没有索引功能	具备主键索引功能
	查询时间为分钟至小时级别	查询时间为秒级

[1]	Lu HAN, Weiyu CHEN, Fei ZHANG, Jianfeng HE, Huaizhen SU. Big data classification method of non relational distributed submission information under differentiated requirements [J]. Telecommunications Science, 2023, 39(6): 114-121.
[2]	Xue HAN. Damage toughness assessment method of power backbone communication network based on power big data [J]. Telecommunications Science, 2023, 39(5): 136-143.
[3]	Yudi SUN. A prediction model of massive 5G network users’ revisit behavior based on telecom big data [J]. Telecommunications Science, 2023, 39(2): 157-162.
[4]	Aihua LI, Xiaobo WU, Chao CHEN, Bin WEI, Yuanyuan SHI. Big data intelligent analysis technology for 5G network [J]. Telecommunications Science, 2022, 38(8): 129-139.
[5]	Gengsheng HUANG, Yuhong HUANG, Manxue GUO, Jianping ZHENG, Xin GE. Architecture of a new infrastructure for data service in DT era [J]. Telecommunications Science, 2022, 38(7): 138-145.
[6]	Panpan LI, Zhengxia XIE, Guangxue YUE, Xin LIU. Research progress and trends of deep learning based wireless communication receiving method [J]. Telecommunications Science, 2022, 38(2): 1-17.
[7]	Haibo ZHAO, Zhijun XIANG, Linsong XIAO. A big data framework for short-term power load forecasting using heterogenous data [J]. Telecommunications Science, 2022, 38(12): 103-111.
[8]	Zhiyong LIU, Zhongjiang HE, Yilong RUAN, Junfeng SHAN, Chao ZHANG. Big data security features and operation practices [J]. Telecommunications Science, 2021, 37(5): 160-169.
[9]	Xiaojun LIU,Juan WU,Xiaoqing XU. Big data architecture analysis and data security fusion technology [J]. Telecommunications Science, 2020, 36(7): 146-155.
[10]	Yujiao LIU,Kunhuang SONG,Xiang WANG. Analysis of economic prosperity index based on big data of electricity [J]. Telecommunications Science, 2020, 36(6): 166-171.
[11]	Weihua CHENG,Xiaorong HE. Quality analysis method of Web browsing based on big data [J]. Telecommunications Science, 2020, 36(11): 174-181.
[12]	Xiaomin TAN,Ai FANG,Bing LIANG,Haojie YANG. Locating causes of abnormality of EPG experience based on decision tree [J]. Telecommunications Science, 2019, 35(8): 158-164.
[13]	Zelin LU,Qiangzhi LI. Evolutionary logic and operational mechanism of E-commerce platform [J]. Telecommunications Science, 2019, 35(7): 152-158.
[14]	Ming LIAO,Ming CHEN,Ji ZHOU,Xiaohua XIANG,Fang LI,Yefen JIAO. DNS log analysis system based on big data fusion algorithm [J]. Telecommunications Science, 2019, 35(5): 129-139.
[15]	Yuanfeng GONG,Yi HUANG. Optimization method of wireless indoor coverage based on big data analysis [J]. Telecommunications Science, 2019, 35(5): 149-154.

Scheme of enhancing real-time business processing capabilities based on Kudu for the big data platform

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 4

Related Articles 15

Metrics

Recommended 0