基于HBase+ElasticSearch的海量交通数据实时存取方案设计

doi:10.11959/j.issn.2096-0271.2017010

大数据 ›› 2017, Vol. 3 ›› Issue (1): 80-89.doi: 10.11959/j.issn.2096-0271.2017010

基于HBase+ElasticSearch的海量交通数据实时存取方案设计

董长青,任女尔,张庆余,田玉靖

北京卡达克数据技术中心软件业务本部，天津 300300

出版日期:2017-01-20 发布日期:2017-03-17
作者简介:董长青（1980-），男，北京卡达克数据技术中心软件业务本部高级工程师，主要研究方向为大数据、车联网。|任女尔（1990-)，女，北京卡达克数据技术中心软件业务本部助理工程师，主要研究方向为大数据、云计算。|张庆余（1991-)，男，北京卡达克数据技术中心软件业务本部助理工程师，主要研究方向为软件架构、云计算。|田玉靖（1987-)，女，北京卡达克数据技术中心软件业务本部中级工程师，主要研究方向为软件架构、编程模式。

Design scheme of massive traffic data real-time access based on HBase and ElasticSearch

Changqing DONG,Nver REN,Qingyu ZHANG,Yujing TIAN

Software Business Department,Beijing CATARC Data ＆ Technology Center,Tianjin 300300,China

Online:2017-01-20 Published:2017-03-17

摘要/Abstract

摘要：

交通流数据具有数据海量、存储和交互速率快等特征，因此其数据的采集、存储及检索成为了车辆远程监控平台中的关键问题。采用LVS集群技术进行数据采集负载均衡，队列缓存处理I/O时延，HBase进行分布式数据存储；针对Hadoop实时在线数据处理不足的问题，整合ElasticSearch并构建了分层索引。通过关键技术的设计和实现，车辆监控由400辆扩展到上万辆，PB级数据在线查询速度提升了10～20倍，验证了方案的高效性。

关键词: Hadoop/HBase, ElasticSearch, Linux虚拟服务器, 海量数据, 实时

Abstract:

Traffic data has the characteristics of massive and real-time,and its massive data acquisition,storage and retrieval has become a key issue in the vehicle remote monitoring platform.According to the study of these problems,the cluster technology of LVS was used to solve the data acquisition load balance,the queue cache model was used to solve I/O delay,and HBase distributed data storage scheme was used to solve the massive data storage.HBase integration ElasticSearch,which was aimed to solve the real-time online data processing problems of Hadoop,was designed to build a hierarchical index.Through the design and implementation of the key technologies,the number of vehicle monitoring had been promoted from 400 to 1 million,online query speed increased about 10 to 20 times based on PB level data.The results verified the efficiency of the scheme.

Key words: Hadoop/HBase, ElasticSearch, Linux virtual server, massive data, real-time

中图分类号:

TP311

董长青, 任女尔, 张庆余, 田玉靖. 基于HBase+ElasticSearch的海量交通数据实时存取方案设计[J]. 大数据, 2017, 3(1): 80-89.

Changqing DONG, Nver REN, Qingyu ZHANG, Yujing TIAN. Design scheme of massive traffic data real-time access based on HBase and ElasticSearch[J]. Big Data Research, 2017, 3(1): 80-89.

图/表 8

图1

图2

图3

图4

图5

图6

图7

图8

参考文献 9

[1]	王颐帅 . 基于LVS 的服务器负载均衡技术[J]. 计算机系统应用, 2014,23(7): 252-255.
	WANG Y S . Server load balancing architecture based on LVS[J]. Computer Systems ＆ Applications, 2014,23(7): 252-255.
[2]	刘敏娜, 张继涛 . 基于LVS+KEEPALIVED的高可用负载均衡研究与应用[J]. 自动化技术与应用, 2014,33(11): 22-27.
	LIU M N , ZHANG J T . The study and application of based on the LVS +KEEPALIVED high avaliablility load balance[J]. Techniques of Automation and Applications, 2014,33(11): 22-27.
[3]	陆婷, 房俊, 乔彦克 . 基于HBase 的交通流数据实时存储系统[J]. 计算机应用, 2015,35(1): 103-107,135.
	LU T , FANG J , QIAO Y K . HBase-based real-time storage system for traffic stream data[J]. Journal of Computer Applications, 2015,35(1): 103-107,135.
[4]	葛微, 罗圣美, 周文辉 . HiBase:一种基于分层式索引的高效HBase 查询技术与系统[J]. 计算机学报, 2015,38(35): 1-15.
	GE W , LUO S M , ZHOU W H . HiBase:a hierarchical indexing mechanism and system for efficient HBase query[J]. Chinese Journal of Computers, 2015,38(35): 1-15.
[5]	白俊, 郭贺彬 . 基于ElasticSearch的大日志实时搜索的软件集成方案研究[J]. 吉林师范大学学报（自然科学版）, 2014(1): 85-87.
	BAI J , GUO H B . The design of software integration for big log data real time search based on ElasticSearch[J]. Jilin Normal University Journal(Natural Science Edition), 2014(1): 85-87.
[6]	钟雨, 黄向东, 刘丹 ,等. 大规模装备监测数据的NoSQL 存储方案[J]. 计算机集成制造系统, 2013,19(12): 3008-3016.
	ZHONG Y , HUANG X D , LIU D ,et al. NoSQL storage solution for massive equipment monitoring data management[J]. Computer Integrated Manufacturing Systems, 2013,19(12): 3008-3016.
[7]	SFAKIANAKIS G ， PATLAKAS I ， NTARMOS N ,et al. Interval indexing and querying on key-value cloud stores[C]// The 29th IEEE International Conference on Data Engineering (ICDE)， April 8-12,2013, Brisbane,Australia. New Jersey: IEEE Press, 2013: 805-816.
[8]	苏命峰, 陈文芳, 李仁发 . LVS 集群负载调度算法研究[J]. 长沙大学学报, 2012(5): 72-74.
	SU M F , CHEN W F , LI R F . Research on LVS cluster load scheduling algorithm[J]. Journal of Changsha University, 2012(5): 72-74.
[9]	BAI J . Feasibility analysis of big log data real time search based on HBase and ElasticSearch[C]// 2013 Ninth International Conference on Natural Computation (ICNC)， January 28-31,2013, San Diego,USA. New Jersey: IEEE Press, 2013: 1166-1170.

基于HBase+ElasticSearch的海量交通数据实时存取方案设计

Design scheme of massive traffic data real-time access based on HBase and ElasticSearch

在线阅读

PDF下载

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 9

相关文章 3

Metrics

推荐阅读 0

[1]	范良, 张晓蓉, 吴亚东, 陈呈, 王昉. 一种基于sort-last架构的非结构网格并行体可视化算法[J]. 大数据, 2021, 7(4): 117-129.
[2]	王腾蛟, 李喜莲. 让大数据分析更可信[J]. 大数据, 2017, 3(2): 31-37.
[3]	姚前, 谢华美, 景志刚, 胡青青, 司恩哲. 基于数据挖掘的个人征信系统异常查询实时监测模型及其应用[J]. 大数据, 2016, 2(4): 83-92.