基于HBase+ElasticSearch的海量交通数据实时存取方案设计

doi:10.11959/j.issn.2096-0271.2017010

Abstract

Abstract:

Traffic data has the characteristics of massive and real-time,and its massive data acquisition,storage and retrieval has become a key issue in the vehicle remote monitoring platform.According to the study of these problems,the cluster technology of LVS was used to solve the data acquisition load balance,the queue cache model was used to solve I/O delay,and HBase distributed data storage scheme was used to solve the massive data storage.HBase integration ElasticSearch,which was aimed to solve the real-time online data processing problems of Hadoop,was designed to build a hierarchical index.Through the design and implementation of the key technologies,the number of vehicle monitoring had been promoted from 400 to 1 million,online query speed increased about 10 to 20 times based on PB level data.The results verified the efficiency of the scheme.

Key words: Hadoop/HBase, ElasticSearch, Linux virtual server, massive data, real-time

CLC Number:

TP311

Changqing DONG, Nver REN, Qingyu ZHANG, Yujing TIAN. Design scheme of massive traffic data real-time access based on HBase and ElasticSearch[J]. Big Data Research, 2017, 3(1): 80-89.

Figures/Tables 8

References 9

[1]	王颐帅 . 基于LVS 的服务器负载均衡技术[J]. 计算机系统应用, 2014,23(7): 252-255.
	WANG Y S . Server load balancing architecture based on LVS[J]. Computer Systems ＆ Applications, 2014,23(7): 252-255.
[2]	刘敏娜, 张继涛 . 基于LVS+KEEPALIVED的高可用负载均衡研究与应用[J]. 自动化技术与应用, 2014,33(11): 22-27.
	LIU M N , ZHANG J T . The study and application of based on the LVS +KEEPALIVED high avaliablility load balance[J]. Techniques of Automation and Applications, 2014,33(11): 22-27.
[3]	陆婷, 房俊, 乔彦克 . 基于HBase 的交通流数据实时存储系统[J]. 计算机应用, 2015,35(1): 103-107,135.
	LU T , FANG J , QIAO Y K . HBase-based real-time storage system for traffic stream data[J]. Journal of Computer Applications, 2015,35(1): 103-107,135.
[4]	葛微, 罗圣美, 周文辉 . HiBase:一种基于分层式索引的高效HBase 查询技术与系统[J]. 计算机学报, 2015,38(35): 1-15.
	GE W , LUO S M , ZHOU W H . HiBase:a hierarchical indexing mechanism and system for efficient HBase query[J]. Chinese Journal of Computers, 2015,38(35): 1-15.
[5]	白俊, 郭贺彬 . 基于ElasticSearch的大日志实时搜索的软件集成方案研究[J]. 吉林师范大学学报（自然科学版）, 2014(1): 85-87.
	BAI J , GUO H B . The design of software integration for big log data real time search based on ElasticSearch[J]. Jilin Normal University Journal(Natural Science Edition), 2014(1): 85-87.
[6]	钟雨, 黄向东, 刘丹 ,等. 大规模装备监测数据的NoSQL 存储方案[J]. 计算机集成制造系统, 2013,19(12): 3008-3016.
	ZHONG Y , HUANG X D , LIU D ,et al. NoSQL storage solution for massive equipment monitoring data management[J]. Computer Integrated Manufacturing Systems, 2013,19(12): 3008-3016.
[7]	SFAKIANAKIS G ， PATLAKAS I ， NTARMOS N ,et al. Interval indexing and querying on key-value cloud stores[C]// The 29th IEEE International Conference on Data Engineering (ICDE)， April 8-12,2013, Brisbane,Australia. New Jersey: IEEE Press, 2013: 805-816.
[8]	苏命峰, 陈文芳, 李仁发 . LVS 集群负载调度算法研究[J]. 长沙大学学报, 2012(5): 72-74.
	SU M F , CHEN W F , LI R F . Research on LVS cluster load scheduling algorithm[J]. Journal of Changsha University, 2012(5): 72-74.
[9]	BAI J . Feasibility analysis of big log data real time search based on HBase and ElasticSearch[C]// 2013 Ninth International Conference on Natural Computation (ICNC)， January 28-31,2013, San Diego,USA. New Jersey: IEEE Press, 2013: 1166-1170.

Metrics

Recommended 0

No Suggested Reading articles found!

Design scheme of massive traffic data real-time access based on HBase and ElasticSearch

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 9

Related Articles 3

Metrics

Recommended 0

[1]	Liang FAN, Xiaorong ZHANG, Yadong WU, Cheng CHEN, Fang WANG. A sort-last architecture based parallel volume visualization algorithm for unstructured grid [J]. Big Data Research, 2021, 7(4): 117-129.
[2]	Tengjiao WANG, Xilian LI. Making big data analysis more credible [J]. Big Data Research, 2017, 3(2): 31-37.
[3]	Qian YAO, Huamei XIE, Zhigang JING, Qingqing HU, Enzhe SI. Real-time data-mining-based anomaly inquiry monitoring model of personal credit reference system and it's application [J]. Big Data Research, 2016, 2(4): 83-92.