基于HBase数据分类的压缩策略选择方法

doi:10.11959/j.issn.1000-436x.2016068

Abstract

Abstract:

Most of the current compression strategies selection methods for HBase data did not consider whether the data was cold or hot. Besides, problem of incompleteness and unreliability existed during selection process. To address the problems above, a compression strategies selection method based on classification of HBase data was put forward. HBase data was classified into cold and hot data according to the access frequency of each data file and an access level would be designated to each file. On this base, an evaluation layer was added and a compression strategies selection method based on access level with integration of neighbor sector and statistic column based selection methods. Simulation experiments and results demonstrate that the proposed compression strategies selection method based on classification of HBase data can not only save storage space but also greatly improve the query performance of HBase data.

Key words: data compression, HBase, compression strategies selection method, cold and hot data

Hai-yan WANG,Cai-hang FU. Compression strategies selection method based on classification of HBase data[J]. Journal on Communications, 2016, 37(4): 12-22.

Figures/Tables 8

References 20

[1]	程学旗，靳小龙，王元卓，等．大数据系统和分析技术综述[J]．软件学报， 2014,25(9): 1889-1908. CHENG X Q , JIN X L , WANG Y Z , et al. Survey on big data system and analytic technology[J]. Journal of Software, 2014,25(9): 1889-1908.
[2]	郭嘉凯．如何存储“冷数据”[J]．软件和信息服务， 2013,23(10): 58-59. GUO J K . How to store“cloud data”[J]. Software and Information Service, 2013,23(10): 58-59.
[3]	王振玺，乐嘉锦，王梅，等．列存储数据区级压缩模式与压缩策略选择方法[J]．计算机学报， 2010,33(8): 1524-1530. WANG Z X , LE J J , WANG M , et al. Sector-based compression and compression strategy selection method for column stores[J]. Chinese Journal of Computers, 2010,33(8): 1524-1530.
[4]	TRONDHEIM N , STONEBRAKER M , ABADI D J , et al. C-store-A column-oriented DBMS[C]// The 31st VLDB Conference. Trondheim, Norway, c2005:553-564.
[5]	YAN K , XIE M Y , ZHU H . Fixed-length string compression for direct operations in column-oriented databases[C]// 2013 Ninth International Conference on Natural Computation(ICNC). Shenyang,China, c2013:1171-1176.
[6]	TOMMY S , SANJAY M . On compressing data in wireless sensor networks for energy efficiency and real time delivery[J]. Distributed and Parallel Databases, 2013,31(2):151-182.
[7]	MEHLA U S , DASGUPTA K S . Hamming distance based reordering and columnwise bit stuffing with difference vector: a etter scheme for test data compression with run length based codes[C]// VLSID 23rd In-ternational Conference on VLSI Design. India,Bangalore, c2010:33-38.
[8]	丘建平，张广艳，舒继武． DMStone：一个分级存储系统性能测试工具[J]．软件学报， 2012,23(4): 987-995. QIU J P , ZHANG G Y , SHU J W . DMStone: a tool for evaluating hie-rarchical storage management systems[J]. Journal of Software, 2012,23(4): 987-995.
[9]	LEVANDOSKI J J , LARSON P A , STOICA R . Identifying hot and cold data in main-memory databases[C]// 2013 IEEE 29th International Conference on Data Engineering(ICDE). Australia, Brisbane, c2013:26-37.
[10]	GAO H B , WANG D F . LSD2H: a novel storage method of linked sensor data based on HBase[C]// 2014 10th International Conference on Semantics, Knowledge and Grids(SKG). Beijing, China, c2014:116-119.
[11]	朱敏，程佳，柏文阳．一种基于HBase的RDF 数据存储模型[J]．计算机研究与发展， 2013,50(Suppl.): 23-31. ZHU M , CHENG J , BAI W Y . A storage model for RDF data based on HBase[J]. Journal of Computer Research and Development, 2013,50(Suppl.): 23-31.
[12]	葛微，罗圣美，周文辉，等． HiBase：一种基于分层式索引的高效HBase查询技术与系统[C]// 2014中国大数据技术大会．中国，北京， c2014 GE W , LUO S M , ZHOU W H , et al. HiBase: a hierarchical ndexing mechanism and system for efficient HBase query[C]// Big Data Tech-nology Conference 2014, China,Beijing, c2014
[13]	IDREOSS.Self-organizing tuple reconstruction in column-stores[C]// Proceedings of the SIGMOD. Providence, Rhode Island,USA, c2009:297-308.
[14]	宁正元，王李近．统计与决策常用算法及其实现[M]．北京：清华大学出版社， 2009:260-345. NING Z Y , WANG L J . Statistical and decision algorithm and its rea-lization[M]. Beijing： Tsinghua University Press， 2009:260-345.
[15]	李光，王亚东，苏小红．隐私保持的决策树分类挖掘[J]．电子学报， 2010,38(1): 204-212. LI G , WANG Y D , SU X H . Privacy preserving data mining on deci-sion tree[J]. Acta Electronica Sinica, 2010,38(1): 204-212.
[16]	崔颖安，李雪，王志晓，等．在线社交媒体数据抽样方法的比较研究[J]．计算机学报． 2014,37(8): 1859-1876. CUI Y A , LI X , WANG Z X , et al. A comparison on methodologies of sampling online social media[J]. Chinese Journal of Computer, 2014,37(8): 1859-1876.
[17]	WEI Y , YONG W F . A Cost-effective and reliable cloud storage[C]// 2014 IEEE International Conference on Cloud Computing. Anchorage, AK, c2014:938-939.
[18]	TPC[EB/OL]. , 2011.
[19]	龙礡涛．列存储数据仓库中压缩技术的研究与实现[D]．上海：东华大学， 2013. LONG B T . Research and implementation of compression technology in column-oriented data warehouse[D]. Shanghai: Donghua University, 2013.
[20]	NYAMAGWA M , LIU J , UEHARA T . Cloud foren: a novel framework for digital forensics in cloud computing[J]. Journal of Harbin Institute of Technology, 2014,21(6):39-45.

Metrics

Recommended 0

No Suggested Reading articles found!

列	Vlevel	列	Vlevel
1	沉默数据	9	沉默数据
2	不活跃数据	10	沉默数据
3	沉默数据	11	活泼数据
4	沉默数据	12	沉默数据
5	沉默数据	13	沉默数据
6	沉默数据	14	沉默数据
7	沉默数据	15	沉默数据
8	次热点数据	16	沉默数据

Compression strategies selection method based on classification of HBase data

RichHTML

PDF

Knowledge

Cited

Abstract

Cite this article

share this article

Figures/Tables 8

References 20

Related Articles 8

Metrics

Recommended 0

[1]	Yimin MAO, Qianhu DENG, Zhigang CHEN. Parallel association rules incremental mining algorithm based on information entropy and genetic algorithm [J]. Journal on Communications, 2021, 42(5): 122-136.
[2]	. DTW-based multi-wavelet data compression algorithm for wireless sensor networks [J]. Journal on Communications, 2014, 35(8): 12-94.
[3]	Si-wang ZHOU,Lan LI. DTW-based multi-wavelet data compression algorithm for wireless sensor networks [J]. Journal on Communications, 2014, 35(8): 86-94.
[4]	Peng JIANG,Jian-feng WU,Bin WU,Lin-xi DONG,Da WANG. Data compression method for wireless sensor networks based on adaptive optimal zero suppression [J]. Journal on Communications, 2013, 34(2): 1-7.
[5]	Wei-ling CHANG,Xiao-chun YUN,Bin-xing FANG,Shu-peng WANG. HitIct:Chinese corpus for the evaluation of lossless compression algorithms [J]. Journal on Communications, 2009, 30(3): 42-47.
[6]	Tie-jun ZHU,Ya-ping LIN,Si-wang ZHOU,Xiao-long XU. Adaptive multiple-modalities data compression algorithm using wavelet for wireless sensor networks [J]. Journal on Communications, 2009, 30(3): 48-53.
[7]	Guo-qing WU,Hong CHEN. Wavelet based data reduction and compression scheme for high volume scientific data [J]. Journal on Communications, 2008, 29(11A): 91-95.
[8]	Sheng-tian YANG,Pei-liang QIU. Several practical coding schemes for a block of integers [J]. Journal on Communications, 2005, 26(7): 1-7.