面向海量电子凭据的分层可扩展存储架构

doi:10.11959/j.issn.1000-436x.2019104

Abstract

Abstract:

The rise of e-commerce and other network services have greatly promoted the development of electronic bill service,while traditional data storage schemes can no longer satisfy the rapid access requirements of massive electronic bill data.To solve these problems,a hierarchical scalable storage architecture for massive electronic bill was proposed,which combined hash modular algorithm and consistent hash algorithm,and supported fast data locating.A horizontal expansion scheme based on hash modular algorithm was designed to reduce the amount of data that needed to migrate when adding or deleting data nodes.Besides,a data caching scheme based on hot data and a load balancing scheme based on access delay were designed and implemented,further improving the speed of data access.Finally,the experiments prove the effectiveness of proposed architecture and schemes.

Key words: massive data, electronic bill, data locating, data caching, load balancing

CLC Number:

TP311

Fenghua LI,Dingyan LI,Wei JIN,Zhu WANG,Yunchuan GUO,Kui GENG. Hierarchical scalable storage architecture for massive electronic bill[J]. Journal on Communications, 2019, 40(5): 79-87.

Figures/Tables 10

References 22

[1]	MAKRIS A , TSERPES K , ANDRONIKOU V ,et al. A classification of NoSQL data stores based on key design characteristics[J]. Procedia Computer Science, 2016,97: 94-103.
[2]	CORBELLINI A , MATEOS C , ZUNINO A ,et al. Persisting big data:the NoSQL landscape[J]. Information Systems, 2017,63: 1-23.
[3]	SIDDIQA A , KARIM A , GANI A . Big data storage technologies:a survey[J]. Frontiers of Information Technology and Electronic Engineering, 2017,18(8): 1040-1070.
[4]	DECANDIA G , HASTORUN D , JAMPANI M ,et al. Dynamo:Amazon’s highly available key-value store[J]. ACM SIGOPS Operating Systems Review, 2007,41(6): 205-220.
[5]	马友忠, 孟小峰 . 云数据管理索引技术研究[J]. 软件学报, 2015,26(1): 145-166.
	MA Y Z , MENG X F . Research on indexing for cloud data management[J]. Journal of Software, 2015,26(1): 145-166.
[6]	VICTOR Z , DIVYAKANT A , AMR E A . Caching at the web scale[J]. Proceedings of the VLDB Endowment, 2017(10): 2002-2005.
[7]	ASAF C , ASSAF E , MOHAMMAD A ,et al. Cliffhanger:scaling performance cliffs in web memory caches[C]// 13th USENIX Symposium on Networked Systems Design and Implementation. USENIX, 2016: 379-392.
[8]	JIANG Y . A survey of task allocation and load balancing in distributed systems[J]. IEEE Transactions on Parallel and Distributed Systems, 2016,27(2): 585-599.
[9]	LIU J , PACITTI E , VALDURIEZ P . A survey of scheduling frameworks in big data systems[J]. International Journal of Cloud Computing, 2018(7): 103-128.
[10]	AKRAM E , LARBI H , ABDERRAHIM M . The main characteristics of five distributed file systems required for big data:a comparative study[J]. Advances in Science,Technology and Engineering Systems Journal, 2017,2(4): 78-91.
[11]	邹立达, 李庆忠, 孔兰菊 ,等. 基于Chord的多租户索引机制研究[J]. 计算机学报, 2016,39(2): 270-285.
	ZOU L D , LI Q Z , KONG L J ,et al. Indexing mechanism of multi-tenant database based on Chord[J]. Chinese Journal of Computers, 2016,39(2): 270-285.
[12]	QAISAR J , AYESHA Z , MUHAMMAD A ,et al. Cache memory:an analysis on replacement algorithms and optimization techniques[J]. Mehran University Research Journal of Engineering and Technology, 2017,36(4): 831-840.
[13]	ZAIDENBERG N , GAVISH L , MEIR Y . New caching algorithms performance evaluation[C]// International Symposium on Performance Evaluation of Computer and Telecommunication Systems. 2015: 1-7.
[14]	CIDON A , EISENMAN A , ALIZADEH M ,et al. Dynacache:dynamic cloud caching[C]// USENIX Conference on Hot Topics in Cloud Computing. USENIX, 2015.
[15]	葛微, 罗圣美, 周文辉 ,等. HiBase:一种基于分层式索引的高效HBase查询技术与系统[J]. 计算机学报, 2016(1): 140-153.
	GE W , LUO S M , ZHOU W H ,et al. HiBase:a hierarchical indexing mechanism and system for efficient HBase query[J]. Chinese Journal of Computers, 2016(1): 140-153.
[16]	RATHORE N . Performance of hybrid load balancing algorithm in distributed web server system[J]. Wireless Personal Communications, 2018(101): 1233-1246.
[17]	REN G , JUEBO W . Dynamic load balancing strategy for cloud computing with ant colony optimization[J]. Future Internet, 2015,7(4): 465-483.
[18]	YANG Z X , ZHANG S Y , JI X Y ,et al. Research on cloud service quality control implementation based on improved load balance algorithm[J]. Journal of Computational Methods in Sciences and Engineering, 2018,18(3): 793-800.
[19]	HERBST N , AMIN A , ANDRZEJAK A ,et al. Online workload forecasting[C]// Self-Aware Computing Systems. 2017: 529-553.
[20]	邢屹 . 大规模键值分布式存储系统的设计与实现[D]. 成都:电子科技大学, 2013.
	XING Y . The design and implementation of a large-scale key-value distributed storage system[D]. Chengdu:University of Electronic Science and Technology of China, 2013.
[21]	佘楚玉, 温武少, 肖扬 ,等. 一种自适应文件系统元数据服务负载均衡策略[J]. 软件学报, 2017,28(8): 1952-1967.
	SHE C Y , WEN W S , XIAO Y ,et al. Adaptive load balancing strategy for file-system metadata service[J]. Journal of Software, 2017,28(8): 1952-1967.
[22]	JOANNA B , MACIEJ D . Comparing load-balancing algorithms for mapreduce under zipfian data skews[J]. Parallel Computing, 2018,72: 14-28.

Metrics

Recommended 0

No Suggested Reading articles found!

设备	CPU	内存	硬盘	操作系统	数据库
g₀	8核，i7 3.4 GHz	8 GB	500 GB，7200 rpm	Centos7，64位	—
g₁	8核，i7 3.4 GHz	8 GB	500 GB，7 200 rpm	Centos7，64位	—
g₂	8核，i7 3.4 GHz	8 GB	500 GB，7 200 rpm	Centos7，64位	—
node₁	4核，i7 3.4 GHz	4 GB	1 TB，7 200 rpm	Centos7，64位	MySQL 5.6，64位
node₂	4核，i7 3.4 GHz	8 GB	1 TB，7 200 rpm	Centos7，64位	MySQL 5.6，64位
node₃	4核，i7 3.4 GHz	4 GB	1 TB，7 200 rpm	Centos7，64位	MySQL 5.6，64位
node₄	4核，i7 3.4 GHz	8 GB	1 TB，7 200 rpm	Centos7，64位	MySQL 5.6，64位

Hierarchical scalable storage architecture for massive electronic bill

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 22

Related Articles 15

Metrics

Recommended 0

[1]	Mingchuan YANG, Guanchang XUE, Qingyi LI. Neighbor satellite load based low orbit satellite distributed routing algorithm [J]. Journal on Communications, 2021, 42(8): 43-51.
[2]	Aijing SUN, Shichang LI, Yicai ZHANG. WSN clustering routing algorithm based on PSO optimized fuzzy C-means [J]. Journal on Communications, 2021, 42(3): 91-99.
[3]	Yingxu LAI,Yewei PU,Jing LIU. Research on switch migration method based on minimum cost path [J]. Journal on Communications, 2020, 41(2): 131-142.
[4]	Bowen CHEN,Xiaodong FU,Yu LEI. Joint optimization method of failure probability and fiber-link load balancing in flexible bandwidth optical network [J]. Journal on Communications, 2019, 40(9): 136-144.
[5]	Guo CHEN,Weifeng ZHANG. ELAB:end-host-based congestion aware load balancing for data center network [J]. Journal on Communications, 2019, 40(3): 196-205.
[6]	Li LI,Guozhen SHI,Kui GENG,Xiuze DONG,Fenghua LI. Scheduling algorithm for stochastic job stream cipher service based on load balancing [J]. Journal on Communications, 2018, 39(6): 11-19.
[7]	Tao HU,Jianhui ZHANG,Weigong KONG,Sen YANG,Lujia CAO. Dynamic deployment algorithm for multi-controllers based on bidirectional matching in software defined networking [J]. Journal on Communications, 2018, 39(1): 159-169.
[8]	Tao HU,Jian-hui ZHANG,Teng MA,Wei ZHAO. Multi-controller balancing deployment strategy based on reliability evaluation in SDN [J]. Journal on Communications, 2017, 38(11): 188-198.
[9]	Bin CAO,Shi-chao XIA,Fang HE,Yun LI. Research of embedding algorithm for wireless network virtualization [J]. Journal on Communications, 2017, 38(1): 35-43.
[10]	Xiao-bo SHI,Yin ZHANG,Shan ZHAO,Deng-ming XIAO. Discrete multi-objective optimization of particle swarm optimizer algorithm for multi-agents collaborative planning [J]. Journal on Communications, 2016, 37(6): 29-37.
[11]	Bo ZHENG,Heng-yang ZHANG,Bao-liang WANG,Wei ZHAO. Load balancing geographic routing strategy for aeronautical ad hoc networks [J]. Journal on Communications, 2016, 37(12): 67-76.
[12]	Kuang-yu QIN,Chuan-he HUANG,Cai-hua WANG,Jiao-li SHI,Di WU,Xi CHEN. Balanced multiple controllers placement with latency and capacity bound in software-defined network [J]. Journal on Communications, 2016, 37(11): 90-103.
[13]	Xiao-jian SHEN,Zhi-gang CHEN,Li LIU. Load balancing multicast routing based on network coding in wireless mesh network [J]. Journal on Communications, 2015, 36(4): 90-95.
[14]	Yong-hong ZHU,En-jie DING. Method of node distribution for heterogeneous WMSN based on load balancing [J]. Journal on Communications, 2015, 36(10): 157-164.
[15]	Peng WANG,Ju-long LAN,Shu-qiao CHEN. Multipath traffic splitting algorithm based on adaptive granularity [J]. Journal on Communications, 2015, 36(1): 211-217.