电网数据存储技术选型的研究

doi:10.11959/j.issn.1000-0801.2018134

Abstract

Abstract:

Unstructured data and real-time data were storaged by state grid company’s big data platform distributed file system (based on HDFS optimized package).Based on HDFS optimized packaging,HDFS was characterized by high fault tolerance,low cost and high throughput.It was suitable for large-scale data set applications and effectively supports large file storage.The unstructured data storage and real-time data storage was introduced,and the selection of grid data storage technology was studied.

Key words: data center, Hadoop, Hadoop distributed file system, HBase

CLC Number:

TP392

Chengling LI,Yuxiang ZHENG,Yiqi HONG,Wen LI,Huimin GUO. Research on the selection of grid data storage technology[J]. Telecommunications Science, 2018, 34(5): 183-191.

Figures/Tables 20

References 11

[1]	GEORGE L . HBase 权威指南[M]. 代志远,刘佳,蒋杰,译. 北京: 人民邮电出版社, 2013.
	GEORGE L . HBase:the definitive guide[M]. Translated by DAI Z Y,LIU J,JIANG J. Beijing: Posts ＆ Telecom Press, 2013.
[2]	蔡斌, 陈湘萍 . Hadoop技术内幕:深入解析Hadoop Common和 HDFS 架构设计与实现原理[M]. 北京: 机械工业出版社, 2013.
	CAI B , CHEN X P . Hadoop internals:in-depths study of common and HDFS[M]. Beijing: China Machine Press, 2013.
[3]	孟鑫, 马延辉, 李立松 . HBase企业应用开发实战[M]. 北京: 机械工业出版社, 2014.
	MENG X , MA Y H , LI L S . Enterprise application development with HBase[M]. Beijing: China Machine Press, 2014.
[4]	皮雄军 . NoSQL 数据库技术实战[M]. 北京: 清华大学出版社, 2015.
	PI X J . NoSQL database technology combat[M]. Beijing: Tsinghua University Press, 2015.
[5]	DIMIDUK N , KHURANA A . HBase实战[M]. 谢磊,译.北京: 人民邮电出版社, 2013.
	DIMIDUK N , KHURANA A . HBase in action[M]. Translated by XIE L. Beijing: Posts ＆ Telecom Press, 2013.
[6]	蒋燚峰 . HBase管理指南[M]. 北京: 人民邮电出版社, 2013.
	JIANG Y F . HBase administration cookbook[M]. Beijing: Posts＆ Telecom Press, 2013.
[7]	SHRIPARV S . Learning HBase[M]. 周彦伟,蒲聪,蒋杰,译. 北京: 电子工业出版社, 2015.
	SHRIPARV S . Learning HBase[M]. Translated by ZHOU Y W,LOU S,PU C. Beijing: Publishing House of Electronics Industry, 2015.
[8]	董西成 . Hadoop技术内幕:深入解析MapReduce架构设计与实现原理[M]. 北京: 机械工业出版社, 2013.
	DONG X C . Hadoop internals:in-depths study of MapReduce[M]. Beijing: China Machine Press, 2013.
[9]	GROVER M , MALASKA T , SEIDMAN J . Hadoop应用架构[M]. 郭文超,译. 北京: 人民邮电出版社, 2017.
	GROVER M , MALASKA T , SEIDMAN J . Hadoop application architecture[M]. Translated by GUO W C. Beijing: Posts ＆Telecom Press, 2017.
[10]	王雪迎 . Hadoop 构建数据仓库实践[M]. 北京: 清华大学出版社, 2017.
	WANG X Y . Practice of Hadoop data warehouse[M]. Beijing: Tsinghua University Press, 2017.
[11]	WHITE T . Hadoop权威指南:大数据的存储与分析(第4版)[M]. 王海,华东,刘喻,等译. 北京: 清华大学出版社, 2017.
	WHITE T . Hadoop:the definitive guide[M]. Translated by WANG H,HUA D,LIU Y,et al. Beijing: Tsinghua University Press, 2017.

Metrics

Recommended 0

No Suggested Reading articles found!

节点名称	节点数据	客户端	类型	时间	加载	等待输入/输出操作进程占比	网络输入/MB	网络输出/MB
BG8S02	BG8S01	BG8S02	写	15 min 27 s	3.2%	43%	22	3.9
BG8S02	BG8S01	BG8S01	写	2 min 20 s	3.3%	76%	93	1.6
BG8S02	BG8S01	BG8S02	读	15 min 37 s	3.4%	42%	2.5	29
BG8S02	BG8S01	BG8S01	读	1 min 53 s	3.3%	81%	1.2	96

节点名称	节点数据	客户端	类型	时间	加载	等待输入/输出操作进程占比	网络输入/MB	网络输出/MB
BG8S02	BG8S01BG8S03	BG8S02	写	15 min 49 s	3.1%	26%	22	2.3
BG8S02	BG8S01 BG8S03	BG8S01	写	4 min	2.6%	39%	37	2.6
BG8S02	BG8S01 BG8S03	BG8S03	写	2 min 22 s	3.3%	42%	51	3.1
BG8S02	BG8S01 BG8S03	BG8S02	读	15 min 33 s	3.9%	23%	2.8	32
BG8S02	BG8S01 BG8S03	BG8S01	读	8 min 56 s	2.9%	32%	3.9	36
BG8S02	BG8S01 BG8S03	BG8S03	读	13 min 3 s	1.8%	26%	4.2	57

节点名称	节点数据	客户端	类型	时间	加载	等待输入/输出操作进程占比	网络输入/MB	网络输出/MB
BG8S02	BG8S01 BG8S03 BG8S04	BG8S02	写	15 min 49 s	3.2%	17%	29	2.9
BG8S02	BG8S01 BG8S03 BG8S04	BG8S01	写	4 min 18 s	3.5%	28%	32	2.5
BG8S02	BG8S01 BG8S03 BG8S04	BG8S03	写	2 min 54 s	3.6%	36%	26	3.6
BG8S02	BG8S01 BG8S03 BG8S04	BG8S04	写	2 min 50 s	2.8%	36%	43	3.1
BG8S02	BG8S01 BG8S03 BG8S04	BG8S02	读	15 min 44 s	2.1%	16%	4.2	30
BG8S02	BG8S01 BG8S03 BG8S04	BG8S01	读	13 min 21 s	3.5%	25%	3.9	35
BG8S02	BG8S01 BG8S03 BG8S04	BG8S03	读	10 min 30 s	3.9%	22%	2.8	37
BG8S02	BG8S01 BG8S03 BG8S04	BG8S04	读	15 min 20 s	3.4%	18%	3.2	31

节点名称	节点数据	客户端	响应	时间	加载	等待输入/输出操作进程占比	网络输入/MB	网络输出/MB
BG8S02	BG8S01 BG8S03 BG8S04	BG8S02	1	15 min 49 s	2.9%	16%	29	3.5
BG8S02	BG8S01 BG8S03 BG8S04	BG8S01	1	4 min 18 s	3.6%	25%	38	4.2
BG8S02	BG8S01 BG8S03 BG8S04	BG8S03	1	2 min 54 s	3.1%	31%	45	3.8
BG8S02	BG8S01 BG8S03 BG8S04	BG8S04	1	2 min 50 s	2.8%	32%	45	2.9
BG8S02	BG8S01 BG8S03 BG8S04	BG8S02	2	16 min 54 s	2.9%	14%	31	3.1
BG8S02	BG8S01 BG8S03 BG8S04	BG8S01	2	17 min 1 s	3.6%	14%	30	3.6
BG8S02	BG8S01 BG8S03 BG8S04	BG8S03	2	17 min 9 s	3.4%	14%	30	3.2
BG8S02	BG8S01 BG8S03 BG8S04	BG8S04	2	16 min 46 s	3.6%	15%	31	3.4
BG8S02	BG8S01 BG8S03 BG8S04	BG8S02	3	17 min 45 s	3.5%	14%	29	4.5
BG8S02	BG8S01 BG8S03 BG8S04	BG8S01	3	19 min 6 s	3.2%	13%	25	4.2
BG8S02	BG8S01 BG8S03 BG8S04	BG8S03	3	19 min 10 s	3.7%	13%	25	3.1
BG8S02	BG8S01 BG8S03 BG8S04	BG8S04	3	19 min 51 s	3.9%	13%	25	2.9

节点名称	节点数据	客户端	响应	时间	加载	等待输入/输出操作进程占比	网络输入/MB	网络输出/MB
BG8S02	BG8S01 BG8S03 BG8S04	BG8S02	1	15 min 22 s	3.2%	14%	4.2	30
BG8S02	BG8S01 BG8S03 BG8S04	BG8S01	1	13 min 21 s	2.6%	16%	4.7	31
BG8S02	BG8S01 BG8S03 BG8S04	BG8S03	1	10 min 30 s	2.9%	18%	4.8	33
BG8S02	BG8S01 BG8S03 BG8S04	BG8S04	1	15 min 20 s	3.5%	14%	3.9	30
BG8S02	BG8S01 BG8S03 BG8S04	BG8S02	2	15 min 22 s	2.9%	14%	3.8	30
BG8S02	BG8S01 BG8S03 BG8S04	BG8S01	2	11 min 42 s	2.8%	17%	4.4	32
BG8S02	BG8S01 BG8S03 BG8S04	BG8S03	2	7 min 2 s	2.5%	21%	3.7	39
BG8S02	BG8S01 BG8S03 BG8S04	BG8S04	2	9 min 16 s	2.9%	19%	3.1	41
BG8S02	BG8S01 BG8S03 BG8S04	BG8S02	3	15 min 35 s	3.5%	14%	3.9	30
BG8S02	BG8S01 BG8S03 BG8S04	BG8S01	3	3 min 42 s	3.6%	25%	3.8	56
BG8S02	BG8S01 BG8S03 BG8S04	BG8S03	3	2 min 29 s	3.4%	26%	4.1	59
BG8S02	BG8S01 BG8S03 BG8S04	BG8S04	3	2 min 5 s	2.9%	26%	3.7	59

Research on the selection of grid data storage technology

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 20

References 11

Related Articles 15

Metrics

Recommended 0

测试	工作节点数/个	总数据量/万	总费时/s	平均总速率/(万·s^-1)	加载	等待输入/输出操作进程占比
1	1	1 000	454.5	2.2	3.9%	1.7%
2	2	1 000	238	4.2	3.8%	1.5%
3	3	1 000	153.8	6.5	3.6%	1.4%

测试	列族名长度	列名长度	平均总速率/(万·s^-1)	加载	等待输入/输出操作进程占比	网络输入/MB	网络输出/MB
1	1	1	6.3	4.2%	1.6%	11.1	1.23
2	2	1	6.1	3.9%	1.4%	10.7	1.12
3	3	1	5.7	3.3%	1.9%	9.2	1.07
5	1	2	5.9	3.7%	1.1%	9.4	1.09
6	1	3	5.7	3.1%	1.1%	9.3	1.06

测试	批量操作数/个	平均总速率/(万·s^-1)	加载	等待输入/输出操作进程占比	网络输入/MB	网络输出
1	1	6.6	2.4%	1.1%	11.7	698 KB
2	100	13.7	2.6%	1.4%	25.8	1.15 MB
3	1 000	15.2	2.3%	1.5%	29.9	1.38 MB
4	10 000	12.4	3.9%	1.8%	23.8	1.59 MB

测试	批量操作数/个	平均总速率/(万·ms^-1)	加载	等待输入/输出操作进程占比	网络输入/KB	网络输出/KB
1	1	0.2	1.2%	0.5%	1.3	986
2	100	0.06	1.9%	0.1%	1.7	693
3	1 000	0.05	1.7%	0.1%	1.3	985
4	10 000	0.06	1.9%	0.1%	587	678

测试	工作节点数/个	表数量/个	单表region数量/个	压缩前数据大小/MB	压缩后数据大小/MB	压缩比
1	1	1	50	361.2	216.8	39.975%

[1]	Kaihui GAO, Dan LI. Data center networks with performance guarantee: a survey [J]. Telecommunications Science, 2023, 39(6): 1-21.
[2]	Shengchao XU, Maohua XIONG, Tianqi ZHOU. Approach of glowworm swarm optimization based virtual machine placement [J]. Telecommunications Science, 2022, 38(3): 172-182.
[3]	Dan XU, Yu ZENG, Weiye MENG, Lika LI. AI-enabled 5G energy-saving technology [J]. Telecommunications Science, 2021, 37(5): 32-41.
[4]	Fan ZHANG, Guangwei XIE, Wei GUO, Hongchao HU, Ruyun ZHANG, Wenyan LIU. Key technologies and implementation methods of endogenous safety and security cloud data center based on mimic architecture [J]. Telecommunications Science, 2021, 37(3): 39-48.
[5]	Jianglong WANG,Bo LEI,Yunpeng XIE,Qi HE,Yunhe LI. Key technology of data center network for cloud network convergence [J]. Telecommunications Science, 2020, 36(4): 125-135.
[6]	ZHU Hong,SHAO Qi. Analysis of value-added services of power communication network in energy internet [J]. Telecommunications Science, 2019, 35(6): 50-59.
[7]	Cui ZHONG,Lei WANG,Xing LUO. SDN solutions for cloud data center [J]. Telecommunications Science, 2018, 34(7): 15-22.
[8]	Lei SHEN,Fang WANG,Lihua WU. Key point of NFV hardware resource pool planning and construction [J]. Telecommunications Science, 2018, 34(6): 107-114.
[9]	Yuan WANG,Hao JIANG,Ming WU,Donggui YAO,Yi ZHANG,Shuwen YI,Hai WANG,Jing WU. A distributed high efficiency similarity matrix computation method based on users’ mobile network access location [J]. Telecommunications Science, 2018, 34(5): 26-38.
[10]	Wei PENG,Xiaodong HE,Xiaoming LU,Chuanxin JI. Optimization and application of the carrier DNS data analysis technology [J]. Telecommunications Science, 2018, 34(3): 138-144.
[11]	Xiaosong LIU,Ting HUANG. Application of inspection robot in internet data center intelligent operation and maintenance [J]. Telecommunications Science, 2018, 34(12): 132-137.
[12]	Haiyong XU,Yan HUANG. Applications of a big data platform based on PaaS [J]. Telecommunications Science, 2018, 34(1): 148-157.
[13]	Chukun LU,Fangfang YAN,Dong LI. An algorithm for embedding oversubscribed virtual data center [J]. Telecommunications Science, 2017, 33(7): 76-87.
[14]	Yongming YAN,Liang ZUO,Bin XU,Haifeng XU,Kechuang YE. Research and implementation of network architecture of LSN DCI EVPN VxLAN [J]. Telecommunications Science, 2017, 33(6): 171-178.
[15]	Teng MA,Yuxiang HU. Fast and live virtual machine migration in data center networks [J]. Telecommunications Science, 2017, 33(6): 64-72.