基于Hadoop+MPP架构的电信运营商网络数据共享平台研究

doi:10.3969/j.issn.1000-0801.2014.04.020

Abstract

Abstract:

Based on using distributed computing and distributed storage technology, the telecom operators' network data sharing platform based on Hadoop +MPP architectures was studied. By integrating various distributed network management data, the network basic data platform, application platform and unified control platform were built.

Key words: distributed, big data, Hadoop, massively parallel processing, data sharing

Huang Xin,Xinghui Yi,Zhenyu Chen. Design of Telecom Operators' Network Data Sharing Platform Based on Hadoop+MPP Architecture[J]. Telecommunications Science, 2014, 30(4): 135-145.

Figures/Tables 15

References 9

1	王峰，雷葆华． Hadoop分布式文件系统的模型分析．电信科学， 2010,26(12)
2	程莹，张云勇，徐雷．等基于Hadoop 及关系型数据库的海量数据分析研究．电信科学， 2010,26(11)
3	田秀霞，周耀君，周耀君等．基于Hadoop架构的分布式计算和存储应用．上海电力学学报， 2011,27(1)
4	陈梦杰，陈勇旭，贾益斌等．基于Hadoop的大数据查询系统简述．计算机与数字工程， 2013(12)
5	单士华，曹社香．基于Hadoop处理大数据分析．创新科技， 2013(12)
6	张雨，蔡鑫，李爱民等．分布式文件系统与MPP 数据库的混搭架构在电信大数据平台中的应用．电信科学， 2013,29(11)
7	吉增瑞．基于MPP结构的计算机平台数据库管理系统设计技术探讨．计算机工程与科学， 1998(Z1)
8	基于Hadoop的大规模数据处理系统． http://wenku.baidu.com/link url=ZNmtQsNXNogXR6 MDZKFT8cri7_eINjCix8I0X_4aucr-GC2X3JQfg-TlYfOPJfY4q93O6ERd_HXQoxZCkvtFXTUUOS-FqITzcneblkCXhSV4G
9	李建中，刘显敏．大数据的一个重要方面：数据可用性．计算机研究与发展， 2013,50(6)

Metrics

Recommended 0

No Suggested Reading articles found!

序号	数据类别	计算处理的实时性
1	设备性能	准实时
2	信令XDR/DPI	准实时
3	MR测量数据	每日
4	拨测数据	准实时
5	路测数据	每日
6	网络日志数据	每日
7	话单数据	每日
8	资源数据	准实时
9	工单数据	每日
10	告警数据	每日
11	终端数据	按需更新
12	用户属性	按需更新

	30天基础明细数据Hadoop存储量推导公式
存储份数	采用Hadoop架构数据存储3份，系数为3
压缩比	采用Hadoop架构数据存储压缩比为1_:3
索引比	原始数据入库增加索引等额外开销20%
存储周期	数据存储时间为30天
算法	基础明细数据层数据=一天基础数据量D×存储份数/压缩比×索引比×存储时长

	30天基础明细数据MPP存储量推导公式
存储份数	采用MPP架构数据存储2份，系数为2
压缩比	采用MPP架构数据存储压缩比为1:5
索引比	原始数据入库增加索引等额外开销20%
存储周期	数据存储时间为30天
算法	基础明细数据层数据=一天基础数据量D×存储份数/压缩比×索引比×存储时长

数据层次	汇总比例	冗余系数	汇总系数	存储时长/天
入库前基础数据	1
细粒度汇总数据	1:10	0.75	7.50%	300
轻度汇总数据	1:75	0.75	1.00%	300

数据层次	汇总比例	冗余系数	汇总系数	存储时长/天
入库前基础数据	1
应用汇总数据	1_:1 800	0.75	0.04%	1 080

Design of Telecom Operators' Network Data Sharing Platform Based on Hadoop+MPP Architecture

RichHTML

PDF下载

Knowledge

Cited

Abstract

Cite this article

share this article

Figures/Tables 15

References 9

Related Articles 15

Metrics

Recommended 0

[1]	Zehua GUO, Haowen ZHU, Tongwen XU. Network modal innovation for distributed machine learning [J]. Telecommunications Science, 2023, 39(6): 44-51.
[2]	Lu HAN, Weiyu CHEN, Fei ZHANG, Jianfeng HE, Huaizhen SU. Big data classification method of non relational distributed submission information under differentiated requirements [J]. Telecommunications Science, 2023, 39(6): 114-121.
[3]	Xue HAN. Damage toughness assessment method of power backbone communication network based on power big data [J]. Telecommunications Science, 2023, 39(5): 136-143.
[4]	Yudi SUN. A prediction model of massive 5G network users’ revisit behavior based on telecom big data [J]. Telecommunications Science, 2023, 39(2): 157-162.
[5]	Aihua LI, Xiaobo WU, Chao CHEN, Bin WEI, Yuanyuan SHI. Big data intelligent analysis technology for 5G network [J]. Telecommunications Science, 2022, 38(8): 129-139.
[6]	Gengsheng HUANG, Yuhong HUANG, Manxue GUO, Jianping ZHENG, Xin GE. Architecture of a new infrastructure for data service in DT era [J]. Telecommunications Science, 2022, 38(7): 138-145.
[7]	Nan CHEN, Jianjun ZHAO, Ping ZHONG, Yongjun HUANG, Tian CHEN. Research on cloud native based distributed IoT operating system architecture [J]. Telecommunications Science, 2022, 38(7): 146-156.
[8]	Feng BI. Research on the enhancement technology based on the 5G wireless access and wireless backhaul integration [J]. Telecommunications Science, 2022, 38(5): 173-178.
[9]	Panpan LI, Zhengxia XIE, Guangxue YUE, Xin LIU. Research progress and trends of deep learning based wireless communication receiving method [J]. Telecommunications Science, 2022, 38(2): 1-17.
[10]	Haibo ZHAO, Zhijun XIANG, Linsong XIAO. A big data framework for short-term power load forecasting using heterogenous data [J]. Telecommunications Science, 2022, 38(12): 103-111.
[11]	Qin LI, Weiyuan LI, Xiaowen SUN, Yushuang HU, Tao SUN. Thinking of native artificial intelligence in 6G networks [J]. Telecommunications Science, 2021, 37(9): 20-29.
[12]	Zhiyong LIU, Zhongjiang HE, Yilong RUAN, Junfeng SHAN, Chao ZHANG. Big data security features and operation practices [J]. Telecommunications Science, 2021, 37(5): 160-169.
[13]	Feiyang LIU, Kun LI, Fei SONG, Huachun ZHOU. Construction of DDoS attacks malicious behavior knowledge base construction [J]. Telecommunications Science, 2021, 37(11): 17-32.
[14]	Yunhe YU, Jun SUN. Research on resource allocation algorithm of centralized and distributed Q-learning in machine communication [J]. Telecommunications Science, 2021, 37(11): 41-50.
[15]	Shuting LI,Jinke LIU,Na CHEN. Design and research of clearing-settlement platform based on blockchain technology [J]. Telecommunications Science, 2020, 36(9): 84-93.