基于OLAP技术的物联网客户数据平台构建

doi:10.11959/j.issn.2096-3750.2020.00141

Abstract

Abstract:

At this stage,the Internet of things (IoT) customer business is developing rapidly.The management problems of weak customer operation ability have gradually exposed in the IoT industry,resulting in the low AP (analytical processing) value of customers,the low profitability and high operating cost.Therefore,it is urgent to enhance the massive data analysis capabilities of the IoT to provide data guidance for business development.To improve the customer value,big data technology,multi-dimensional analysis technology and massive query technology were used to make portraits of IoT enterprise customers,and realize the construction of the customer data platform with refined and efficient operation services.It was introduced how to construct the data operation platform from the aspect of technology implementation.

Key words: big data, IoT customer operation, on line analytical processing (OLAP)

CLC Number:

TP39

Yue GUO,Yue LONG,Xun ZHANG,Yang QU. Construction of Internet of things customer data platform based on OLAP technology[J]. Chinese Journal on Internet of Things, 2020, 4(2): 122-128.

Figures/Tables 11

References 10

[1]	丁飞 . 物联网开放平台[M]. 北京: 电子工业出版社, 2018.
	DING F . Internet of things open platform[M]. Beijing: Publishing House of Electronics IndustryPress, 2018.
[2]	SANDY R , URI L , SEAN O,等 . Spark高级数据分析[M]. 北京: 人民邮电出版社, 2017.
	SANDY R , URI L , SEAN O ,et al. Advanced analysis with Spark[M]. Beijing: Posts and Telecom PressPress, 2017.
[3]	ISMAIL B I , GOORTANI E M , KARIM M B A ,et al. Evaluation of docker as edge computing platform[C]// IEEE Conference on Open Systems. IEEE, 2015: 130-135.
[4]	TAKASE W , NAKAMURA T , WATASE Y ,et al. A solution for secure use of Kibana and Elasticsearch in multi-user environment[J]. arXiv:1706.10040, 2017
[5]	苏敏坚 . 大数据是运营商复兴的“起搏器”[J]. 中国电信业, 2017(4): 40-41.
	SU M J . Big data is the pacemaker of operators’ renaissance[J]. China Telecommunication Trade, 2017(4): 40-41.
[6]	范铭 . 运营商进入调速换挡关键时期大数据创造新机遇[J]. 通信世界, 2017(16):54.
	FAN M . Operators enter the key period of speed shift and big data create new opportunities[J]. Communications World, 2017(16):54.
[7]	JD-Presto研发团队.Presto技术内幕[M]. 北京: 电子工业出版社, 2016.
	JD-Presto研发团队. Presto technology insider[M]. Beijing: Publishing House of Electronics IndustryPress, 2016.
[8]	蒋守壮 . 基于 Apache Kylin 构建大数据分析平台[M]. 北京: 清华大学出版社, 2017.
	JIANG S Z . Building a big data analytics platform based on Apache Kylin[M]. Beijing: Tsinghua University PressPress, 2017.
[9]	Apache Kylin核心团队. Apache Kylin权威指南[M]. 北京: 机械工业出版社, 2007.
	Apache Kylin Core Team. Apache Kylin authoritative guide[M]. Beijing: China Machine PressPress, 2007.
[10]	王建民 . 工业大数据技术综述[J]. 大数据, 2017,3(6): 3-14.
	WANG J M . Survey on industrial big data[J]. Big Data Research, 2017,3(6): 3-14.

Metrics

Recommended 0

No Suggested Reading articles found!

测试用例编号	GN-01
测试用例名称	各个OLAP引擎性能对比
测试目的	测试Presto、Kylin1.5、Kylin2.0在数据量分别为千万规模和亿规模下的查询性能
场景描述	计数、求和与10个维度的过滤
预设条件测试过程	Kylin已经进行了预计算
	序号	测试步骤	检查点
	1	在Presto、Kylin1.5、Kylin2.0上执行计数运算	记录各个引擎的运算时间
	2	在Presto、Kylin1.5、Kylin2.0上执行求和运算	记录各个引擎的运算时间
	3	在where后面增加10个维度条件	记录各个引擎的运算时间

展示内容	数据指标	评价分数
运营建议评价体系	高活跃、高价值客户	9～10
	高价值客户	7～8
	普通客户	5～6
	待维系客户	3～4
	低价值客户	1～2
	僵尸客户	0
客户经营特征	连接数	—
	连接数上月增长	—
	活跃率上月增长	—
	上月出账金额	—
客户行为特征	激活率	—
	订购情况	—
	累计离网率	—
	API使用次数	—
	活跃率	—
客户基本属性	客户名称	—
	主要行业	—
	重点产品	—
	细分产品	—
	客户编号	—
	系统内标识	—

名称	配置	与实际差异	备注
机架式服务器	E5-2620V4 8 Core×2	模拟	实验室环境性能弱于实际环境
	128 G(8×16 G、4×32 G)
	系统盘：240 G s3510以上SSD
	硬盘：4 T(6×12)
	网卡：10 GE×4 GE×2 GE×1(IPMI) RAID卡
负载均衡器	Nginx	模拟
光纤交换机	—	无法模拟

软件名称	版本	与实际差异
操作系统	CentOS 7	一致
数据库系统	MySQL 5.7	一致
查询工具	Hive 3.1.0	一致
Hadoop集群	Cloudera CDH V5.7	一致
数据仓库	HBase V1.4.10	一致
Kylin	V2.6	一致
Presto	V0.225	一致

测试项	测试语句	测试结果	耗时
数据量全分区查询	select count(*) from dm_custom_2018	总记录条数：16 718 581条	1 min 7 s
全量数据多维度查询	SELECT BUS_ACC_YEAR,AREA,PROVINCE FROM dm_custom_2018 a WHERE STR_TO_DATE(BUS_ACC_YEAR,'%Y-%m-%d')<=STR_TO_DATE('20180801','%Y%m%d') AND STR_TO_DATE(BUS_ACC_YEAR ,'%Y-%m-%d')>= (SELECT DATE_ SUB (STR_TO_DATE ('20180831','%Y%m%d') GROUP BY a.BUS_ACC_YEAR,AREA,PROVINCE	总记录条数：122 536条	测试3次：1.845 s 1.731 s 1.703 s
全量数据多维度聚合查询	SELECT BUS_ACC_YEAR,AREA,PROVINCE,SUM (a.CONNECTIONS) AS VALUE FROM dm_custom_2018 a WHERE STR_TO_DATE(BUS_ACC_YEAR,'%Y-%m-%d')<= STR_TO_DATE ('20180801','%Y%m%d') AND STR_TO_DATE(BUS_ACC_YEAR ,'%Y-%m-%d')>= (SELECT DATE_SUB (STR_TO_ DATE('20180831','%Y%m%d') GROUP BY a.BUS_ACC_YEAR,AREA,PROVINCE	总记录条数：122 536条	测试3次：2.166 s 1.932 s 2.021 s
分区数据多维度聚合查询	SELECT a.'拍照月融合类型',a.sumUserG,a.sumUserP,(a.sumUserG/ a.sumUserP)as rateY from (SELECT ls.'拍照月融合类型',sum(ls.`拍照月用户数1`) as sumUserG,sum (ls.'拍照月用户数') as sumUserP from 'dm_201801` ls where ls.'省份'='北京' and ls.'拍照月融合类型'!='2I'GROUP BY ls.'拍照月融合类型') as a SELECT a.'拍照月融合类型',a.sumUserG,a.sumUserP,(a.sumUserG/a.sumUserP)as rateY from (SELECT ls.'拍照月融合类型',sum(ls.'拍照月用户数 1') as sumUserG ,sum(ls.'拍照月用户数') as sumUserP from 'dm_201801' ls where ls.'省份'='北京' and ls.'入网渠道' ='集团渠道' GROUP BY ls.'拍照月融合类型') as a;	总记录条数：1 167 533条	测试3次：3.776 s 2.932 s 3.221 s

Construction of Internet of things customer data platform based on OLAP technology

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 10

Related Articles 8

Metrics

Recommended 0

[1]	Zhuxi ZHANG,Wang TIAN,Shaochuan ZHU,Hongyan LIU,Xi ZHU. Research on real-time fusion method of multi-source heterogeneous flight trajectory data stream [J]. Chinese Journal on Internet of Things, 2020, 4(3): 60-68.
[2]	Tao YU,Lijun LIU,Hongjun CHEN,Yao YU. Big data mining and application of long-distance oil and gas pipeline [J]. Chinese Journal on Internet of Things, 2020, 4(3): 112-119.
[3]	Shulei GONG,Kun LI,En TONG,Hongde GUO,Yi ZHOU,Ye WANG,Fei DING. Smart factory solutions based on cellular industrial Internet of things [J]. Chinese Journal on Internet of Things, 2019, 3(2): 108-114.
[4]	Qiuying ZHANG,Le ZHOU,Jingyao TANG,Luoyi FU,Xinbing WANG. Internet of everything:interconnection,mining and visualization of academic data [J]. Chinese Journal on Internet of Things, 2018, 2(4): 56-60.
[5]	Xuli YE,Qiang ZENG,Liyu LIU,Yu SU,Meng KANG. Building end to end service guarantee system of IoT covering“terminal pipe and cloud” based on signaling big data [J]. Chinese Journal on Internet of Things, 2018, 2(2): 85-93.
[6]	Chen YU,Lijuan ZHANG,Hai JIN. Research progress and trend of big data-driven intelligent transportation system [J]. Chinese Journal on Internet of Things, 2018, 2(1): 56-63.
[7]	Enjie DING,Qingsong HU. Design ideas of the top layer of the mine Internet of things [J]. Chinese Journal on Internet of Things, 2018, 2(1): 69-75.
[8]	WU He-quan. Technology and application progress on Internet of Things [J]. Chinese Journal on Internet of Things, 2017, 1(1): 1-6.