一种关联感知的大数据导入方法

doi:10.11959/j.issn.1000-0801.2016044

Telecommunications Science ›› 2016, Vol. 32 ›› Issue (3): 130-134.doi: 10.11959/j.issn.1000-0801.2016044

• Operating technology wide-angle • Previous Articles Next Articles

An associated perception import method for big data

Huaiyu GONG¹,Jinsong XU²,Pan WANG²

¹ Jiyuan Branch of China Telecom Co.，Ltd.，Jiyuan 454650，China
² Nanjing University of Posts and Telecommunications，Nanjing 210003，China

Online:2016-03-20 Published:2016-03-28
Supported by:
Jiangsu Province Project of Natural Science Foundation;Jiangsu Province Education Project of Natural Science Foundation;2013 Six Talent Peaks Project in Jiangsu Province;2013 Information Security Special Funds of the National Development and Reform Commission;State Grid 2014 Science and Technology Project：Research and Application of Network Traffic Prediction and Smart Pipe Key Technologies for Electric Power Information Communication Network;2015 Industrialization，Education and Research Joint Prospective Project

Abstract

Abstract:

Against the background of the existing database to the large data migration，Apache introduced the Sqoop as the main tool for the relational database to the big data migration.Sqoop simply cut the data table and randomly store it on diffe rent nodes.Being aimed at the problem of low efficiency of the query of the relationship between the Hadoop，a method of data importing and preprocessing was designed.To reduce the network transmission delay and improve the performance of the system，the high correlation degree was kept in the adjacent nodes.The contrast experiment shows that the performance of the data query can be improved greatly by the same or adjacent nodes.

Key words: big data, Sqoop, Hadoop, NoSQL

Huaiyu GONG,Jinsong XU,Pan WANG. An associated perception import method for big data[J]. Telecommunications Science, 2016, 32(3): 130-134.

Figures/Tables 3

References 9

[1]	中国大数据发展调查研究结果[EB/OL].（2015-07-29）[2015-09-01]. .Research report of China big data development[EB/OL].（2015-07-29）[2015-09-01]. .
[2]	Apache Sqoop[EB/OL].[2015-09-01]. .
[3]	BALMIN A ， KALDEWEY T ， TATA S . Clydesdale：structured data processing on Hadoop[C]// 2012 ACM SIGMOD International Conference on Management of Data， May 20-24，2012， Scottsdale，AZ，USA. New York： ACM Press， 2012：705-708.
[4]	BALMIN A ， KALDEWEY T ， TATA S . Clydesdale：structured data processing on MapReduce[C]// 2012 International Conference on Extending Database Technology， March 27-30，2012， Berlin，German. New York： ACM Press， 2012：15-25.
[5]	THUSOO A ， SARMA J S ， JAIN N ， et al. Hive - a warehousing solution over a MapReduce framework[J]. PVLDB， 2009，2（2）：1626-1629.
[6]	LEE R B ， LUO T ， HUAI Y ， et al. YSmart：yet another SQL-to-MapReduce translator[C]// 2011 International Conference on Distributed Computing Systems， June 20-24，2011， Minneapolis，Minnesota，USA. New Jersey： IEEE Press， 2011：25-36.
[7]	LYNDEN S ， TANIMURA Y ， KOJIMAL ， et al. Dynamic data redistribution for MapReduce joins[C]// 2011 IEEE International Conference on Coud Computing Technology and Science， November 29-December 1，2011， Athens，Greece. New Jersey： IEEE Press， 2011：717-723.
[8]	ALPER O ， MIREK R ， . Processing theta-joins using MapReduce[C].// 2011 ACM SIGMOD Internati onal Conference on Management of Data， June 12-16，2011， Athens，Greece. New Jersey： IEEE Press， 2011：949-960.
[9]	JIANG D W ， TUNG A K H ， CHEN G . Map-join-reduce：toward scalable and efficient data analysis on large clusters[J]. IEEE Transactions on knowledge and Data Engineering， 2011，23（9）：1299-1311.

Metrics

Recommended 0

No Suggested Reading articles found!

[1]	Lu HAN, Weiyu CHEN, Fei ZHANG, Jianfeng HE, Huaizhen SU. Big data classification method of non relational distributed submission information under differentiated requirements [J]. Telecommunications Science, 2023, 39(6): 114-121.
[2]	Xue HAN. Damage toughness assessment method of power backbone communication network based on power big data [J]. Telecommunications Science, 2023, 39(5): 136-143.
[3]	Yudi SUN. A prediction model of massive 5G network users’ revisit behavior based on telecom big data [J]. Telecommunications Science, 2023, 39(2): 157-162.
[4]	Aihua LI, Xiaobo WU, Chao CHEN, Bin WEI, Yuanyuan SHI. Big data intelligent analysis technology for 5G network [J]. Telecommunications Science, 2022, 38(8): 129-139.
[5]	Gengsheng HUANG, Yuhong HUANG, Manxue GUO, Jianping ZHENG, Xin GE. Architecture of a new infrastructure for data service in DT era [J]. Telecommunications Science, 2022, 38(7): 138-145.
[6]	Panpan LI, Zhengxia XIE, Guangxue YUE, Xin LIU. Research progress and trends of deep learning based wireless communication receiving method [J]. Telecommunications Science, 2022, 38(2): 1-17.
[7]	Haibo ZHAO, Zhijun XIANG, Linsong XIAO. A big data framework for short-term power load forecasting using heterogenous data [J]. Telecommunications Science, 2022, 38(12): 103-111.
[8]	Zhiyong LIU, Zhongjiang HE, Yilong RUAN, Junfeng SHAN, Chao ZHANG. Big data security features and operation practices [J]. Telecommunications Science, 2021, 37(5): 160-169.
[9]	Xiaojun LIU,Juan WU,Xiaoqing XU. Big data architecture analysis and data security fusion technology [J]. Telecommunications Science, 2020, 36(7): 146-155.
[10]	Yujiao LIU,Kunhuang SONG,Xiang WANG. Analysis of economic prosperity index based on big data of electricity [J]. Telecommunications Science, 2020, 36(6): 166-171.
[11]	Weihua CHENG,Xiaorong HE. Quality analysis method of Web browsing based on big data [J]. Telecommunications Science, 2020, 36(11): 174-181.
[12]	Xiaomin TAN,Ai FANG,Bing LIANG,Haojie YANG. Locating causes of abnormality of EPG experience based on decision tree [J]. Telecommunications Science, 2019, 35(8): 158-164.
[13]	Zelin LU,Qiangzhi LI. Evolutionary logic and operational mechanism of E-commerce platform [J]. Telecommunications Science, 2019, 35(7): 152-158.
[14]	Ming LIAO,Ming CHEN,Ji ZHOU,Xiaohua XIANG,Fang LI,Yefen JIAO. DNS log analysis system based on big data fusion algorithm [J]. Telecommunications Science, 2019, 35(5): 129-139.
[15]	Yuanfeng GONG,Yi HUANG. Optimization method of wireless indoor coverage based on big data analysis [J]. Telecommunications Science, 2019, 35(5): 149-154.

An associated perception import method for big data

RichHTML

PDF下载

Knowledge

Cited

Abstract

Cite this article

share this article

Figures/Tables 3

References 9

Related Articles 15

Metrics

Recommended 0