基于用户移动网络接入位置的高效分布式相似矩阵计算方法

doi:10.11959/j.issn.1000-0801.2018136

Abstract

Abstract:

A distributed similarity matrix computation method based on users’ mobile network access location was proposed.MapReduce computing framework in the Hadoop ecosystem was used,and users’ mobile network access location information was considered as prior knowledge,then users were divided based on geographic location information and the similarity calculation was performed.The experimental results show that the computational efficiency of the proposed method is increased by nearly 25 times compared with the existing similarity matrix computation method.As for the outcome of the community detection,the proposed method get nearly the same results as the existing similar matrix computation method,the agreement rate is nearly 99.9%.

CLC Number:

TP393

Yuan WANG,Hao JIANG,Ming WU,Donggui YAO,Yi ZHANG,Shuwen YI,Hai WANG,Jing WU. A distributed high efficiency similarity matrix computation method based on users’ mobile network access location[J]. Telecommunications Science, 2018, 34(5): 26-38.

Figures/Tables 10

References 26

[1]	SCRIPPS J , TREFFTZ C . Parallelizing an algorithm to find communities using the Jaccard metric[C]// IEEE International Conference on Electro/Information Technology,June 8-12,2015,London,UK. Piscataway:IEEE Press, 2015.
[2]	HURLEY N , DURIAKOVA E . Reformulations of the map equation for community finding and blockmodelling[C]// IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining,August 25-28,2015,Paris,France. Piscataway:IEEE Press, 2015.
[3]	RESTREPO A , SOLANO A , SCRIPPS J ,et al. High-performance implementations of a clustering algorithm for finding network communities[C]// IEEE International Conference on Electro/Information Technology,May 6-8,2012,Indianapolis,USA. Piscataway:IEEE Press, 2012.
[4]	GUNTHER J H , HOFFMAN K H . Numerische mathematik[M]. Berlin: SpringerPress, 1991.
[5]	STEWART G W . Jampack:a Java package for matrix computations[J]. 2017.
[6]	JOE H , CLEVE M , PETER W . JAMA:a Java matrix package[EB]. 2017.
[7]	NGUYEN D K , LAVALLEE I , BUI M ,et al. A general scalable parallelizing of strassen’s algorithm for matrix multiplication on distributed memory computers[C]// ACIS International Conference on Computer and Information Science,July 14-16,2005,Washington,DC,USA. Piscataway:IEEE Press, 2005.
[8]	LIN C , HUANG Z H , YANG F ,et al. Identify content quality in online social networks[J]. IET Communications, 2012,6(12): 1618-1624.
[9]	孙远帅, 陈垚, 官新均 ,等. 基于Hadoop的大矩阵乘法处理方法[J]. 计算机应用, 2013,33(12): 3339-3344.
	SUN Y S , CHEN Y , GUAN X J ,et al. Approach of large matrix multiplication based on Hadoop[J]. Journal of Computer Applications, 2013,33(12): 3339-3344.
[10]	REZA M , SINHA A , NAG R ,et al. CUDA-enabled Hadoop cluster for sparse matrix vector multiplication[C]// IEEE International Conference on Recent Trends in Information Systems,July 9-11,2015,Kolkata,India. Piscataway:IEEE Press, 2015.
[11]	MALYSIAK D , KOPINSKI T . A generic and adaptive approach for workload distribution in multi-tier cluster systems with an application to distributed matrix multiplication[C]// IEEE International Symposium on Computational Intelligence and Informatics,November 19-21,2015,Budapest,Hungary. Piscataway:IEEE Press, 2015.
[12]	GIZA-BELCIUG F , PENTIUC S G . Parallelization of similarity matrix calculus in ontology mapping systems[C]// Roedunet International Conference-Networking in Education and Research,September 24-26,2015,Craiova,Romania. Piscataway:IEEE Press, 2015.
[13]	ZHANG R , WANG Y . An enhanced agglomerative fuzzy k-means clustering method with MapReduce implementation on Hadoop platform[C]// International Conference on Progress in Informatics and Computing,May 16-18,2014,Shanghai,China. Piscataway:IEEE Press, 2014.
[14]	MANN K S , KAUR N . Cloud-deployable health data mining using secured framework for clinical decision support system[C]// International Conference and Workshop on Computing and Communication,October 15-17,2015,Vancouver,BC,Canada. Piscataway:IEEE Press, 2015.
[15]	SHAHRIVARI S , JALILI S . Single-pass and linear-time k-means clustering based on MapReduce[J]. Information Systems, 2016(60): 1-12.
[16]	LU S , TONG W , CHEN Z . Implementation of the KNN algorithm based on Hadoop[C]// International Conference on Smart and Sustainable City and Big Data,July 26-27,2015,Shanghai,China. Birmingham:IET Press, 2015.
[17]	SONG G , ROCHAS J , BEZE L ,et al. K nearest neighbour joins for big data on MapReduce:a theoretical and experimental analysis[J]. IEEE Transactions on Knowledge ＆ Data Engineering, 2016,28(9): 2376-2392.
[18]	MIEGHEM P V , BLENN N , DOERR C . Lognormal distribution in the digg online social network[J]. European Physical Journal B, 2011,83(2): 251-261.
[19]	MAHANTI A , CARLSSON N , MAHANTI A ,et al. A tale of the tails:power-laws in internet measurements[J]. IEEE Network, 2013,27(1): 59-64.
[20]	ZHOU C , JIANG H , CHEN Y ,et al. TCB:a feature transformation method based central behavior for user interest prediction on mobile big data[J]. International Journal of Distributed Sensor Networks, 2016,12(9).
[21]	WU L , JIANG H , ZHENG H ,et al. Long tail and small world characteristic of mobile internet traffic dynamics[C]// IEEE International Conference on Systems,Man and Cybernetics,October 5-8,2014,San Diego,CA,USA. Piscataway:IEEE Press, 2014.
[22]	WU L , LI Y , ZHOU C ,et al. Statistic analysis of data access behavior in the mobile internet[C]// IEEE/CIC International Conference on Communications in China,August 12-14,2013,Xi’an,China. Piscataway:IEEE Press, 2013.
[23]	ZHANG N , ZHENG G , CHEN H ,et al. HBaseSpatial:a scalable spatial data storage based on HBase[C]// IEEE International Conference on Trust,Security and Privacy in Computing and Communications,September 24-26,2014,Beijing,China. Piscataway:IEEE Press, 2014.
[24]	王荣 . 基于三元组表表示的稀疏矩阵的快速转置算法及其改进[J]. 现代电子技术, 2008,31(22): 78-79.
	WANG R . Improvement on fast transposition algorithm to sparse matrix expressed by triple list[J]. Modern Electronics Technique, 2008,31(22): 78-79.
[25]	AFSHANG M , DHILLON H S , CHONG P H J . Fundamentals of cluster-centric content placement in cache-enabled device-to-device networks[J]. IEEE Transactions on Communications, 2015,64(6): 2511-2526.
[26]	ZHOU B , CUI Y , TAO M . Stochastic content-centric multicast scheduling for cache-enabled heterogeneous cellular networks[J]. IEEE Transactions on Wireless Communications, 2016,15(9): 6284-6297.

Metrics

Recommended 0

No Suggested Reading articles found!

RowKey			columnFamily
RowKey	sTime	eTime	lacID	cellID	x	y	url
uid1+startTime1	startTime1	endTime1	lacID1	cellID1	x1	y1	url1
uid2+startTime2	startTime2	endTime2	lacID2	cellID2	x2	y2	url2
…	…	…	…	…	…	…	…
uidn+startTimen	startTimen	endTime n	lacIDn	cellID n	xn	yn	urln

字段名	含义
uid	加密后的用户ID，唯一标识用户身份
lacID	基站所在地区的区域代码
cellID	扇区编号（lacID和cellID唯一标识基站）
sTime	数据流起始时间
eTime	数据流持续时间
url	截短后的访问内容地址
longitude	基站所在位置经度
latitude	基站所在位置纬度

对比项	划分前	划分后	百分比
社区内边数/条	16 438 105	7 315 487	44.5%
好边数/条	16 438 105	7 277 559	44.3%
正确用户数/个	20 846	20 824	99.9%

对比项	划分前	划分后	百分比
社区内边数/条	109 498	74 581	68.1%
好边数/条	109 498	56 400	51.5%
正确用户数/个	3 468	3 327	95.9%

A distributed high efficiency similarity matrix computation method based on users’ mobile network access location

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 26

Related Articles 15

Metrics

Recommended 0

[1]	Chengling LI,Yuxiang ZHENG,Yiqi HONG,Wen LI,Huimin GUO. Research on the selection of grid data storage technology [J]. Telecommunications Science, 2018, 34(5): 183-191.
[2]	Wei PENG,Xiaodong HE,Xiaoming LU,Chuanxin JI. Optimization and application of the carrier DNS data analysis technology [J]. Telecommunications Science, 2018, 34(3): 138-144.
[3]	Haiyong XU,Yan HUANG. Applications of a big data platform based on PaaS [J]. Telecommunications Science, 2018, 34(1): 148-157.
[4]	Ai FANG,Yuzhong HANG,Xiong XU,Duo JIN. Scheme and practice of IPTV user viewing behavior analysis based on MapR [J]. Telecommunications Science, 2017, 33(2): 138-144.
[5]	Baoyou WANG,Jing QIAN,Shijin YUAN. Research and implementation on acquisition scheme of telecom big data based on Hadoop [J]. Telecommunications Science, 2017, 33(1): 135-142.
[6]	Nanhai LIU,Lei LEI,Rui WANG. Practice and thinking on the transition of telecom operator analysis support system in big data era [J]. Telecommunications Science, 2016, 32(8): 146-158.
[7]	Zaole TAN,Zhifeng HAO,Ruichu CAI,Xiaojun XIAO,Yu LU. Hadoop bottleneck detection algorithm based on information gain [J]. Telecommunications Science, 2016, 32(7): 115-120.
[8]	Yuanning LI,Sen LIU,Shijun ZHANG,Feng CHEN,Zhiying WANG. Practice and application of distributed data quality management system in power enterprise [J]. Telecommunications Science, 2016, 32(4): 169-174.
[9]	Huaiyu GONG,Jinsong XU,Pan WANG. An associated perception import method for big data [J]. Telecommunications Science, 2016, 32(3): 130-134.
[10]	Hongxun GU,Ke YANG. Mobile user behavior analysis system and applications based on big data [J]. Telecommunications Science, 2016, 32(3): 139-146.
[11]	Huimin ZHAO,Jiangtao LUO,Junchao YANG,Zheng XU,Xiao LEI,Lin LUO. Research and application of prediction model based on ensemble BP neural network [J]. Telecommunications Science, 2016, 32(2): 60-67.
[12]	Zhongwei Wang,Yefang Chen,Siyou Xiao,Jiangbo Qian. An AkNN Algorithm for High-Dimensional Big Data [J]. Telecommunications Science, 2015, 31(7): 52-62.
[13]	Jianxin Ren,Huahui Chen. An Adaptive Subspace Similarity Search Approach [J]. Telecommunications Science, 2015, 31(7): 63-74.
[14]	Ai Fang,Xiong Xu,Bing Liang,Yuzhong Zhang,Yiping Yang. Comparison of Open-Source Distributed Computing Framework for Big Data [J]. Telecommunications Science, 2015, 31(7): 152-157.
[15]	Juan Yu. Discussion on Integration of Data Warehouse and Big Data [J]. Telecommunications Science, 2015, 31(3): 159-163.