分布式环境下话题发现算法性能分析

doi:10.11959/j.issn.1000-436x.2018136

Abstract

Abstract:

Social network has become a way of life,therefore more and more people choose social network to express their views and feelings.Quickly find what people are talking about in big data gets more and more attention.And a lot of related methods of topic detection spring up in this situation.The performance analysis project was proposed based on the characteristics of social network.According to the project,the performances of some typical topic detection algorithms were tested and compared in large-scale data of Sina Weibo.What’s more,the advantages and disadvantages of these algorithms were pointed out so as to provide references for later applications.?

Key words: topic detection, distributed environment, social network, performance analysis

CLC Number:

TP391

Lu DENG,Yan JIA,Binxing FANG,Bin ZHOU,Tao ZHANG,Xin LIU. Performance analysis of topic detection algorithms in distributed environment[J]. Journal on Communications, 2018, 39(8): 176-184.

Figures/Tables 12

References 18

[1]	中国互联网络信息中心第41次《中国互联网络发展状况统计报告》[R]. 2018.
	China Internet Network Information Center The 41th statistical report on Internet development in China[R]. 2018.
[2]	DHILLON I S , MODHA D S . Concept decompositions for large sparse text data using clustering[C]// Machine Learning. 2001: 143-175.
[3]	KUMMAMURU K , DHAWALE A , KRISHNAPURAM R . Fuzzy co-clustering of documents and keywords[C]// The IEEE International Conference on Fuzzy Systems. 2003: 772-777.
[4]	ZHAO Y , KARYPIS G . Soft clustering criterion functions for partitional document clustering:a summary of results[C]// Thirteenth ACM International Conference on Information ＆ Knowledge Management. 2004: 246-247.
[5]	MAKKONEN J , AHONENMYKA H , SALMENKIVI M . Topic detection and tracking with spatio-temporal evidence[C]// European Conference on Ir Research. 2003: 251-265.
[6]	WU C , WANG B . Extracting topics based on Word2Vec and improved jaccard similarity coefficient[C]// IEEE Second International Conference on Data Science in Cyberspace. 2017: 389-397.
[7]	HOFMANN T , . Probabilistic latent semantic indexing[C]// International ACM SIGIR Conference on Research and Development in Information Retrieval. 1999: 50-57.
[8]	BLEI D M , NG A Y , JORDAN M I . Latent dirichlet allocation[J]. J Machine Learning Research Archive, 2003,3: 993-1022.
[9]	STEYVERS M , GRIFFITHS T . Probabilistic topic models[J]. Handbook of Latent Semantic Analysis, 2007,427(7): 424-440.
[10]	BLEI D , CARIN L , DUNSON D . Probabilistic topic models[C]// ACM SIGKDD International Conference Tutorials. 2011:1.
[11]	BERNHARD S , JOHN P , THOMAS H . A collapsed variational bayesian inference algorithm for latent dirichlet allocation[C]// The Twentieth Conference on Neural Information Processing Systems. 2006: 1353-1360.
[12]	GRIFFITHS T L , STEYVERS M . Finding scientific topics[J]. National Academy of Sciences of the United States of America, 2004: 5228-5235.
[13]	RAMAGE D , . Characterizing microblogs with topic models[C]// International AAAI Conference on Weblogs and Social Media. 2010: 130-137.
[14]	CHEN Z , LIU B . Mining topics in documents:standing on the shoulders of big data[C]// ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2014: 1116-1125.
[15]	LIN T , TIAN W , MEI Q ,et al. The dual-sparse topic model:mining focused topics and focused terms in short text[C]// International Conference on World Wide Web. 2014: 539-550.
[16]	ZHAI K , BOYD G J , ASADI N ,et al. MrLDA:a flexible large scale topic modeling package using variational inference in MapReduce[C]// International Conference on World Wide Web. 2012: 879-888.
[17]	ARONSSON F . Large scale cluster analysis with Hadoop and Mahout[J]. Technology ＆ Engineering, 2015.
[18]	MENG X R , BRADLEY J , BURAK Y ,et al. MLlib:machine learning in apache spark[J]. Journal of Machine Learning Research, 2015,17(1): 1235-1241.

Metrics

Recommended 0

No Suggested Reading articles found!

集群规模	迭代次数
	10			30			50
	MrLDA/min	Mahout LDA/min	SparkLDA/min	MrLDA/min	Mahout LDA/min	SparkLDA/min	MrLDA/min	Mahout LDA/min	SparkLDA/min
4	632.73	16.90	1.30	1 981.47	27.52	3.10	2 714.32	44.21	4.80
8	303.60	9.99	1.30	807.37	25.95	3.30	1 334.95	41.06	6.50
16	73.33	10.92	1.50	220.90	27.20	3.40	252.38	44.25	5.60
32	41.63	11.26	1.60	126.50	28.33	4.50	145.78	45.40	5.60

集群规模	迭代次数
	10		30		50
	Mahout LDA/min	Spark LDA/min	Mahout LDA/min	Spark LDA/min	Mahout LDA/min	Spark LDA/min
4	159.07	5.30	244.13	14.00	684.24	22.00
8	71.55	5.60	187.18	14.00	293.10	22.00
16	62.52	6.50	164.67	16.00	260.21	25.00
32	57.19	7.40	164.09	17.00	271.38	28.00

集群规模	迭代次数
	10		30		50
	Mahout LDA/min	Spark LDA/min	Mahout LDA/min	Spark LDA/min	Mahout LDA/min	Spark LDA/min
4	338.07	34.00	903.15	84.00	1454.67	138.00
8	145.56	33.00	384.55	84.00	618.98	138.00
16	125.15	38.00	329.17	96.00	493.88	150.00
32	82.71	44.00	233.39	108.00	378.67	216.00

Performance analysis of topic detection algorithms in distributed environment

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 18

Related Articles 15

Metrics

Recommended 0

[1]	Xuan WANG, Yu ZHANG, Junfeng ZHOU, Ziyang CHEN. Influence maximization algorithm based on social network [J]. Journal on Communications, 2022, 43(8): 151-163.
[2]	Tao PENG, Wentao ZHONG, Guojun WANG, Entao LUO, Jinbo XIONG, Yining LIU, Wang Hao. Privacy-preserving precise profile matching in mobile social network [J]. Journal on Communications, 2022, 43(11): 90-103.
[3]	Sai LI, Liang YANG, Qimei CUI, Siyuan YU. Performance analysis of RIS-assisted mixed RF/THz system [J]. Journal on Communications, 2022, 43(1): 49-58.
[4]	Zheng GUAN,Lei XIONG,Yao JIA,Min HE,Zhijun YANG. Research on scheduled WLAN MAC protocol with failure retries on RoF-DAS architecture [J]. Journal on Communications, 2020, 41(3): 102-111.
[5]	Zhen LIU,Na’na WANG,Xiaodong WANG,Yongqi SUN. Spectral clustering and embedding-enhanced POI recommendation in location-based social network [J]. Journal on Communications, 2020, 41(3): 197-206.
[6]	Qiuyang GU, Chunhua JU, Gongxing WU. Social network link prediction method based on subgraph evolution and improved ant colony optimization algorithm [J]. Journal on Communications, 2020, 41(12): 21-35.
[7]	Jing CHEN,Ziyi QI. Research on social network influence maximization algorithm based on time sequential relationship [J]. Journal on Communications, 2020, 41(10): 211-221.
[8]	Junlei BI,Zhiyuan LI. Time-variant interest community based query message routing algorithm in opportunity social network [J]. Journal on Communications, 2019, 40(9): 86-94.
[9]	Fenghua LI,Zhe SUN,Ben NIU,Jin CAO,Hui LI. Privacy-preserving photo sharing framework cross different social network [J]. Journal on Communications, 2019, 40(7): 1-13.
[10]	Haiping HUANG,Kai WANG,Xiong TANG,Dongjun ZHANG. Differential privacy protection scheme based on edge betweenness model [J]. Journal on Communications, 2019, 40(5): 88-97.
[11]	Xiaomeng WANG, Binxing FANG, Hongli ZHANG, Xing WANG. TSL:predicting popularity of Facebook content based on tie strength [J]. Journal on Communications, 2019, 40(10): 1-9.
[12]	Guanghui YAN, Meng ZHANG, Hao LUO, Shikui LI, Ting LIU. Identifying vital nodes algorithm in social networks fusing higher-order information [J]. Journal on Communications, 2019, 40(10): 109-118.
[13]	Yu FU, Yihan YU, Xiaoping WU. Differential privacy protection technology and its application in big data environment [J]. Journal on Communications, 2019, 40(10): 157-168.
[14]	Yancui SHI,Yuan WANG,Qing ZHAO,Xiankun ZHANG. Research status of community detection based on local expansion [J]. Journal on Communications, 2019, 40(1): 149-162.
[15]	Yong ZENG,Lingjie ZHOU,Zhongyuan JIANG,Zhihong LIU,Jianfeng MA. Security analysis of weighted network anonymity based on singular value decomposition [J]. Journal on Communications, 2018, 39(5): 23-33.