Larbin体系结构的研究与优化

doi:10.11959/j.issn.2096-109x.2016.00076

Abstract

Abstract:

Web crawler is an important part of the search engine,its performance will directly affect the accuracy and timeliness of the search engine.Larbin is an efficient and simple open source crawler with relatively perfect in functions.Several typical open-source crawler were firstly introduced and a multi-dimensional comparison was made among them.Then,the system architecture and working mechanism of Larbin were given in detail.Its short-comings in the program structure and process were pointed out,and improved programs were proposed.Experimen-tal results show that improved program is better in speed and performance.

Key words: search engine, Web crawler, Larbin, open source, optimization

CLC Number:

TP393

Xuan WANG,Yi-xia HUO,Yun-fei CI,Guo-zhen SHI,Li LI. Study and optimization on system architectures of Larbin[J]. Chinese Journal of Network and Information Security, 2016, 2(8): 74-83.

Figures/Tables 8

References 13

[1]	BRIN S , PAGE L . Reprint of:the anatomy of a large-scale hyper-textual Web search engine[J]. Computer Networks, 2012, 56(18): 3825-3833.
[2]	孙骏雄 . 基于网络爬虫的网站信息采集技术研究[D]. 大连: 大连海事大学， 2014.
	SUN J X . The study on technology of website information collec-tion based on Web crawler[D]. Dalian: Dalian Maritime University, 2014.
[3]	吕阳 . 分布式网络爬虫系统的设计与实现[D]. 成都: 电子科技大学， 2013.
	LV Y . Distributed Web crawler system design and implementation[D]. Chengdu: University of Electronic Science and Technology of China, 2013.
[4]	单月光 . 基于微博的网络舆情关键技术的研究与实现[D]. 成都: 电子科技大学， 2013.
	SHAN Y G . The research and implementation of network public opinion's key techniques based on microblog[D]. Chengdu: University of Electronic Science and Technology of China, 2013.
[5]	李跃健，朱程荣 . 基于 Larbin 的网络爬虫体系结构的研究与改进[J]. 计算机技术与发展， 2012, 22(7): 147-150.
	LI Y J , ZHU C R . Study and improvement on system architectures of Larbin Web cawler[J]. Computer Technology and Development, 2012, 22(7): 147-150.
[6]	罗浩 . 基于CLucene和Larbin的企业搜索引擎的研究与实现[D]. 成都: 电子科技大学， 2010.
	LUO H . The research and implementation of enterprise search en-gine based on CLucene and Larbin[D]. Chengdu: University of Electronic Science and Technology of China, 2010.
[7]	杜一平 . 主题搜索网络爬虫的设计与研究[D]. 合肥: 中国科学技术大学， 2009.
	DU Y P . Design and implementation of topical search engine Web crawler[D]. Hefei: University of Electronic Science and Technol-ogy of China, 2009.
[8]	CAFARELLA M , CUTTING D . Building nutch:open source search.[J]. Queue, 2004, 2(2): 54-61.
[9]	敖东阳, 刘好杰 . Larbin分析与Windows平台下移植[J]. 智能计算机与应用， 2009(4): 23-24.
	AO D Y , LIU H J . Analysis of Larbin and transplantation to Win-dows system[J]. Computer Study, 2009(4): 23-24.
[10]	张敏，孙敏 . 基于 Heritrix 限定爬虫的设计与实现[J]. 计算机应用与软件， 2013, 30(4): 33-35.
	ZHANG M , SUN M . Design and implementation of qualified spi-der based on Heritrix[J]. Computer Applications and Software, 2013, 30(4): 33-35.
[11]	赵本本，殷旭东，王伟 . 基于 Scrapy 的 GitHub 数据爬虫[J]. 电子技术与软件工程， 2016(6): 199-202.
	ZHAO B B , YIN X D , WANG W . GitHub Web crawler based on Scrapy[J]. Journal of Electronic Technology And Software Engi-neering, 2016(6): 199-202.
[12]	KARKI R , GENNERT M A . Fresh analysis of streaming media stored on the Web [J]. 2011.
[13]	TONG W , XIE X Y . A research on a defending policy against the WebCrawler's attack[C]// International Conference on Anti- Coun-terfeiting,Security,and Identification in Communication. c2009: 363-366.

Metrics

Recommended 0

No Suggested Reading articles found!

开源框架	二次开发	开发语言	支持分布式	镜像保存	优点
Larbin	易	C++	否	是	高效、高度定制化
Nutch	难	Java	是	否	支持分布式、提供抓取和索引、提供插件扩展
Heritrix	易	Java	否	是	高度可扩展、高度可控性
Scrapy	一般	Python	是	否	可用性高、功能完善

分类	工具
Python	Beautiful Soup、html5lib
Java	Jsoup、htmlparser、JTidy
C/C++	Gumbo、Htmlcxx、Streaming HTML parser、libhtml

Study and optimization on system architectures of Larbin

RichHTML

PDF下载

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 13

Related Articles 13

Metrics

Recommended 0

[1]	Xiangdong HU, Lingling TANG. Method on intrusion detection for industrial internet based on light gradient boosting machine [J]. Chinese Journal of Network and Information Security, 2023, 9(2): 46-55.
[2]	Ganqin LIU, Hui LI, Hui ZHU, Yukun HUANG, Xingdong LIU. Public key cryptographic algorithm SM2 optimized implementation on low power embedded platform [J]. Chinese Journal of Network and Information Security, 2022, 8(6): 29-38.
[3]	Rongna XIE, Zhuhong MA, Zongyu LI, Ye TIAN. Encrypted traffic classification method based on convolutional neural network [J]. Chinese Journal of Network and Information Security, 2022, 8(6): 84-91.
[4]	Mingying ZHANG, Bing HUA, Yuguang ZHANG, Haidong LI, Mohong ZHENG. Robust reinforcement learning algorithm based on pigeon-inspired optimization [J]. Chinese Journal of Network and Information Security, 2022, 8(5): 66-74.
[5]	Baoqin ZHAI, Jian WANG, Lei HAN, Jiqiang LIU, Jiahao HE, Tianhao LIU. Hierarchical proxy consensus optimization for IoV based on blockchain and trust value [J]. Chinese Journal of Network and Information Security, 2022, 8(3): 142-153.
[6]	Cheng HUANG, Mingxu SUN, Renyu DUAN, Susheng WU, Bin CHEN. Vulnerability identification technology research based on project version difference [J]. Chinese Journal of Network and Information Security, 2022, 8(1): 52-62.
[7]	Pei CHEN, Fenghua LI, Zifu LI, Yunchuan GUO, Lin CHENG. Using rule association to generate data collection policies [J]. Chinese Journal of Network and Information Security, 2021, 7(5): 132-148.
[8]	Wenchao WU, Zhiyu REN, Xuehui DU. Permission clustering-based attribute value optimization [J]. Chinese Journal of Network and Information Security, 2021, 7(4): 175-182.
[9]	Tao WANG, Hongchang CHEN. Multi-objective optimization placement strategy for SDN security controller considering Byzantine attributes [J]. Chinese Journal of Network and Information Security, 2021, 7(3): 72-84.
[10]	Dier GU, Hua LU, Renchao XIE, Tao HUANG. Survey on open source edge computing platforms [J]. Chinese Journal of Network and Information Security, 2021, 7(2): 22-34.
[11]	Qiang CHEN,Caixia LIU,Lingshu LI. 5G network slicing function migration mechanism based on particle swarm optimization algorithm [J]. Chinese Journal of Network and Information Security, 2018, 4(8): 47-55.
[12]	Congyue YIN,Xingming ZHANG,Quan REN,Shuai WEI. Rapid IO routing strategy based on improved glowworm swarm optimization algorithm [J]. Chinese Journal of Network and Information Security, 2018, 4(6): 52-61.
[13]	Ying-jun LI,Hong-li ZHANG,Xing WANG. Temporal relation recognition method based on news event fragments [J]. Chinese Journal of Network and Information Security, 2017, 3(6): 33-41.