Web Intelligence and Intelligent Agent Technology, IEEE/WIC/ACM International Conference on (2008)
Dec. 9, 2008 to Dec. 12, 2008
Web page crawlers are an essential component in a number of web applications. The sheer size of the Internet can pose problems in the design of web crawlers. All currently known crawlers implement approximations or have limitations so as to maximize the throughput of the crawl, and hence, maximize the number of pages that can be retrieved within a given time frame. This paper proposes a distributed crawling concept which is designed to avoid approximations, to limit the network overhead, and to run on relatively inexpensive hardware. A set of experiments, and comparisons highlight the effectiveness of the proposed approach.
Distributed Crawler, complete crawl, Web crawler
Milly Kc, Ah Chung Tsoi, Markus Hagenbuchner, "A Scalable Lightweight Distributed Crawler for Crawling with Limited Resources", Web Intelligence and Intelligent Agent Technology, IEEE/WIC/ACM International Conference on, vol. 03, no. , pp. 663-666, 2008, doi:10.1109/WIIAT.2008.234