Web Congress, Joint Conference Brazilian Symposium on Multimedia and the Web & Latin America (2004)
Ribeir?o Preto-SP, Brazil
Oct. 12, 2004 to Oct. 15, 2004
Carlos Castillo , Universidad de Chile
Mauricio Marin , Universidad de Magallanes
Andrea Rodriguez , Universidad de Concepción
Ricardo Baeza-Yates , Universidad de Chile
This article presents a comparative study of strategies for Web crawling. We show that a combination of breadth-first ordering with the largest sites first is a practical alternative since it is fast, simple to implement, and able to retrieve the best ranked pages at a rate that is closer to the optimal than other alternatives. Our study was performed on a large sample of the Chilean Web which was crawled by using simulators, so that all strategies were compared under the same conditions, and actual crawls to validate our conclusions. We also explored the effects of large scale parallelism in the page retrieval task and multiple-page requests in a single connection for effective amortization of latency times.
R. Baeza-Yates, A. Rodriguez, C. Castillo and M. Marin, "Scheduling Algorithms for Web Crawling," Web Congress, Joint Conference Brazilian Symposium on Multimedia and the Web & Latin America(LA-WEBMEDIA), Ribeir?o Preto-SP, Brazil, 2004, pp. 10-17.