Pyropus - a scalable distributed web crawler

27.05.2004

Abstract
The Web has han estimated size of several billion pages, and it is presumed that it doubles in size roughly every two years. To obtain knowledge about content or structure in this vast material, special applications called web crawlers traverse the Web and record structure and data as it downloads documents it discovers.

Because of the web's size, crawler designs aiming at crawling significant portions of the web must be able to combine the power of multiple computers, making the crawler a distributed system. This calls for investigating new ways to parallelize and distribute the task of crawling.

In this thesis we investigate some of the problems connected to web crawling, and we present the design and implementation of the Pyropus, a scalable distributed parallel crawler. We examine ways to partition the web among several crawler nodes, yet maximize the amount of locality within the partitions. This is done by organizing websites based on DNS domains. Further, we examine ways of organizing the meta data and the data itself, needed for book-keeping to avoid multiple downloads of the same content, as well as download scheduling policies to avoid overly stressing single webservers. Also, we examine programming models for highly I/O intensive applications.