This paper describes Mercator, a scalable, extensible web crawler written entirely in Java. Scalable web crawlers are an important component of many web services, but their design is not well-documented in the literature. We enumerate the major components of any scalable web crawler, comment on alternatives and tradeoffs in their design, and describe the particular components used in Mercator. We also describe Mercatorâ€™s support for extensibility and customizability. Finally, we comment on Mercatorâ€™s performance, which we have found to be comparable to that of other crawlers for which performance numbers have been published.