MX Interview Question

How would you build a web crawler?