Crawlr Web Crawler

Fetching one page is ten lines. Fetching ten thousand pages in parallel without crawling the same URL twice, without hammering one host, without leaking threads, and while stopping cleanly on Ctrl-C is a real concurrency exercise. Virtual threads make the blocking style cheap, but they do not remove the need to bound concurrency, throttle per host, and decide what cancellation means.

Target behaviour

$ crawlr --depth 3 --concurrency 20 --rps 5 --timeout 10s \
         --same-host --out sitemap.json https://example.com

crawling https://example.com  depth=3 concurrency=20 rps/host=5
  200  /                      14 links    82ms
  200  /docs                  31 links   120ms
  404  /old-page               0 links    41ms
  --   /admin                 blocked by robots.txt
...
done  pages=842 ok=820 errors=22 skipped=17 unique_urls=1,204 elapsed=48s
wrote sitemap.json (842 pages, 3,118 edges)

^C  -> cancelling: interrupting 20 fetches... flushed partial results to sitemap.json

Architecture

 seed URLs
    |
    v
 Crawler.submit(url, depth) --- visited.add(normalised)? --no--> drop
    |  yes (first time)
    v
 virtual thread per page (pending counter tracks the whole tree)
    |
    +--> Semaphore(concurrency)        global bound on in-flight fetches
    +--> RobotsCache.allowed(url)      per host, fetched once, cached
    +--> HostLimiter.await(host)       min gap between requests to one host
    +--> HttpClient.send(request)      per-request timeout, deadline from CLI
    +--> jsoup parse -> links -> normalise -> filter -> submit(child, depth+1)
    |
    v
 ResultSink (ConcurrentLinkedQueue<PageResult>) -> sitemap.json

Why a thread per page but a semaphore on top: virtual threads are free to create, so the number of threads is not your limit. The Semaphore limits sockets and memory, the HostLimiter limits politeness, and the visited set limits total work. Three independent bounds, three different reasons.

The core of it

Start with the data and the dedupe key. Normalisation matters because https://Example.com/a#top and https://example.com/a are one page.

public record Task(URI url, int depth) {}
public record PageResult(URI url, int status, int links, long millis, String error) {}

static URI normalise(URI base, String href) {
    try {
        URI u = base.resolve(href.strip()).normalize();
        if (!"http".equals(u.getScheme()) && !"https".equals(u.getScheme())) return null;
        String path = u.getPath() == null || u.getPath().isEmpty() ? "/" : u.getPath();
        return new URI(u.getScheme(), null, u.getHost().toLowerCase(), u.getPort(),
                       path, u.getQuery(), null);   // null fragment drops #anchor
    } catch (URISyntaxException | IllegalArgumentException | NullPointerException e) {
        return null;
    }
}

The visited set must make "check and mark" one atomic step. Set.add on a concurrent set returns false if the element was already present, which is exactly that.

private final Set<URI> visited = ConcurrentHashMap.newKeySet();
private final AtomicInteger pending = new AtomicInteger();
private final CountDownLatch done = new CountDownLatch(1);

void submit(URI url, int depth) {
    if (depth > maxDepth || !visited.add(url)) return;
    pending.incrementAndGet();                    // count BEFORE starting, never inside the task
    pool.submit(() -> {
        try { process(new Task(url, depth)); }
        finally { if (pending.decrementAndGet() == 0) done.countDown(); }
    });
}

Per-host politeness: reserve the next slot under a lock and sleep outside it, so many threads waiting on one host do not serialise on the lock longer than needed.

final class HostLimiter {
    private final Map<String, Long> nextFree = new HashMap<>();
    private final long gapNanos;
    HostLimiter(double rps) { this.gapNanos = (long) (1_000_000_000L / rps); }

    void await(String host) throws InterruptedException {
        long wait;
        synchronized (this) {
            long now = System.nanoTime();
            long slot = Math.max(now, nextFree.getOrDefault(host, now));
            nextFree.put(host, slot + gapNanos);
            wait = slot - now;
        }
        if (wait > 0) TimeUnit.NANOSECONDS.sleep(wait);
    }
}

Fetching combines the semaphore, the limiter, a request timeout, and the cancellation signal. Interrupting a virtual thread blocked in HttpClient.send throws InterruptedException, which is your cancellation path.

void process(Task t) throws InterruptedException {
    if (!robots.allowed(t.url())) { sink.add(new PageResult(t.url(), -1, 0, 0, "robots")); return; }
    permits.acquire();
    try {
        limiter.await(t.url().getHost());
        long start = System.nanoTime();
        var req = HttpRequest.newBuilder(t.url()).timeout(Duration.ofSeconds(10))
                .header("User-Agent", "crawlr/1.0").GET().build();
        HttpResponse<String> res = client.send(req, BodyHandlers.ofString());
        List<URI> links = extractLinks(t.url(), res.body());
        sink.add(new PageResult(t.url(), res.statusCode(), links.size(), ms(start), null));
        links.stream().filter(this::inScope).forEach(l -> submit(l, t.depth() + 1));
    } catch (IOException e) {
        sink.add(new PageResult(t.url(), 0, 0, 0, e.toString()));
    } finally { permits.release(); }
}

robots.txt is fetched once per host. Cache the Future in a ConcurrentHashMap with computeIfAbsent so 20 threads arriving at once cause one request, and treat a 404 as "allow all".

boolean allowed(URI u) throws InterruptedException {
    String origin = u.getScheme() + "://" + u.getAuthority();
    Rules rules = cache.computeIfAbsent(origin, o -> fetchRules(URI.create(o + "/robots.txt")));
    return rules.allows(u.getPath());
}

Do not hold a semaphore permit while waiting on the limiter for a long time if you can avoid it; in this version the permit is acquired first for simplicity. Cancellation is the last piece: the main thread awaits done with the overall deadline, and on timeout or Ctrl-C it calls pool.shutdownNow(), which interrupts every in-flight fetch.

Runtime.getRuntime().addShutdownHook(new Thread(() -> { pool.shutdownNow(); writer.flush(); }));
boolean finished = done.await(deadline.toSeconds(), TimeUnit.SECONDS);
if (!finished) pool.shutdownNow();

Testing

Run a real local server with WireMock so tests cover real HTTP, redirects and timeouts without touching the internet.

@WireMockTest
class CrawlerTest {
    @Test
    void crawlsOnlyOnceAndHonoursRobots(WireMockRuntimeInfo wm) throws Exception {
        stubFor(get("/robots.txt").willReturn(ok("User-agent: *\nDisallow: /private")));
        stubFor(get("/").willReturn(ok("<a href='/a'>a</a><a href='/a#x'>a2</a><a href='/private'>p</a>")));
        stubFor(get("/a").willReturn(ok("<a href='/'>home</a>")));

        var results = new Crawler(Config.of(wm.getHttpBaseUrl(), 3, 4, 100.0)).run();

        assertThat(results).extracting(r -> r.url().getPath()).containsExactlyInAnyOrder("/", "/a", "/private");
        verify(1, getRequestedFor(urlEqualTo("/a")));
        verify(0, getRequestedFor(urlEqualTo("/private")));
    }
}

Add a second test with a stub that delays 5 seconds, a 200 ms client timeout, and an assertion that the result carries a timeout error and the run still terminates.

Bugs you will hit

• Counting pending work inside the task. If the increment happens after submit returns, the counter can hit zero while children are about to be queued, and done fires early with half the crawl missing. Increment before the task starts, as above.
• Check-then-act on the visited set. if (!visited.contains(u)) visited.add(u) lets two threads both pass and crawl the page twice. Use the return value of add.
• Swallowing InterruptedException. A bare catch (Exception e) in the worker eats the interrupt, and shutdownNow appears to do nothing. Rethrow or call Thread.currentThread().interrupt(), and never catch InterruptedException into an error result silently.
• Holding synchronized across sleep or I/O. Blocking inside a monitor serialises every thread and, on Java 21, pins the carrier thread. Reserve the slot inside the lock and sleep outside it, or use ReentrantLock.
• Unbounded crawl traps. Calendars and session ids generate infinite unique URLs. Cap total pages, strip tracking parameters in normalise, and stop at maxDepth.

Interview angle

• Why do you still need a Semaphore if virtual threads are cheap? Threads are no longer the scarce resource; sockets, memory per response, and the remote server are. The semaphore caps in-flight requests so you do not open 50,000 connections.
• How does cancellation work with blocking calls? Interrupting a virtual thread unblocks HttpClient.send with InterruptedException. Cancellation is cooperative, so every blocking call must propagate the interrupt rather than swallow it.
• What is the difference between ExecutorService.shutdown and shutdownNow? shutdown stops accepting tasks and lets queued ones finish; shutdownNow also interrupts running tasks and returns the ones never started. A crawler that must stop on Ctrl-C needs the second.
• How would StructuredTaskScope change this design? Each page's children become subtasks of a scope, so a deadline or a failure cancels the whole subtree and no task outlives its parent. It is preview in Java 21, so most production code still uses an executor with an explicit pending counter.