Filestat CLI

A single command that takes a path and prints files scanned, total bytes, lines, words, the largest files and a breakdown by extension. It sounds trivial until you meet symlink loops, unreadable directories, files larger than memory, non-UTF-8 bytes, and the fact that reading 200,000 small files one at a time is dominated by I/O wait. That last point is why this is a good first concurrency project: the work is blocking I/O, which is exactly what virtual threads are cheap at.

Target behaviour

Build until your program prints this. The numbers will differ, the layout should not.

$ filestat ./myproject
PATH  ./myproject          FILES 1,284     SIZE 42.1 MB     TIME 310ms

BY EXTENSION              FILES      LINES       BYTES
  .java                     842    118,402      3.9 MB
  .json                      61      9,882      1.2 MB
  .md                        18      2,104      212 KB

LARGEST
  1.  12.1 MB  ./testdata/dump.json
  2.   3.4 MB  ./src/gen/Schema.java

$ filestat --json --top 5 --ext .java --workers 8 ./myproject | jq .totals
$ filestat --exclude 'target,node_modules,.git' .
$ filestat --version
filestat 1.0.0

Architecture

  Main (picocli)        args -> Config, then Runner.run(config, PrintStream)
     |
  walker (producer)     Files.walkFileTree -> BlockingQueue<Path>     [1 thread]
     |
  workers (fan-out)     N virtual threads: read + count -> FileStat
     |
  collector (fan-in)    merge into Report (single owner, no locks)
     |
  renderer              TextRenderer or JsonRenderer -> PrintStream

Why a PrintStream parameter and not System.out: the renderer becomes testable with a ByteArrayOutputStream, and --json is a one-line switch. Why a single collector: each worker returns an immutable record, and only one thread mutates the aggregate, so no synchronized is needed.

The core of it

Model the per-file result and the whole report as records. A record gives you equals, toString and Jackson serialisation for free.

public record FileStat(Path path, String ext, long bytes, long lines, long words) {}

public record ExtTotals(long files, long lines, long words, long bytes) {
    ExtTotals plus(FileStat s) {
        return new ExtTotals(files + 1, lines + s.lines(), words + s.words(), bytes + s.bytes());
    }
    static final ExtTotals ZERO = new ExtTotals(0, 0, 0, 0);
}

Count by streaming through a buffered reader, never Files.readString, so a 4 GB log does not become a 4 GB heap. Malformed bytes would make a strict decoder throw, so decode leniently.

static FileStat count(Path p) throws IOException {
    long lines = 0, words = 0;
    boolean inWord = false;
    var decoder = StandardCharsets.UTF_8.newDecoder()
        .onMalformedInput(CodingErrorAction.REPLACE)
        .onUnmappableCharacter(CodingErrorAction.REPLACE);
    try (var in = new BufferedReader(new InputStreamReader(Files.newInputStream(p), decoder), 1 << 16)) {
        int c;
        while ((c = in.read()) != -1) {
            if (c == '\n') lines++;
            boolean space = Character.isWhitespace(c);
            if (!space && !inWord) words++;
            inWord = !space;
        }
    }
    return new FileStat(p, extOf(p), Files.size(p), lines, words);
}

The walker skips excluded directories entirely by returning SKIP_SUBTREE, and records unreadable entries instead of aborting the walk. It does not follow symlinks unless asked, which also avoids cycles.

Files.walkFileTree(root, new SimpleFileVisitor<>() {
    @Override public FileVisitResult preVisitDirectory(Path d, BasicFileAttributes a) {
        return excluded.contains(d.getFileName().toString()) ? SKIP_SUBTREE : CONTINUE;
    }
    @Override public FileVisitResult visitFile(Path f, BasicFileAttributes a) throws IOException {
        if (a.isRegularFile()) put(queue, f);
        return CONTINUE;
    }
    @Override public FileVisitResult visitFileFailed(Path f, IOException e) {
        errors.add(f + ": " + e.getMessage());
        return CONTINUE;
    }
});

Workers run on a virtual-thread-per-task executor. Bound the in-flight work with the queue size and a fixed worker count, so the walker is slowed by a full queue (backpressure) instead of buffering every path in memory. A poison pill (Path.of("") sentinel) ends each worker.

try (var pool = Executors.newVirtualThreadPerTaskExecutor()) {
    List<Future<List<FileStat>>> parts = new ArrayList<>();
    for (int i = 0; i < workers; i++) parts.add(pool.submit(() -> drain(queue)));
    walk(root, queue);
    for (int i = 0; i < workers; i++) queue.put(POISON);
    Report report = new Report();
    for (var f : parts) f.get().forEach(report::add);
    return report;
}

Output uses Jackson for --json. Put the CLI together with picocli annotations; Callable<Integer> makes the return value the exit code.

@Command(name = "filestat", mixinStandardHelpOptions = true, version = "filestat 1.0.0")
public class Main implements Callable<Integer> {
    @Parameters(index = "0", defaultValue = ".") Path root;
    @Option(names = "--json") boolean json;
    @Option(names = "--top", defaultValue = "10") int top;
    @Option(names = "--workers", defaultValue = "8") int workers;
    @Option(names = "--exclude", split = ",") Set<String> exclude = Set.of(".git");

    public Integer call() throws Exception {
        Report r = Runner.scan(root, workers, exclude);
        (json ? new JsonRenderer() : new TextRenderer()).render(r, top, System.out);
        return r.errors().isEmpty() ? 0 : 1;
    }
    public static void main(String[] a) { System.exit(new CommandLine(new Main()).execute(a)); }
}

Testing

Build a tiny tree in a JUnit @TempDir, run the scanner, and assert on the exact aggregate. No mocks are needed because the file system is the real dependency.

class RunnerTest {
    @Test
    void countsLinesWordsAndExtensions(@TempDir Path dir) throws Exception {
        Files.writeString(dir.resolve("a.java"), "one two\nthree\n");
        Files.writeString(dir.resolve("b.md"), "hello\n");
        Files.createDirectory(dir.resolve(".git"));
        Files.writeString(dir.resolve(".git/config"), "ignored\n");

        Report r = Runner.scan(dir, 4, Set.of(".git"));

        assertThat(r.totalFiles()).isEqualTo(2);
        assertThat(r.byExt().get(".java").lines()).isEqualTo(2);
        assertThat(r.byExt().get(".java").words()).isEqualTo(3);
        assertThat(r.errors()).isEmpty();
    }
}

Add a second test that runs the scan with 1 worker and with 16 workers and asserts the reports are equal. If it ever flakes, you have shared mutable state.

Bugs you will hit

• Using Files.walk and losing errors. The stream throws UncheckedIOException on the first unreadable directory and stops the whole scan. Use walkFileTree with visitFileFailed, or accept that one Permission denied ends the run.
• Never closing the stream from Files.lines or Files.walk. Both hold an open directory or file handle. Wrap in try-with-resources or you hit Too many open files on a large tree.
• MalformedInputException on binary files. Files.readAllLines and Files.lines decode strictly as UTF-8 and throw on the first bad byte. Use a decoder with REPLACE, or skip files whose first 8 KB contains a NUL byte.
• Symlink cycles and double counting. walkFileTree does not follow links by default, which is right. If you add --follow, pass FileVisitOption.FOLLOW_LINKS and handle FileSystemLoopException in visitFileFailed.
• Mutating a HashMap from worker threads. It appears to work on small trees and corrupts counts on large ones. Return immutable FileStat records and merge them in one thread, or use ConcurrentHashMap.merge.

Interview angle

• Why virtual threads here instead of a fixed pool sized to the CPU count? The work is blocking file I/O, so CPU cores are idle most of the time. A platform thread pool needs tuning to hide latency; virtual threads park cheaply on blocking calls and let the carrier thread run another task. For pure CPU counting, a pool of availableProcessors() is still the right tool.
• Does a virtual thread make file reads non-blocking? Not at the OS level. Regular file I/O still blocks a carrier thread in many cases, and the JDK compensates by temporarily adding carriers. The win is the programming model, not magic asynchronous file reads.
• How do you stop one slow producer from exhausting memory? Use a bounded ArrayBlockingQueue. put blocks the walker when workers fall behind, which is backpressure. An unbounded LinkedBlockingQueue hides the problem until an OutOfMemoryError.
• Files.walk or walkFileTree? Files.walk is a lazy stream and convenient, but error handling and pruning are awkward. walkFileTree gives per-directory pruning and per-file failure callbacks, which is what a robust tool needs.