File I/O & Serialization

InputStream/OutputStream move raw bytes; Reader/Writer move chars and perform charset decoding/encoding. Use streams for binary data (images, archives) and readers for text.

• InputStreamReader and OutputStreamWriter are the bridges from bytes to chars and need a charset.
• FileReader and FileWriter use the platform default charset before Java 18 and UTF-8 from Java 18 (JEP 400).

Every unbuffered read()/write() on a FileInputStream is a system call. Wrapping in BufferedInputStream/BufferedReader (default 8 KB) batches them and reduces syscalls dramatically.

• BufferedWriter needs flush() or close() to push data out.
• Bulk reads such as read(byte[]) with a large array already amortize syscalls; extra buffering adds little.
• Files.newBufferedReader returns a buffered reader by default.

Streams wrap other streams to add behaviour: new DataInputStream(new BufferedInputStream(new FileInputStream(f))). Each layer adds one concern (buffering, primitive decoding, compression, encryption).

• Closing the outermost wrapper closes the whole chain.
• Order matters: put GZIPOutputStream between the file and the buffer or the buffer in front, depending on what you want compressed.

Resources implementing AutoCloseable declared in the header are closed automatically in reverse order of declaration, whether the block completes normally or throws.

try (var in = Files.newInputStream(src);
     var out = Files.newOutputStream(dst)) {
    in.transferTo(out);
}

• If both the body and close() throw, the body exception is primary and the close exception is attached via addSuppressed.
• Java 9+ allows effectively final variables: try (in).

An exception from close() in finally replaces the original exception, hiding the root cause, and nested resources require nested try blocks to avoid leaks.

• Try-with-resources preserves the primary exception and suppresses the rest.
• A null resource is allowed in the header and simply skipped on close.

A charset maps characters to bytes. Reading with the wrong charset yields garbage (mojibake) or replacement characters. Always specify one explicitly via StandardCharsets.UTF_8.

• Before Java 18 the default came from file.encoding/OS locale; since Java 18 it is UTF-8 for APIs like FileReader, String.getBytes().
• System.out and console use stdout.encoding (Java 19+), which can still differ.
• Files.readString and Files.writeString default to UTF-8 regardless of platform.

InputStreamReader and new String(bytes, cs) replace malformed bytes with U+FFFD silently. Files.readString and Files.newBufferedReader throw MalformedInputException. To control behaviour use CharsetDecoder.onMalformedInput(CodingErrorAction.REPORT).

var dec = StandardCharsets.UTF_8.newDecoder()
    .onMalformedInput(CodingErrorAction.REPORT);
try (var r = new BufferedReader(new InputStreamReader(in, dec))) { /* ... */ }

File is a legacy class with poor error reporting (booleans), no symlink support and limited metadata. Path plus Files (NIO.2, Java 7) is the modern API with exceptions, attributes, symlinks, atomic moves and walking.

• Convert with file.toPath() and path.toFile().
• Path is immutable and filesystem-aware (zip filesystems, in-memory test filesystems).

resolve joins paths (an absolute argument replaces the base); relativize computes the path between two paths of the same type; normalize removes . and .. lexically.

Path base = Path.of("/data/app");
base.resolve("logs/a.log");          // /data/app/logs/a.log
base.resolve("/etc/passwd");         // /etc/passwd
Path.of("/a/b/../c").normalize();    // /a/c
Path.of("/a/b").relativize(Path.of("/a/c/d")); // ../c/d

readString, writeString, readAllBytes, readAllLines, lines, copy, move, delete, deleteIfExists, createDirectories, walk, list, newBufferedReader/Writer, size, getLastModifiedTime.

• readAllBytes and readString load the whole file into memory; unsuitable for big files.
• Files.lines, Files.walk and Files.list return lazy streams that hold an open file handle.

These streams wrap an open file or directory handle, released only when the stream is closed. Leaking them eventually causes "Too many open files".

try (Stream<String> lines = Files.lines(p)) {
    long n = lines.filter(l -> l.contains("ERROR")).count();
}

• Files.lines wraps checked IOException in UncheckedIOException during iteration.

list is one directory level, walk is a lazy depth-first stream to a max depth, and walkFileTree uses a FileVisitor with callbacks and control flow (SKIP_SUBTREE, TERMINATE) and per-file error handling.

• Use walkFileTree for recursive delete or copy where you must handle postVisitDirectory and permission errors.
• walk aborts the whole stream on an AccessDeniedException (wrapped in UncheckedIOException).

By default both fail with FileAlreadyExistsException if the target exists. REPLACE_EXISTING overrides it; ATOMIC_MOVE requests an atomic rename and throws AtomicMoveNotSupportedException if the filesystem cannot guarantee it.

• Copying a directory copies only the empty directory, not its content.
• Moves across filesystems degrade to copy plus delete, which is not atomic.
• COPY_ATTRIBUTES keeps timestamps where supported.

Register a directory with a WatchService for ENTRY_CREATE, ENTRY_MODIFY, ENTRY_DELETE; then loop on take()/poll() to receive WatchKeys, process pollEvents() and call key.reset().

try (WatchService ws = FileSystems.getDefault().newWatchService()) {
    dir.register(ws, StandardWatchEventKinds.ENTRY_CREATE);
    WatchKey key = ws.take();
    for (WatchEvent<?> e : key.pollEvents()) {
        System.out.println(e.context());
    }
    key.reset();
}

• It uses inotify on Linux, but on macOS the JDK polls (about 10 s delay by default, tunable via SensitivityWatchEventModifier in com.sun).
• Watching is not recursive; register each subdirectory yourself.

FileChannel is bidirectional, supports positional reads/writes, transferTo/transferFrom, memory mapping, locking and force(). It works with ByteBuffers, optionally direct ones.

• Positional read read(buf, position) is thread-safe and does not move the shared position.
• ByteBuffer requires flip() between write and read modes.
• Obtain via FileChannel.open(path, options) or stream.getChannel().

ByteBuffer.allocateDirect allocates native memory outside the heap, letting the OS do I/O without an intermediate copy from the heap array. It is costly to allocate and freed only when GC collects the buffer object.

• Good for long-lived, reused buffers on network or file channels.
• Capped by -XX:MaxDirectMemorySize (defaults to max heap); exceeding it throws OutOfMemoryError: Direct buffer memory.
• Heap buffers passed to channels are copied to a temporary direct buffer internally.

FileChannel.map(mode, position, size) returns a MappedByteBuffer whose contents are backed by the OS page cache. Reads and writes become memory accesses; the OS faults pages in lazily and writes dirty pages back.

try (FileChannel ch = FileChannel.open(p, StandardOpenOption.READ)) {
    MappedByteBuffer buf = ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size());
    byte first = buf.get(0);
}

• A single mapping is limited to Integer.MAX_VALUE bytes (2 GB) via ByteBuffer; map in windows for larger files.
• force() flushes dirty pages. Unmapping is tied to GC (no public unmap) before the Java 22 MemorySegment API with Arena.

It shines for random access to large read-mostly files and for sharing data between processes. It is not magically faster for sequential scans, where buffered FileChannel reads with a large buffer are comparable.

• Page faults are hidden I/O stalls, so latency becomes unpredictable and blocking cannot be interrupted.
• Many mappings consume virtual address space and are slow to release on 32-bit JVMs.
• There is no error handling per read; I/O errors surface as InternalError.

transferTo(position, count, target) lets the kernel move bytes directly (e.g., sendfile on Linux) between file and socket or file without copying through user space.

• It may transfer fewer bytes than requested; loop until done.
• Used by servers (Netty FileRegion, Tomcat) for static content delivery.
• Does not work with TLS since data must be encrypted in user space (unless kTLS).

Stream it: read line by line (or in chunks) with bounded memory, process incrementally and write results incrementally. Never use readAllBytes, readAllLines or readString on files that may not fit in the heap.

• BufferedReader.readLine or Files.lines for text; FileChannel plus a reusable buffer for binary.
• For parallelism, split by byte ranges aligned to newline boundaries or use Files.lines(...).parallel() cautiously (the spliterator on file-backed streams splits poorly before Java 9; improved for UTF-8 and ASCII line streams).
• Memory use must be independent of file size.

RandomAccessFile supports seek(pos) with reads and writes at arbitrary offsets in modes r, rw, rws and rwd. It suits fixed-record files and tailing files.

• rws forces both content and metadata to disk on each write; rwd forces content only. Both are slow.
• It is unbuffered, so wrap or use FileChannel for performance.

flush() only moves data from Java buffers to the OS. To persist, call FileChannel.force(true) (or FileDescriptor.sync()), which issues fsync. Even then, disk write caches can lie.

• Writes with StandardOpenOption.SYNC or DSYNC sync on every write.
• For a new file also fsync the parent directory (not possible in pure Java on some platforms), otherwise the directory entry may be lost on crash.
• Databases batch writes and fsync once per commit group for throughput.

Write to a temporary file in the same directory (same filesystem), fsync it, then Files.move(tmp, target, ATOMIC_MOVE). Readers see either the old or the new file, never a partial one.

Path tmp = Files.createTempFile(target.getParent(), "cfg", ".tmp");
try {
    Files.writeString(tmp, content);
    try (FileChannel ch = FileChannel.open(tmp, StandardOpenOption.WRITE)) {
        ch.force(true);
    }
    Files.move(tmp, target, StandardCopyOption.ATOMIC_MOVE);
} finally {
    Files.deleteIfExists(tmp);
}

Files.createTempFile and createTempDirectory create a uniquely named file atomically, with owner-only permissions (0600 / 0700) on POSIX. The location is java.io.tmpdir.

• Delete explicitly or use StandardOpenOption.DELETE_ON_CLOSE; File.deleteOnExit() keeps a list in memory that grows forever in long-running servers.
• The old File.createTempFile created files with default permissions.

FileChannel.lock() (blocking) or tryLock() returns a FileLock, either exclusive or shared, over a region. Locks are held on behalf of the whole JVM, not individual threads, and are advisory on most Unix systems.

• Within one JVM a second lock on an overlapping region throws OverlappingFileLockException.
• Behaviour on NFS and some network filesystems is unreliable.
• Typical use is a lock file to ensure a single application instance or serialize writers across processes.

Resolve the user input against a base directory, normalize and verify the result still starts with the base. Prefer ignoring client paths and generating your own filenames.

Path base = Path.of("/srv/uploads").toRealPath();
Path target = base.resolve(userName).normalize();
if (!target.startsWith(base)) {
    throw new SecurityException("Path traversal: " + userName);
}

• startsWith on Path compares name elements, so /srv/uploads-evil does not match /srv/uploads (unlike string prefix).
• Symlinks inside the base can still escape; check toRealPath on the existing target.

A class implementing the marker interface Serializable can be written by ObjectOutputStream.writeObject and rebuilt by ObjectInputStream.readObject. The runtime walks the object graph by reflection, writing class metadata and non-static, non-transient field values.

• Shared references and cycles are preserved via back-references within one stream.
• Constructors of serializable classes are not called on deserialization; the no-arg constructor of the first non-serializable superclass is.

It is a static final long that versions the serialized form. On read, the stream's UID must equal the local class's UID or InvalidClassException is thrown.

• If omitted, the JVM computes a hash from class name, interfaces, fields and methods; almost any change (even adding a method) changes it and breaks compatibility. Compilers may also compute it differently.
• Declare it explicitly: private static final long serialVersionUID = 1L;.
• Compatible changes: adding fields, adding classes in the hierarchy. Incompatible: removing a field type change, changing a class from Serializable to Externalizable.

transient excludes a field from default serialization. After deserialization it holds the default (null/0/false), so you must recompute it in readObject or readResolve.

• Use for caches, derived values, thread handles, passwords and file handles.
• Final transient fields cannot be reassigned in readObject without reflection, which makes them awkward.

Declare private methods writeObject(ObjectOutputStream) and readObject(ObjectInputStream) with exact signatures. Call defaultWriteObject()/defaultReadObject() first, then handle extra data.

private void readObject(ObjectInputStream in)
        throws IOException, ClassNotFoundException {
    in.defaultReadObject();
    if (age < 0) throw new InvalidObjectException("age");
    this.cache = new HashMap<>();
}

• readObject is effectively a public constructor: validate invariants there.
• The methods are found by reflection, so a wrong signature or visibility silently disables them.

writeReplace substitutes another object before writing; readResolve substitutes the deserialized object with another (often the canonical instance). readResolve preserves singleton and enum-like guarantees.

private Object readResolve() { return INSTANCE; }

• Without readResolve a serializable singleton can be duplicated by deserialization.
• writeReplace returning a small serialization proxy is the safest pattern (Effective Java).

The outer class's writeReplace returns a private static nested SerializationProxy that holds the logical state. The outer class's readObject throws InvalidObjectException to block direct deserialization, and the proxy's readResolve rebuilds the real object via its public constructor/factory.

• Invariants are enforced by the normal constructor, defeating forged streams.
• Allows fields to be final and decouples the serialized form from the implementation.
• java.time classes use it (Ser), as do immutable collections.

If a superclass is Serializable, so is the subclass and the superclass fields are serialized. If the superclass is not, its fields are not saved and on deserialization its no-arg constructor runs (which must be accessible, else InvalidClassException).

• A subclass can prevent serialization by throwing NotSerializableException from writeObject/readObject.
• Superclass state must be re-initialised via that constructor.

Externalizable extends Serializable with writeExternal/readExternal, giving full control of the format. The class must have a public no-arg constructor, which is called first and then readExternal fills in the state.

• Nothing is saved automatically, including superclass fields.
• Can be more compact and faster, but you own versioning and correctness.

readObject instantiates classes named in the stream and runs their readObject/readResolve/finalize-like code before the caller can check the type. With "gadget" classes on the classpath (commons-collections, Spring, etc.) an attacker can chain calls to achieve remote code execution or DoS.

• The cast after readObject happens too late; the damage is done during deserialization.
• Famous cases: Apache Commons Collections InvokerTransformer, many app-server CVEs.
• Also DoS: deeply nested or huge object graphs, hash collision payloads.

ObjectInputFilter (Java 9+, backported to 8u121) lets you allow or reject classes and cap depth, references, array length and stream bytes while reading. Configure per stream or process-wide.

ObjectInputFilter f = ObjectInputFilter.Config.createFilter(
    "com.acme.dto.*;java.base/*;maxdepth=10;maxbytes=1000000;!*");
ois.setObjectInputFilter(f);

• Pattern order matters and !* rejects everything not allowed (allow-list).
• Global: -Djdk.serialFilter=... or jdk.serialFilter in java.security.
• JEP 415 (Java 17) adds context-specific filter factories (ObjectInputFilter.Config.setSerialFilterFactory).

It is insecure by design, couples the wire format to class internals, is fragile across versions, is Java-only, verbose and slow. Oracle has called it a "horrible mistake" and long-term plans aim to remove it.

• Prefer explicit formats: JSON, Protobuf, Avro, with schemas and DTOs.
• If you must use it, restrict to trusted data, use filters, declare UID and consider serialization proxies.

A record implementing Serializable is serialized from its components only, and deserialization calls the canonical constructor, so constructor validation applies. readObject, writeObject, serialVersionUID matching and Externalizable customizations are ignored or restricted.

record Point(int x, int y) implements Serializable {
    Point { if (x < 0) throw new IllegalArgumentException(); }
}

• serialVersionUID can be declared, but its matching is not required (the default is 0L unless declared).
• This closes the "forged stream bypasses constructor" hole of normal classes.
• Jackson 2.12+ supports records natively.

JSON is human-readable, schema-less and verbose. Protobuf is compact binary with a compiled .proto schema and field numbers. Avro is compact binary where the schema travels with the data (or a registry) and is resolved at read time.

• JSON: easy debugging, browsers, loose typing, no built-in schema evolution rules.
• Protobuf: fast, tag-based forward/backward compatibility, great for gRPC.
• Avro: no field tags in data, needs writer's schema to read, strong for Kafka plus Schema Registry and big-data files.

Protobuf identifies fields by number: add optional fields with new numbers, never reuse or renumber, and mark deleted ones reserved. Unknown fields are skipped. Avro resolves the writer schema against the reader schema by field name: new fields need defaults, and removed fields must have had defaults for backward compatibility.

• Changing a Protobuf type (e.g., int32 to string) breaks compatibility; some integer widening is wire compatible.
• Avro supports aliases for renaming.
• Use a registry (Confluent) with compatibility modes BACKWARD, FORWARD, FULL.

ObjectMapper uses reflection (or generated accessors with modules) to map getters/fields to JSON properties and constructors/setters/@JsonCreator for deserialization. Create one ObjectMapper and reuse it; it is thread-safe once configured.

ObjectMapper om = new ObjectMapper()
    .registerModule(new JavaTimeModule())
    .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
String json = om.writeValueAsString(dto);
Dto back = om.readValue(json, Dto.class);

@JsonProperty (rename), @JsonIgnore, @JsonIgnoreProperties, @JsonCreator, @JsonFormat, @JsonInclude, @JsonTypeInfo/@JsonSubTypes for polymorphism, @JsonView, @JsonAnyGetter.

• Entities with bidirectional relationships cause infinite recursion; use @JsonManagedReference/@JsonBackReference, DTOs or @JsonIdentityInfo.
• Dates: without JavaTimeModule, java.time types fail (2.12+) or serialize as arrays; disable WRITE_DATES_AS_TIMESTAMPS for ISO strings.
• Prefer DTOs over serializing JPA entities (lazy proxies).

Enabling default typing (enableDefaultTyping, @JsonTypeInfo(use = Id.CLASS)) allows the JSON to name any class to instantiate, enabling gadget attacks similar to Java deserialization (many Jackson CVEs).

• Use @JsonTypeInfo(use = Id.NAME) with explicit @JsonSubTypes.
• If default typing is needed, use PolymorphicTypeValidator allow-lists (2.10+).
• Keep Jackson patched and avoid Object/Serializable typed fields from untrusted JSON.

Use the streaming API (JsonParser/JsonGenerator) or ObjectMapper.readerFor(T).readValues(parser) for arrays so only one element is in memory at a time.

try (JsonParser p = om.createParser(in)) {
    if (p.nextToken() != JsonToken.START_ARRAY) throw new IOException("array");
    while (p.nextToken() == JsonToken.START_OBJECT) {
        Event e = om.readValue(p, Event.class);
        handle(e);
    }
}

• readValue(file, List.class) loads everything.
• Jackson also enforces StreamReadConstraints (2.15+) such as max string length to guard against abusive payloads.

Jackson is the de-facto standard (Spring), feature-rich, with streaming, modules and afterburner/blackbird optimisations. Gson is simpler, uses fields by default and has fewer features. JSON is a poor fit for high-volume internal messaging, binary payloads (base64 overhead 33%), exact numeric precision and schema enforcement.

• Large long values lose precision in JavaScript consumers; send as strings.
• BigDecimal needs USE_BIG_DECIMAL_FOR_FLOATS to avoid double conversion on read.

Use ZipInputStream/ZipOutputStream for streaming, ZipFile for random access to entries, or the zip FileSystem provider (FileSystems.newFileSystem(path)) to treat an archive as a directory using Files.

try (ZipOutputStream zos = new ZipOutputStream(Files.newOutputStream(out))) {
    zos.putNextEntry(new ZipEntry("a/data.txt"));
    Files.copy(src, zos);
    zos.closeEntry();
}

• GZIPOutputStream compresses a single stream (no entries), commonly used for .gz logs.
• ZipFile reads the central directory, so it is faster for selective access; ZipInputStream must scan sequentially.

Zip Slip is a path traversal where an archive entry name like ../../etc/cron.d/x makes extraction write outside the target directory. Always resolve each entry against the destination, normalize, and verify it stays inside.

Path dest = destDir.toRealPath();
Path out = dest.resolve(entry.getName()).normalize();
if (!out.startsWith(dest)) throw new IOException("Bad entry: " + entry.getName());

• Also handle symlink entries in tar archives and absolute names.
• Create parent directories explicitly, and skip directory entries correctly.

A small archive that expands to gigabytes (or nests archives) to exhaust disk, memory or CPU. Defend by limiting total uncompressed bytes, per-entry size, entry count, nesting depth and the compression ratio while extracting.

• Do not trust ZipEntry.getSize(): it can be -1 or forged; count the bytes actually read.
• Abort once limits are exceeded and delete partial output.
• Run extraction in a quota-limited directory.

Stream rows using a real CSV parser (OpenCSV, Apache Commons CSV, univocity) with a BufferedReader, handling each record and discarding it. split(",") is wrong for quoted fields, embedded commas, newlines inside quotes and escaped quotes.

try (var reader = Files.newBufferedReader(p);
     CSVParser parser = CSVFormat.DEFAULT.builder().setHeader().setSkipHeaderRecord(true)
         .build().parse(reader)) {
    for (CSVRecord r : parser) { process(r.get("id")); }
}

• readLine-based parsing breaks on quoted multi-line fields.
• Batch inserts (e.g., 1000 rows per JDBC batch) rather than per-row.
• For export, write through BufferedWriter/response stream and never build a giant List first.

ObjectOutputStream keeps a strong reference to every object written in its handle table to encode back-references, so writing millions of distinct objects to one stream leaks memory. Call reset() periodically (or use writeUnshared) to clear the table; also mind that mutating an object after writing and re-writing it sends only a back-reference, not the new state.

• Same issue on the read side: ObjectInputStream retains deserialized objects.
• A new ObjectOutputStream writes a header; appending to an existing file requires a custom subclass overriding writeStreamHeader.

Since Java 9 InputStream.transferTo(OutputStream) copies everything with an internal buffer (16 KB, later 8 KB variants). Files.copy(InputStream, Path) and Files.copy(Path, OutputStream) cover the common cases.

• Before Java 9 write the loop: read into byte[], write the count read.
• Always use the returned count from read; writing the full array writes stale bytes.

read() returns an int 0-255 or -1 at EOF; read(byte[]) returns the number of bytes actually read (possibly fewer than the array size) or -1. Ignoring the return value is the classic bug.

• Use readNBytes(n) (Java 9+) or DataInputStream.readFully when you need exactly n bytes.
• available() is only an estimate of non-blocking bytes, not the file size or the end marker.
• skip(n) may skip fewer bytes than asked.

Stream line by line and keep no state beyond what you need. Memory stays constant, regardless of file size.

try (BufferedReader r = Files.newBufferedReader(log, StandardCharsets.UTF_8);
     BufferedWriter w = Files.newBufferedWriter(out)) {
    String line;
    while ((line = r.readLine()) != null) {
        if (line.contains("ERROR")) { w.write(line); w.newLine(); }
    }
}

• Files.readAllLines would throw OutOfMemoryError (and exceed the 2 GB array limit).
• Run time is bounded by disk throughput; grep may be faster for a one-off.

Split the file into byte ranges, adjust each start forward to the next newline and each end to finish the last line, then process ranges on a thread pool, each with its own FileChannel positional reads or RandomAccessFile. Merge partial results.

• Aligning boundaries: seek to the range start, skip to after the next \n (except range 0), and read until passing the range end.
• Gains are limited by disk throughput; on SSD/NVMe and cached files it scales with cores, on HDD random reads hurt.
• Use FileChannel.read(buf, pos) which is thread-safe.

The code probably called file.getBytes() or read the whole body into a byte[]/String. Stream the InputStream straight to disk or object storage instead.

@PostMapping("/upload")
void upload(@RequestParam MultipartFile file) throws IOException {
    try (InputStream in = file.getInputStream()) {
        Files.copy(in, storageDir.resolve(UUID.randomUUID().toString()));
    }
}

• Set spring.servlet.multipart.max-file-size and max-request-size; multipart parsers spool above file-size-threshold to disk.
• For real big uploads use chunked/resumable uploads or pre-signed S3 URLs so bytes bypass your JVM.
• Do not trust file.getOriginalFilename() for paths.

Never overwrite in place. Write a temp file in the same directory, force it, then ATOMIC_MOVE over the target. Readers then see either the old or the new complete file.

• Files.writeString(target, ...) truncates first, so a crash mid-write leaves an empty or partial file.
• Keep one backup generation (config.json.bak) for safety.
• Validate (parse/checksum) on load and fall back to the backup.

Open with StandardOpenOption.APPEND, which on POSIX makes each write atomic positioned at EOF, and write each record in a single write call. For multi-write records or non-local filesystems use FileLock, or better route logs through one writer.

• BufferedWriter can split a line across multiple write syscalls when its buffer fills; build the complete line (with newline) first and write once.
• Atomicity of appends is only guaranteed up to a size (PIPE_BUF applies to pipes, not files); NFS has no atomic append.
• Best practice: one logging process or a logging framework with file appender locking.

Output: abcd, then efcd, then 2. The second read fills only two bytes; c and d remain from the previous read. Ignoring the returned count prints stale data.

• Use new String(buf, 0, n).
• Real streams (sockets, files) may return fewer bytes than requested even before EOF.

The bytes are in a different encoding than the one used to decode them. UTF-8 text decoded as Latin-1 yields é for é. Windows-1252 text decoded as UTF-8 yields U+FFFD or exceptions.

• Ask the producer for UTF-8 (Excel's "CSV UTF-8" adds a BOM) and decode explicitly.
• Strip a leading  if present.
• Never rely on the platform default (before Java 18 it differed across servers).

JEP 400 made UTF-8 the default charset in Java 18+, whereas Java 11 used file.encoding from the OS locale (e.g., Windows-1252 or ISO-8859-1). Existing files written with the old platform default are now read as UTF-8, corrupting non-ASCII characters.

• Fix by specifying the charset on all constructors/methods, explicitly using the legacy charset for old files.
• Compatibility switch: -Dfile.encoding=COMPAT restores the legacy behaviour.
• native.encoding system property exposes the OS charset.

File descriptors are leaked: streams, readers or Files.list/walk/lines streams are not closed on every path (especially exceptions). The process hits the ulimit -n limit.

// leak
Files.list(dir).forEach(this::handle);
// fix
try (Stream<Path> s = Files.list(dir)) { s.forEach(this::handle); }

• Inspect with lsof -p <pid> or ls /proc/<pid>/fd | wc -l.
• Raising ulimit hides the leak; fix the code.
• GC finalisation/Cleaner may eventually close leaked streams, which makes failures intermittent.

Windows prevents deleting or renaming a file that is still open without share-delete semantics, so a leaked stream or an active memory mapping makes Files.delete throw FileSystemException ("being used by another process"). On Linux unlinking an open file succeeds; the data persists until the last descriptor closes.

• Ensure all handles are closed first; mapped buffers stay mapped until GC.
• On Linux, deleting an open log file still consumes disk space until the process closes it (common "disk full but du says empty" case).

Treat the archive as hostile: extract into a fresh temp directory, validate each entry, limit sizes, then process.

• Zip Slip: dest.resolve(name).normalize().startsWith(dest).
• Zip bomb: cap entry count, per-entry and total uncompressed bytes (counted as read), nesting and ratio.
• Reject or ignore symlinks, absolute paths, device-like names; sanitise names for the OS.
• Use a dedicated quota-limited directory, run with minimal privileges, delete on failure.
• Scan for malware before use and never execute extracted content.

By default FAIL_ON_UNKNOWN_PROPERTIES is true. Make consumers tolerant: disable it globally or annotate the type with @JsonIgnoreProperties(ignoreUnknown = true).

• Tolerant reader is a core rule for compatible evolution: ignore unknown fields, never repurpose existing ones.
• Spring Boot's auto-configured ObjectMapper already disables the feature, so the failure often comes from a hand-built mapper.

For gRPC/RPC choose Protobuf. For Kafka pipelines with many producers and consumers choose Avro (or Protobuf) with a schema registry enforcing compatibility. Choose JSON for public/browser APIs and low volume where debuggability matters.

• Measure payload size and CPU; binary formats save bandwidth and parse time.
• Registry plus CI compatibility check blocks breaking schema changes before deployment.
• Provide a debug tool (e.g., kafka-avro-console-consumer) to offset binary opacity.

The class changed and, with no explicit serialVersionUID, the computed UID changed, so old cache entries cannot be read by the new version. Declare a fixed UID and only make compatible changes, or version cache keys and flush on deploy.

• Better: store JSON/Protobuf in Redis with a version in the key.
• During rolling deploys old and new nodes share the cache; incompatible shapes cause errors on one side.

Deserialization prints Base() only. Base is not serializable, so its no-arg constructor runs and b is reset to 5; Child() is not invoked. The result has c=2 (restored) and t=0 (transient, default value, field initialisers not run).

• Serialization time prints Base() and Child() once from the original construction.

Best: remove native deserialization from untrusted input and switch to JSON/Protobuf with explicit DTOs. If it cannot be removed immediately, apply an allow-list ObjectInputFilter, limit depth/size, authenticate and sign payloads, and remove gadget libraries or update them.

ObjectInputFilter f = ObjectInputFilter.Config.createFilter(
    "com.acme.api.Request;java.base/java.lang.*;java.base/java.util.*;maxdepth=5;maxbytes=65536;!*");
ois.setObjectInputFilter(f);

• Use HMAC signatures if only your own services write the data; never encrypt-only.
• Monitor with -Djdk.serialFilter plus logging to learn which classes appear.

Jackson 2.12+ calls the canonical constructor with null for the missing component, so the compact constructor's checks run and throw; Jackson wraps it in ValueInstantiationException.

record User(String name) {
    User { Objects.requireNonNull(name, "name"); }
}
// {} -> ValueInstantiationException caused by NullPointerException: name

• Without a check, the record silently holds null.
• Bean Validation annotations are not enforced by Jackson; validate separately.

No. For records the serialVersionUID match requirement is waived; the stream is matched by component names and types, then the canonical constructor is invoked. The default UID for a record is 0L unless declared.

• Compatibility depends on component names and types: adding a component gives default value (null/0) for old streams, which is passed to the constructor.
• Constructor validation may then reject old data.

ENTRY_CREATE fires when the file appears, not when the copy completes, and ENTRY_MODIFY may fire repeatedly. Do not process on first event.

• Producers should write to a temp name or directory and atomically rename into the watched folder.
• Otherwise debounce: wait until size and mtime are stable, or until a .done marker appears, or try an exclusive lock/open.
• Handle OVERFLOW by rescanning the directory; keep a processed-files ledger for idempotency.

Linux uses inotify (event-driven). The JDK on macOS has no native FSEvents backend for the default WatchService and falls back to a polling implementation with a default interval of about 10 seconds.

• Use the com.sun.nio.file.SensitivityWatchEventModifier.HIGH modifier (about 2 s; deprecated and removed in recent JDKs) or a library using FSEvents (e.g., directory-watcher).
• Production services typically run on Linux, but dev environments differ; tests should not assume timing.

Use an exclusive FileLock on a dedicated lock file (not the data file you atomically replace) that both sides acquire, or use an atomic write so concurrent readers are safe and one writer wins.

try (FileChannel ch = FileChannel.open(lockFile, CREATE, WRITE);
     FileLock lock = ch.lock()) {
    // read-modify-write under lock
}

• Lock the stable lock file because renaming the data file replaces the inode and the lock on the old one no longer protects the new one.
• The other process must use the same protocol (advisory locks).

Open a lock file and call tryLock(); if it returns null another process holds it, so exit. Keep the channel and lock referenced for the process lifetime (a GC'd channel releases the lock).

FileChannel ch = FileChannel.open(Path.of("/var/run/app.lock"), CREATE, WRITE);
FileLock lock = ch.tryLock();
if (lock == null) { System.exit(1); }

• Do not delete the lock file on exit as that causes races; the lock, not the file existence, is the guard.
• Not reliable on NFS.

Temp files are created per request but never deleted; deleteOnExit only runs at JVM shutdown (and also leaks its internal path list), so a long-lived server accumulates them.

• Delete in finally/try-with-resources, or open with DELETE_ON_CLOSE.
• Use a dedicated subdirectory per job and a scheduled cleaner by age.
• Configure -Djava.io.tmpdir to a quota-limited volume; tmpfs /tmp counts against RAM.

For files above about 2 GB it throws OutOfMemoryError: Required array size too large since arrays are indexed by int. Under the limit it may still exhaust the heap.

• Stream with a buffer and process chunks.
• Memory-map in windows when random access is needed.
• Use a FileChannel with a reused ByteBuffer.

map throws IllegalArgumentException: Size exceeds Integer.MAX_VALUE. Map the file in multiple windows (e.g., 1 GB each) and index across them, or use the Java 22+ FFM API where FileChannel.map(mode, offset, size, Arena) returns a MemorySegment with long offsets.

long win = 1L << 30;
for (long pos = 0; pos < size; pos += win) {
    MappedByteBuffer b = ch.map(MapMode.READ_ONLY, pos, Math.min(win, size - pos));
    scan(b);
}

• Records straddling a window boundary need overlap handling.
• Unmapping is GC-driven for MappedByteBuffer; many large windows can hit address-space or OutOfMemoryError: Map failed.

The mapping is shared via the page cache, so writes by one process are visible to the other, but without proper ordering. Use a file layout with a sequence number or header, write payload first and then publish the length/flag with ordered or volatile semantics (VarHandle) and handle torn reads.

• Java gives no cross-process mutexes in mapped memory; use atomic fields via VarHandle on a ByteBuffer, or a lock file.
• force() is for durability, not visibility between processes.
• Libraries like Chronicle Queue and Aeron implement this robustly.

Write directly to the response stream (StreamingResponseBody in Spring MVC, or HttpServletResponse.getOutputStream()), fetching rows with a streaming cursor, and flush periodically.

@GetMapping("/export")
StreamingResponseBody export() {
    return out -> {
        try (var w = new BufferedWriter(new OutputStreamWriter(out, UTF_8))) {
            repo.streamAll(row -> write(w, row));
        }
    };
}

• Use DB fetch size/cursor (setFetchSize, Stream return types) in a transaction.
• Streaming means errors after the first bytes cannot change the HTTP status; the client sees a truncated response.
• Configure async timeouts and a task executor.

split(",") yields four fields (1, "Smith, John", NY) because it ignores quotes. Use a proper CSV parser that implements RFC 4180 quoting and escaped double quotes ("").

• Also embedded newlines within quotes break readLine loops.
• Trailing empty fields are dropped by split(",") without a limit argument (split(",", -1) keeps them).

Stream the file, batch JDBC inserts (500-5000 rows per executeBatch and commit), use rewriteBatchedStatements=true (MySQL) or reWriteBatchedInserts=true (PostgreSQL), or the DB bulk loader (COPY, LOAD DATA).

• Drop or defer secondary indexes/constraints during the load, then rebuild.
• Parse and insert in a pipeline: reader thread to bounded queue to writer threads.
• Track progress (last committed line) for restartability.

It prints body 1. The body exception is primary, and the close exception is attached as a suppressed exception. With old try/finally, close's exception would have replaced body.

• Log e.getSuppressed() in diagnostics; many loggers print them automatically.

If the GZIPInputStream constructor throws (e.g., file is not gzip, header read fails, ZipException/EOFException), the FileInputStream was already created but never assigned to a resource, so it leaks.

try (var fin = new FileInputStream(path);
     var gz = new GZIPInputStream(fin)) { /* read */ }

• Files.newInputStream plus a separate declaration avoids the leak.
• Most JDK stream constructors that read headers can throw.

Never use client names as paths. Generate a server-side name (UUID) and store the original name as metadata. If you must use the name, take only Path.of(name).getFileName(), normalize and check startsWith(base).

• Reject names with NUL, separators, reserved Windows names.
• Store uploads outside the web root and without execute permission.
• Validate content type by sniffing, enforce size limits.

normalize() is purely lexical so uploads/link/passwd passes startsWith(base) but actually reads /etc/passwd. Resolve real paths and use LinkOption.NOFOLLOW_LINKS.

Path real = target.toRealPath();
if (!real.startsWith(base.toRealPath())) throw new SecurityException();
Files.readString(target); // or open with NOFOLLOW_LINKS

• Still racy (TOCTOU): a link can be swapped between check and use; open with NOFOLLOW_LINKS and prefer a directory the app alone controls.
• Zip extraction must not create symlinks from entries.

Write to report.csv.tmp (or a hidden name) in the same directory and Files.move with ATOMIC_MOVE to report.csv once complete. Readers only match final names, never *.tmp.

• Alternative: write a report.csv.done marker last.
• Do not rely on file size stabilising as proof of completion.

Renaming an open file is fine on POSIX since the handle follows the inode, so the writer would keep writing to the renamed file. Either let the logging framework (Logback/Log4j2 rolling appenders) close and reopen, or use copy-truncate (which can lose lines written between the copy and truncation).

• With external logrotate, use create plus a signal to reopen, or copytruncate as a last resort.
• On Windows, renaming an open file fails.

Entries sit in the 8 KB Java buffer (and then the OS page cache) until flushed. A crash or kill -9 loses buffered data; a power loss loses even flushed-but-not-fsynced data.

• Call flush() per record for process-crash safety.
• For power-loss durability use FileChannel.force or SYNC open options, balanced against throughput (group commit).
• Audit logs often go to a log pipeline or DB with transactional guarantees instead.

The console encoding is derived from the container's locale (often POSIX/ANSI_X3.4-1968), so unmappable characters become ?. The data in memory is correct.

• Set LANG=C.UTF-8, -Dstdout.encoding=UTF-8 (Java 19+) or -Dfile.encoding=UTF-8 on older JVMs, or write bytes via new PrintStream(System.out, true, UTF_8).
• For files, always specify the charset when writing.

The stream records fully qualified class names. Renaming or moving a class breaks reads of old data.

• Workarounds: keep the old class as a stub, override ObjectInputStream.resolveClass to map old names to new classes, or migrate data with a converter job.
• Avoid storing Java-serialized blobs long-term; use schema'd formats.

NotSerializableException for the first non-serializable field encountered. Mark such fields transient and re-create them lazily, or better, do not make resource-holding classes serializable; serialize a DTO with the needed state.

• Inner classes capture the outer instance, which can drag a non-serializable outer object into the graph.
• Lambdas are serializable only if the target interface is and captured values are serializable.

Deserialization creates a new instance bypassing the private constructor. Add private Object readResolve() { return INSTANCE; } or, better, implement the singleton as a single-element enum.

• Reflection can also break a private-constructor singleton; enums defeat both.
• Make fields transient so no forged state leaks through.

Unbuffered single-byte reads issue about a billion system calls and can be 100-1000x slower than buffered or bulk reads. A BufferedInputStream or read(byte[64k]) reduces it to tens of thousands of calls.

• The JIT cannot remove syscalls; the kernel transition dominates.
• Optimal buffer size is typically 8-64 KB; very large buffers give diminishing returns.

Chain the decompressing stream into a reader; decompression is streamed with constant memory.

try (var in = new GZIPInputStream(Files.newInputStream(p), 64 * 1024);
     var r = new BufferedReader(new InputStreamReader(in, UTF_8))) {
    r.lines().filter(l -> l.contains("ERROR")).forEach(System.out::println);
}

• Gzip is not seekable, so parallel splitting needs bgzip or block-based formats (zstd frames, Parquet).
• CPU is likely the bottleneck rather than disk.

Most likely an incompatible schema change (new required field without default, type change, removed field without default) or the consumer cannot fetch the writer schema from the registry.

• With a registry, enforce BACKWARD (or FULL) compatibility so registration of a breaking schema is rejected at publish time.
• Consumers use the writer schema id embedded in the message to resolve against their reader schema.
• Deploy consumers first for backward compatible changes, producers first for forward compatible.

Old messages or old senders still emit the old number with its old type. The new reader interprets the bytes using the new type: the data is silently corrupted or fails with parse errors. Field numbers must never be reused.

message User {
  reserved 3;
  reserved "legacy_email";
  string email = 4;
}

• reserved makes protoc reject accidental reuse.
• Same wire type reuse (e.g., int32 to int64) may appear to work but still has semantic risks.

JavaScript numbers are IEEE 754 doubles with safe integers up to 2^53 - 1. A 64-bit id such as 9007199254740993 is rounded when parsed. Serialize ids as strings (@JsonSerialize(using = ToStringSerializer.class) or JsonFormat.Shape.STRING).

• Same problem with BigDecimal money values parsed by JS Number.
• Use ISO-8601 strings for dates and instants.

JavaTimeModule writes dates as numeric arrays when WRITE_DATES_AS_TIMESTAMPS is enabled (default). Disable the feature to get ISO-8601 text like "2026-10-04T09:30:00".

ObjectMapper om = new ObjectMapper()
    .registerModule(new JavaTimeModule())
    .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);

• Spring Boot sets this by default (spring.jackson.serialization.write-dates-as-timestamps=false).
• Prefer Instant/OffsetDateTime over LocalDateTime for wire formats to avoid timezone ambiguity.

readValue builds all Event objects (plus parser overhead) in memory. Iterate using the streaming parser or MappingIterator, handling one event at a time.

try (MappingIterator<Event> it =
         om.readerFor(Event.class).readValues(file)) {
    while (it.hasNext()) handle(it.next());
}

• This works for concatenated objects (NDJSON). For a top-level array use a JsonParser positioned after START_ARRAY.
• For files with one huge object, consider the pull parser and extract only needed fields.

It is real. Default typing lets payloads name arbitrary classes, which allows gadget chain RCE with vulnerable classes on the classpath. Replace with explicit subtype registration or a restrictive PolymorphicTypeValidator.

PolymorphicTypeValidator ptv = BasicPolymorphicTypeValidator.builder()
    .allowIfSubType("com.acme.events.")
    .build();
om.activateDefaultTyping(ptv, ObjectMapper.DefaultTyping.NON_FINAL);

• Better still: @JsonTypeInfo(use = Id.NAME) and @JsonSubTypes.
• Keep Jackson up to date; it blocks known gadget classes.

ZipInputStream/ZipEntry expose modified time but not Unix permission bits. Apache Commons Compress (ZipArchiveEntry.getUnixMode, TarArchiveEntry.getMode) reads them, and you then apply with Files.setPosixFilePermissions. Never apply setuid bits from untrusted archives.

• Set Files.setLastModifiedTime from the entry.
• For executables you ship yourself, restore mode 755 explicitly.
• Windows ignores POSIX attributes (UnsupportedOperationException for POSIX views).

Checks like canWrite()/Files.isWritable are advisory snapshots: permissions, ACLs, mount options and locks can change or differ from the effective access when you open the file. Just try the operation and handle the exception (EAFP).

• AccessDeniedException is a subclass of FileSystemException with the reason; legacy File methods return just false.
• Check-then-act sequences are racy (exists then create); use CREATE_NEW for atomic creation.

Across filesystems Files.move is a copy followed by a delete, not an atomic rename. A crash mid-copy leaves a partial target, and a crash between copy and delete leaves duplicates. ATOMIC_MOVE would fail with AtomicMoveNotSupportedException instead of degrading.

• Copy to target.tmp on the destination filesystem, verify size or checksum, atomically rename within that filesystem, then delete the source.
• Make the job idempotent: skip when the destination exists with matching checksum.

Avoid Java serialization (slow, fragile). Use a compact schema'd binary format (Protobuf/Avro/custom length-prefixed records) written as a stream, with a header (magic, version, count) and a trailing checksum, and load it in a streaming fashion.

• Write atomically: temp file, fsync, rename.
• Compress with LZ4/zstd if disk-bound.
• For random access and zero deserialisation consider memory-mapped fixed layouts (as Chronicle Map or Lucene do).
• Keep a version field so a loader can reject or migrate old snapshots, and fall back to a cold start on corruption.