Java Networking

TCP is a connection-oriented, reliable, ordered byte stream with flow and congestion control. UDP is connectionless, sends independent datagrams, and gives no delivery, ordering or duplicate protection.

• TCP: three-way handshake, retransmission, no message boundaries.
• UDP: low overhead, preserves datagram boundaries, suited to DNS, metrics, streaming and QUIC (which builds reliability on top).
• Java: Socket/ServerSocket/SocketChannel for TCP, DatagramSocket/DatagramChannel for UDP.

Handshake: client sends SYN, server replies SYN-ACK, client sends ACK. Teardown uses FIN/ACK in each direction (four segments), so each side closes independently (half-close).

• connect() returns after the SYN-ACK/ACK exchange; accept() returns connections already completed by the kernel.
• An abrupt close sends RST instead of FIN.
• The side that sends the first FIN enters TIME_WAIT.

ServerSocket binds to a port and accept() blocks until a client connects, returning a Socket. Each Socket exposes an InputStream and OutputStream for blocking reads and writes.

try (ServerSocket server = new ServerSocket(8080)) {
    while (true) {
        Socket s = server.accept();
        new Thread(() -> handle(s)).start();
    }
}

• new ServerSocket(port, backlog) sets the accept-queue length hint.
• Always close sockets (try-with-resources) to release the file descriptor.

• SO_TIMEOUT (setSoTimeout): max blocking time of a read/accept, throws SocketTimeoutException.
• SO_KEEPALIVE: OS sends probes on idle connections (default idle 2 hours on Linux).
• TCP_NODELAY: disables Nagle's algorithm so small writes go out immediately.
• SO_REUSEADDR: lets a server rebind a port with connections in TIME_WAIT.
• SO_LINGER: controls close() behaviour; linger 0 sends RST instead of FIN.

Nagle buffers small writes until the previous data is acknowledged, reducing tiny packets. Combined with delayed ACK on the peer, it can add up to ~40 ms latency to request/response protocols that issue several small writes.

• Disable with socket.setTcpNoDelay(true) for latency-sensitive RPC.
• Better fix: coalesce writes into one buffer or flush once.

Each connection holds a thread blocked in read. It is simple but each platform thread costs ~1 MB of stack reserve plus scheduling overhead, so tens of thousands of idle connections are impractical.

• Thread pools cap resource usage but then idle connections can starve active ones.
• Java 21 virtual threads let you keep the blocking style: a blocking socket read parks the virtual thread, not the carrier.

NIO (java.nio) uses channels, buffers and selectors. Classic IO is stream-oriented and blocking; NIO is buffer-oriented, can be non-blocking, and one thread can multiplex many channels.

• Channels (SocketChannel, ServerSocketChannel, FileChannel, DatagramChannel) are bidirectional.
• Data is read into / written from ByteBuffers.
• Selector reports which channels are ready.

A buffer has capacity (fixed), limit (first index not to read/write) and position (next index). After writing into a buffer, flip() sets limit=position and position=0 to read it.

• clear(): position=0, limit=capacity (ready to write; data not erased).
• compact(): copies unread bytes to the start and sets position after them, ready to write more; use after a partial read.
• rewind(): position=0, limit unchanged.

ByteBuffer b = ByteBuffer.allocate(8);
b.putInt(7);      // pos=4
b.flip();         // pos=0 limit=4
int v = b.getInt();   // 7

A heap buffer wraps a byte[] in the Java heap; a direct buffer lives in native memory. Channel I/O on a heap buffer copies through a temporary direct buffer, so direct buffers avoid one copy for I/O.

• Direct allocation is slower and costs more, and it is freed only when the buffer object is GC'd (via a cleaner), limited by -XX:MaxDirectMemorySize.
• Use direct buffers for long-lived, reused I/O buffers; pool them.
• OutOfMemoryError: Direct buffer memory signals exhaustion.

A channel configured non-blocking is registered with a Selector for interest ops (OP_ACCEPT, OP_CONNECT, OP_READ, OP_WRITE). select() blocks until at least one is ready, then you iterate selectedKeys().

Selector sel = Selector.open();
ServerSocketChannel ssc = ServerSocketChannel.open();
ssc.bind(new InetSocketAddress(8080));
ssc.configureBlocking(false);
ssc.register(sel, SelectionKey.OP_ACCEPT);
while (true) {
    sel.select();
    var it = sel.selectedKeys().iterator();
    while (it.hasNext()) {
        SelectionKey k = it.next(); it.remove();
        // handle accept/read
    }
}

• On Linux the implementation uses epoll, on macOS kqueue.

One or more event-loop threads wait on a selector, dispatch ready events to handlers, and handlers must not block. Multi-reactor variants use a boss loop to accept and worker loops to read/write.

• Netty NioEventLoopGroup, Tomcat NIO connector acceptor/poller and Vert.x use this model.
• Blocking work (DB, disk) is offloaded to a separate pool.

Java 7 AsynchronousSocketChannel/AsynchronousServerSocketChannel start an operation and report completion via Future or CompletionHandler, executed on an AsynchronousChannelGroup thread pool.

• Proactor style: you supply the buffer up front; the OS (IOCP on Windows, epoll-based emulation on Linux) completes it.
• Rarely used in production; Netty on selectors dominates.

Blocking vs non-blocking is whether the call waits when no data is ready. Synchronous vs asynchronous is whether the caller itself performs the read after readiness (sync) or is notified once the operation completed (async).

• Java IO: sync blocking. NIO with selector: sync non-blocking (multiplexed). NIO.2 async: asynchronous.

Netty is an asynchronous event-driven network framework over NIO (or epoll/io_uring natives). It hides selector complexity and gives pipelines, codecs, pooled buffers and backpressure hooks.

• Core pieces: EventLoopGroup, Channel, ChannelPipeline, ChannelHandler, ByteBuf, Bootstrap/ServerBootstrap.
• Powers Spring WebFlux (Reactor Netty), gRPC-Java, Cassandra and Elasticsearch clients.

The pipeline is an ordered chain of inbound and outbound handlers per channel. Inbound events (read, active) flow head to tail; outbound operations (write, flush) flow tail to head.

• Typical chain: SslHandler, HttpServerCodec, HttpObjectAggregator, business handler.
• ByteBuf has separate reader and writer indexes (no flip), is reference-counted and usually pooled.
• You must release() a ByteBuf you consume (or use SimpleChannelInboundHandler, which does it).

java.net.http.HttpClient is the standard modern HTTP client: immutable, thread-safe, supports HTTP/1.1 and HTTP/2, WebSocket, sync send and async sendAsync returning CompletableFuture.

HttpClient client = HttpClient.newBuilder()
    .connectTimeout(Duration.ofSeconds(3)).build();
HttpRequest req = HttpRequest.newBuilder(URI.create("https://example.com/api"))
    .timeout(Duration.ofSeconds(5)).GET().build();
HttpResponse<String> res = client.send(req, HttpResponse.BodyHandlers.ofString());
System.out.println(res.statusCode());

• Build one client and reuse it; it owns the connection pool and selector thread.

HttpClient.Builder.connectTimeout limits connection establishment; HttpRequest.Builder.timeout limits the wait for the response (throws HttpTimeoutException). There is no built-in read-idle or total-body timeout.

• For total deadlines, use CompletableFuture.orTimeout on sendAsync or cancel the future.
• Cancelling the future does not necessarily abort the in-flight request in older JDKs; JDK 16+ improved cancellation.

HTTP/2 is a binary, multiplexed protocol: many concurrent streams over one TCP connection, HPACK header compression, stream prioritisation and server push (now deprecated). HTTP/1.1 is textual with one in-flight request per connection.

• HTTP/1.1 pipelining exists but suffers head-of-line blocking and is effectively unused; browsers open ~6 connections per host.
• HTTP/2 removes application-level HOL blocking but TCP-level HOL blocking remains (solved by HTTP/3 over QUIC).
• Browsers require TLS for HTTP/2; h2c is the cleartext variant.

HttpClient defaults to Version.HTTP_2 and falls back to HTTP/1.1 automatically. Over HTTPS the protocol is chosen through TLS ALPN; over plain HTTP it tries an Upgrade: h2c header.

• Force HTTP/1.1 with .version(HttpClient.Version.HTTP_1_1).
• Check response.version() to see what was actually used.
• Many servers ignore h2c upgrade, so cleartext usually falls back to 1.1.

TLS authenticates the server through its certificate chain, negotiates cipher suites and keys (ephemeral ECDHE in TLS 1.2/1.3), then encrypts the stream. Java exposes it through SSLContext, SSLSocket, SSLEngine.

• Trust store (javax.net.ssl.trustStore) decides whom to trust; key store holds your private key for mutual TLS.
• Defaults: JDK 11+ enables TLS 1.3; the default trust store is cacerts.
• Hostname verification must remain enabled; with HttpClient it is on unless the endpoint identification algorithm is cleared.

TLS 1.2 needs 2 round trips after TCP; TLS 1.3 needs 1 (and 0-RTT on resumption). Asymmetric crypto adds CPU cost on the server.

• Reduce with connection reuse (keep-alive, HTTP/2), session resumption (tickets), and TLS 1.3.
• Terminate TLS at a load balancer to offload CPU.
• OCSP stapling avoids a blocking revocation lookup.

URI is a pure syntactic parser/value object for identifiers. URL additionally knows a protocol handler and can open a connection; URL.equals/hashCode may perform DNS lookups.

• Prefer URI, and convert with uri.toURL() only when needed. Java 20 deprecated new URL(String) constructors in favour of URI.create(...).toURL().
• URI.create throws IllegalArgumentException for invalid input; new URI throws a checked URISyntaxException.

Use URLEncoder.encode(value, StandardCharsets.UTF_8) for query component values (form encoding, space becomes +) and a URI multi-arg constructor or a builder for paths (space becomes %20).

• Encode each component separately, never the whole URL: that would encode ://, / and ?.
• Double-encoding (%2520) is a common bug when a framework already encodes.
• Spring: UriComponentsBuilder and UriUtils.

InetAddress.getByName calls the OS resolver and caches results inside the JVM. Positive results are cached for 30 seconds by default (when no security manager) and negative results for 10 seconds.

• Controlled by security properties networkaddress.cache.ttl and networkaddress.cache.negative.ttl in java.security or set at startup via Security.setProperty.
• -1 means cache forever, 0 means no caching.
• getAllByName returns all A/AAAA records; resolution order follows java.net.preferIPv4Stack / preferIPv6Addresses.

java.security.Security.setProperty("networkaddress.cache.ttl", "10");

Failover and blue/green deployments that rely on DNS changes will not reach a long-lived JVM, which keeps connecting to the old IP. Cloud services (load balancers, RDS, S3) rotate IPs frequently.

• When a security manager is installed (older setups), the default TTL is infinite, a classic production trap.
• Set a small positive TTL (for example 30 s) and make clients re-resolve on reconnect.
• Pooled connections also pin an old IP until closed; set max connection lifetime.

Connect timeout limits TCP connection establishment. Read (socket) timeout limits the wait between bytes while reading, not the total response time.

• A request streaming one byte every 29 seconds never trips a 30 s read timeout.
• Also needed: connection-acquire (pool) timeout, request/total timeout, and TLS handshake timeout.
• Without a connect timeout, the OS default (about 2 minutes via SYN retries on Linux) applies.

A pool keeps established connections (TCP + TLS done) and leases them to requests, avoiding handshake cost per call and bounding concurrency toward a backend.

• Key knobs: max total, max per route/host, acquire timeout, idle eviction, max lifetime, validation.
• Apache HttpClient: PoolingHttpClientConnectionManager (default max 20 total, 2 per route in 4.x). OkHttp: ConnectionPool (5 idle, 5 min). JDK HttpClient: internal pool, tuned via system properties such as jdk.httpclient.connectionPoolSize and jdk.httpclient.keepalive.timeout.
• A response body must be fully read or closed to return the connection.

HTTP keep-alive (persistent connections, the default in HTTP/1.1) reuses one TCP connection for multiple requests. TCP keep-alive (SO_KEEPALIVE) is an OS probe on idle sockets that detects dead peers. They are unrelated mechanisms.

• Servers close idle connections after a timeout (Keep-Alive: timeout=5); a client reusing a just-closed connection gets a reset.
• Set the client idle eviction below the server and load balancer idle timeout.
• Connection: close disables reuse.

After actively closing, a TCP endpoint stays in TIME_WAIT for 2xMSL (60 s on Linux) so late duplicate segments die and the final ACK can be retransmitted. The side that closes first holds the state.

• Thousands of TIME_WAIT sockets are normal for a client making many short connections.
• They consume ephemeral ports (the 4-tuple stays reserved), not much memory.
• Best fix: reuse connections (keep-alive, pooling), not shortening TIME_WAIT.

The OS assigns a source port from a range (Linux net.ipv4.ip_local_port_range, default 32768-60999, about 28k ports) to each outgoing connection. A connection is identified by (src IP, src port, dst IP, dst port), so exhaustion occurs per destination.

• Symptom: BindException: Cannot assign requested address on connect.
• Causes: no connection reuse, many TIME_WAIT sockets, NAT gateways with limited ports.
• Mitigations: pool connections, widen the port range, add source IPs, tcp_tw_reuse for outgoing connections.

A forward proxy acts for clients (egress control, caching, anonymity). A reverse proxy acts for servers (TLS termination, routing, load balancing). Java supports proxies through java.net.Proxy, ProxySelector and system properties.

HttpClient c = HttpClient.newBuilder()
    .proxy(ProxySelector.of(new InetSocketAddress("proxy.corp", 3128)))
    .build();

• System properties: https.proxyHost, https.proxyPort, http.nonProxyHosts.
• HTTPS through a forward proxy uses CONNECT host:443 to create a tunnel; the proxy cannot see the content.

L4 balancers (AWS NLB, IPVS) route by IP/port at the TCP/UDP level without parsing HTTP. L7 balancers (ALB, NGINX, Envoy) understand HTTP and can route by path/header, terminate TLS, retry and balance per request.

• With HTTP/2 or gRPC over an L4 balancer, all requests of one long-lived connection go to a single backend, so load is uneven; use L7 or client-side balancing.
• Algorithms: round robin, least connections, consistent hashing, power of two choices.

Proxies append the client IP to X-Forwarded-For (and set X-Forwarded-Proto/Host) so the origin knows the real client. Clients can send their own forged value.

• Only trust the entries added by your own proxies: take the rightmost trusted hop, not the first value.
• Spring Boot: server.forward-headers-strategy=native or framework with a trusted proxy.
• The standardised alternative is Forwarded (RFC 7239).

CORS is a browser mechanism that relaxes the same-origin policy: the server declares via Access-Control-Allow-* headers which origins may read its responses. It is enforced by the browser, not the server.

• Simple requests are sent directly; the browser checks Access-Control-Allow-Origin on the response.
• Non-simple requests (custom headers, JSON Content-Type, PUT/DELETE) trigger a preflight OPTIONS request.
• Access-Control-Allow-Credentials: true requires an explicit origin, not *.

Before a non-simple request the browser sends OPTIONS with Origin, Access-Control-Request-Method and Access-Control-Request-Headers. The server answers with Allow-Origin, Allow-Methods, Allow-Headers, and optionally Access-Control-Max-Age.

• Max-Age caches the preflight result (browser caps, for example 2 hours in Chrome).
• The preflight must succeed with 2xx and must not require authentication cookies.
• Spring: @CrossOrigin or CorsConfigurationSource, applied before Spring Security's filter chain.

WebSocket provides a full-duplex, message-oriented channel over a single TCP connection. It starts as an HTTP/1.1 GET with Upgrade: websocket and Sec-WebSocket-Key; the server replies 101 Switching Protocols with Sec-WebSocket-Accept.

• After upgrade, framed messages (text, binary, ping, pong, close) flow in both directions.
• Browsers cannot set custom headers on the handshake, so auth uses cookies, query token or a first message.
• wss:// is WebSocket over TLS.

Client: HttpClient.newWebSocketBuilder() with a WebSocket.Listener. Server: Jakarta WebSocket (@ServerEndpoint) or Spring WebSocketHandler / STOMP.

WebSocket ws = HttpClient.newHttpClient().newWebSocketBuilder()
    .buildAsync(URI.create("wss://example.com/feed"), new WebSocket.Listener() {
        public CompletionStage<?> onText(WebSocket w, CharSequence d, boolean last) {
            System.out.println(d);
            w.request(1);
            return null;
        }
    }).join();

• The JDK client is demand driven: you must call request(n) to receive more messages.

WebSocket: bidirectional, low-latency (chat, games). Server-Sent Events: one-way server push over plain HTTP with auto-reconnect (notifications, feeds). Long polling: fallback with highest overhead.

• SSE works through HTTP/2 and proxies easily but is text only and unidirectional.
• WebSockets need sticky sessions or a shared broker to scale beyond one node.

Backpressure is a way for a slow consumer to signal a fast producer to slow down so buffers do not grow unbounded. TCP has it built in: the receiver's window shrinks and the sender's write blocks or becomes not-writable.

• Blocking IO: OutputStream.write blocks when the send buffer is full.
• NIO: write returns fewer bytes (maybe 0); register OP_WRITE and retry.
• Netty: check channel.isWritable() and the write-buffer water marks.
• Reactive Streams: subscriber request(n).

Flow control protects the receiver: it advertises a receive window so the sender never exceeds receiver buffers. Congestion control protects the network: the sender limits in-flight data with a congestion window (slow start, congestion avoidance, CUBIC/BBR).

• Effective send window = min(receiver window, congestion window).
• Throughput per connection is bounded by window / RTT (bandwidth-delay product).
• A zero window stalls the sender until a window update.

Linux has a SYN queue for half-open connections and an accept queue for completed handshakes awaiting accept(). The Java backlog argument sets the accept-queue limit, capped by net.core.somaxconn (default 4096 on newer kernels, 128 on older).

• If the app does not call accept() fast enough, the queue fills and new connections are dropped/retried; clients see connect timeouts.
• Check with ss -lnt (Recv-Q is the queue length on a listener, Send-Q the limit).
• Tomcat: acceptCount (default 100); Netty: ChannelOption.SO_BACKLOG.

Use a length prefix, a delimiter, or fixed-size records, because TCP delivers a byte stream. A read may return part of a message or several messages.

• Length-prefix: read 4 bytes, then loop until that many bytes are read (DataInputStream.readFully).
• Delimiter (newline for line protocols): beware of delimiter bytes inside payloads.
• Netty: LengthFieldBasedFrameDecoder, LineBasedFrameDecoder.
• Always cap the max frame length to prevent memory attacks.

Each client has a selector manager thread (HttpClient-N-SelectorManager) for I/O and uses an executor (default: cached thread pool) to run completion stages and handlers. You can supply your own via .executor(...).

• sendAsync returns immediately; chained thenApply callbacks run on the client executor unless you use ...Async with another executor.
• Blocking in a callback consumes a pool thread.
• A client lives until it is unreachable and idle (JDK 21 adds close()/AutoCloseable).

Choose a BodyHandler: ofString, ofByteArray, ofInputStream, ofLines, ofFile, or a subscriber for streaming. HttpClient does not retry on its own beyond the safe internal retry of idempotent requests on connection reset, and non-2xx status codes are not exceptions.

• Check statusCode() yourself.
• Retry only idempotent requests (GET, PUT, DELETE) with exponential backoff and jitter; for POST use an idempotency key.
• ofInputStream must be closed to free the connection.

Redirects: followRedirects(Redirect.NORMAL) follows them except HTTPS to HTTP downgrades; the default is NEVER. Cookies need cookieHandler(new CookieManager()). Authentication uses .authenticator(...) for Basic/Digest on 401/407.

• Setting an Authorization header manually is required for pre-emptive Basic auth, since the authenticator only runs after a challenge.
• Redirect.ALWAYS also follows HTTPS to HTTP.

InetAddress resolution returns both A and AAAA records; Java sorts by preference (java.net.preferIPv6Addresses, default false so IPv4 first) and Socket.connect tries only a single address. The JDK does not implement RFC 8305 Happy Eyeballs; Netty and others add it separately.

• A broken IPv6 route with AAAA first causes long connect timeouts.
• -Djava.net.preferIPv4Stack=true disables IPv6 sockets entirely.
• URL/HttpClient may try only the first resolved address, so failures on a dead first address surface as errors rather than falling back.

1xx informational, 2xx success, 3xx redirect, 4xx client error, 5xx server error. Safe methods (GET, HEAD, OPTIONS) do not change state; idempotent methods (those plus PUT, DELETE) can be repeated with the same effect.

• 502 Bad Gateway: proxy got an invalid upstream response; 503: overloaded or unavailable; 504: upstream timeout.
• POST and PATCH are not idempotent by definition, so automatic retries are risky.
• 429 with Retry-After is the standard throttling signal.

Cache-Control: max-age lets a cache reuse a response without contacting the origin. After expiry, a conditional request with If-None-Match (ETag) or If-Modified-Since returns 304 Not Modified with no body when unchanged.

• no-cache means revalidate before use; no-store means never store.
• private restricts to browser caches; public allows shared caches.
• Vary lists request headers that make cached variants distinct.

Content-Length declares the body size up front. Transfer-Encoding: chunked sends the body as length-prefixed chunks ending with a zero-length chunk, used when the size is unknown (streaming responses).

• A wrong Content-Length leads to truncated bodies or connection desync.
• Disagreement between proxy and backend over these headers underlies HTTP request smuggling.
• HTTP/2 has no chunked encoding; it frames data natively.

Each socket consumes a file descriptor; the per-process limit (ulimit -n, often 1024 by default) caps concurrent connections. Exceeding it throws SocketException: Too many open files.

• Raise with ulimit -n, systemd LimitNOFILE, or container settings.
• Count with ls /proc/<pid>/fd | wc -l or lsof -p.
• Leaks (unclosed sockets, response bodies) show as growing CLOSE_WAIT counts.

CLOSE_WAIT means the peer sent FIN and your application has not yet closed its socket. A growing count means your code is leaking connections.

• Typical causes: missing close() on error paths, forgetting to close response bodies, a pool not evicting dead connections.
• Unlike TIME_WAIT, it never times out by itself; the OS keeps the socket until the process closes it.
• Diagnose with ss -tanp state close-wait and a thread/heap dump.

All indicate the peer sent TCP RST. Connection reset is thrown on a read when an RST arrives; Broken pipe (EPIPE) occurs on a write after the connection was already closed/reset; Connection reset by peer is the OS wording of the same RST.

• Common triggers: server closed an idle keep-alive connection, the process crashed, a firewall/NAT dropped state, a LB timeout, or sending data to a closed socket.
• Fix: retry idempotent calls on stale pooled connections, evict idle connections earlier than server timeout, check LB idle timeouts.

Calling threads block indefinitely in read, the worker pool drains, and the service stops serving unrelated requests: a cascading failure from one slow dependency.

• Defaults: HttpURLConnection and Socket have infinite connect/read timeouts (connect limited only by the OS); JDK HttpClient has none either.
• Fix: set connect, read/request and pool-acquire timeouts, add a circuit breaker and bulkheads.
• Confirm with a thread dump: many threads in SocketInputStream.socketRead0 / NioSocketImpl.read.

SO_TIMEOUT applies per blocking read, not to the total. If the server trickles a byte every few seconds, each read succeeds before 5 s and the call continues.

• Also the timeout does not cover connect, DNS lookup, TLS handshake or pool wait.
• Enforce a total deadline: track elapsed time between reads, use Future.get(timeout) and close the socket, or a higher-level client with a request timeout.
• Closing the socket from another thread makes the blocked read throw SocketException.

Without a timeout, connect retries SYNs with exponential backoff and fails with ConnectException: Connection timed out after about 2 minutes on Linux (tcp_syn_retries=6). With connect(addr, 3000) you get SocketTimeoutException: Connect timed out after 3 s.

• Silent drops (firewall DROP, security group) cause timeouts; an active reject or closed port gives Connection refused immediately.
• Distinguish them: refused means host reachable but nothing listens; timed out means packets vanish.

A pooled keep-alive connection was closed by the server or an intermediary (idle timeout) while the client still considered it open. The next write hits a dead connection and gets RST.

• Fix: set client idle eviction/max-idle shorter than the server and LB idle timeout, enable validate-after-inactivity (Apache setValidateAfterInactivity), or retry idempotent requests once.
• Example: AWS ALB idle timeout 60 s; server Tomcat keepAliveTimeout 20 s; client must evict before 20 s.
• Timing pattern (failures after N seconds of idle) is the signature.

The JVM caches the lookup (infinite if a security manager is present or networkaddress.cache.ttl=-1), and pooled connections keep using the old address.

• Fix: set networkaddress.cache.ttl to 10-60 s via java.security or Security.setProperty before the first lookup, set max connection lifetime on pools, and verify with jcmd / test.
• For long-lived connections (gRPC, DB), use client-side re-resolution or periodic reconnect.

Security.setProperty("networkaddress.cache.ttl", "30");
Security.setProperty("networkaddress.cache.negative.ttl", "5");

The client opens a new connection per request and closes it, consuming ephemeral ports faster than TIME_WAIT frees them. Fix the root cause by reusing connections.

• Use a shared pooled client with keep-alive; HTTP/2 multiplexing reduces connections further.
• Check ss -s, sysctl net.ipv4.ip_local_port_range.
• Stopgaps: widen port range, net.ipv4.tcp_tw_reuse=1 (client side outgoing only), spread over more destination IPs.
• Avoid SO_LINGER 0 hacks that send RST: data loss risk.

Threads wait for a pooled connection, so the pool is exhausted. Either demand exceeds maxPerRoute or connections are leaked because response entities are not consumed/closed.

• Check pool stats (getTotalStats(): leased/available/pending).
• Always close the response in try-with-resources or use EntityUtils.consume.
• Set a connectionRequestTimeout so callers fail fast instead of waiting forever.
• Default maxPerRoute is 2 in Apache 4.x, which is rarely enough.

Not blindly. A timeout means the outcome is unknown: the server may have processed it. Retrying a non-idempotent POST can double charge.

• Use an idempotency key header that the server de-duplicates on.
• Otherwise query the status of the original operation first.
• Retry with exponential backoff and jitter, with a cap on attempts and total deadline.

Nagle's algorithm holds the second small write until the first is acked, while the receiver delays its ACK (up to ~40 ms) hoping to piggyback it. The interaction stalls the body.

• Fix: set TCP_NODELAY, or better write header and body in one buffer / BufferedOutputStream and flush once.
• Verify with a packet capture showing the pause between segments.

The code forgets to remove the key and ignores the -1 end-of-stream result.

while (true) {
    selector.select();
    for (SelectionKey k : selector.selectedKeys()) {
        SocketChannel ch = (SocketChannel) k.channel();
        ByteBuffer buf = ByteBuffer.allocate(256);
        ch.read(buf);
        buf.flip();
        ch.write(buf);
    }
}

• selectedKeys() is never cleared, so keys are re-processed on every loop (busy spinning on stale keys).
• After the peer closes, read returns -1 forever and the key stays ready: CPU spin; you must ch.close() on -1.
• write may be partial; unwritten bytes must be kept and OP_WRITE registered.
• Allocating a buffer per event is wasteful.

The historic JDK epoll spin bug (and similar misuse): select() returns 0 repeatedly, burning CPU. Older JDKs on Linux had it; Netty works around it by counting premature returns and rebuilding the selector.

• Check for other causes: a registered OP_WRITE interest on an always-writable channel, or OP_CONNECT left set after connect completes.
• Use a mature framework (Netty), a recent JDK, and profile the event loop thread.

The query blocks the event-loop thread, so every other channel assigned to that loop stalls (latency spikes, timeouts, heartbeats missed).

• Fix: add the handler to a separate EventExecutorGroup (pipeline.addLast(group, handler)) or hand work to a dedicated thread pool and write results back via the channel.
• Detect with Netty's BlockHound or logs of long event-loop tasks.
• Core rule: event loop threads only do non-blocking work.

The server writes faster than the client reads; unwritten data accumulates in the channel's outbound buffer because the app ignores channel.isWritable().

• Configure WRITE_BUFFER_WATER_MARK (low/high); when the high mark passes, isWritable() turns false and channelWritabilityChanged fires.
• Pause producing until writable again, or use ChunkedWriteHandler/ChunkedFile.
• Also check for ByteBuf leaks with leak detection.

First call pays DNS lookup, TCP connect, TLS handshake, class loading of the SSL/HTTP stack and JIT warm-up, plus possibly certificate chain validation and OCSP.

• Verify by timing phases (curl -w timing variables, -Djavax.net.debug=ssl:handshake, or client event listeners in OkHttp).
• Mitigate with warm-up calls at startup, connection pre-warming, and an app-level health check that hits dependencies.
• Entropy-related slow SecureRandom was a classic cause on old VMs.

A 5 s delay matches the default DNS resolver timeout (glibc timeout:5): a lost UDP DNS packet or a parallel A/AAAA query race in resolv.conf, notoriously seen in Kubernetes with conntrack races on UDP DNS.

• Check with time getent hosts name and tcpdump port 53.
• Mitigations: options single-request-reopen or use-vc in resolv.conf, NodeLocal DNSCache, lower ndots, caching in JVM.
• Excessive ndots:5 makes short names trigger many search-domain queries.

The JVM cannot build a chain from the server certificate to a trusted root in its trust store: a private/internal CA, missing intermediate certificate, or an outdated cacerts.

• Import the CA: keytool -importcert -alias corp-ca -file ca.pem -keystore $JAVA_HOME/lib/security/cacerts, or a dedicated truststore via -Djavax.net.ssl.trustStore.
• The server must send the intermediate certificates too.
• Do not disable validation; trust-all managers defeat TLS.
• Debug with -Djavax.net.debug=ssl:handshake.

Java verifies that the host in the URL matches a certificate Subject Alternative Name. Connecting by IP when the certificate only lists DNS names fails (CN is ignored when SANs are present).

• Connect with the DNS name, or issue a certificate with an IP SAN.
• For tests, override resolution (hosts file) rather than disabling verification.
• A mismatched name behind a proxy or load balancer URL is the common case.

Newer JDKs disable legacy protocols and ciphers via jdk.tls.disabledAlgorithms in java.security: SSLv3, TLS 1.0 and 1.1 (disabled in JDK 8u291 / 11.0.11), weak ciphers, SHA-1 certs.

• Best fix: upgrade the server to TLS 1.2+.
• Temporary: edit jdk.tls.disabledAlgorithms (security risk), or force protocols with -Dhttps.protocols=TLSv1.2.
• Diagnose by -Djavax.net.debug=ssl:handshake and openssl s_client -connect host:443 -tls1_2.

Classic keep-alive mismatch: the backend closes an idle connection at the same moment the LB reuses it, producing 502. The backend idle timeout must be longer than the LB idle timeout.

• Example: ALB idle 60 s requires Tomcat/Netty/NGINX keep-alive > 60 s (for example 65 s).
• Also check backend crashes, slow responses beyond LB timeout (504), and request size limits.
• Look at LB access logs (elb_status_code vs target_status_code).

ClusterIP balances per TCP connection (kube-proxy). gRPC multiplexes all calls over a single long-lived HTTP/2 connection, so one pod receives everything.

• Fix: a headless service plus client-side round-robin (dns:/// target with round_robin policy), an L7 proxy/mesh (Envoy, Linkerd), or periodically recycling connections (maxConnectionAge).
• Re-resolving DNS interacts with networkaddress.cache.ttl in Java.

Postman does not enforce the same-origin policy; the browser does. The response lacks Access-Control-Allow-Origin for the page's origin, or the preflight failed.

• Inspect the OPTIONS request in dev tools: status, Allow-Methods, Allow-Headers covering the headers (Authorization, Content-Type).
• Fix server-side: allow the specific origin, methods and headers; errors (4xx/5xx) must also carry CORS headers.
• A redirect on the preflight also fails.

allowedOrigins("*") together with allowCredentials(true) is rejected at startup/runtime in Spring (IllegalArgumentException), because browsers refuse wildcard origin with credentials.

registry.addMapping("/api/**")
    .allowedOrigins("*")
    .allowCredentials(true);

• Use allowedOriginPatterns("https://*.example.com") or an explicit list.
• With Spring Security, enable http.cors(Customizer.withDefaults()) so the CORS filter runs before authentication.
• Reflecting any Origin with credentials enabled is a security hole.

A proxy or load balancer closes idle connections after its idle timeout (NGINX proxy_read_timeout 60 s, ALB 60 s default). Locally nothing sits in between.

• Fix: send application or protocol ping/pong frames more often than the timeout, and raise the proxy timeout for upgrade routes.
• The NGINX config must pass Upgrade and Connection headers and use HTTP/1.1 to the backend.
• Add client-side reconnect with exponential backoff and jitter.

Each instance holds only its own connections in memory. A message published on instance A is never delivered to sockets on B.

• Fix: a shared pub/sub backbone (Redis pub/sub, Kafka, RabbitMQ with Spring STOMP broker relay) so every instance fans out to its local sessions.
• Load balance with least-connections; sticky sessions are needed only for fallback transports like SockJS.
• Store presence in a shared store with TTLs; handle reconnect and resume with message IDs.

ofString buffers the whole body in memory. For large payloads stream it: ofInputStream, ofLines, or ofFile.

HttpResponse<Path> r = client.send(req,
    HttpResponse.BodyHandlers.ofFile(Path.of("out.bin")));
System.out.println(r.body());   // out.bin

• With ofInputStream, close the stream in try-with-resources.
• Limit expected sizes (Content-Length check) to avoid untrusted huge bodies.

Possible limits: the pool max per route (Apache), the server's HTTP/2 MAX_CONCURRENT_STREAMS (JDK client queues extra streams or opens more connections), the server worker threads, or the ephemeral port/FD limits.

• JDK HttpClient with HTTP/1.1 opens up to many connections; use jdk.httpclient.maxstreams and a semaphore on the caller to cap concurrency.
• Backpressure in the producer: bound concurrency (for example Semaphore(50)) rather than blasting everything.
• Look at the server side queue and thread dump of the client.

413 Payload Too Large: body exceeds the proxy limit (NGINX client_max_body_size, default 1 MB), or the app server (Spring spring.servlet.multipart.max-file-size). 504 Gateway Timeout: upstream did not answer within the proxy's read timeout (NGINX proxy_read_timeout 60 s).

• Fix 504 by making slow work asynchronous (return 202 and poll) rather than only raising timeouts.
• Keep timeouts ordered: client > proxy > app > downstream per hop to avoid duplicate work.

HTTPS via a forward proxy uses CONNECT host:443. The JVM may only have http.proxyHost set (not https.proxyHost), or the proxy blocks CONNECT/needs authentication (407), or HttpClient was never given a ProxySelector.

• Set both http.* and https.* properties, http.nonProxyHosts.
• Java disables Basic auth for HTTPS tunneling by default: jdk.http.auth.tunneling.disabledSchemes="" to allow it.
• JDK HttpClient needs .proxy(ProxySelector.getDefault()) and an Authenticator.

If nothing is listening (process restarting), the kernel sends RST: refused. If the process listens but accept queue overflows, behaviour depends on tcp_abort_on_overflow (default 0: drop, causing client timeouts and SYN retries).

• Use graceful shutdown with readiness gates (deregister from LB, drain in-flight, then stop).
• Increase backlog/somaxconn, and make the accept loop fast.
• Check netstat -s | grep -i listen for overflow counters.

If the client machine died (power loss, cable pull) no FIN/RST is sent, so the server's read blocks indefinitely: a half-open connection. A process crash differs, because the OS sends FIN/RST.

• Defences: SO_TIMEOUT, application heartbeats, TCP keep-alive (tune tcp_keepalive_time, intvl, probes; Java 11+ ExtendedSocketOptions.TCP_KEEPIDLE).
• Servers should have idle-connection timeouts.

Check whether the server actually responded in time, and whether the delay is before or after the first byte: use tcpdump/Wireshark on both ends, server access logs with request duration, and curl -w timings.

• Server slow (GC pause, DB, thread pool queue): response leaves late.
• Network/proxy slow: server log shows fast, client sees slow; look at LB, retransmissions (ss -ti), MTU.
• Client slow: its own GC pause or exhausted executor before reading.

They list sockets by state and peer, so you can see which remote endpoints a JVM holds many connections to and in what state (ESTABLISHED, CLOSE_WAIT, TIME_WAIT).

ss -tanp | awk '{print $1}' | sort | uniq -c
lsof -p <pid> -iTCP -sTCP:CLOSE_WAIT | wc -l

• Growth in ESTABLISHED to one host: unreleased pool leases or missing close.
• Growth in CLOSE_WAIT: peer closed, app did not.
• Combine with jcmd <pid> Thread.print and heap dump histogram of Socket/SocketImpl objects.

Throughput is limited by window/RTT. With 100 Mbit/s and 100 ms RTT the bandwidth-delay product is about 1.25 MB, so a 64 KB window yields about 5 Mbit/s.

• Enable window scaling (default on) and let autotuning grow buffers; avoid hard-coding setReceiveBufferSize low values.
• Use parallel connections or HTTP/2 multiplexing to fill the pipe, or a congestion algorithm such as BBR.
• Check packet loss with ss -ti retransmit counters.

It serves them one after the other: the second client connects (the kernel completes the handshake) but is not served until the first finishes, because handle runs on the accept thread.

try (ServerSocket ss = new ServerSocket(9000)) {
    while (true) {
        try (Socket s = ss.accept()) {
            handle(s);   // blocks while client 1 is served
        }
    }
}

• Client 2 may appear connected (writes succeed into the kernel buffer) yet gets no response.
• Fix: hand each socket to an executor (or virtual threads on Java 21: Executors.newVirtualThreadPerTaskExecutor()).

Parsing breaks randomly: TCP can coalesce two writes into one read or split one write into several. It appears to work on localhost with small messages and fails under load or on real networks.

• Fix with length-prefixed framing and DataInputStream.readFully:

int len = in.readInt();
if (len < 0 || len > 1_000_000) throw new IOException("bad frame");
byte[] body = new byte[len];
in.readFully(body);

The first write after the peer's FIN usually succeeds (data goes out, peer answers RST). The next write fails with SocketException: Broken pipe (Linux) or Connection reset by peer/Connection reset.

• Seen in web apps as ClientAbortException / AsyncRequestNotUsableException when the browser navigates away mid-response.
• Handle by treating them as client disconnects (log at debug), stop producing, and release resources.

get(timeout) only stops waiting and throws TimeoutException; the request keeps running, holding the connection. You must call future.cancel(true) or use orTimeout and cancel.

var f = client.sendAsync(req, BodyHandlers.ofString());
try {
    f.get(2, TimeUnit.SECONDS);
} catch (TimeoutException e) {
    f.cancel(true);
}

• Cancellation closes the stream for HTTP/2 or the connection for HTTP/1.1 (reliable since JDK 16).
• The more direct approach is HttpRequest.timeout.

The default SimpleClientHttpRequestFactory (backed by HttpURLConnection) reuses connections only if responses are fully read and closed and keep-alive allowed. Often the response is not consumed, or Connection: close is sent, or a new client is created per call.

• Configure a pooled factory (Apache HttpComponentsClientHttpRequestFactory, JDK JdkClientHttpRequestFactory), share one bean.
• WebClient (Reactor Netty) pools by default but dedicates a ConnectionProvider; set max idle time below the server idle timeout.
• Verify handshakes with packet capture or -Djavax.net.debug=ssl:handshake counts.

The remote address seen by the app is the proxy's. Use X-Forwarded-For (set by NGINX with proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for) and tell the framework to trust it.

• Spring Boot: server.forward-headers-strategy=framework and configure server.tomcat.remoteip.internal-proxies.
• Do not trust the header from untrusted sources, or rate limiting and IP allowlists are bypassed.

Differences between curl and the JVM: JVM proxy properties (-Dhttp.proxyHost applied to an internal host without nonProxyHosts), IPv6 vs IPv4 preference (AAAA resolves but is unreachable), a stale DNS cache entry, or a different trust/SNI setup.

• Compare InetAddress.getAllByName(host) with dig/getent.
• Test with -Djava.net.preferIPv4Stack=true and -Dhttp.nonProxyHosts=*.internal.
• Run -Djava.net.debug=all-style debugging (-Djdk.httpclient.HttpClient.log=errors,requests,headers) to see the resolved address and proxy.

The server or a middlebox is rejecting the ClientHello: unsupported protocol version or cipher list, missing/incorrect SNI, client-certificate required (mutual TLS) but not offered, or a WAF resetting suspicious clients.

• Reproduce: openssl s_client -connect host:443 -servername host -tls1_2.
• Check -Djavax.net.debug=ssl:handshake for which messages were sent and the last one before reset.
• Send SNI: with SSLParameters.setServerNames; the HttpClient does it automatically from the URI host.

Load the client key and certificate into a KeyManagerFactory, the trusted CAs into a TrustManagerFactory, build an SSLContext, and hand it to the client.

KeyStore ks = KeyStore.getInstance("PKCS12");
try (var in = Files.newInputStream(Path.of("client.p12"))) {
    ks.load(in, "changeit".toCharArray());
}
KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
kmf.init(ks, "changeit".toCharArray());
SSLContext ctx = SSLContext.getInstance("TLS");
ctx.init(kmf.getKeyManagers(), null, null);   // null: default trust managers
HttpClient c = HttpClient.newBuilder().sslContext(ctx).build();

• Passing null for trust managers uses the default cacerts; supply a custom TrustManagerFactory for private CAs.

Plain round-robin ignores per-request cost and backend slowness; a slow instance accumulates in-flight requests. Use least-outstanding-requests or power-of-two-choices, outlier detection that ejects slow/failing hosts, and bounded retry budgets.

• Retries amplify overload (retry storms); cap retries and use backoff.
• Active health checks should reflect real capacity, not just process liveness.
• Slow-start for new instances prevents a cold JVM being flooded.

When the peer host vanishes without closing (killed VM, dropped route, NAT entry expired), the client's connection is half-open and a read with no timeout blocks until TCP keep-alive or retransmission failure (up to ~15 minutes for retransmission, hours for keep-alive).

• Prevent: always set read timeouts, tune tcp_keepalive_* or TCP_KEEPIDLE options, and tcp_retries2 / TCP_USER_TIMEOUT for faster failure detection.
• For JDBC set socketTimeout/tcpKeepAlive; for pools use validation queries.

Streams and connections were not closed on all paths (including error streams from getErrorStream()), so FDs leak.

• Close getInputStream() or getErrorStream(), use try-with-resources, and call disconnect() if you do not want keep-alive reuse.
• Verify growth with lsof -p <pid> | wc -l.
• Raise ulimit -n only after the leak is fixed.

UDP offers no delivery guarantee: drops happen when the receiver socket buffer fills (SO_RCVBUF), at routers, or when datagrams exceed the MTU and fragment.

• Keep payloads under ~1400 bytes to avoid fragmentation.
• Raise SO_RCVBUF and the OS cap net.core.rmem_max; check netstat -su for receive errors.
• If loss is unacceptable, use TCP or add sequence numbers, acks and retries.

localhost may resolve to both 127.0.0.1 and ::1; Java chooses by preferIPv6Addresses (default IPv4 first) and tries only that address, so a service bound to ::1 yields Connection refused on 127.0.0.1.

• Bind to 0.0.0.0 / :: (dual-stack), or connect using the explicit loopback the service uses.
• Check with ss -ltn for the bound address.
• -Djava.net.preferIPv6Addresses=true flips the order.

Stop accepting new connections, deregister from the load balancer (readiness off), let in-flight requests finish within a deadline, then close channels and release event loops.

bossGroup.shutdownGracefully(0, 30, TimeUnit.SECONDS);
workerGroup.shutdownGracefully(2, 30, TimeUnit.SECONDS).sync();

• shutdownGracefully(quietPeriod, timeout) waits for a quiet period with no tasks before terminating.
• For HTTP/1.1 send Connection: close on the final responses; for HTTP/2 send GOAWAY.
• Kubernetes: handle SIGTERM and use a preStop sleep to let endpoints deprogram.

An MTU/PMTU black hole: large packets with the DF bit set are dropped because a tunnel reduces the MTU and ICMP "fragmentation needed" messages are blocked. Small packets (handshakes, headers) pass; the first full-size data segment never arrives.

• Diagnose with ping -M do -s 1400 host and varying sizes, tracepath.
• Fixes: lower the interface MTU, enable TCP MSS clamping on the gateway, allow ICMP type 3 code 4.

Reactor Netty sent a request on a pooled connection that the server had already closed, or the server closed during processing. It is the same stale keep-alive problem as in blocking clients.

• Set ConnectionProvider.builder("p").maxIdleTime(Duration.ofSeconds(20)).maxLifeTime(...).evictInBackground(Duration.ofSeconds(30)) below the server idle timeout.
• Retry idempotent requests with retryWhen on this exception.
• Confirm with server logs that it did not crash or hit max request limits.

For Java 21+, virtual threads with blocking code give most of the scalability of reactive with simpler stack traces, and they work with existing blocking libraries. Reactive/Netty still wins for streaming, strict backpressure and very high connection counts with minimal memory.

• Virtual threads: simple code, but cap concurrency to protect downstreams (semaphore/bulkhead); watch pinning in synchronized blocks on older JDKs (fixed in 24).
• Reactive: operators for backpressure, but harder debugging and colored functions.
• Pool limits, timeouts and circuit breakers are needed in all models.

Each upstream timeout must exceed the sum of what it waits for downstream, and a total deadline must propagate. If A's timeout is shorter than B's, A gives up while B (and C) keep working, wasting capacity.

• Rule of thumb: A timeout > B timeout x attempts + overhead; budget the deadline (gRPC deadlines, X-Request-Deadline headers) and pass the remaining time down.
• Limit retries to one layer to avoid multiplicative retries (3 x 3 x 3 = 27 calls).
• Use circuit breakers and bulkheads; trace with correlation IDs to find the slow hop.