Logging, Monitoring, Tracing, Prometheus & Grafana

Metrics tell you that something is wrong and how much (aggregated numbers over time), logs tell you what happened for a specific event (discrete records), and traces tell you where time went across services for one request.

• Metrics are cheap and good for alerting; high-cardinality detail is expensive.
• Logs are detailed but costly to search at scale.
• Traces connect services; they need propagation of context across every hop.
• Monitoring answers known questions (dashboards/alerts); observability lets you ask new questions of telemetry.

Structured logs are machine-parseable key-value records so they can be filtered and aggregated by field. MDC (Mapped Diagnostic Context) is a thread-local map whose entries are added to every log line on that thread, ideal for request or trace IDs.

@Component
class RequestIdFilter extends OncePerRequestFilter {
  @Override
  protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain)
      throws ServletException, IOException {
    String id = Optional.ofNullable(req.getHeader("X-Request-Id")).orElse(UUID.randomUUID().toString());
    MDC.put("requestId", id);
    res.setHeader("X-Request-Id", id);
    try { chain.doFilter(req, res); } finally { MDC.clear(); }
  }
}

• Spring Boot 3.4+ supports structured logging natively: logging.structured.format.console=ecs (also logstash, gelf).
• Always clear MDC in finally; pooled threads would leak values to later requests.

Micrometer is the metrics facade (like SLF4J for metrics) with registries for Prometheus, Datadog and others. Actuator exposes operational endpoints (/actuator/health, /metrics, /prometheus) and auto-configures Micrometer instrumentation for HTTP server/client, JVM, datasource, caches and more.

management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  endpoint:
    health:
      probes:
        enabled: true
  metrics:
    tags:
      application: ${spring.application.name}

• Add micrometer-registry-prometheus; the endpoint is /actuator/prometheus.
• Expose only needed endpoints, ideally on a separate management port not publicly routed (management.server.port).

Counter (monotonic count), Gauge (current value that goes up and down), Timer (latency count + total time + optionally histogram), DistributionSummary (distribution of non-time values), LongTaskTimer (in-flight long operations).

@Service
class PaymentService {
  private final Counter failures;
  private final Timer latency;
  PaymentService(MeterRegistry registry) {
    this.failures = Counter.builder("payments.failures").tag("provider", "stripe").register(registry);
    this.latency = Timer.builder("payments.latency").publishPercentileHistogram().register(registry);
  }
  void pay() { latency.record(() -> { /* call */ }); }
}

• Prometheus names become payments_failures_total, payments_latency_seconds_*.
• Use publishPercentileHistogram() (aggregable histogram buckets) rather than client-side percentiles, which cannot be averaged across instances.
• Gauge holds a weak reference to the observed object; keep a strong reference.

Prometheus pulls (scrapes) text-format metrics over HTTP from targets at an interval and stores them as time series identified by a metric name plus labels. Targets are found through static config or service discovery (Kubernetes).

scrape_configs:
  - job_name: orders
    metrics_path: /actuator/prometheus
    scrape_interval: 15s
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"

• Types: counter, gauge, histogram, summary.
• Pull gives built-in target-health (up metric) and central control; short-lived jobs use the Pushgateway.
• With the Prometheus Operator, use ServiceMonitor/PodMonitor instead of raw config.

Counters only increase, so you query their per-second rate over a window, then aggregate by labels.

# requests per second per service
sum by (application) (rate(http_server_requests_seconds_count[5m]))

# error ratio
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m]))

# p99 latency from histogram buckets
histogram_quantile(0.99,
  sum by (le) (rate(http_server_requests_seconds_bucket[5m])))

• rate handles counter resets and extrapolates; window should be at least 4x the scrape interval.
• increase(x[1h]) is total increase; irate is instantaneous and noisy, for graphs of volatile series only.
• Always aggregate with sum by (le) before histogram_quantile.

RED is for request-driven services: Rate, Errors, Duration. USE is for resources: Utilization, Saturation, Errors. Use RED per service endpoint and USE for CPU, memory, disk, connection pools, thread pools.

• RED example for Spring: request rate, 5xx ratio, p95/p99 latency from http.server.requests.
• USE example: Hikari pool active/max (utilization), pending threads (saturation), timeouts (errors).
• Add the "Four Golden Signals" saturation view for services as well: latency, traffic, errors, saturation.

A layered dashboard that answers "is it broken, where, and why" top-down: SLO/summary row first, then RED, then dependencies and resources, with variables for environment/service/pod.

• Top row: traffic, error %, p95/p99, SLO burn.
• Middle: per-endpoint breakdown, downstream calls, DB pool, JVM heap/GC/threads.
• Use template variables ($application, $namespace), annotations for deploys, and links to logs/traces filtered by the same labels.
• Provision dashboards as code (JSON/Jsonnet/Grafonnet via Grafana provisioning or Terraform) so they are versioned and reviewable.
• Avoid dashboard sprawl and vanity graphs; each panel must support a decision.

Prometheus evaluates alert rules and sends firing alerts to Alertmanager, which deduplicates, groups, silences, inhibits and routes them to receivers (Slack, PagerDuty).

groups:
  - name: orders
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_server_requests_seconds_count{application="orders",status=~"5.."}[5m]))
          / sum(rate(http_server_requests_seconds_count{application="orders"}[5m])) > 0.05
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "orders 5xx ratio above 5% for 10m"
          runbook_url: https://runbooks.acme.io/orders-5xx

• for suppresses flapping; too long delays detection.
• Alert on symptoms (user impact) rather than every cause; every page must be actionable and have a runbook.

An SLI is a measured indicator (ratio of good requests), an SLO the target (99.9% over 30 days), and the error budget the allowed failure (0.1%). Burn rate is how fast you consume the budget relative to even spending; a burn rate of 1 uses the whole budget exactly at 30 days.

Multi-window multi-burn-rate alert (page when 14.4x over 1h and 5m, meaning 2% of monthly budget in an hour):

(
  sum(rate(http_server_requests_seconds_count{status=~"5.."}[1h]))
  / sum(rate(http_server_requests_seconds_count[1h])) > (14.4 * 0.001)
)
and
(
  sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
  / sum(rate(http_server_requests_seconds_count[5m])) > (14.4 * 0.001)
)

• Typical set: 14.4x (1h/5m) page, 6x (6h/30m) page, 1x-3x (1d-3d) ticket.
• The short window makes the alert reset quickly after recovery.

A trace is a tree of spans (timed operations) sharing a trace ID. Context propagation passes the trace ID and parent span ID between services via headers (W3C traceparent), so each service continues the same trace.

# Spring Boot 3: Micrometer Tracing bridge to OpenTelemetry
management:
  tracing:
    sampling:
      probability: 0.1
  otlp:
    tracing:
      endpoint: http://otel-collector:4318/v1/traces

• Dependencies: micrometer-tracing-bridge-otel and opentelemetry-exporter-otlp.
• Use an OpenTelemetry Collector between apps and the backend (Tempo/Jaeger) for batching, sampling and routing.
• Spring Cloud Sleuth is gone in Boot 3; replaced by Micrometer Tracing.

Put the trace ID into every log line so you can jump from a trace to its logs and back. With Micrometer Tracing, traceId and spanId are placed into MDC automatically.

logging:
  pattern:
    correlation: "[${spring.application.name:},%X{traceId:-},%X{spanId:-}] "
  include-application-name: false

• Return the trace ID in an error response or header so support can quote it.
• For messaging, propagate the header in message metadata (Kafka headers), and use observation-enabled templates.
• A business correlation ID (order ID) is complementary but not a replacement for trace IDs.

Heap usage after GC, GC pause time/frequency, thread counts, and CPU. Micrometer exposes them as jvm_* and process_*.

# heap after GC trend (leak signal: rising floor)
jvm_memory_used_bytes{area="heap"}

# GC pause time per second
sum(rate(jvm_gc_pause_seconds_sum[5m]))

# live threads
jvm_threads_live_threads

# pool saturation
hikaricp_connections_active / hikaricp_connections_max

• A rising floor of heap after each major GC suggests a memory leak.
• Time in GC above ~5-10% indicates memory pressure.
• Container-level: container_memory_working_set_bytes against the limit to anticipate OOMKill (heap is only part of RSS; metaspace, threads, direct buffers count).

Each unique combination of label values is a separate time series. Labels with unbounded values (user ID, full URL, trace ID) create millions of series, exploding memory and query time.

• Use URI templates (/orders/{id}), which Spring MVC does for http.server.requests; avoid raw paths in custom meters.
• Inspect the top cardinalities: topk(10, count by (__name__) ({__name__=~".+"})) and the TSDB status page.
• Drop labels or metrics at scrape with metric_relabel_configs, and set sample_limit per job.
• Put high-cardinality identifiers in traces/logs, not metrics.

Go top-down and narrow: scope, then dependency, then code.

1. Scope: is it all endpoints or one, all pods or one, started at a deploy? (RED dashboard split by uri, pod; deploy annotations.)
2. Compare p99 vs p50: a tail-only problem suggests contention, GC or a slow dependency.
3. Check saturation: Hikari pending, thread pool queue, CPU throttling, GC pauses.
4. Open slow traces (exemplars or Tempo search duration > 2s) to see which span dominates (DB, HTTP call).
5. Pivot to logs by traceId for errors/retries in that span.

histogram_quantile(0.99, sum by (le, uri) (rate(http_server_requests_seconds_bucket{application="orders"}[5m])))

• Mitigate before root-causing: rollback a recent deploy, scale out, or disable a flag.

OOMKilled means the container's total memory (RSS) exceeded its limit; the JVM heap is only a fraction.

• Compare container_memory_working_set_bytes vs jvm_memory_used_bytes by area; look at non-heap, jvm_buffer_memory_used_bytes (direct buffers; Netty), jvm_threads_live_threads, metaspace.
• Check -XX:MaxRAMPercentage: 75% heap on a small container leaves little for the rest; try 50-60%.
• Enable -XX:NativeMemoryTracking=summary and run jcmd <pid> VM.native_memory.
• Thread-per-request with thousands of threads costs stack memory; each thread ~1MB reserved.
• Distinguish from java.lang.OutOfMemoryError (JVM-reported, in logs) vs exit code 137 (kernel kill).

resources:
  requests: { memory: 1Gi }
  limits:   { memory: 1Gi }
env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=60"

Move from cause-based, threshold-twitchy alerts to symptom/SLO-based ones, and delete or downgrade what is not actionable.

• Audit: for each alert, did a human take action? If not, remove or send to a ticket/chat channel.
• Add for: durations and multi-window burn rates instead of instantaneous thresholds.
• Group and inhibit in Alertmanager (a node down should not also page for every pod on it).
• Every page needs a runbook link and an owner; review alert quality in the on-call handover.

route:
  group_by: [alertname, application]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: pagerduty
    - receiver: slack-warnings

Context was not propagated across a hop. Common causes: the HTTP client was created with new RestTemplate() rather than the injected builder, an async executor lost the context, the gateway strips traceparent, or services use different propagation formats (B3 vs W3C).

@Bean
RestClient restClient(RestClient.Builder builder) {   // builder bean carries observation/propagation
  return builder.baseUrl("http://payments").build();
}

• For @Async, wrap executors with context propagation (ContextPropagatingTaskDecorator, Boot 3.2+).
• Verify with curl -v -H 'traceparent: 00-<32hex>-<16hex>-01' and check the same trace ID downstream.
• Ensure sampling decision is parent-based so a sampled parent leads to sampled children.

Almost certainly a cardinality explosion from a new label with unbounded values (user ID, request path with ids, exception message).

• Check prometheus_tsdb_head_series growth, and the TSDB status page for top label names and metrics.
• Find the offender: topk(10, count by (__name__)({__name__=~".+"})).
• Fix in code (remove or bound the tag), then drop at ingestion as a stopgap.

metric_relabel_configs:
  - source_labels: [__name__]
    regex: "orders_lookup_.*"
    action: drop

• Guardrail: sample_limit and label_limit in scrape config so a bad deploy fails scrapes instead of OOM-ing Prometheus.
• Churn (pods recreated) also inflates series; avoid pod label on application-level metrics where not needed.

Compare the new version's RED against the old one using a version label (from pod labels via relabeling or Micrometer common tags), then gate the rollout on it.

sum by (version) (rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/
sum by (version) (rate(http_server_requests_seconds_count[5m]))

histogram_quantile(0.95, sum by (le, version)
  (rate(http_server_requests_seconds_bucket[5m])))

• Add the version tag: management.metrics.tags.version=${app.version}.
• Check readiness/restarts too: increase(kube_pod_container_status_restarts_total[15m]) > 0.
• Automate with Argo Rollouts/Flagger analysis rather than a human staring at graphs.

Find the top noisy loggers, fix levels, then sample and tier.

• Query volume per logger/level (Loki: sum by (logger) (count_over_time({app="orders"}[1h]))).
• Set DEBUG/TRACE off in production; adjust per package at runtime via Actuator /loggers when debugging, with automatic revert.
• Do not log per-item inside loops, request/response bodies, or health-check hits.
• Log errors once, at the boundary, with the stack trace; avoid log-and-rethrow duplicates.
• Retention tiers: hot short window, cheap archive for long; derive metrics instead of searching logs for counts.

curl -X POST localhost:8080/actuator/loggers/com.acme.orders \
  -H 'Content-Type: application/json' -d '{"configuredLevel":"DEBUG"}'

The failure is occurring before the app records it (ingress/LB/connection layer) or the app is dying/not ready. Check the layer that emits the 502.

• Ingress/LB metrics and logs: upstream connect errors, upstream_response_time, 502/504 counts per backend.
• Pod restarts, readiness flaps and rollout timing: kube_pod_container_status_restarts_total, termination during deploys.
• Graceful shutdown: pod removed from endpoints after SIGTERM races causes resets; need preStop sleep and server.shutdown=graceful.
• Keep-alive mismatch: LB idle timeout longer than Tomcat keep-alive-timeout closes connections the LB reuses.
• App-side metric http.server.requests only counts requests the app handled.

lifecycle:
  preStop:
    exec: { command: ["sleep", "10"] }

Metrics, structured correlated logs, traces and a few alerts, all from configuration.

management:
  endpoints.web.exposure.include: health,info,prometheus
  endpoint.health.probes.enabled: true
  metrics:
    tags: { application: ${spring.application.name} }
    distribution:
      percentiles-histogram:
        http.server.requests: true
      slo:
        http.server.requests: 100ms,300ms,1s
  tracing.sampling.probability: 0.1
logging.structured.format.console: ecs

• Dashboards: RED + JVM + Hikari + downstream calls, provisioned from Git.
• Alerts: burn-rate on availability and latency SLO, plus pod restarts and saturation warnings.
• Runbook link on each alert; trace ID in error responses.

Capture 3 thread dumps ~10s apart and see what the blocked threads wait on; commonly they wait for a DB connection (pool exhausted), a lock or a slow downstream without timeouts.

jcmd <pid> Thread.print > td1.txt
# look for: "parking to wait for <HikariPool>", "BLOCKED", or "Found one Java-level deadlock"

• Metrics confirm: hikaricp_connections_pending > 0, hikaricp_connections_active == max, hikaricp_connections_timeout_total rising.
• Cause patterns: long transactions holding connections while calling remote services, leaked connections, missing HTTP client timeouts.
• Fix: timeouts everywhere, shorter transactions, bulkheads, right-sized pool.

A ratio over tiny traffic is noisy (1 error in 2 requests = 50%), and total outage yields division by zero (no data) so the alert may not fire at all. Add a minimum-traffic condition and a separate absence/traffic-drop alert.

# error ratio only when there is meaningful traffic
(
  sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
  / sum(rate(http_server_requests_seconds_count[5m]))
  > 0.05
)
and sum(rate(http_server_requests_seconds_count[5m])) > 1

# outage: traffic vanished compared to the same time last week
sum(rate(http_server_requests_seconds_count[5m])) < 0.1 * sum(rate(http_server_requests_seconds_count[5m] offset 1w))

• Use absent(up{job="orders"} == 1) to catch missing targets.
• Black-box probes (blackbox_exporter) cover total outages from the outside.