Deploying Spring Boot on AWS

Pick the lowest-operations option that meets your control, scale and cost needs.

• EC2: full control of VM and JVM, cheapest at steady load, you patch and scale it (ASG).
• ECS on Fargate: serverless containers, no nodes to manage, pay per vCPU/GB-second; the usual default for teams without Kubernetes needs.
• ECS on EC2: containers on your own nodes, better bin-packing and GPU/special instances.
• EKS: managed Kubernetes control plane; choose for portability, ecosystem (Helm, operators) or existing K8s skills.
• Lambda: event-driven, scale to zero; JVM cold start and 15-minute limit matter (SnapStart, native image help).
• Elastic Beanstalk / App Runner: PaaS-like, fastest to start, less control.

An ALB is a Layer 7 load balancer: it terminates TLS, routes by host/path/header rules to target groups, and health-checks targets. Targets can be instances, IPs (Fargate/EKS pods) or Lambda.

• Health check path: /actuator/health/readiness; healthy/unhealthy thresholds and interval determine how fast bad tasks leave rotation.
• Idle timeout defaults to 60 s; set the app's keep-alive (Tomcat server.tomcat.keep-alive-timeout) higher than the ALB's to avoid 502s on reused connections.
• X-Forwarded-* headers: set server.forward-headers-strategy=framework (or native) so redirects and request.getScheme() reflect HTTPS.
• Deregistration delay (default 300 s) lets in-flight requests drain on deploy.
• NLB (Layer 4) is for TCP/TLS passthrough, static IPs, extreme throughput.

Attach an IAM role to the compute resource (instance profile for EC2, task role for ECS, IRSA/Pod Identity for EKS, execution role for Lambda); the AWS SDK's default credentials chain obtains short-lived credentials automatically. No access keys in code, env vars or config.

• ECS has two roles: the execution role (pull image from ECR, write logs, read secrets at launch) and the task role (what your application code can call).
• Apply least privilege: scope to the specific bucket/queue/secret ARN.
• SDK v2 credentials chain: env vars, system props, web identity token, profile, container credentials, instance profile.

S3Client s3 = S3Client.create(); // credentials and region resolved from the environment

Multi-AZ VPC with public subnets for the ALB/NAT and private subnets for compute and data.

• Public subnets: route to an Internet Gateway; hold ALB and NAT Gateways.
• Private app subnets: outbound via NAT Gateway (or VPC endpoints); no public IPs.
• Isolated data subnets: RDS/ElastiCache with no internet route.
• Security groups (stateful) chain trust: ALB SG to app SG on 8080 to DB SG on 5432; reference SGs, not CIDRs.
• NACLs are stateless subnet-level filters, rarely needed beyond defaults.
• VPC endpoints: gateway endpoints for S3/DynamoDB (free), interface endpoints (PrivateLink) for ECR, Secrets Manager, CloudWatch, which also cut NAT data charges.
• Span at least 2 AZs.

RDS manages patching, backups, failover and replicas. Multi-AZ gives a synchronous standby with automatic failover (same endpoint, DNS flips in ~1-2 minutes); read replicas scale reads asynchronously.

• Connect via the endpoint DNS name, never an IP; keep JVM DNS TTL low (networkaddress.cache.ttl=30) so failover is picked up.
• Size HikariCP so pool size x instances stays under max_connections; consider RDS Proxy for many short-lived clients (Lambda).
• Force TLS (sslmode=verify-full with the RDS CA bundle).
• Use IAM database auth or Secrets Manager rotation for credentials.
• Aurora: cluster writer/reader endpoints, faster failover (~30 s), storage auto-scaling; Aurora Serverless v2 scales ACUs.

spring.datasource.hikari.maximum-pool-size: 10
spring.datasource.hikari.max-lifetime: 1500000   # below proxy/LB idle limits

ElastiCache provides managed Redis/Valkey (or Memcached) inside your VPC, used via Spring Data Redis / Spring Cache.

spring:
  data.redis:
    host: my-cache.xxxxxx.use1.cache.amazonaws.com
    port: 6379
    ssl.enabled: true
  cache.type: redis

• Private subnets only; SG allows 6379 from the app SG only.
• Enable in-transit encryption (TLS) and at-rest encryption; AUTH token or IAM auth.
• Replication group with Multi-AZ and automatic failover for HA; cluster mode for sharding.
• Always set TTLs; cache is an optimization, not the source of truth.
• Use the primary endpoint (or configuration endpoint for cluster mode) so failover is transparent.

Use AWS SDK v2 (or Spring Cloud AWS S3Template). For big uploads/downloads avoid streaming through the app: issue pre-signed URLs so clients talk to S3 directly.

S3Presigner presigner = S3Presigner.create();
PresignedPutObjectRequest req = presigner.presignPutObject(r -> r
    .signatureDuration(Duration.ofMinutes(10))
    .putObjectRequest(p -> p.bucket("shop-uploads").key("u/42/avatar.png")));
String url = req.url().toString();

• Multipart upload for objects over ~100 MB (required above 5 GB); S3TransferManager handles it.
• Block public access; encrypt with SSE-S3 or SSE-KMS; use bucket policies and versioning.
• S3 is strongly consistent for reads after writes since Dec 2020.
• Use a VPC gateway endpoint to avoid NAT costs.

Both are managed stores. Secrets Manager adds automatic rotation (native for RDS), cross-region replication and costs per secret per month plus API calls. SSM Parameter Store is cheaper (standard tier is free) for config and simple secrets (SecureString with KMS).

• Spring Cloud AWS 3.x imports them: spring.config.import=aws-secretsmanager:prod/shop/db or aws-parameterstore:/shop/prod/.
• On ECS, inject at container start via the task definition secrets field (needs execution role permission); on EKS use External Secrets Operator or Secrets Store CSI driver.
• Secrets are read at startup; rotation requires restart or a refresh mechanism, unless you use a driver that fetches credentials dynamically.
• Never put secrets in task definition plain environment, images, or Git.

"secrets": [{ "name": "SPRING_DATASOURCE_PASSWORD",
  "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/db-AbCdEf:password::" }]

Scale out/in the number of instances or tasks based on metrics.

• EC2 Auto Scaling Group: target-tracking policy (e.g. average CPU 60%), scheduled actions, health checks via ELB; uses a launch template.
• ECS Service Auto Scaling: target tracking on ECSServiceAverageCPUUtilization, memory, or ALBRequestCountPerTarget.
• EKS: HPA for pods plus Cluster Autoscaler or Karpenter for nodes.
• Lambda: scales per concurrent request; control with reserved/provisioned concurrency.
• Tune cooldowns: JVM warm-up means new tasks are not ready for 30-60 s, so scale out aggressively and scale in slowly.
• Request-count-per-target is often a better signal than CPU for I/O-bound services.

Rolling replaces instances/tasks gradually; blue/green runs the new version (green) alongside the old (blue) and shifts traffic, allowing instant rollback.

• ECS native rolling: minimumHealthyPercent/maximumPercent with the deployment circuit breaker (auto-rollback on failed deployment).
• ECS blue/green via CodeDeploy: two target groups on the ALB, traffic shifted all-at-once, linear or canary (e.g. 10% for 5 min), with test listener and alarms triggering rollback. ECS also offers built-in blue/green now.
• EC2: ASG instance refresh, or CodeDeploy with ALB.
• Lambda: aliases with weighted traffic shifting.
• Database changes must be backward compatible (expand/contract with Flyway), since both versions run concurrently.

"deploymentConfiguration": {
  "minimumHealthyPercent": 100, "maximumPercent": 200,
  "deploymentCircuitBreaker": { "enable": true, "rollback": true }
}

Lambda runs a handler per event; for HTTP you front it with API Gateway/Function URL and adapt via spring-cloud-function-adapter-aws or AWS Serverless Java Container. A JVM cold start of a full Spring context takes 5-10+ seconds.

Mitigations:

• SnapStart (Java 11+; snapshot of the initialized runtime, restored on cold start, often under 1-2 s). Beware uniqueness: random seeds, connections and timestamps cached at init must be re-established via CRaC hooks.
• GraalVM native image on provided.al2023 runtime: very fast start, small memory.
• Provisioned concurrency: pre-initialized environments, costs money.
• Smaller context: Spring Cloud Function instead of full MVC, lazy beans, more memory (CPU scales with memory).
• DB connections: use RDS Proxy; each concurrent Lambda opens its own connection.

ECS is simpler and AWS-native; EKS is standard Kubernetes with a richer ecosystem and more operational surface.

• ECS: task definitions/services, deep IAM and ALB integration, no control plane fee, Fargate integration is seamless; vendor lock-in.
• EKS: Kubernetes APIs, Helm/operators/service mesh, portable across clouds; control plane fee (~$0.10/h per cluster), upgrades every ~14 months of support, node management (managed node groups, Karpenter, Fargate).
• Choose ECS when the team is small and AWS-only; EKS when you need K8s tooling, multi-cloud or many platform teams.
• Both pull from ECR, log to CloudWatch (awslogs/Fluent Bit), and use IAM roles per workload.

Log JSON to stdout; the platform ships it to CloudWatch Logs. Export metrics with Micrometer, and traces with OpenTelemetry/X-Ray.

• ECS: awslogs driver (or FireLens/Fluent Bit); EKS: Fluent Bit/CloudWatch agent daemonset; EC2: CloudWatch agent.
• CloudWatch Logs Insights for queries; set retention (default is never expire, which costs money).
• Micrometer micrometer-registry-cloudwatch2 or Prometheus + AMP/Grafana; Actuator for health.
• Include a trace/correlation ID in MDC (Micrometer Tracing) to follow a request through services.
• Alarms: ALB 5xx, target response time, unhealthy hosts, task count, RDS CPU/connections/free storage.

Right-size first, then commit, then optimize data transfer.

• Right-size memory/CPU from real metrics; a JVM often runs at 20% CPU on over-provisioned instances.
• Savings Plans / Reserved Instances for the steady baseline (up to ~70% off), Spot for stateless/batch (up to 90%, handle 2-minute interruption), Graviton (ARM64) instances ~20% cheaper per performance; multi-arch images required.
• Fargate Spot for tolerant tasks; scale down non-prod at night.
• NAT Gateway: hourly + per-GB; use VPC endpoints, one NAT per AZ only for production.
• Cross-AZ and internet egress data transfer, CloudWatch log ingestion, idle load balancers, unattached EBS volumes.
• Use Cost Explorer, tags and budgets with alerts.

Tasks start but fail target-group health checks, so ECS stops them and starts new ones. Work through the chain:

1. ECS service events and stopped-task reason: Task failed ELB health checks, or Essential container exited.
2. CloudWatch logs: startup exception, bad datasource URL, missing secret.
3. Health check config: path correct (/actuator/health), port matches the container port, matcher 200, and healthCheckGracePeriodSeconds larger than app start-up (e.g. 120 s).
4. Security groups: ALB SG must reach task SG on the container port.
5. App binds to 0.0.0.0 (Boot does by default) and on the port in the task definition.
6. Health endpoint requires authentication (Spring Security 401), so the check fails; permit it.

aws ecs describe-services --cluster prod --services api --query 'services[0].events[:5]'
aws ecs describe-tasks --cluster prod --tasks <id> --query 'tasks[0].stoppedReason'

137 is SIGKILL, usually the container exceeded its hard memory limit (OutOfMemoryError: Container killed due to memory usage). Compare task memory with heap plus non-heap.

• Check the container memory (hard limit) vs memoryReservation (soft) in the task definition and Fargate task size.
• Set -XX:MaxRAMPercentage=70 instead of a fixed -Xmx near the limit.
• Look at CloudWatch Container Insights MemoryUtilized trends; sawtooth to limit suggests a leak or too small a limit.
• Also rule out health-check kill: ECS sends SIGTERM first and SIGKILL after stopTimeout (default 30 s) if the app does not exit.

Locally the SDK uses your developer credentials; in ECS it uses the task role, which lacks permission, or the wrong role was used (execution vs task).

• Confirm which identity the app uses: aws sts get-caller-identity from the container (or log StsClient).
• Add least-privilege policy to the task role for app calls; to the execution role only for pulling images and launch-time secrets injection.
• Check resource ARN specifics: bucket ARN vs object ARN (arn:aws:s3:::b and arn:aws:s3:::b/*), KMS key policy permitting decrypt.
• Check for a permissions boundary, SCP or bucket policy deny.

{ "Effect": "Allow", "Action": ["s3:GetObject","s3:PutObject"],
  "Resource": "arn:aws:s3:::shop-uploads/*" }

A timeout (as opposed to "connection refused" or auth failure) almost always means network blocking.

1. RDS security group inbound rule: port 5432/3306 from the app's SG (not just a CIDR).
2. Same VPC (or peering/TGW routes) and subnet routing; RDS "Publicly accessible" is irrelevant inside the VPC.
3. Correct endpoint and port in spring.datasource.url; not a stale instance endpoint after a restore.
4. Network ACLs allow ephemeral return ports.
5. Test from the task: nc -zv <endpoint> 5432 using ECS Exec / a debug container.
6. If it connects but fails: password authentication failed points to the Secrets Manager value, SSL required to sslmode.

On failover the endpoint's DNS record flips to the standby. The app has dead pooled connections and may cache the old IP.

• Set JVM DNS cache TTL low: -Dnetworkaddress.cache.ttl=30 (or networkaddress.cache.ttl in java.security); the default is about 30 s, but infinite if a security manager is installed.
• HikariCP: max-lifetime below network idle limits, connection-timeout ~3-5 s, keepaliveTime, so bad connections are evicted quickly.
• Use the AWS Advanced JDBC Wrapper (failover plugin) to detect topology changes in seconds, or RDS Proxy which holds client connections across failover.
• Aurora's cluster endpoint and faster failover (~30 s).
• Make clients retry idempotent operations (Spring Retry/Resilience4j).

Cold start of the JVM plus Spring context init exceeds the API Gateway/Lambda timeout. In order of cost-effectiveness:

1. Enable SnapStart on a published version/alias (invoke via alias), and add CRaC hooks for resources opened at init.
2. Increase memory (e.g. 512 MB to 2048 MB) to get more CPU; often halves init time.
3. Slim the app: Spring Cloud Function, remove unused starters, lazy initialization, disable component scanning of large packages.
4. Native image with custom runtime.
5. Provisioned concurrency for latency-critical endpoints.
6. If traffic is steady, move to Fargate; Lambda is wrong for constantly warm, long-lived JVM workloads.

During rolling or blue/green deploys old and new code run simultaneously, so the schema must support both: expand/contract.

1. Release 1 (expand): add the new column full_name, write to both columns, read old; backfill existing rows in batches.
2. Release 2: read new column, still dual-write.
3. Release 3 (contract): stop writing the old column; later migration drops it.

-- V12__expand_full_name.sql
ALTER TABLE customer ADD COLUMN full_name varchar(200);
UPDATE customer SET full_name = name WHERE full_name IS NULL;

A direct RENAME COLUMN breaks every old task instantly. Run migrations as a separate pre-deploy step (not by every instance racing on start) and make them non-blocking (e.g. CREATE INDEX CONCURRENTLY).

Three-tier, multi-AZ, least privilege.

• Route 53 to CloudFront/WAF (optional) to public ALB in 2+ public subnets (ACM TLS cert).
• ECS Fargate service (min 2 tasks across AZs) in private subnets; task role scoped to needed resources; secrets from Secrets Manager.
• RDS PostgreSQL Multi-AZ (or Aurora) in isolated subnets, encrypted with KMS, automated backups plus PITR; read replica for reporting.
• ElastiCache Redis replication group, Multi-AZ, TLS.
• VPC endpoints for ECR, S3, Secrets Manager, CloudWatch; NAT only for other egress.
• SG chain ALB to app to DB/cache; no inbound from internet except ALB.
• Autoscaling on request count; alarms on 5xx/latency/unhealthy hosts; CloudWatch dashboards; centralized logs.
• IaC (Terraform/CDK) and CI/CD with image scanning and blue/green deployment.

Use Cost Explorer grouped by service and usage type (tag your resources).

• Fargate: tasks over-sized (4 vCPU/8 GB where the JVM uses 20%); right-size from Container Insights; run Graviton (ARM) tasks, ~20% cheaper; Fargate Spot for non-critical.
• Too many always-on tasks in dev/staging: scale to zero out of hours.
• NAT Gateway data processing: image pulls from Docker Hub/ECR and S3 via NAT; add ECR/S3 endpoints.
• CloudWatch Logs ingestion from verbose logs; set retention and level INFO.
• ALB LCUs, cross-AZ data transfer.
• Commit with Compute Savings Plans (cover Fargate, EC2, Lambda).

Expected: right-sizing plus endpoints plus Savings Plan typically yields 30-50% savings.

The task cannot reach ECR (or Docker Hub) from a private subnet, or lacks permission.

• No route: add a NAT Gateway route or, better, VPC interface endpoints ecr.api, ecr.dkr, plus an S3 gateway endpoint (ECR layers are stored in S3) and logs.
• Endpoint security group must allow 443 from the task SG; enable private DNS.
• Execution role needs ecr:GetAuthorizationToken, ecr:BatchGetImage, ecr:GetDownloadUrlForLayer.
• Wrong image tag or repo/account/region; the architecture mismatch (arm64 image on x86 task) shows a different error (exec format error).
• Docker Hub rate limits: mirror to ECR (pull-through cache).

Total connections = tasks x Hikari max pool size. Autoscaling to 40 tasks with pool 20 asks for 800 connections.

• Compute the budget: max_connections (RDS default depends on instance memory) minus reserved for admin/replication; divide by max task count.
• Lower the per-task pool (10 is plenty for most; throughput is bounded by DB cores, not pool size).
• Add RDS Proxy to multiplex many app connections onto fewer DB connections (essential for Lambda).
• Offload reads to replicas, add caching, shorten transactions (do not hold a connection during remote calls).
• Alarm on DatabaseConnections and pool pending metrics (hikaricp_connections_pending).

spring.datasource.hikari:
  maximum-pool-size: 10
  minimum-idle: 10
  connection-timeout: 3000

Use ECS with the CodeDeploy blue/green controller and a canary config, wired to CloudWatch alarms.

• Two target groups (blue/green) behind the ALB with a production listener and a test listener.
• Deployment config such as CodeDeployDefault.ECSCanary10Percent5Minutes: 10% of traffic to green for 5 minutes, then the remainder.
• Alarms: HTTPCode_Target_5XX_Count, TargetResponseTime p99 on the green target group, plus a synthetic check.
• autoRollbackConfiguration with DEPLOYMENT_STOP_ON_ALARM, and DEPLOYMENT_FAILURE.
• Lifecycle hook (AfterAllowTestTraffic) Lambda runs smoke tests against the test listener before shifting real traffic.
• The old task set is kept for terminationWaitTimeInMinutes for fast rollback.

Scale-in terminates instances without regard for in-flight work unless you drain.

• Enable ALB deregistration delay (e.g. 60-120 s) so connections drain.
• Add an ASG termination lifecycle hook that runs a script: stop accepting traffic, wait for server.shutdown=graceful to finish, then complete the hook.
• Make the service run under systemd with TimeoutStopSec above the graceful timeout and KillSignal=SIGTERM.
• Enable instance scale-in protection for instances running critical jobs; make long jobs resumable (queue-based).

[Service]
ExecStart=/usr/bin/java -jar /opt/app/app.jar
SuccessExitStatus=143
TimeoutStopSec=40

Do not proxy large payloads through the app. API Gateway has a 10 MB payload limit; the ALB has a 60 s idle timeout and the app holds buffers.

Design: client asks the API for a pre-signed (or multipart) upload; uploads directly to S3; S3 event triggers processing.

1. POST /uploads returns a pre-signed PUT URL (or multipart upload ID plus per-part URLs).
2. Client uploads to S3 with a bucket CORS rule allowing the origin.
3. S3 event notification to SQS; a worker validates (size, type, virus scan) and records metadata.
4. Bucket: block public access, SSE-KMS, lifecycle rules for abandoned multipart uploads.

If streaming through the app is unavoidable, stream (InputStream to S3TransferManager) and set spring.servlet.multipart.max-file-size and temp dir sizes; never buffer into a byte[].

Locate the time with data rather than guessing.

1. ALB metrics: TargetResponseTime vs the app's own timer; large gap means network/queueing (target SG, connection reuse).
2. Tracing (X-Ray/OTel) to split time among app, DB, Redis and downstream calls.
3. RDS: Performance Insights for top SQL and waits, CPUUtilization, ReadIOPS, burst balance on gp2/T-class, connection count.
4. App: Hikari pending threads, GC pauses, CPU throttling on small Fargate sizes, thread pool saturation.
5. Cache: hit ratio, evictions, big keys; network cross-AZ latency (place app and cache in same AZ where possible).
6. Slow only on cold tasks: JIT warm-up; use autoscaling warm-up and readiness gating.

Typical root causes: N+1 queries, missing index, undersized DB, exhausted connection pool, cache stampede.

ECS resolves the task definition, not the registry, on deployment. With an unchanged task definition (still repo:latest), no new deployment is triggered, and a new task may reuse a cached image on the host.

• Use immutable tags (Git SHA, build number) and register a new task definition revision per release, so the deployment is explicit and rollback is a revision change.
• Enable ECR tag immutability.
• If forced: aws ecs update-service --force-new-deployment pulls latest again, but you lose traceability.
• Prefer the digest: repo@sha256:....

aws ecs register-task-definition --cli-input-json file://td.json
aws ecs update-service --cluster prod --service api --task-definition api:57