Kubernetes Basics

A Pod is one or more tightly coupled containers sharing a network namespace (one IP, localhost between them) and optionally volumes, scheduled together on one node. Kubernetes schedules and restarts Pods, not bare containers.

• Pods are ephemeral: when one dies it is replaced by a new Pod with a new IP and name.
• Multi-container patterns: sidecar (log shipper, proxy), init container (runs to completion before app containers, e.g. DB migrations or waiting for a dependency).
• Do not create bare Pods for apps; use a controller (Deployment).

A Deployment declares the desired state of stateless Pods; it manages ReplicaSets, and each ReplicaSet maintains a number of identical Pods for one Pod template version. A template change creates a new ReplicaSet and shifts replicas gradually.

apiVersion: apps/v1
kind: Deployment
metadata: { name: api }
spec:
  replicas: 3
  selector: { matchLabels: { app: api } }
  template:
    metadata: { labels: { app: api } }
    spec:
      containers:
        - name: api
          image: reg.io/shop/api:1.4.0
          ports: [{ containerPort: 8080 }]

• kubectl rollout history/undo deployment/api uses retained ReplicaSets (revisionHistoryLimit, default 10).
• selector is immutable after creation and must match the template labels.

A Service gives a stable virtual IP and DNS name in front of a changing set of Pods selected by labels; kube-proxy (iptables/IPVS) load-balances to ready endpoints only.

• ClusterIP (default): in-cluster only; DNS api.shop.svc.cluster.local.
• NodePort: opens a port (30000-32767) on every node.
• LoadBalancer: provisions a cloud LB (ELB/NLB on AWS).
• ExternalName: DNS CNAME alias.
• Headless (clusterIP: None): no VIP, DNS returns Pod IPs; used with StatefulSets.

apiVersion: v1
kind: Service
metadata: { name: api }
spec:
  selector: { app: api }
  ports: [{ port: 80, targetPort: 8080 }]

An Ingress is an L7 routing rule set (host and path to Service) implemented by an Ingress controller (NGINX, AWS Load Balancer Controller, Traefik); the resource alone does nothing without a controller. It consolidates many Services behind one load balancer and handles TLS termination.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: api }
spec:
  ingressClassName: nginx
  tls: [{ hosts: [api.shop.com], secretName: api-tls }]
  rules:
    - host: api.shop.com
      http:
        paths:
          - { path: /, pathType: Prefix, backend: { service: { name: api, port: { number: 80 } } } }

• On AWS, the AWS Load Balancer Controller turns Ingress into an ALB (and Service type=LoadBalancer into an NLB).
• The newer Gateway API is the successor for richer routing.

ConfigMaps hold non-sensitive config, Secrets hold sensitive values (base64 encoded, not encrypted by default). Inject as env vars or mounted files; Spring Boot's relaxed binding maps SPRING_DATASOURCE_URL to spring.datasource.url.

envFrom:
  - configMapRef: { name: api-config }
env:
  - name: SPRING_DATASOURCE_PASSWORD
    valueFrom: { secretKeyRef: { name: db-secret, key: password } }

• Mounted ConfigMap/Secret files update in running Pods (eventually, ~1 min); env vars do not update until Pod restart.
• spring.config.import=optional:configtree:/etc/secrets/ reads each file as a property.
• Enable encryption at rest for etcd, restrict RBAC, or use External Secrets / Secrets Store CSI with a cloud vault.

• Liveness: is the process stuck? Failure makes kubelet restart the container.
• Readiness: can it serve traffic now? Failure removes the Pod from Service endpoints without restarting.
• Startup: gates the other two until the app has started; protects slow-starting apps from being killed by liveness.

startupProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30        # up to 150 s to start
livenessProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8080 }
  periodSeconds: 5

• Spring Boot provides liveness/readiness state groups (management.endpoint.health.probes.enabled=true; auto on Kubernetes).
• Liveness should check only the app's internal health, never downstream dependencies.
• Readiness may include critical dependencies, but beware cascading removal of all Pods.

Requests are what the scheduler reserves on a node and what CPU shares are weighted by; limits are hard caps enforced by the kernel. Exceeding the memory limit kills the container (OOMKilled); exceeding the CPU limit throttles it.

resources:
  requests: { cpu: "500m", memory: "1Gi" }
  limits:   { memory: "1Gi" }

• QoS: Guaranteed (requests == limits for all containers), Burstable (some requests/limits), BestEffort (none). Under node pressure BestEffort, then Burstable are evicted first.
• Common JVM pattern: memory request = limit, CPU request set, CPU limit generous or absent to avoid throttling.
• Units: m millicores, Mi/Gi binary (M/G decimal).
• Set -XX:MaxRAMPercentage relative to the memory limit.

HPA periodically (default 15 s) computes desired = ceil(current * currentMetric / targetMetric) and updates the Deployment's replicas within min/max bounds. CPU utilization is measured as a percentage of the Pod's request, requiring metrics-server and CPU requests to be set.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: api }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 65 } }
  behavior:
    scaleDown: { stabilizationWindowSeconds: 300 }

• Custom/external metrics (requests/s, queue length) via Prometheus Adapter or KEDA.
• HPA scales Pods; the Cluster Autoscaler/Karpenter scales nodes when Pods are Pending.
• Do not set replicas in the Deployment manifest when using HPA via GitOps, or it will fight the HPA.

The default strategy RollingUpdate creates new Pods and terminates old ones gradually, governed by maxSurge (extra Pods allowed) and maxUnavailable (Pods allowed down). A new Pod counts as available only after readiness passes (plus minReadySeconds).

strategy:
  type: RollingUpdate
  rollingUpdate: { maxSurge: 25%, maxUnavailable: 0 }
minReadySeconds: 10
progressDeadlineSeconds: 300

• maxUnavailable: 0 with maxSurge: 1 gives zero-downtime but needs spare capacity.
• Roll back: kubectl rollout undo deployment/api; watch with kubectl rollout status.
• Recreate strategy kills all Pods first (downtime) and fits apps that cannot run two versions.
• A hung rollout exceeds progressDeadlineSeconds and is marked failed but is not rolled back automatically.

A PDB limits how many Pods of an app can be down due to voluntary disruptions (node drain, cluster upgrade, autoscaler scale-down), keeping minimum availability.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: api }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: api } }

• kubectl drain evicts through the Eviction API and respects the PDB; it blocks if eviction would violate it.
• It does not protect against involuntary disruptions (node crash, OOM) or deletion with kubectl delete pod.
• A PDB with minAvailable equal to replica count blocks node drains forever; leave headroom (maxUnavailable: 1).
• Combine with topology spread/anti-affinity so replicas are on different nodes/AZs.

Helm is the package manager for Kubernetes: a chart is a set of Go-templated manifests plus values.yaml defaults; a release is an installed instance of a chart with its revision history.

# templates/deployment.yaml (excerpt)
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
replicas: {{ .Values.replicaCount }}

helm upgrade --install api ./chart -n shop -f values-prod.yaml --set image.tag=1.4.1 --atomic --wait
helm rollback api 3
helm template api ./chart -f values-prod.yaml   # render without installing

• Layered values: chart defaults, environment file, --set.
• --atomic rolls back automatically if the release fails; hooks run Jobs (migrations) pre/post-upgrade.
• Add checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} as a Pod annotation to roll Pods on ConfigMap changes.
• Alternative: Kustomize overlays (built into kubectl -k).

CoreDNS creates records for every Service: <svc>.<namespace>.svc.cluster.local. Pods resolve short names via the search path in /etc/resolv.conf, so within the same namespace http://orders works, and across namespaces use orders.billing.

• Env vars <SVC>_SERVICE_HOST are injected too but only for Services existing at Pod creation.
• Spring Cloud Kubernetes or plain Service DNS replaces Eureka in a K8s-only environment.
• ndots:5 default causes extra DNS lookups for external names; use FQDN with trailing dot or tune dnsConfig.
• Headless Services return individual Pod A records.

Namespaces partition a cluster (per team/environment) for names, quotas and RBAC. Labels are key/value tags on objects; selectors query them (Services, Deployments, NetworkPolicies depend on this). RBAC (Role/ClusterRole plus bindings) controls which identities can perform which verbs on which resources.

• ResourceQuota and LimitRange cap and default resources per namespace.
• NetworkPolicy restricts Pod-to-Pod traffic (needs a CNI that enforces it); default is allow-all.
• Each Pod runs as a ServiceAccount; grant it only the permissions it needs, and on EKS map it to an IAM role (IRSA / Pod Identity).

CrashLoopBackOff means the container keeps exiting and kubelet restarts it with exponential backoff (10 s up to 5 min). Find why it exits.

kubectl get pod api-7d9f -o wide
kubectl logs api-7d9f --previous          # logs of the crashed container
kubectl describe pod api-7d9f             # Last State, Exit Code, Events
kubectl get events --sort-by=.lastTimestamp

• Exit code 1: application error (bad config, missing env, DB unreachable at startup).
• 137: SIGKILL, often OOMKilled or liveness kill; 143: SIGTERM.
• 126/127: command not executable/not found (wrong entrypoint, missing shell).
• Liveness probe failing during slow start: add a startupProbe or raise initialDelaySeconds.
• Debug: kubectl debug -it pod/api --image=busybox --target=api, or temporarily override command to sleep.

The container's cgroup memory limit was exceeded and the kernel killed it. Distinguish a leak from a mis-sized limit.

kubectl describe pod api-7d9f | grep -A5 "Last State"
kubectl top pod api-7d9f --containers

• Heap too close to limit: replace -Xmx with -XX:MaxRAMPercentage=70.
• Non-heap growth: threads, direct buffers, metaspace; use NMT (-XX:NativeMemoryTracking=summary) and jcmd.
• Raise the memory limit (and request equal to it) if working set legitimately needs it; the scheduler must then find nodes that fit.
• Plot container_memory_working_set_bytes against the limit; a steady climb means leak, sawtooth hitting limit means undersized.
• If the heap itself overflows, you get a Java OutOfMemoryError instead and the container may stay running; add -XX:+ExitOnOutOfMemoryError.

Pending means the scheduler has not placed it (or images/volumes are not ready). kubectl describe pod Events say why.

• 0/5 nodes are available: Insufficient memory/cpu: requests too large; lower them, add nodes, or let Cluster Autoscaler/Karpenter scale up.
• node(s) had untolerated taint: add a toleration or use another node pool.
• didn't match Pod's node affinity/selector: fix nodeSelector/affinity.
• unbound immediate PersistentVolumeClaims: no matching StorageClass/PV or zone mismatch (EBS volumes are AZ-bound).
• Quota exceeded in the namespace (ResourceQuota) appears at creation time (Deployment events), not as Pending.
• ContainerCreating/ImagePullBackOff are different later states (image/registry/secret issues).

kubectl describe pod web-0 | sed -n '/Events/,$p'
kubectl get nodes; kubectl describe node <n> | grep -A8 Allocated

Kubelet cannot pull the image, and retries with backoff. Check kubectl describe pod for the pull error.

• Typo in image name or tag, or the tag does not exist.
• Private registry without credentials: create a docker-registry Secret and reference it via imagePullSecrets (or attach to the ServiceAccount). On EKS, node role needs ECR read permissions.
• Registry rate limits (Docker Hub) or network egress blocked.
• Architecture mismatch gives exec format error at run time rather than pull failure.

kubectl create secret docker-registry regcred \
  --docker-server=reg.io --docker-username=ci --docker-password="$TOKEN"

spec:
  imagePullSecrets: [{ name: regcred }]

Walk from Pod outward.

1. kubectl get pods: READY 1/1? Not ready Pods are excluded.
2. kubectl get endpoints api (or endpointslices): empty means selector mismatch or unready Pods.
3. Compare Service.spec.selector with Pod labels; compare targetPort with the port the app listens on (Boot default 8080, not 80).
4. Test directly: kubectl port-forward pod/api-x 8080:8080, then from another Pod curl http://api.shop/actuator/health.
5. Check the app binds 0.0.0.0 (not 127.0.0.1, e.g. server.address set to localhost).
6. NetworkPolicy blocking, or Ingress serviceName/port wrong; check the controller logs.

kubectl get endpoints api -o wide
kubectl run tmp --rm -it --image=curlimages/curl -- curl -sv http://api/actuator/health

On termination, Kubernetes concurrently sends SIGTERM and removes the Pod from endpoints. Load balancers/kube-proxy may still route to the Pod for a few seconds; if the app stops accepting immediately, requests fail.

Fixes:

• preStop sleep to delay SIGTERM until endpoint removal propagates.
• Spring graceful shutdown to finish in-flight requests.
• terminationGracePeriodSeconds greater than preStop plus shutdown timeout.
• Readiness probe that goes refusing traffic on shutdown (Boot does this when graceful shutdown is on).
• maxUnavailable: 0 and sufficient surge.

terminationGracePeriodSeconds: 60
containers:
  - name: api
    lifecycle:
      preStop: { exec: { command: ["sleep", "10"] } }

server.shutdown: graceful
spring.lifecycle.timeout-per-shutdown-phase: 30s

kubectl describe hpa api shows the current metrics and conditions.

• <unknown>/65%: metrics-server not installed/unhealthy, or Pods have no CPU requests.
• Load is not CPU-bound (I/O wait on DB): use request rate or latency custom metrics.
• Already at maxReplicas, or ScalingLimited condition.
• Pods Pending after scale-up: no node capacity; need Cluster Autoscaler/Karpenter.
• The CPU limit throttles the Pod, so usage stays under target; utilization is relative to the request.
• Stabilization windows delay decisions (5 min for scale down by default; scale up is immediate).
• A Deployment manifest hard-coding replicas overwritten on every apply.

kubectl top pods -l app=api
kubectl describe hpa api

New Pods are not becoming Ready, so the rollout cannot proceed (and old Pods are kept, which protects availability).

kubectl rollout status deployment/api
kubectl get rs -l app=api
kubectl describe pod <new-pod>
kubectl logs <new-pod>

• Readiness probe failing: wrong path/port, the endpoint requires auth, app slow to start (add startupProbe).
• CrashLoopBackOff, ImagePullBackOff or Pending (insufficient resources; maxSurge Pods cannot schedule).
• Missing ConfigMap/Secret reference (CreateContainerConfigError).
• Roll back: kubectl rollout undo deployment/api. After progressDeadlineSeconds (default 600 s) the condition ProgressDeadlineExceeded is set.

Env vars are fixed at container start; mounted Secret files are refreshed by kubelet after a delay but Spring does not re-read them unless told to. A Pod restart is the simple fix.

• kubectl rollout restart deployment/api.
• Helm: annotate the Pod template with a checksum of the Secret/ConfigMap so a change triggers a rollout automatically.
• Tools like Reloader watch ConfigMaps/Secrets and restart workloads.
• Mounts with subPath never update.
• For dynamic refresh: Spring Cloud Kubernetes config reload or Actuator /refresh with @RefreshScope, but a DataSource password change still requires re-creating the pool.

Combine probes, resources, security context, graceful termination and spreading.

spec:
  replicas: 3
  strategy: { rollingUpdate: { maxSurge: 1, maxUnavailable: 0 } }
  template:
    spec:
      terminationGracePeriodSeconds: 60
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: { matchLabels: { app: api } }
      securityContext: { runAsNonRoot: true, runAsUser: 10001 }
      containers:
        - name: api
          image: reg.io/shop/api:1.4.0
          env: [{ name: JAVA_TOOL_OPTIONS, value: "-XX:MaxRAMPercentage=70" }]
          resources:
            requests: { cpu: "500m", memory: 1Gi }
            limits:   { memory: 1Gi }
          readinessProbe: { httpGet: { path: /actuator/health/readiness, port: 8080 } }
          livenessProbe:  { httpGet: { path: /actuator/health/liveness, port: 8080 } }
          lifecycle: { preStop: { exec: { command: ["sleep","10"] } } }

Plus: PDB, HPA, NetworkPolicy, ServiceAccount with least privilege, pinned image tags, readOnlyRootFilesystem with /tmp emptyDir.

Under load the health endpoint responded slowly (thread pool saturation, long GC, CPU throttling) and exceeded timeoutSeconds for failureThreshold times, so kubelet killed a working but busy Pod, shifting load to others and cascading.

• Make liveness trivial and cheap; Boot's /actuator/health/liveness reflects only internal state.
• Raise timeoutSeconds (default 1 s) and failureThreshold; use a startupProbe instead of long initialDelaySeconds.
• Run management endpoints on a separate port/thread pool (management.server.port=8081) so app saturation does not starve probes (Tomcat shares its pool otherwise).
• Fix real saturation: autoscale earlier, remove CPU limit throttling, tune pools.
• Use readiness (not liveness) for "temporarily overloaded".

Keep one chart and separate environment values files, with the image tag injected by CI.

# values.yaml (defaults)
replicaCount: 1
image: { repository: reg.io/shop/api, tag: "" }
resources: { requests: { cpu: 250m, memory: 512Mi } }
autoscaling: { enabled: false }
# values-prod.yaml
replicaCount: 3
autoscaling: { enabled: true, minReplicas: 3, maxReplicas: 15 }
resources: { requests: { cpu: 500m, memory: 1Gi }, limits: { memory: 1Gi } }

helm upgrade --install api ./chart -n prod -f values-prod.yaml \
  --set image.tag=$GIT_SHA --atomic --timeout 5m

• Secrets are not in values files; reference ExternalSecrets or use sops/helm-secrets.
• helm lint and helm template | kubeconform in CI; use GitOps (Argo CD/Flux) for drift control.
• Pin chart and image versions; promote the same artifact through environments.