Kubernetes

Scope: workloads, scheduling, networking, autoscaling, resource and admission policy, upgrades, and debugging for current Kubernetes clusters.

Version and support policy

• Upstream supports roughly the three most recent minor releases; managed offerings (EKS, GKE, AKS, and equivalents) track their own windows and often lag or lead by a minor. Pin clusters to a supported minor and upgrade at least every two minors so you never face a forced multi-minor jump. Verify the current supported set upstream before planning.
• Version skew: kubelet and kube-proxy may be older than the API server; kubectl should be within one minor of the cluster. Skew bounds relax slowly; check the current policy upstream.
• Feature lifecycle: do not build production dependencies on alpha APIs. Beta APIs can still change shape. Many capabilities that were once feature-gated are now on by default (Pod Security Admission, Gateway API core, ValidatingAdmissionPolicy CEL-based validation). Confirm graduation status per minor upstream.

Workload primitives

| Primitive | Use for | Avoid when | |---|---|---| | Deployment | Stateless HTTP/gRPC workers, rolling updates | Ordered identity or stable storage needed | | StatefulSet | Databases, queues, anything needing stable network ID or per-replica PVCs | Stateless workloads (headless service overhead is pointless) | | DaemonSet | Per-node agents: CNI, CSI, log shippers, node exporters | You only want one replica total | | Job | Run-to-completion batch, migrations | Long-lived processes | | CronJob | Scheduled batch | Sub-minute schedules or jobs longer than the interval | | ReplicaSet | Managed by Deployments only | Direct use (no rollout strategy) | | Bare Pod | Debugging only | Production (no self-healing, no rollout) |

Set startingDeadlineSeconds and concurrencyPolicy on CronJobs; overlapping jobs are the default and a common source of duplicate work.

Pod design rules

• Always set CPU and memory requests. Set limits for memory; be careful with CPU limits (throttling) — many teams drop CPU limits for latency-sensitive services and rely on requests plus namespace quotas. Document the choice.
• QoS classes: Guaranteed (requests equal limits) for latency-critical, Burstable for most, BestEffort never in production.
• Probes: use startupProbe for slow boot, readinessProbe for traffic gating, livenessProbe only for true deadlock detection. Never make liveness depend on a downstream dependency; that turns a partial outage into a restart storm.
• Handle SIGTERM: stop accepting new work, drain in-flight requests, exit before terminationGracePeriodSeconds. Add a preStop sleep when the service was just removed from endpoints.
• Security context baseline: runAsNonRoot: true, seccompProfile: RuntimeDefault, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities.drop: ["ALL"].
• Spread replicas with topologySpreadConstraints (maxSkew: 1, whenUnsatisfiable: DoNotSchedule) across zones and nodes. Prefer it over required pod anti-affinity, which is rigid and scheduler-expensive at scale.
• Attach a PodDisruptionBudget to every multi-replica service; without one, node drains can take all replicas at once.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels: { app: api }
  template:
    metadata:
      labels: { app: api }
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: api
          image: registry.example.com/api@sha256:REPLACE_WITH_DIGEST
          ports: [{ containerPort: 8080 }]
          resources:
            requests: { cpu: 200m, memory: 256Mi }
            limits: { memory: 512Mi }
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }

Scheduling

• Order of decisions: node selector/affinity narrows candidates; taints and tolerations filter; topology spread balances; priority and preemption resolve contention.
• nodeAffinity required rules exclude nodes permanently — good for architecture (arm64, GPU) but brittle for soft preferences. Use preferredDuringSchedulingIgnoredDuringExecution for preferences.
• Taints keep workloads off nodes: NoSchedule for dedicated pools (GPU, spot, ingress), NoExecute for eviction on node problems. Tolerate only what is necessary; blanket tolerations defeat the point.
• Use PriorityClass so critical system components win under pressure. Do not assign system-cluster-critical to user workloads.
• Node provisioning: Karpenter (NodePools/NodeClaims, just-in-time instances, consolidation, spot) is the common choice on AWS and expanding elsewhere; Cluster Autoscaler remains viable for mixed environments. Configure consolidation carefully — aggressive consolidation causes churn for long-running jobs.
• Use the descheduler or Karpenter drift/consolidation to rebalance after traffic shifts, not as a first-line scheduler fix.
• Avoid fragmentation: standardize a few instance shapes and keep headroom per zone to absorb failures.

Networking

• CNI: Cilium (eBPF dataplane, network policy, Hubble observability, service mesh modes) and Calico are the mainstream self-managed choices; cloud CNIs are fine but often lack rich policy. NetworkPolicy support is table stakes — verify enforcement, not just API acceptance.
• Services: ClusterIP for internal, headless for StatefulSets, NodePort for dev only, LoadBalancer with cloud annotations for external. Enable internalTrafficPolicy: Local when you want node-local routing and lower latency.
• DNS: CoreDNS scales with cluster size; ndots: 5 causes upstream search-path queries — use fully qualified names with a trailing dot (api.prod.svc.cluster.local.) or lower ndots for hot paths. NodeLocal DNSCache reduces conntrack pressure on large clusters.
• North-south: Gateway API is the forward-looking standard (GatewayClass, Gateway, HTTPRoute, GRPCRoute, TCPRoute); Ingress v1 is stable but feature-frozen. Use one gateway implementation per cluster where possible and attach certs via cert-manager or cloud certificate services.
• Egress: control and audit outbound traffic with explicit egress gateways or static egress IPs; NAT gateway costs scale with data volume, so route private traffic over VPC endpoints/private service connect.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: prod }
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

Then add explicit allow policies for DNS egress and required peers; default-deny without a DNS allow rule is the most common self-inflicted outage.

Autoscaling

| Layer | Tool | Scales on | Notes | |---|---|---|---| | Pod replicas | HPA | CPU, memory, custom, external | Stabilization windows prevent flapping | | Pod resources | VPA | Historical usage | Run in recommendation mode in production unless auto updates are proven | | Pod replicas (events) | KEDA | Queue depth, lag, cron, any event source | Enables scale-to-zero with request buffering | | Nodes | Karpenter / Cluster Autoscaler | Pending pods, utilization | Pair with PDBs and topology spread | | Cluster-proportional | CPA | Cluster size | For DNS, ingress, and similar per-cluster controllers |

• HPA formula: desiredReplicas = ceil(currentReplicas * currentMetricValue / targetMetricValue). Always set behavior.scaleDown.stabilizationWindowSeconds (for example 300s) to avoid churn.
• Do not point HPA and VPA at the same metric; VPA also fights memory-based HPA. Split: HPA on requests-per-second or queue depth, VPA on resources.
• Scale-to-zero on Kubernetes requires a buffer (KEDA HTTP add-on, Knative, or your own queue) — otherwise the user pays the cold-start latency. See ./05-serverless.md for cold-start tradeoffs.
• Node scale-down requires PDBs to be honored; pod-level PDBs plus empty budget handling can block consolidation and silently raise cost.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: api }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
  minReplicas: 3
  maxReplicas: 30
  behavior:
    scaleDown: { stabilizationWindowSeconds: 300 }
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }

Resource management and policy

• ResourceQuota caps a namespace's total requests/limits and object counts. Pair it with LimitRange defaults so pods that omit requests still get sane values instead of blocking on quota.
• PodDisruptionBudget: minAvailable percentages scale better than integers for replica counts that change; understand that PDBs only cover voluntary disruptions (drains, upgrades), not crashes.
• Admission order: mutating webhooks first, then validating. Keep webhook timeoutSeconds short (for example 5-10s), scope rules narrowly, and use failurePolicy: Ignore only for non-security-critical policies — Fail on a broad rule can lock out the whole cluster if the webhook goes down.
• Policy engines: Pod Security Admission labels namespaces (enforce: restricted, audit, warn) and needs no extra components; Kyverno is Kubernetes-native YAML policies; Gatekeeper/OPA suits organizations already invested in Rego. ValidatingAdmissionPolicy with CEL covers simple invariants without a webhook.
• With GitOps, never hand-write policy exceptions: policy exceptions are code, reviewed and expiring. See ./08-supply-chain-security.md for image verification policies.

Upgrades

1. Read the release notes for every minor you cross; check removed APIs (kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis, Pluto, or similar scanners).
2. Upgrade staging with production-like add-ons first; run conformance or smoke suites.
3. Upgrade the control plane (managed: one click/API; self-managed: one minor at a time, never skip).
4. Upgrade nodes with surge capacity: add new nodes, cordon and drain old ones, respect PDBs, then remove. Blue/green node pools are the safest pattern.
5. Upgrade add-ons in dependency order: CNI, CSI, DNS, metrics, ingress/gateway, policy, mesh, autoscaler.
6. Watch for 24-48h after; keep the previous node pool available for rollback until confident.
7. Control plane rollback is generally unsupported upstream — the real rollback is rebuilding or restoring the previous version deliberately. Test that path.

Debugging playbook

| Symptom | Likely cause | Check | |---|---|---| | Pending | Insufficient resources, taints, affinity, PVC unbound | kubectl describe pod events, quota, node capacity | | ImagePullBackOff | Missing imagePullSecret, wrong digest, rate limit | Events, node network, registry creds | | CrashLoopBackOff | App error, bad config, failing liveness | kubectl logs --previous, config mounts | | OOMKilled | Memory limit too low or leak | kubectl top pod, memory profile | | Evicted | Node disk/memory pressure | Node conditions, ephemeral storage usage | | Readiness never true | Probe misconfigured, slow dependency | Probe command, endpoint state | | DNS failures | CoreDNS overload, NetworkPolicy, conntrack | DNS test pod, CoreDNS metrics | | Stuck Terminating | Finalizer, volume detach, unresponsive kubelet | kubectl get pod -o yaml finalizers |

• Reach for kubectl debug (ephemeral containers) before SSH; nodes should be immutable.
• kubectl get events --sort-by=.lastTimestamp and kubectl describe cover most first-pass diagnosis.
• For control-plane issues, check admission webhooks first: a hung or failing webhook presents as broad API timeouts.

Anti-patterns

• Bare pods, or Deployments without probes and requests.
• latest image tags and no digest pinning.
• One giant namespace shared by unrelated teams with cluster-admin service accounts.
• Liveness probes that call downstream services.
• hostPath mounts, privileged containers, and hostNetwork without a documented reason.
• CPU limits on every latency-sensitive service causing throttling during bursts.
• PDB with minAvailable: 0 (renders it useless) or maxUnavailable: 0 (blocks all drains).
• Ad-hoc kubectl apply against production outside GitOps.

Checklist

• [ ] All workloads have requests, appropriate limits, and probes.
• [ ] Replicas >= 2 with topology spread across zones; PDBs attached.
• [ ] Namespace labels enforce Pod Security Admission at restricted (or documented exception).
• [ ] NetworkPolicy default-deny plus explicit DNS and peer allows.
• [ ] Images pinned by digest; admission verifies signatures (./08-supply-chain-security.md).
• [ ] Quota and LimitRange set per namespace.
• [ ] Autoscaling configured with stabilization and sane min/max.
• [ ] Upgrade path tested in staging, including add-on order.
• [ ] Runbooks exist for the top failure modes above.

GitOps

Scope: pull-based delivery design with Argo CD and Flux — repository structure, environment promotion, drift handling, secrets, and progressive delivery.

Principles

1. Declarative: the desired state of every environment is expressed as manifests.
2. Versioned and immutable: the source of truth is a Git commit; history is the audit log.
3. Pulled automatically: an in-cluster agent reconciles; CI does not need cluster credentials.
4. Continuously reconciled: divergence is detected and either corrected or surfaced.

The practical payoff is rollback by git revert and a reviewable trail of every production change. The practical cost is that everything, including secrets and generated manifests, needs a strategy. See ./08-supply-chain-security.md for signing and verification of what Git deploys.

Tool selection

| Dimension | Argo CD | Flux | |---|---|---| | Model | Application CRD + controller; UI-centric | Toolkit of controllers (source, kustomize, helm, notification) | | UI | Rich web UI, diff, sync, rollback | CLI-first; optional UI add-ons (Weave GitOps) | | Multi-cluster | ApplicationSets; hub-and-spoke, cluster generator | Kustomization per cluster; flux bootstrap per cluster | | Progressive delivery | Argo Rollouts | Flagger | | Secrets | Any (SOPS, Sealed Secrets, ESO) | SOPS first-class; Sealed Secrets, ESO also supported | | Helm | Native support plus hooks | HelmRelease via helm-controller | | Learning curve | Lower for app teams, ops needs to plan projects/RBAC | Lower for platform teams comfortable with composition | | Best fit | Manual gates, multi-tenant platform with UI expectations | Fully automated fleets, GitOps as a library of controllers |

Both are CNCF-graduated; the decision is organizational, not technical. Do not run both in the same cluster.

Repository structures

| Structure | Shape | Tradeoffs | |---|---|---| | Environment dirs in one repo | envs/{dev,staging,prod}/... | Simplest; watch RBAC and review noise | | App + env split | App repo has code and chart; config repo has env values | Clean ownership; two PRs per release | | App-of-apps | One root Application pointing at child Applications | Bootstrap and grouping; can become a snowflake | | ApplicationSet | Generator produces Applications from clusters, dirs, PRs | Scales multi-cluster; generators must be understood | | Flux hierarchy | Root Kustomization -> infra -> apps, each with dependsOn | Explicit ordering; deep nesting is hard to debug | | Per-team repos | Team owns its manifests; platform owns bootstrap | Strong tenancy; platform must standardize interfaces |

Conventions that prevent pain:

• One directory per environment; never one branch per environment. Branches diverge and defeat promotion.
• Keep rendered manifests out of Git unless you need auditability of the exact applied bytes; if you do render, do it in CI and commit the result to a clearly marked path.
• Cluster-specific configuration (domain, account IDs, regions) lives in the leaf env directory, not in shared templates.
• Bootstrap is code too: the root Application or Flux sync lives in a repo and is applied by a documented bootstrap procedure.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-prod
  namespace: argocd
spec:
  project: team-a
  source:
    repoURL: https://github.com/example/config.git
    targetRevision: main
    path: envs/prod/api
  destination:
    server: https://kubernetes.default.svc
    namespace: api
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: ["CreateNamespace=true", "ServerSideApply=true"]

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: api, namespace: flux-system }
spec:
  interval: 10m
  sourceRef: { kind: GitRepository, name: config }
  path: ./envs/prod/api
  prune: true
  wait: true
  dependsOn: [{ name: infra }]

Environment promotion

Promotion classes, from most to least desirable:

1. PR-based promotion: CI opens a PR that bumps the image digest or chart version in the next environment. Humans or policy approve. Best auditability; slightly slow.
2. Automated promotion with gates: a controller (Argo CD Image Updater, Renovate, Flux image automation) writes the bump, and policy plus health gates decide whether to continue.
3. Release-branch promotion: environment branches/tags; acceptable but watch drift between branches.
4. Build-once per environment: rebuilding for prod breaks artifact identity. Never.

Mechanics:

• Promote immutable references: image digest plus chart version. Tags can be moved; digests cannot.
• Keep environment differences in overlays (kustomize) or values files (Helm), not in duplicated manifests.
• Promotion should be a single commit across environments in order: dev -> staging -> prod.
• Record the source commit/build ID in annotations so any running workload traces back to code.
• Database migrations are their own pipeline step with backward-compatible sequencing; see ./06-iac.md for infrastructure promotion parity.

Drift detection and remediation

• Argo CD: automatic sync plus selfHeal corrects drift; OutOfSync diff view explains it. Turn on ServerSideApply for large CRDs. Set ignoreDifferences deliberately and narrowly (for example HPA-managed replicas), never wholesale.
• Flux: every Kustomization reconciles on an interval; drift is corrected on the next cycle. Use --watch style short intervals for critical namespaces and longer intervals elsewhere.
• Sources of legitimate drift: HPA replicas, cluster autoscaler, mutating admission webhooks (sidecars injected after apply), defaulting by controllers. Encode the exception once.
• Alert on drift that persists: an environment that continuously self-heals is hiding an uncontrolled mutation path.
• Never grant humans direct apply rights except documented break-glass; after break-glass, reconcile Git the same day.

Secrets in GitOps

| Approach | How it works | Use when | Caveats | |---|---|---|---| | SOPS + age/KMS | Encrypt values in Git; controller decrypts | Monorepo-first teams; Flux native | Key distribution and rotation are on you | | Sealed Secrets | Cluster controller public key encrypts | Single-cluster simplicity | Re-encrypt per cluster; rotation is awkward | | External Secrets Operator | Git holds references; secrets live in Vault/Cloud SM | Multi-cluster, centralized rotation | Requires secret-store IAM design | | Secrets Store CSI | Pods mount secrets from external store | Runtime injection without K8s Secret | Mount semantics differ per provider | | Plain Sealed/encrypted with shared key in CI | Avoid | — | CI key becomes a universal secret |

Rules: no plaintext secrets in Git ever; encrypt at the leaf, not the root; scope encryption keys per environment; prefer external stores when rotation matters; audit who can read decrypted values.

Progressive delivery

• Argo Rollouts: Rollout CRD replaces Deployment; canary steps, pauses, analysis templates querying Prometheus/Datadog; blue-green with preview services.
• Flagger: automates canary/blue-green for Deployments with either controller; uses service mesh or ingress traffic splitting.
• Gate promotion on service-level metrics from ./04-sre.md (error rate, latency), not on pod readiness alone.
• Always define abort behavior and a maximum canary duration; a canary that never finishes is an incident with no owner.
• Supported split backends: service mesh (Istio/Linkerd), Gateway API, and ingress controllers. Plain ClusterIP alone cannot split traffic — pick a backend first.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: api }
spec:
  replicas: 5
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }
      analysis:
        templates: [{ templateName: error-rate }]
        startingStep: 1
  selector: { matchLabels: { app: api } }
  template: { metadata: { labels: { app: api } }, spec: { containers: [{ name: api, image: api@sha256:REPLACE }] } }

Multi-cluster and tenancy

• Choose a topology: hub with remote clusters (Argo CD) or self-managing clusters (Flux). Hybrid ("management cluster runs Argo CD, clusters run Flux") is common and adds a second control plane to operate.
• Cluster registration is code: Argo CD Secrets/ApplicationSets or Flux bootstrap manifests. Manual registration drifts.
• Use AppProjects / Flux tenancy (--watch-label-selector or per-namespace service accounts) to scope what each team's pipeline can touch.
• For cross-cluster promotion, keep ordering explicit: clusters with dependsOn or sync waves; never infer ordering from alphabetical names.
• Disaster recovery: the Git repo plus bootstrap is the recovery artifact. Test rebuilding a cluster from Git quarterly. If rebuild takes days, the platform is not GitOps.

Anti-patterns

• Environment branches as the promotion mechanism.
• CI credentials with cluster-admin; push-based deploys mixed with pull reconciliation on the same resources.
• Committing rendered manifests and hand-editing them.
• Blanket ignoreDifferences to silence noisy diffs.
• Secrets encrypted with a key every developer holds.
• One giant Application per cluster with no ownership boundaries.
• Canary analysis on pod readiness only.
• No prune for years, then a prune-enabled sync deleting dozens of orphaned resources at once. Enable prune deliberately and review orphans regularly.

Checklist

• [ ] Single source of truth per environment; no branch-based env promotion.
• [ ] Bootstrap documented and reproducible from an empty cluster.
• [ ] Promotion uses digests; environment diffs live in overlays/values.
• [ ] Drift alerts exist and ignoreDifferences entries have owners.
• [ ] Secrets strategy chosen; no plaintext in Git; rotation documented.
• [ ] Progressive delivery backend selected with analysis metrics.
• [ ] Sync waves/dependsOn encode ordering, including CRDs before CRs.
• [ ] RBAC: teams scoped to their projects/namespaces; break-glass audited.
• [ ] Cluster rebuild tested.

Service Mesh and Networking

Scope: east-west and north-south service networking — Istio ambient and sidecar, Linkerd, Cilium, Gateway API, mTLS, traffic shifting, and multi-cluster.

Do you need a mesh?

| Requirement | Mesh needed? | Reasonable default | |---|---|---| | Encrypt all service-to-service traffic (zero trust) | Yes | Istio ambient or Linkerd | | Per-route retries, timeouts, circuit breaking | Yes (or gateway-only) | Mesh, or Gateway API at the edge for north-south | | Canary traffic splitting inside the cluster | Yes | Mesh or a service-mesh-capable ingress | | Network policy and flow observability only | No | Cilium NetworkPolicy + Hubble | | A handful of services on one cluster | No | Ingress/Gateway plus app-level libraries | | Compliance requiring cryptographic workload identity | Yes | SPIFFE-based mesh (Istio/Linkerd) |

Default posture: run a mesh when you have more than roughly a dozen services or a hard mTLS requirement; otherwise the operational overhead outweighs the benefit. Avoid running two meshes or a mesh plus per-app service discovery libraries with overlapping responsibilities.

Data plane models

| Model | How traffic is intercepted | Cost | Fits | |---|---|---|---| | Sidecar proxy (Envoy per pod) | iptables redirect per pod | Per-pod memory/CPU; restarts per upgrade | Maximum features and maturity; existing Istio fleets | | Ambient (ztunnel per node + waypoint per namespace/service) | eBPF plus node-level proxy for L4; optional L7 waypoints | Lower overhead; no pod restarts for L4 | New Istio installs; teams wanting mTLS without sidecar tax | | eBPF mesh (Cilium) | Kernel-level, no user-space proxy for L4 | Lowest overhead | Teams already on Cilium wanting policy plus L4 identity | | Lightweight Rust/proxy (Linkerd) | Sidecar with micro-proxy | Low per-pod overhead | Simplicity-focused teams; stable L7 feature set |

Istio ambient mode is GA in recent releases (verify upstream) and is the recommended greenfield Istio mode; sidecar mode remains fully supported and is still the choice when you need L7 features on every pod without waypoint management.

Istio

• Components: istiod (control plane), ztunnel (ambient L4), waypoints (ambient L7), gateways (north-south). Use the istioctl CLI for install, analyze, and proxy debugging.
• Installation: prefer the official Helm charts or the operator in a dedicated istio-system namespace; pin the minor version and use revision-based upgrades (install new revision, migrate, remove old).
• PeerAuthentication sets mTLS mode per mesh/namespace/workload; set STRICT mesh-wide only after non-mesh traffic is handled, or use PERMISSIVE during migration. DestinationRule with ISTIO_MUTUAL is needed on the client side for sidecar mode.
• Authorization: AuthorizationPolicy works on identity (service account) and L7 attributes in sidecar mode and at waypoints. Default-deny plus explicit allows mirrors NetworkPolicy design.
• Ingress/egress: use one Gateway implementation (Istio Gateway or Gateway API) per cluster. Duplicated ingress paths across istio-ingressgateway and a second ingress controller are a top source of 502s and stale DNS.
• Traffic shifting: VirtualService weights or Gateway API HTTPRoute weights; see the snippet below. Prefer Gateway API for new north-south routes, VirtualService for sidecar-era east-west routing.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: prod }
spec:
  mtls: { mode: STRICT }

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: api, namespace: prod }
spec:
  parentRefs: [{ name: prod-gateway }]
  hostnames: ["api.example.com"]
  rules:
    - backendRefs:
        - { name: api-v1, port: 80, weight: 90 }
        - { name: api-v2, port: 80, weight: 10 }

Linkerd

• Choose Linkerd for a small operational surface: a Rust micro-proxy, automatic mTLS on by default, and a stable feature set. It does not attempt to replace your gateway or provide Envoy extensibility.
• Install via the CLI/Helm; inject by namespace annotation or per-workload annotation. Verify with linkerd check and watch success rate via linkerd viz.
• Linkerd supports Gateway API HTTPRoute for traffic splitting when configured with its gateway; otherwise use HTTPRoute attached to your existing gateway and let Linkerd handle identity and observability.
• Multicluster: gateway-based linking between clusters; services remain addressable across clusters with mTLS.
• Tradeoff: no WASM filters or EnvoyFilter escape hatch. If you need deep L7 customization, that is an Istio/Envoy argument, not a Linkerd one.

Cilium

• eBPF dataplane provides NetworkPolicy (including L7 via Envoy where needed), transparent encryption (WireGuard/IPsec), and Hubble flow visibility.
• Cilium service mesh features cover L4 identity-based policy and, with CiliumEnvoyConfig, L7 routing. This is not a drop-in replacement for a full L7 mesh; it is the right answer when you want policy plus encryption plus observability with minimum latency.
• Use Hubble for flow-level debugging: hubble observe --namespace prod --verdict DROPPED answers "why can't A reach B" faster than most mesh tooling.
• Gateway API support exists for ingress; verify feature maturity for your version upstream.

Gateway API

• Core resources: GatewayClass (infrastructure), Gateway (listener + TLS), HTTPRoute/GRPCRoute (routing rules). Stable channel covers the core; experimental channel adds TCP/UDP/TLS routes and more.
• One Gateway per cluster or per tenant; routes attach via parentRefs with optional section names. This separation lets platform teams own listeners and app teams own routes.
• Migration from Ingress: map host/path rules to HTTPRoute, annotations to policy attachments or implementation-specific resources. Keep both only during a bounded migration window.
• GAMMA (Gateway API for Mesh) is the emerging east-west standard; adoption varies by mesh. Verify support upstream before standardizing on it.
• TLS: terminate at the gateway with cert-manager or cloud certificates; for passthrough, use TLSRoute with SNI. Redirect HTTP to HTTPS at the gateway, not in every app.

Traffic management and resilience

• Timeouts and retries belong at one layer. If the mesh retries and the app retries and the client retries, you get retry storms. Set mesh retries small (1-2) with budget, and disable app-level retries on idempotent-safe paths.
• Circuit breaking: Envoy outlierDetection (sidecar) or route-level policies; eject unhealthy hosts and let them recover. Without outlier detection, one bad pod absorbs traffic until a human notices.
• Load balancing: default round-robin is fine for most; locality-aware routing (localityLbSetting) reduces cross-zone cost and latency. See ./09-cost-platform-engineering.md for the cost angle.
• Rate limiting: apply at the gateway for external traffic and in the mesh for internal abuse; a shared Redis-backed limiter beats per-pod limits that multiply with replicas.
• Health checks and retries must agree: a mesh retry after the app has already timed out doubles load on a struggling dependency.

Multi-cluster

• Options: single mesh across clusters (Istio multicluster primary-remote, Linkerd multicluster links), or gateway-to-gateway federation (east-west gateways), or no mesh with cloud private networking.
• Requirements before multi-cluster traffic: shared trust (same root CA or federated trust bundles), unique cluster/network identities, non-overlapping pod/service CIDRs, and DNS resolution across clusters.
• Failure modes: partial connectivity partitions, trust-bundle rotation breaking links silently, and version skew between clusters. Test link failure and trust rotation deliberately.
• Keep the blast radius explicit: multi-cluster east-west is usually for a small set of services, not the whole mesh.

Gateway operations

• TLS: terminate with cert-manager (ACME for public certs, internal CA for private) or cloud certificate services. Rotate certificates automatically; monitor expiry as an SLI because an expired gateway cert takes down everything behind it.
• Listeners: one Gateway per environment is usually enough. Separate listeners by port/protocol; use allowedRoutes to control which namespaces may attach routes (Same vs All).
• Timeouts belong on the route or listener where the client behavior is defined; keep backend timeouts longer than gateway timeouts so the gateway gives up first and returns a clean 504 instead of a hanging connection.
• Header and path rewrites: do them at the gateway, not in every application. Keep one canonical scheme-to-host mapping documented; shadow routes (duplicate hostnames across gateways/ingress) are a leading cause of intermittent 404s.
• External DNS and health checks: automate DNS records from Gateway/Service annotations where your platform supports it; health-check the gateway from outside the cluster so a bad listener is detected before users report it.
• Rate limiting and WAF: apply coarse protection at the edge (CDN/WAF) and identity-aware limits at the gateway. A mesh is not a WAF.

Mesh upgrade and operations

• Control plane upgrades: install the new revision alongside the old, migrate namespaces by relabeling, verify, then remove the old revision. Never upgrade the control plane and data plane in one uncontrolled step.
• Sidecar-mode upgrades restart pods; schedule them with PDBs and surge capacity the same way you would a deployment rollout.
• Ambient upgrades touch ztunnel (node-level) and waypoints; node drains rolling ztunnels cause brief L4 reconnection for pods on that node — plan for it.
• Configuration hygiene: run istioctl analyze (or the Linkerd equivalent) in CI against rendered manifests; catch misapplied AuthorizationPolicies before they lock out a namespace.
• Keep a known-good rollback revision and the previous chart values in Git so a failed upgrade is one revert away. See ./02-gitops.md.

Observability

| Layer | Istio | Linkerd | Cilium | |---|---|---|---| | Metrics | Prometheus, Kiali | Prometheus, linkerd viz | Prometheus, Hubble | | Traces | Envoy/OpenTelemetry integration | OpenTelemetry integration | Hubble plus OTel | | Flow logs | Access logs per proxy | linkerd viz tap | hubble observe | | Topology | Kiali graph | linkerd viz graph | Hubble UI |

Instrument at the mesh for service-to-service RED metrics, but keep application-level traces for business context. See the sibling observability pack for metric and alert design.

Traffic capture and debugging

• Start with identity: istioctl proxy-config (clusters, listeners, routes) for sidecars and waypoints; linkerd viz tap and hubble observe for live flows.
• To answer "is mTLS actually on?", check the peer identity in proxy stats/metrics rather than trusting configuration; configuration can be accepted but not enforced due to mesh/namespace scope or revision mismatch.
• Packet-level debugging is a last resort: capture inside the proxy/netns (kubectl debug plus tcpdump) rather than on the host, and remember encrypted traffic shows nothing useful.
• The three most common mesh incidents: a route attached to the wrong gateway/listener; an AuthorizationPolicy applied to the wrong workload selectors; and a control plane/data plane version mismatch after a partial upgrade.
• Keep a synthetic canary (one client calling one server every minute) whose result is exported to your monitoring; it distinguishes "mesh broken" from "app broken" within seconds.
• Log levels: raise proxy logging only for the namespace under investigation, with a time limit and a cleanup commit; global debug logging in Envoy can generate gigabytes per hour.

Anti-patterns

• Enabling a mesh before a single team understands its failure modes and debugging commands.
• STRICT mTLS flipped before every client is injected; the resulting connection failures get misdiagnosed as app bugs.
• Layering mesh retries, gateway retries, and application retries on the same call path.
• Running two ingress controllers for the same hostnames.
• Sidecar injection on Jobs and batch workloads where overhead and restart semantics hurt.
• Using a mesh as a substitute for NetworkPolicy in clusters where enforcement is already available natively.
• Ignoring control plane version skew during mesh upgrades.

Checklist

• [ ] Mesh justified against the decision table above.
• [ ] One data plane model chosen (ambient, sidecar, or eBPF); no mixed-mode without a migration plan.
• [ ] mTLS rollout staged with visibility before enforcement.
• [ ] Authorization default-deny with explicit allows per service.
• [ ] Gateway API (or one gateway implementation) owns north-south; no overlapping hostnames.
• [ ] Retry/timeout policy assigned to exactly one layer.
• [ ] Outlier detection or equivalent ejection configured for critical paths.
• [ ] Upgrade procedure tested in a non-production cluster, including trust-bundle rotation.
• [ ] Debugging runbook exists (istioctl analyze, linkerd check, hubble observe).

Site Reliability Engineering

Scope: reliability targets, error budgets, toil reduction, incident response, chaos engineering, and capacity planning.

SLI, SLO, SLA

| Term | Definition | Example | |---|---|---| | SLI | A measured aspect of service behavior | Fraction of requests served < 300ms | | SLO | Target for an SLI over a window | 99.9% over 28 days | | SLA | Contractual commitment with consequences | 99.9% or service credits | | Error budget | Allowed failure implied by the SLO | 0.1% of requests |

Rules of thumb:

• Pick a small number of SLIs per user journey: availability, latency, and for pipelines freshness/throughput. Correctness and durability are SLIs too where data matters.
• Express SLIs as good-events / valid-events so the math is a ratio, not an average of averages.
• Start with 99.9% for user-facing services; reserve 99.99% for services whose failure directly blocks revenue or safety. Every nine costs money and slows delivery.
• Window: rolling 28 days (four weeks) balances responsiveness and stability; some teams use calendar months for reporting. Pick one and stay consistent.
• Make SLOs of dependencies stricter than dependents: if your SLO is 99.9%, dependencies should target 99.95%+.
• Do not promise 100% - user devices and networks are outside your control.

Error budget policy

Write it down before the first breach and get it signed by product and engineering:

1. Budget intact (> 25% remaining): ship features normally.
2. Budget low (< 25%): prioritize reliability work; freeze risky launches.
3. Budget exhausted: freeze all changes except reliability fixes and security patches until the budget recovers.
4. Every freeze has an owner, an exit criterion, and a review date.
5. Budget spend is a signal, not a failure: teams that never spend the budget are under-shipping.

Escalation path must be explicit. A policy that nobody enforces is worse than no policy because it teaches the organization that SLOs are theater.

Burn-rate alerting

Burn rate = rate of budget consumption relative to the SLO. A burn rate of 1 exhausts the budget exactly at the end of the window; 14.4x exhausts a 28-day budget in about two days.

Multi-window, multi-burn-rate alerts catch both fast outages and slow bleeds without paging on every blip:

| Severity | Burn rate | Long window | Short window | Budget consumed | |---|---|---|---|---| | Page | 14.4 | 1h | 5m | ~2% in 1h | | Page | 6 | 6h | 30m | ~5% in 6h | | Ticket | 1 | 3d | 6h | ~10% in 3d |

# Page: 14.4x burn over 1h, confirmed by 5m window
(
  sum(rate(http_requests_total{code=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > (14.4 * 0.001)
)
and
(
  sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > (14.4 * 0.001)
)

• Alert on symptoms users feel (error ratio, latency), never on causes (CPU, disk) except where a cause has no symptom proxy yet.
• Every alert links a runbook and a dashboard. If a page has no action, delete it.
• Review alert quality monthly: pages per shift, false-positive rate, time to acknowledge. Target fewer than two pages per on-call shift outside real incidents.

Toil

Toil is manual, repetitive, automatable, tactical work that scales with service growth and produces no enduring value.

How to find it: track work in half-hour granularity for two weeks; interview on-call. Common sources: manual deploys, certificate and secret rotation, access grants, capacity tickets, data restores, false-positive alert triage, one-off dashboards.

How to kill it:

• Automate the top five recurring tasks per quarter; require a runbook to be executed twice before automation (the second execution is the spec).
• Self-service replaces ticket queues (see ./09-cost-platform-engineering.md for internal developer platforms).
• Cap toil per SRE at 50% of time; the rest goes to engineering and reliability projects.
• Watch automation rot: scripts with no owner become new toil. Every automation has an owner and a test.

Incident management

Severity model

| Severity | Impact | Response | Comms | |---|---|---|---| | SEV1 | Major outage, data loss risk, safety | Page immediately, all hands | Status page + exec update every 30m | | SEV2 | Degraded for many users or a key customer | Page on-call, incident channel | Internal updates hourly | | SEV3 | Minor degradation, workaround exists | Ticket or next business day | Channel only | | SEV4 | Cosmetic, no user impact | Backlog | None |

Roles

• Incident Commander (IC): owns the response, not the fix. Decides mitigation, delegates, calls severity.
• Operations lead: coordinates technical work, owns the timeline.
• Communications lead: updates stakeholders and status page; shields responders from inbound questions.
• Scribe: records decisions and timestamps for the postmortem.

For small teams one person may hold multiple roles, but the IC must not be debugging. Rotate and practice.

Response principles

1. Mitigate before root cause: rollback, scale up, fail over, rate limit. Customers do not care why yet.
2. Declare early and downgrade later. Under-declaring is the normal failure.
3. One channel, one source of truth for status.
4. Change one thing at a time and record it.
5. Handover explicitly at shift change with current state and next hypothesis.

Postmortems

• Blameless and written within five business days; published internally by default.
• Cover: impact in user terms, timeline, contributing factors (not "human error"), what went well, what was lucky, and action items.
• Action items have owners and dates; no more than three per incident so they actually get done.
• Track action-item completion as a reliability metric. Recurring causes mean the items were too shallow.
• Review patterns quarterly across incidents; fix systemic causes, not just the last one.

Chaos engineering

Principles:

• Define steady state as measurable output (SLO burn, throughput), not internal counters.
• Hypothesize the system will stay in steady state, then inject a realistic fault.
• Run in staging before production; start with a low-blast-radius production experiment only when staging has proven the tooling and the safety valves.
• Automate experiments and put them in CI where possible.
• Minimize blast radius: kill one pod, one zone, one dependency; never "all at once" until the program is mature.

Tooling: LitmusChaos and Chaos Mesh for Kubernetes-native experiments; cloud provider fault injection services (AWS FIS and equivalents) for infrastructure-level faults; load generators plus failure injection to test degradation paths.

Starter experiment catalog:

1. Kill a random pod of a critical deployment; verify PDB and restart budget.
2. Inject 500ms latency on the database dependency; verify timeouts and degraded responses.
3. Fail one availability zone; verify capacity headroom and failover.
4. Exhaust a dependency (DNS, auth); verify circuit breakers and cached behavior.
5. Roll a bad config; verify rollout abort and alerting.

Game days: schedule them, invite the whole team, and record what surprised you. The debrief is the product, not the experiment.

Capacity and load

• Model capacity as peak-arrival-rate -> resource need -> fail-over headroom. Two numbers matter: normal peak, and peak during failure of one zone or one node group.
• Maintain headroom to survive N-1 infrastructure failure plus traffic growth between provisioning cycles. Stateless services: target 40-60% utilization at peak; databases need more.
• Queueing reality: latency rises non-linearly as utilization approaches 100%. The knee usually appears above 60-70% for shared resources. Autoscaling reacts after the knee; provision for the knee, do not chase the cliff.
• Little's law: concurrency = arrival rate x latency. Use it to translate throughput targets into worker/connection requirements and to sanity-check load-test numbers.
• Load testing: k6, Locust, Gatling, or vegeta for HTTP; use production-shaped data and think time; ramp until the SLO breaks and record that point as the safe ceiling with margin.
• Run a load test after every architectural change and before every major event (launch, sale, migration).
• Database and third-party quotas are capacity too: connection pools, API rate limits, and NAT ports exhaust before CPU does.
• Review capacity monthly: forecast 3-6 months out based on growth, not just current usage.

Anti-patterns

• SLOs with no error budget policy or budget that is never enforced.
• Paging on every individual failure instead of on burn rate.
• Incident response with no IC, causing parallel heroes and no decisions.
• Postmortem action items that are all "be more careful" or "add a test" with no owner.
• Chaos experiments without steady-state measurement or a kill switch.
• Capacity planning from average utilization instead of peaks and failure headroom.
• Reliability owned by one team while product changes ship freely.

Checklist

• [ ] Every user-facing service has 1-3 SLIs with defined sources and windows.
• [ ] SLOs published; error budget policy signed and enforced.
• [ ] Burn-rate alerts in place for page and ticket severities.
• [ ] On-call rotation exists with escalation and handover procedure.
• [ ] Incident severity matrix and roles documented; IC trained.
• [ ] Postmortems blameless, timely, and action items tracked.
• [ ] At least one chaos experiment per quarter against a critical path.
• [ ] Capacity headroom tested for N-1 zone failure.
• [ ] Load-test baseline recorded and re-run before major launches.

Serverless

Scope: function platforms, serverless containers, edge runtimes, cold starts, event integration, local development, and platform limits.

Choosing the execution model

| Model | Unit of deploy | Billing | Best for | Watch out for | |---|---|---|---|---| | Functions (FaaS) | One function | Per invocation and duration | Event handlers, glue, spiky APIs | Cold starts, runtime limits, local dev gap | | Serverless containers | Container image | Per request or per vCPU-second | HTTP APIs, workers, jobs, migrations | Scale-to-zero latency, per-request concurrency rules | | Edge runtime | Worker/script at PoP | Per request | Latency-sensitive routing, auth, personalization | Restricted APIs, tiny bundle limits | | Kubernetes | Pod | Per node-hour | Stateful systems, long-running, GPU, custom networking | Operational overhead; see ./01-kubernetes.md |

Default path: start with serverless containers for HTTP services and FaaS for event handlers. Move to Kubernetes only when a concrete constraint (runtime, networking, state, GPU, cost curve at steady high utilization) forces it. See ./09-cost-platform-engineering.md for the crossover math.

Function platforms

| Platform | Packaging | Notable traits | Local tooling | |---|---|---|---| | AWS Lambda | zip or OCI image | Mature event integrations, SnapStart for supported runtimes, response streaming, provisioned concurrency | SAM CLI, sam local, LocalStack | | Google Cloud Run functions | zip (source) or image | Gen2 runs on Cloud Run infrastructure; integrates with Eventarc | functions-framework, gcloud functions deploy --local | | Azure Functions | zip, container | Durable Functions for orchestrations; Flex Consumption plans | Azure Functions Core Tools | | Cloudflare Workers | JS/WASM module | V8 isolates, near-zero cold start, D1/R2/KV bindings | wrangler dev |

Guidance:

• Keep handlers thin: parse input, call a service module, return. Business logic belongs in code you can test without the platform.
• One function per responsibility but not per endpoint by default; packaging many tiny functions multiplies cold starts and IAM surface.
• Use container-image packaging when you need native libraries or large dependencies. Image size affects cold start and deploy time.
• Version runtimes explicitly; language runtimes reach end of support on a published schedule (verify upstream). Platform deprecations are a recurring maintenance cost.

# handler.py - thin function, logic elsewhere
import os
from service import process_order

def handler(event, context):
    for record in event.get("Records", []):
        process_order(record["body"], table=os.environ["ORDERS_TABLE"])
    return {"batchItemFailures": []}

# template.yaml (AWS SAM, minimal)
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Resources:
  Consumer:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.13
      Handler: handler.handler
      MemorySize: 512
      Timeout: 30
      Architectures: ["arm64"]
      Environment: { Variables: { ORDERS_TABLE: !Ref OrdersTable } }
      Events:
        Queue: { Type: SQS, Properties: { Queue: !GetAtt OrdersQueue.Arn, BatchSize: 10 } }

Serverless containers

• Cloud Run: request-driven, concurrency per instance configurable, scale to zero, jobs for batch. Set min-instances > 0 only when cold-start latency is user-visible. Set CPU always-allocated only for background work.
• Fargate: run containers without nodes; pair with ALB for HTTP or queue workers; slower scale-out than request-based platforms and billed per second per task.
• Azure Container Apps: Kubernetes-based (KEDA-driven), revision-based traffic splitting, scale rules from event sources.
• Knative: the portable Kubernetes-native option; use when you need on-prem/portable serverless containers and can operate it.
• Patterns: graceful shutdown (handle SIGTERM, drain), readiness gate before traffic, request timeouts shorter than platform timeouts, and a /healthz that does not call dependencies.
• Concurrency: platform concurrency per instance is a tuning lever. High concurrency reduces instance count and cost but risks noisy-neighbor latency inside the instance; set it deliberately and load test.

# Cloud Run service (excerpt; apply with gcloud run services replace)
apiVersion: serving.knative.dev/v1
kind: Service
metadata: { name: api, annotations: { run.googleapis.com/ingress: internal-and-cloud-load-balancing } }
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "0"
        autoscaling.knative.dev/maxScale: "50"
        run.googleapis.com/cpu-throttling: "true"
        run.googleapis.com/startup-cpu-boost: "true"
    spec:
      containerConcurrency: 40
      containers:
        - image: us-docker.pkg.dev/PROJECT/api@sha256:REPLACE
          resources: { limits: { cpu: "1", memory: "512Mi" } }

Edge runtimes

• Use for: auth at the edge, A/B routing, header/redirect logic, caching, personalization, and lightweight aggregation. Do not use for heavy compute or code needing filesystem, raw sockets, or long-lived connections.
• Constraints: bundle size limits, restricted APIs, per-request CPU limits, and a KV/object model rather than general storage. Verify current limits upstream; they change.
• Data locality: edge KV/object stores are eventually consistent in some regions. Design for read-mostly or accept staleness.
• Observability is often weaker than regional platforms; make sure you can trace edge -> origin with one ID before adopting.

Cold starts

Causes: container/runtime initialization, dependency loading, JIT warmup, VPC/network setup, image pull.

Mitigations, roughly in order of effect:

1. Choose a lighter runtime (compiled or dynamically fast-start engines start in tens of milliseconds; managed runtimes reset the clock on major language versions).
2. Minimize package size; lazy-load heavy dependencies; use arm64 where supported.
3. Increase memory: many platforms allocate CPU proportionally, so more memory means faster start.
4. Enable platform features: SnapStart for supported JVM/Python runtimes (restores from a snapshot; note the need to handle post-restore uniqueness), provisioned/min instances for latency-critical paths.
5. Keep VPC attachments lean; prefer newer networking integrations and private endpoints over legacy NAT-heavy designs.
6. Pre-warm deliberately only as a last resort: scheduled pings reduce cold starts but cost money and can mislead autoscaling.

Budget rule: if p99 cold start plus normal latency violates the SLO, either pay for warm capacity or move that path to serverless containers with min instances.

Events and integration

Patterns:

• Queue worker: queue -> function with batch size, partial-batch failure reporting, DLQ after N receives, and concurrency cap to protect downstream.
• Fan-out: pub/sub topic -> one queue per consumer -> function per consumer. Never point multiple consumers at one queue expecting broadcast.
• Stream processing: shards/partitions -> ordered per key. Increase parallelism with partitions, not batch size, when ordering matters.
• Event bus routing: rule-based routing to targets (EventBridge and equivalents) for decoupled domain events with schema registries.
• Schedule: cron expressions with timezone awareness; idempotent handlers because retries happen.

Reliability rules:

1. Every event handler is idempotent; at-least-once delivery is the norm, exactly-once is rare and still requires idempotency for side effects.
2. Use the outbox pattern or a transaction log to publish events exactly with the state change; do not write the DB and publish separately without it.
3. Poison messages go to a DLQ with enough context to replay; review DLQs on an alarm, not by manual glance.
4. Backpressure: bound concurrency; a function that consumes faster than the dependency can absorb converts a queue into an outage.
5. Timeouts: handler timeout < upstream client timeout < platform limit, so retries are intentional and bounded.
6. Ordering assumptions must be per-key; global ordering is expensive or unavailable.

Local development and testing

• Run the same handler locally with the official emulator: SAM CLI, Functions Framework, wrangler dev, or the Azure Core Tools. Local invoke validates packaging, env, and IAM gaps early.
• Integration tests use containers: LocalStack for AWS APIs, emulators for queues/objects; testcontainers for local dependencies.
• Contract-test event payloads with schema validation in CI; schema drift is the most common production surprise.
• Keep a staging environment with the real event sources; emulators drift from real behavior (IAM, timeouts, cold starts).
• Deploy pipelines: build once as an image/zip, sign it, and promote the same artifact; see ./08-supply-chain-security.md.

Limits and quotas to design around

| Limit class | Examples | Design response | |---|---|---| | Execution | Timeout ceilings, memory caps, ephemeral disk | Long work -> jobs or containers; stream large payloads via object storage | | Payload | Request/response size, event size | Pass references (S3/queue), not blobs | | Concurrency | Account/region concurrency, per-function reserved | Request quota increases early; cap downstream with queues | | Connections | DB connection limits per instance count | Use a pooler/proxy, not one connection per concurrent invocation | | Networking | VPC, static egress, private endpoints | Plan egress before compliance requires it |

Assume every quota will be hit at the worst time; alarm on approaching limits (concurrency, queue depth, throttles) rather than discovering them during an incident.

Anti-patterns

• One function per HTTP route with business logic duplicated per function.
• Synchronous function chains (function calls function calls function) multiplying latency and failure modes.
• Nondeterministic handlers with side effects and no idempotency key.
• Infinite retries with no DLQ; a poison message loop burning budget.
• min-instances left at default for a latency-critical path, then surprise cold-start SLO misses.
• Long-lived DB connections opened per invocation without pooling.
• Storing state on local disk or in memory across invocations.
• Secrets in environment variables committed to the repo; use a secret manager and workload identity.

Checklist

• [ ] Execution model chosen against constraints, not fashion.
• [ ] Runtime and platform versions pinned and tracked for end of support.
• [ ] Cold-start budget measured at p99 and accepted or mitigated.
• [ ] Handlers idempotent, with DLQ and replay procedure.
• [ ] Concurrency bounded to protect downstream dependencies.
• [ ] Timeouts ordered: handler < upstream client < platform.
• [ ] Connection pooling for databases.
• [ ] Local emulation plus staging tests against real services.
• [ ] Artifacts signed and promoted unchanged (./08-supply-chain-security.md).
• [ ] Cost model reviewed against steady-state Kubernetes crossover (./09-cost-platform-engineering.md).

Infrastructure as Code

Scope: tooling choices, module design, state management, policy enforcement, and testing for provisioning cloud infrastructure.

Tool selection

| Tool | Language | State model | Best for | Tradeoffs | |---|---|---|---|---| | OpenTofu | HCL | State file (self-managed backends) | Teams wanting the Terraform workflow under a permissive license | Smaller provider ecosystem nuance; verify provider compatibility upstream | | Terraform | HCL | State file (self-managed backends) | Largest registry, managed Cloud/Enterprise features | Business Source License; verify terms before standardizing | | Pulumi | TS/Python/Go/C#/Java | Managed or self-managed backends | App teams owning infra in general-purpose languages; rich testing | Smaller module ecosystem; language runtime adds dependencies | | Crossplane | Kubernetes CRDs + YAML | Kubernetes etcd (no state file) | In-cluster self-service, platform APIs, drift reconciliation | Control plane is Kubernetes; providers lag cloud APIs | | Cloud-native (CloudFormation, Bicep, Deployment Manager) | JSON/YAML/DSL | Service-managed | Single-cloud shops with compliance constraints | Weak multi-cloud story; limited abstraction | | Ansible | YAML | Stateless (imperative) | Configuration of existing hosts and network devices | Not a provisioning or drift tool |

Decision guidance:

• OpenTofu is the default for new HCL projects unless a dependency requires Terraform-specific features or registry content. The two are command-compatible for the core workflow; divergence grows over time, so pick one per repository.
• Choose Pulumi when infrastructure code is written by application teams and unit testing, code review, and reuse matter more than registry breadth.
• Choose Crossplane when the platform's product is a Kubernetes API that application teams consume, and you want reconciliation and self-service without pipelines.
• Never mix tools for the same resource. Overlapping ownership causes state fights and destructive applies.

Module design

A module is a product with an interface:

• Inputs: required variables minimal and explicit; validate with validation blocks. Prefer structured objects over long flat variable lists.
• Outputs: expose only what consumers need; every output is API surface you must keep stable.
• Versioning: publish to a registry or tag in Git with semantic versions; consumers pin (version = "~> 3.0"). Unpinned module sources are a supply-chain and stability risk.
• Ownership boundary: a module provisions a coherent unit (network, database, service). A module that provisions a whole environment becomes unmaintainable.
• No provider configuration inside modules: modules declare required_providers; the root configures providers and passes aliases. Provider blocks in modules prevent aliasing and multi-region composition.
• No remote state reads across environments as the default integration path; pass values as inputs or use explicit data sources sparingly. Cross-state coupling is the hardest thing to untangle.
• Conditionals: avoid count/for_each on whole modules to toggle behavior. Instead expose two smaller modules or use for_each over maps for genuine repetition. for_each beats count because addresses stay stable when items are removed from the middle.
• Document each module with a README (purpose, inputs, outputs, examples) generated from the code where possible.

terraform {
  required_version = ">= 1.7" # OpenTofu floors differ; verify upstream
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }
  backend "s3" {
    bucket       = "org-tfstate-prod"
    key          = "network/terraform.tfstate"
    region       = "eu-central-1"
    encrypt      = true
    use_lockfile = true # modern locking; DynamoDB locking is the legacy pattern
  }
}

module "network" {
  source  = "git::https://github.com/example/tf-modules.git//network?ref=v3.2.0"
  cidr    = "10.20.0.0/16"
  azs     = ["eu-central-1a", "eu-central-1b", "eu-central-1c"]
  tags    = local.tags
}

State management

• One state per environment and per lifecycle boundary (network, data, apps). Many small states reduce blast radius and apply time; one giant state makes every change risky.
• Remote backends with encryption and locking are mandatory: object storage plus lockfile/DynamoDB-style locking, or the vendor's managed backend. Never commit state; it contains secrets and destroys your ability to plan safely.
• Access control: state read access equals secret read access. Restrict the state bucket to the pipeline identity and a small break-glass role.
• Workspaces vs directories: directories per environment are easier to reason about, review, and grant access to. CLI workspaces are acceptable for ephemeral parallel copies, not for prod/dev separation.
• Drift: run scheduled plan in CI and alert on unexpected diffs. Console/CLI changes are incidents; import them deliberately or revert them.
• Imports and moves: use import blocks and moved blocks so refactors are reviewable and don't destroy/recreate resources. Never hand-edit state except in a documented emergency, with a backup first.
• Deletion protection: enable provider-level deletion protection on databases and buckets, and use prevent_destroy for crown jewels.

Secrets and sensitive data

• Infrastructure code frequently needs secrets: database passwords, API keys, TLS material. Never put them in variables committed to Git or in state in plaintext.
• Generate secrets inside the provider/managed service where possible (RNG resources, managed secret versions), or fetch at apply time from Vault/Cloud Secret Manager via data sources.
• Mark variables and outputs sensitive; this masks CLI output but does not encrypt state. State encryption (supported natively in newer OpenTofu releases; verify upstream) or a backend with encryption at rest is still required.
• Use workload identity federation from CI to cloud (OIDC) instead of long-lived cloud keys in CI secrets. This removes the most valuable credential from your CI system.
• Rotate anything that ever appeared in a plan log, state file, or CI artifact.

Policy as code

| Layer | Tools | Enforces | |---|---|---| | Static pre-plan | Checkov, Trivy config, tflint, cdk-nag | Misconfigurations in source before apply | | Plan-time | OPA/Conftest, Sentinel (Terraform Cloud/Enterprise), Pulumi policies | Rules on planned changes and diffs | | Post-apply | Cloud native config rules, Config/Guard, Security Command Center | Drift and runtime configuration | | Admission | Kyverno/Gatekeeper in Kubernetes | Workload-level policy for in-cluster resources |

Practices:

• Every rule starts in warn/report mode, then becomes blocking after a cleanup window.
• Policies have owners and expiry for exceptions; an exception without an expiry becomes permanent.
• Test policies like code: fixtures that must pass and fixtures that must fail.
• Keep the policy set small and high-signal (tagging, public access, encryption, approved regions, approved instance classes). A wall of 300 rules gets bypassed wholesale.
• Policy must run where the decision is made: PR checks for developer feedback, CI on plan as the authoritative gate, admission for cluster resources.

Testing IaC

• Unit/static: validate, fmt -check, tflint, provider linting in pre-commit; fast and catches most syntax and style issues.
• Native testing: Terraform's test framework and OpenTofu's test command run plan/apply assertions against ephemeral resources. Prefer this for module contract tests; it is the lowest-friction path to real assertions.
• Integration: Terratest or a language-native test runner to apply a module in a sandbox account and assert behavior; always destroy afterwards.
• Ephemeral environments: per-PR stacks in a dedicated account/subscription/project with a TTL and a reaper. This is the only reliable way to test networking and IAM end to end.
• Migration tests: when refactoring, run plan against a copy of production state and require a no-op or expected diff before merging.

# tests/network.tftest.hcl (native framework, sketch)
variables { cidr = "10.99.0.0/16" }

run "creates_three_subnets" {
  command = plan
  assert {
    condition     = length(module.network.subnet_ids) == 3
    error_message = "expected one subnet per AZ"
  }
}

CI/CD for IaC

1. PR: format, validate, lint, policy, plan against the target environment; post the plan as a comment.
2. Human review focuses on the plan diff, not the HCL.
3. Merge to main: apply with a lock, using OIDC-federated short-lived credentials.
4. Prod applies require environment protection/approval; concurrent applies to the same state are serialized.
5. Nightly: drift detection plan; report and triage.
6. Keep apply output archived with the commit SHA so incidents can be correlated with changes.

Migration notes

• Terraform <-> OpenTofu: the core workflow is command-compatible for typical configurations. Before switching, verify provider versions used by your modules are available in the target registry and test plan against a copy of state. State file formats are compatible in practice, but do not round-trip a state file between tools repeatedly; pick one as the writer.
• Tool replacement (any direction): never import everything at once. Pick one low-risk stack, migrate it with import blocks plus a plan that shows no destructive changes, run it in parallel ownership-free for a release, then cut over and delete the old state. Repeat.
• Module refactors: use moved blocks in the same commit as the resource rename so reviewers see intent; a plan that shows destroy/create on a database is a stop sign.
• CloudFormation/Bicep to HCL: export existing resources, then rewrite rather than mechanically converting. Templates encode provider-specific semantics; a mechanical translation tends to reproduce those semantics in awkward HCL.

Anti-patterns

• A single monolithic state shared by all environments.
• terraform apply from laptops with personal admin credentials; no audit trail.
• Manual console changes followed by "we'll import it later" (never happens).
• -auto-approve in production without a plan review.
• Modules that embed provider configs, remote state reads, or environment names.
• Secrets passed as plain variables and printed in CI logs.
• count on lists of resources, causing asymmetric destroy/recreate.
• 300 policy rules in warn mode forever.
• Applying IaC in CI without state locking.

Checklist

• [ ] One tool per resource; ownership boundaries documented.
• [ ] Remote state with encryption and locking; access restricted.
• [ ] One state per environment/lifecycle; no cross-state reads without justification.
• [ ] Modules versioned, pinned, documented, and provider-agnostic.
• [ ] moved/import blocks used for refactors; deletion protection on crown jewels.
• [ ] OIDC federation for CI credentials; no long-lived cloud keys.
• [ ] Policy checks at pre-plan, plan, and admission layers with expiring exceptions.
• [ ] Native unit tests plus at least one integration test per critical module.
• [ ] Nightly drift detection with triage owner.
• [ ] Plan review is the required approval artifact for production changes.

Multi-Cloud

Scope: running across two or more clouds — networking, identity federation, data gravity, portability tradeoffs, and exit strategy.

Why multi-cloud at all?

| Motivation | Genuine multi-cloud? | Better alternative | |---|---|---| | Regulatory requirement for a second provider | Yes | Region/sovereign-cloud variants of one provider may satisfy this | | Resilience against provider-wide failure | Sometimes | Multi-region within one provider covers most realistic failures at lower cost | | Best-of-breed services per domain | Yes, at integration cost | One cloud plus SaaS usually wins on total cost | | Negotiation leverage | Rarely worth it alone | Real, but rarely justifies the engineering bill | | M&A brought another cloud | Yes, temporarily | Set a consolidation deadline or a clear boundary | | Avoiding lock-in for its own sake | No | Design exit options without operating two stacks |

The honest default: multi-cloud is a cost, not a feature. Adopt it for a decision that survives a five-year horizon, and write down which of the rows above is the real driver.

Portability layers

| Layer | Portable option | Lock-in risk | |---|---|---| | Compute | Kubernetes, OCI containers, serverless containers | Managed add-ons (IAM, LB, autoscaling) differ | | Networking | Gateway API, mesh, overlays | Load balancers, DNS, private connectivity are cloud-specific | | Data | PostgreSQL/MySQL, Kafka-compatible, S3-compatible APIs | Managed extensions, replicas, and IAM bindings | | Identity | OIDC, SPIFFE/SPIRE, OPA | IAM policy languages and role models | | Observability | OpenTelemetry, Prometheus remote write | Vendor agents and query languages | | IaC | OpenTofu/Terraform providers, Pulumi | Provider feature lag and defaults | | Delivery | GitOps controllers | Managed GitOps services |

Standardize on the portable layer only where the abstraction cost is low. For data and identity, the abstraction usually leaks; accepting a per-cloud adapter is often cheaper than forcing uniformity.

Networking

• Topology: hub-and-spoke per cloud, connected by a cross-cloud path. Keep the number of VPCs/VNets small; each one is a firewall policy, DNS zone, and quota surface.
• Connectivity options: provider interconnects (Direct Connect, ExpressRoute, Cloud Interconnect) for production, site-to-site VPN for low volume or backup, and SD-WAN/overlay vendors when you need a uniform policy plane. Cross-cloud traffic egresses; model the cost.
• DNS: own the public zones in one registrar/DNS provider with health-checked failover; keep private zones per cloud with clear forwarding rules. Avoid split-brain private DNS across clouds.
• CIDR planning is a first-class constraint: allocate non-overlapping supernets from a global IPAM before the second cloud exists. Retrofitting non-overlap after peering is painful.
• Service-to-service across clouds: prefer mesh/gateway federation or an API gateway as the boundary; never extend one cluster's pod network across clouds without a deliberate design.
• Egress and NAT: static egress IPs per cloud, private endpoints for cloud services, and a documented data path for every cross-cloud flow.

Reference architecture sketch:

Cloud A (primary)             Cloud B (secondary)
  hub VPC  <-> interconnect <->  hub VNet
    |                                |
  spokes (prod, nonprod)         spokes (DR, analytics)
    |                                |
  DNS zone A  <-> conditional forwarders / replicated zone  <-> DNS zone B

Identity federation

• Human identity: one IdP (Okta, Entra ID, Google Workspace) federated to every cloud via SAML/OIDC and SCIM. No local cloud users except break-glass, which is monitored and rotated.
• Workload identity: each cloud's workload identity (IAM roles for service accounts, managed identities, service account impersonation) wins locally. Across clouds, federate:
  • OIDC trust between CI and each cloud (short-lived credentials, no static keys).
  • Workload identity federation so a workload in cloud A can assume a role in cloud B without a key.
  • SPIFFE/SPIRE when you need one cryptographic identity across clusters and clouds.
• Authorization: keep a single source of truth for group membership (the IdP) and map groups to per-cloud roles. Do not maintain parallel user lists.
• Break-glass: sealed, monitored, tested. Two-person rule for its use.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "accounts.google.com" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": { "accounts.google.com:aud": "REPLACE_CLIENT_ID" },
      "StringLike": { "accounts.google.com:sub": "system:serviceaccount:prod:api" }
    }
  }]
}

Sketch only: tighten sub matching, add azp/issuer conditions, and verify the current provider docs upstream.

Data gravity

• Data is the heaviest asset: once terabytes and dependent analytics live in one cloud, compute follows the data, not the reverse.
• Costs of moving: egress fees, transfer time, dual-running during migration, and application rewrites. Estimate all four before promising a migration.
• Placement rules: keep data in the cloud where the majority of readers and writers run; replicate only derived or read-mostly datasets.
• Residency and sovereignty: map data classes to regions and clouds; some data must never leave a jurisdiction regardless of architecture elegance.
• Shared data services: a cross-cloud Kafka or object store can centralize data but adds a network dependency for every consumer. Choose it deliberately, not by default.
• Backups across clouds: cross-cloud backup copies are a strong ransomware and provider-failure control; verify restore times, encryption, and who can delete them.

Operating model and fleet management

• Treat each cloud as a region of one platform: same golden paths, same pipelines, same policy, different adapters.
• Fleet inventory: one system of record for clusters, accounts/subscriptions/projects, owners, and criticality. Without it, multi-cloud becomes ungovernable.
• Configuration consistency: reconcile the same baseline (logging, policy agents, baseline network rules) everywhere via GitOps or IaC, with per-cloud exceptions explicit.
• Observability: one pane of glass with OpenTelemetry and a shared metrics backend; per-cloud exporters normalized to common labels (cloud, region, account, service).
• Cost: one allocation model across clouds with consistent labels and unit economics; see ./09-cost-platform-engineering.md.
• On-call: one rotation that can operate every cloud. If nobody on-call can debug the secondary cloud, it is not a resilience asset; it is a liability.

Exit strategy

Exit is a design property, not a project. Maintain it continuously:

1. Classify workloads by portability: portable (containers, standard protocols), adaptable (managed service with an open equivalent), and sticky (deep provider-specific integration). Keep sticky surface area intentional and small.
2. Two-way doors: prefer services with credible open equivalents (PostgreSQL, Kafka, S3 API, Kubernetes). Price the migration before adopting the sticky version.
3. Data exportability: verify you can extract data in a standard format at a known throughput and cost. Test a sample export yearly.
4. Infrastructure parity: keep the OpenTofu/Terraform layer capable of targeting a second provider for tier-1 services; even an untested plan surfaces hidden assumptions.
5. DNS and traffic control: own your DNS and TLS material so cutover does not require the incumbent's cooperation.
6. People: cross-train at least two engineers per cloud; a cloud only one person understands is a single point of failure.
7. Decision triggers: define what would trigger an exit (price change, reliability, compliance) and the pre-agreed first steps so the decision is not made in a crisis.

Portability budget rule: spend abstraction effort only up to the expected savings of the exit option it preserves. Perfect portability is a fantasy; credible exit in weeks for tier-1 services is a reasonable target.

Common topologies

| Topology | Shape | Use when | Main risk | |---|---|---|---| | Primary + DR | One active cloud, one warm/cold standby | Provider-failure or regulatory resilience | DR rots untested; data replication lag | | Cloud-per-domain | Each cloud owns distinct capabilities (for example analytics vs serving) | Best-of-breed with clear boundaries | Cross-cloud traffic and identity sprawl | | Cloud-per-BU | Business units choose independently | Acquisition or autonomy | Duplicated platforms and skills | | Active-active across clouds | Both serve production traffic | Extreme availability requirements | Data consistency and latency usually kill it | | Cloud-per-environment | Non-prod in one, prod in another | Rare; cost arbitrage or sandboxing | Environment parity breaks in subtle ways |

Rules that apply to all of them:

• The data layer determines the real topology. Decide replication direction, consistency, and failover ownership before drawing compute.
• Failover must be exercised: run a documented drill at least twice a year with a measurable RTO/RPO result. A standby that has never taken traffic is a hypothesis, not a control.
• Keep the number of cross-cloud synchronous calls near zero; asynchronous replication and events travel better than request/response across providers.
• Define one system of record per data class. Two writable primaries for the same dataset is a conflict-resolution project disguised as an architecture.

Migration playbook

When moving a workload between clouds:

1. Inventory dependencies: managed services, private endpoints, IAM bindings, DNS names, and license/contract constraints.
2. Classify each dependency as portable, adaptable, or sticky (see the exit-strategy section), then estimate migration cost per class.
3. Build the target with IaC from day one and deploy the same artifact by digest (./08-supply-chain-security.md).
4. Run dual-write or replica-based data migration with verification jobs comparing counts and checksums.
5. Shadow traffic to the target where feasible; compare responses before shifting any users.
6. Cut over with DNS plus a rollback window; keep the source writable-but-frozen long enough to revert.
7. Decommission deliberately: delete credentials, DNS, and network paths so the old path cannot silently attract traffic.
8. Update runbooks, on-call training, and cost allocation in the same change window as the cutover.

Anti-patterns

• Multi-cloud for resume value or negotiation without a workload-level reason.
• Lowest-common-denominator services that lose the reliability and features that made each cloud useful.
• Separate tooling, pipelines, and identities per cloud with no shared golden path.
• Lifting a VM-based architecture unchanged and calling it multi-cloud.
• Assuming a second cloud gives active-active by default; the data layer usually remains single-cloud and defines the real availability.
• No CIDR plan, followed by overlapping networks that block peering.
• Cross-cloud keys in CI configs instead of federation.
• A secondary cloud that nobody tests; it silently rots until the day it is needed.

Checklist

• [ ] Written driver for multi-cloud; cost and benefit signed off.
• [ ] Global CIDR/IPAM allocation before the second cloud is built.
• [ ] Cross-cloud connectivity with documented data paths and costs.
• [ ] Single human IdP with federation; break-glass monitored and tested.
• [ ] Workload identity federation everywhere; no static cross-cloud keys.
• [ ] Data classification, residency, and cross-cloud backup verified with restore tests.
• [ ] Common baseline applied by GitOps/IaC; exceptions documented.
• [ ] One observability stack and one on-call rotation covering all clouds.
• [ ] Portability classification per workload; exit plan and triggers documented.

Supply-Chain Security

Scope: securing the path from source to running workload — provenance, signing, SBOMs, scanning, admission verification, hardening, and secrets.

Threat model

| Stage | Threats | Primary controls | |---|---|---| | Source | Compromised developer account, malicious dependency, typo-squatting | SSO+MFA, branch protection, dependency review, lockfiles | | Build | Poisoned CI, tampered build steps, credential theft | Ephemeral runners, OIDC, isolated builds, provenance | | Artifact | Registry tampering, tag mutation, unsigned images | Digests, signing, registry immutability | | Deploy | Unauthorized images, policy bypass, secret exposure | Admission verification, least privilege, workload identity | | Runtime | Vulnerable dependency exploitation, lateral movement | Scanning, patching cadence, network policy, hardening |

The goal is not zero vulnerabilities; it is that every deployed artifact can be traced to a reviewed source commit and a trusted build.

SLSA and provenance

SLSA (Supply-chain Levels for Software Artifacts) provides a common vocabulary. Use the tracks/levels model current in 2026 rather than memorized level numbers; verify the spec upstream.

Practical progression:

1. Build L1-L2: builds run in CI with a documented, versioned build definition; provenance is generated and available.
2. Build L3: hardened, ephemeral build environment; provenance is non-forgeable by the build itself and signed.
3. Distribute: artifacts are signed and consumers verify before deploy.

What to implement first, in order:

• One build definition in Git per artifact; the pipeline is code.
• Provenance generated by the CI platform or a dedicated builder (for example in-toto attestations), signed, and stored alongside the artifact.
• Digest-addressed artifacts; registries with immutability enabled so a tag cannot be re-pointed.
• Verification in admission and, where possible, at runtime.

Sigstore and cosign

• cosign signs and verifies OCI artifacts, blobs, and attestations. Keyless signing uses your CI's OIDC identity and Fulcio; Rekor provides a transparency log.
• Key-based signing still works (KMS keys) where keyless is not permitted; KMS-backed keys beat file-based keys and passphrases.
• Sign the digest, then attach SBOM and provenance attestations to the same digest.
• Verification policy must bind identity: --certificate-identity and --certificate-oidc-issuer for keyless, or the public key for key-based. Verifying a signature without checking who signed it is security theater.
• Store signatures as OCI artifacts in the registry (.sig, .att tags) so verification does not need outbound access at admission time; mirror the transparency log entries you depend on.

# Sign an image in CI (keyless; uses the workflow OIDC identity)
cosign sign --yes ghcr.io/example/api@sha256:REPLACE_DIGEST

# Verify with identity binding
cosign verify ghcr.io/example/api@sha256:REPLACE_DIGEST \
  --certificate-identity "https://github.com/example/api/.github/workflows/release.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com"

# Attach an SBOM attestation
cosign attest --yes --predicate sbom.spdx.json --type spdxjson ghcr.io/example/api@sha256:REPLACE_DIGEST

SBOM

• Generate an SBOM per artifact at build time: SPDX or CycloneDX, in a machine-readable form, stored and signed with the artifact.
• Generation tools (syft, Trivy, buildkit plugins, or CI-native generators) should enumerate OS packages, language dependencies, and licenses.
• Use the SBOM for: vulnerability scanning of shipped components, license compliance, incident response ("are we affected by CVE-X?"), and customer questionnaires.
• Keep SBOMs per digest, not per release name; rebuilds produce new SBOMs.
• VEX (Vulnerability Exploitability eXchange) lets you state "not affected" with reasoning; publish VEX for accepted risks so scanners and customers stop re-reporting them.
• SBOM quality varies; validate that the generator includes transitive dependencies and package versions, or the document is compliance theater.

Vulnerability scanning and gates

| Scope | Tools | Cadence | |---|---|---| | Dependencies (SCA) | Dependabot/Renovate, Trivy, Grype, OSV scanners | Every PR and nightly | | Container image | Trivy, Grype, cloud registry scanning | On build and continuously on running digests | | IaC and manifests | Trivy config, Checkov, kube-linter | Every PR | | Runtime | Admission-blocking of new criticals, runtime security agent | Continuous | | Secrets | gitleaks, trufflehog, provider secret scanning | Pre-commit, PR, continuous |

Gate design:

• Block on critical/high with a known fix; time-box exceptions with owner and expiry.
• Do not block on unfixable base-image noise; instead track base image age and rebuild cadence.
• Separate the "can we ship?" gate (build-time, priority-based) from the "what is running?" report (continuous).
• Rebuild and redeploy when base images or dependencies receive fixes; scanning without a rebuild path changes nothing.

Admission verification

Verify signatures and attestations before the pod is admitted:

• Kyverno verifyImages: checks signatures, keys/identities, and attestations; can also mutate the image to its digest.
• Sigstore policy-controller: ClusterImagePolicy with keyless identity rules and attestation checks.
• Ratify: verification with external policy engines and multiple verifiers.
• Admission webhooks themselves must be highly available; a failing verification policy should fail closed for security but you must operate it with redundancy and scoped rules.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-images }
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-signature
      match:
        any:
          - resources: { kinds: ["Pod"] }
      verifyImages:
        - imageReferences: ["ghcr.io/example/*"]
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/example/*"
                    issuer: "https://token.actions.githubusercontent.com"

Pair admission policy with runtime policy: image verification at admission plus continuous scanning of running images catches drift and registry-side changes.

Workload hardening

• Base images: distroless or minimal (scratch, alpine-style) with only needed packages; pin by digest. Rebuild regularly to pick up patched bases.
• Run as non-root with a fixed high UID; read-only root filesystem; drop all capabilities; allowPrivilegeEscalation: false; seccompProfile: RuntimeDefault.
• Pod Security Admission at restricted for namespace defaults; exceptions are per-workload and documented. See ./01-kubernetes.md.
• Disable service account token automounting unless needed; use projected tokens with audience and expiry.
• No hostPath, hostNetwork, or privileged containers without a documented, reviewed exception.
• Keep an SBOM-linked runtime inventory so "which workloads run log4j?" is a query, not an investigation.

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/api /api
USER 65532:65532
ENTRYPOINT ["/api"]

Secrets management

• Source of truth: a secret manager (Vault, cloud secret manager) or encrypted Git (SOPS) with a documented decision. See ./02-gitops.md for GitOps patterns.
• Delivery into workloads: External Secrets Operator or Secrets Store CSI; avoid secrets in plain Kubernetes Secrets created by hand.
• Identity: workloads authenticate to the secret store with workload identity, not static tokens.
• No secrets in environment variables where avoidable (they leak through kubectl describe, crash dumps, and child processes); prefer files with tight permissions or in-memory fetch via SDK.
• Rotation: automated, short-lived where possible (database IAM auth, OIDC tokens), scheduled otherwise. Test rotation in staging; a rotation procedure that has never run is broken.
• CI: OIDC to cloud roles, no repository secrets for cloud credentials; mask anything that might appear in logs and rotate what leaks.

Anti-patterns

• Signing images but not binding an identity in verification.
• Scanning only at build time and never rebuilding so fixes never reach production.
• Deploying mutable tags while claiming digest immutability.
• One shared signing key across all repositories, stored as a CI secret.
• Admission policy with a verification webhook on all resources and no high availability.
• Ignoring base image age; "we scan" while running a two-year-old base.
• Secrets duplicated into Kubernetes Secrets and never rotated.
• SBOMs generated but never used in an actual incident or customer response.

Checklist

• [ ] Build pipeline defined in Git with ephemeral runners and OIDC.
• [ ] Provenance generated and signed for every artifact.
• [ ] Artifacts signed (keyless or KMS) with identity-bound verification.
• [ ] Registry immutability enabled; deploys reference digests.
• [ ] SBOM produced, stored, and signed per digest; VEX for accepted risks.
• [ ] Scanning gates on PR, build, and running workloads; rebuild cadence defined.
• [ ] Admission verification enforced with HA webhooks and scoped rules.
• [ ] Hardening baseline applied cluster-wide with documented exceptions.
• [ ] Secrets from a central store with workload identity and tested rotation.
• [ ] Incident query "which workloads contain component X" answerable in minutes.

Cost and Platform Engineering

Scope: FinOps practice, cost visibility and allocation, rightsizing, and internal developer platforms with golden paths.

FinOps operating model

Three phases, run continuously rather than as a project:

| Phase | Question | Activities | Artifacts | |---|---|---|---| | Inform | What are we spending and why? | Tagging, allocation, showback, unit economics | Dashboards, reports by team/service | | Optimize | Where can we spend less for the same outcome? | Rightsizing, commitments, storage lifecycle, waste removal | Savings backlog | | Operate | How do we keep it that way? | Budgets, anomaly detection, policy, reviews, incentives | Guardrails, reviews, targets |

Personas: engineering owns usage decisions, finance owns accounting and forecast, product owns unit economics targets, platform owns tooling and guardrails. No single team "does FinOps" alone.

Rules that make the program real:

• Cost is discussed in engineering terms: per service, per environment, per customer, per request.
• Every optimization has an owner and an SLO impact check before and after.
• Savings targets are set as a percentage of addressable spend, not a round number invented by finance.
• Cost anomalies page or ticket the owning team, not the platform team.

Visibility and allocation

• Native sources of truth: AWS Cost and Usage Report (CUR) or Cost Explorer, GCP billing export to BigQuery, Azure Cost Management exports. Build one normalized table with account/subscription/project, service, region, and your label dimensions.
• Kubernetes: OpenCost and Kubecost translate pod usage into cost by namespace, workload, and label using cloud pricing plus overhead split. Run one of them per cluster and federate to the central report.
• Label schema (enforce with policy; see ./06-iac.md):

| Label | Example values | Required | |---|---|---| | app / service | checkout, search | Yes | | owner or team | team-payments | Yes | | env | dev, staging, prod | Yes | | cost-center | cc-1234 | Yes | | component | api, worker, cache | Recommended | | managed-by | gitops, tofu | Recommended |

• Showback first: publish accurate per-team cost for one quarter before charging anyone. Chargeback without trust produces arguments, not savings.
• Shared costs (ingress, DNS, observability, CI, control planes) have a written allocation rule: proportional to usage where measurable, equal split otherwise, reviewed quarterly.
• Unit economics: cost per order, cost per active tenant, cost per 1k requests. Track the ratio month over month; absolute spend growth means little without the denominator.
• Budgets and forecasts: per team and per environment, with alerts at 80%/100% of monthly budget and a forecast that flags end-of-month overruns by mid-month.
• Anomaly detection: daily cost-change alerts above a threshold against the trailing baseline, routed to the owner label.

Rightsizing

• The core problem: requests are set once at launch and never revisited, so the scheduler reserves capacity nobody uses. Target 60-80% average utilization on the reserved footprint for stateless services.
• Process: collect 2-4 weeks of usage, set requests near p95 (plus margin), set limits to protect the node, re-measure monthly. Do it service by service; batch it with active owners.
• Tools: VPA in recommendation mode (never auto in production until proven), Goldilocks dashboards, cloud native rightsizing recommendations, and kubectl top for spot checks.
• Honor the QoS consequence: moving from Guaranteed to Burstable saves capacity but changes eviction order; critical services keep Guaranteed or explicit priorities.
• Instance and architecture mix: latest generation and arm64 often give 20-40% better price/performance; validate with the same load test before switching. See ./04-sre.md for the load-test discipline.
• Spot/preemptible for stateless, fault-tolerant work: workers, batch, CI, some ingress. Never for stateful or single-replica critical services. Handle interruption notices and diversify instance types.
• Storage lifecycle: tier to infrequent/cold/archive by access pattern, snapshot before delete, set retention policies, and delete orphaned volumes and snapshots monthly.
• Commitments (savings plans, reserved instances, committed use discounts): cover the stable baseline, not the peaks. Model before buying; unused commitments are worse than on-demand.

Waste checklist to sweep monthly:

• Idle load balancers, NAT gateways in unused AZs, unattached IPs and volumes.
• Non-production environments running 24/7 and oversized dev databases.
• Orphaned snapshots, old AMIs/images, log buckets without lifecycle rules.
• Kubernetes namespaces with zero traffic but running replicas.
• Duplicated observability data (high-cardinality metrics nobody queries) and verbose logs shipped twice.
• Container registries, CI caches, and artifact stores growing without retention policies.
• Cross-AZ and cross-region data transfer from chatty designs (./03-mesh-networking.md locality routing helps).

Internal developer platforms (IDPs)

An IDP is a product, not a portal. Its job: reduce cognitive load and make the paved path the fastest path.

| Component | Examples | Purpose | |---|---|---| | Service catalog | Backstage, Port, OpsLevel | Inventory, ownership, scorecards | | Golden paths / templates | Scaffolding with CI, CD, observability pre-wired | New service to production in hours, not weeks | | Self-service infra | Crossplane, platform APIs, Terraform modules, environment requests | No tickets for routine provisioning | | Delivery | GitOps controllers, pipelines | Consistent promotion (see ./02-gitops.md) | | Guardrails | Policy engines, quotas, admission | Safe defaults and enforcement | | Docs | Reference + how-to by task | Reduce tribal knowledge |

Design principles:

• Treat developers as customers: interview them, measure time-to-first-deploy, adoption, and satisfaction.
• Golden paths are opinionated and maintained; they are not the only path, but leaving them must cost something explicit.
• Every template ships production-ready: health endpoints, metrics, structured logs, tracing, dashboards, alerts, security baseline, ownership metadata.
• Self-service with guardrails beats approvals. Approvals are reserved for irreversible or expensive actions.
• Measure adoption and outcome: percentage of services on the path, deployment frequency, change failure rate, MTTR, and time from commit to production (the DORA set).
• Deprecate templates explicitly with migration guides; unmaintained scaffolding is worse than none.
• Scorecards drive hygiene (has SLO, has owner, has on-call, no critical CVEs, uses golden CI) with visibility rather than punishment first.

Platform team operating rhythm

| Cadence | Activity | Output | |---|---|---| | Weekly | Cost anomaly review, savings backlog grooming | Assigned actions | | Monthly | Rightsizing sweep, waste sweep, adoption review | Changes applied, metrics | | Quarterly | Commitments review, allocation rule review, roadmap with users | Forecasts, priorities | | Quarterly | DR/restore and exit-path checks (./07-multicloud.md) | Tested procedures |

Cost-aware architecture patterns

• Prefer scale-to-zero for non-production and bursty workloads; a staging environment that sleeps outside business hours is often the single largest quick win.
• Locality: keep chatty services in one zone, use topology-aware routing, and avoid cross-region synchronous calls. Cross-AZ traffic is cheap relative to latency and failure risk; cross-region is neither.
• Caching and CDN change the unit economics more than instance tuning: a well-cached read path can cut origin compute by an order of magnitude. Always measure hit ratio as a cost metric.
• Data: compress at rest, choose the right storage class, partition to prune scans, and avoid duplicating datasets across layers (raw, staged, curated) without lifecycle rules.
• Async where the user does not wait: queues absorb bursts and let workers run on cheaper interruptible capacity.
• Batching beats per-request overhead for analytics and ML jobs; a nightly batch on spot can cost a fraction of a real-time cluster.
• Serverless vs Kubernetes crossover: model break-even at sustained utilization; below it, serverless usually wins once you price the platform team. See ./05-serverless.md and ./04-sre.md for headroom costs.

Platform metrics that matter

• Adoption: share of services created from golden paths; share of services with complete ownership metadata.
• Flow: time to first deploy for a new service; lead time from merge to production; deployment frequency; change failure rate; MTTR. Publish per team where it helps, never as a leaderboard for blame.
• Reliability of the platform itself: internal SLOs for CI, artifact registry, GitOps reconciliation, and self-service APIs. When the platform breaks, every product team stops.
• Cost of the platform: platform spend per developer or per service, reported like any other unit cost.
• Support load: tickets per 100 developers; recurring tickets are missing golden-path features, not user error.

Procurement and commitment hygiene

• Own a rolling 12-month view: current on-demand baseline, current commitment coverage, expiring commitments, and forecast growth. Review monthly with finance, not yearly.
• Commitment levers: compute savings plans / committed use discounts for steady baseline, storage reservations for stable capacity, and negotiated private pricing at meaningful spend. Each has different flexibility when the architecture changes.
• Coverage target: commit to roughly 60-80% of the stable baseline. Higher coverage maximizes discount but removes the ability to absorb architecture changes; lower coverage leaves money on the table.
• Term length: shorter terms (1 year) preserve flexibility for fast-moving workloads; longer terms (3 years) only for capacity you are confident about, such as core networking and data platforms.
• Spot strategy: diversify instance families and zones, handle interruption notices, and keep a small on-demand floor so interruption does not cascade. Treat spot savings as a bonus that funds reliability.
• Marketplace and private offers can beat list prices for committed spend, but read the true-up and minimum-spend terms; an over-committed private deal can cost more than on-demand.
• Serverless and managed services often do not benefit from commitments; check the pricing model before including them in a coverage model.
• Track realized savings from every optimization and commitment in the same report; projected savings that never appear in the billing data are fiction.

Anti-patterns

• Chargeback before showback, or allocation by spreadsheet politics.
• Chasing a percentage cost cut by degrading reliability without checking SLO impact.
• Every workload at Guaranteed QoS "to be safe", locking 3x the capacity needed.
• Buying commitments for peak usage, then paying for idle reservation.
• Building a portal no developer asked for; measuring success by click counts.
• Golden paths that stop at "service created" without day-2 operations.
• Cost tooling only visible to finance, never to the engineers who choose instance sizes.
• Treating the platform team as a ticket queue instead of a product team.

Checklist

• [ ] One normalized cost dataset across clouds, refreshed daily.
• [ ] Required labels enforced by policy; untagged spend is < 5% and tracked.
• [ ] Showback published monthly per team/service with shared-cost rules documented.
• [ ] Unit economics tracked for top revenue or usage paths.
• [ ] Budgets and 80%/100% alerts per team and environment; anomalies routed to owners.
• [ ] Rightsizing sweep monthly; VPA recommendations reviewed.
• [ ] Spot/arm64/latest-gen adoption evaluated with load tests.
• [ ] Storage lifecycle and orphan cleanup automated.
• [ ] Commitments cover the stable baseline with a review cadence.
• [ ] Golden path exists, is adopted, and reports DORA metrics.
• [ ] Self-service provisioning with guardrails; approvals only for irreversible actions.