Containers

Scope: image build strategy (multi-stage, base image selection, caching), image hygiene and runtime behavior, buildpacks, multi-arch publishing, and supply-chain scanning and attestation.

Build Model

| Approach | Use when | Trade-offs | |---|---|---| | Multi-stage Dockerfile | You need control over layers, tools, and filesystem contents | You own base-image patching and build logic | | Buildpacks | A standard runtime is sufficient and you want fewer hand-written Dockerfiles | Less control over exact layers; builder updates are a dependency | | Monolithic Dockerfile | Prototypes only | Huge images, leaked build tools, slow rebuilds | | VM image build | Non-container targets | Out of scope for this reference; same principles on immutability and provenance |

The deliverable of the build is an immutable image identified by digest. Every deploy references the digest, never a floating tag. Tags (latest, branch names, short SHAs) are discovery aids, not identity.

Multi-Stage Builds

The pattern: one stage builds, a second stage carries only the runtime artifact.

# syntax=docker/dockerfile:1
FROM node:22-bookworm AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM deps AS build
COPY . .
RUN npm run build && npm prune --omit=dev

FROM gcr.io/distroless/nodejs22-debian12 AS runtime
WORKDIR /app
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
USER 65532:65532
CMD ["dist/server.js"]

Rules that make this work:

• Pin the builder and runtime to the same language version line; mismatched builder/runtime is a class of production-only bugs.
• Copy dependency manifests first, install, then copy source. This is what makes layer caching effective.
• Prune development dependencies in the build stage; never install build tooling in the runtime stage.
• Copy artifacts explicitly (COPY --from=build) — never COPY . . into the runtime stage.
• Set WORKDIR, USER, and the entrypoint explicitly. The runtime image should be boring and declarative.
• Use --mount=type=cache for package-manager caches. Cache mounts speed rebuilds without adding layers.

Base Image Selection

| Base | Size / attack surface | Use when | Watch out | |---|---|---|---| | scratch | Minimal | Static binaries (Go, Rust) | No shell, no CA certs, no tzdata unless copied | | Distroless | Small | JVM, Node, Python, Go runtimes | Debugging requires an ephemeral debug container | | Alpine | Small | You need a shell and package manager | musl vs glibc differences; native modules must build for musl | | slim Debian/Ubuntu | Medium | Native dependencies that assume glibc | More packages to patch; keep image updated | | Full distro | Large | Legacy workloads only | Treat size and CVE count as debt |

• Pin to a digest for reproducible builds and depend on an automated updater (for example Renovate or Dependabot) to move the digest. Human-managed digest pins rot silently.
• Prefer a distroless or scratch runtime for network services; the absence of a shell removes a whole class of post-exploitation behavior.
• If you need curl for a health check or debugging, you picked the wrong health-check mechanism, not the wrong base. See ./07-observability-rollback.md.
• Multi-stage and minimal bases do not remove the need to patch: rebuild regularly even when your code did not change.

Layer Caching

Layer cache is the difference between a one-minute and a twenty-minute pipeline.

• Order instructions from least to most frequently changing.
• Use .dockerignore aggressively: .git, node_modules, virtualenvs, build output, test fixtures, local env files. A leaked .env in the build context is a secret incident.
• Prefer COPY of exact files over globs when the glob is unstable.
• Use RUN --mount=type=cache rather than committing package caches into layers.
• Use --mount=type=bind for files needed only during a RUN step, not in the final layer.
• Cache invalidation is transitive: any change to an earlier layer invalidates all later ones. Put lockfiles before source.
• In CI, share cache between runs with registry-backed or provider-native layer caching. See ./02-ci.md.
• Verify cache effectiveness with --progress=plain and build logs; a cache that never hits is worse than no cache because it hides the cost in upload time.

Image Hygiene

| Check | Rule | |---|---| | User | Run as a non-root numeric UID; set USER in the Dockerfile and enforce via admission policy | | Filesystem | Read-only root filesystem where possible; mount only needed writable volumes | | Capabilities | Drop all, add back only what is required | | Secrets | None in any layer, including deleted files (they persist in earlier layers) | | Signals | Use exec-form ENTRYPOINT; ensure PID 1 handles SIGTERM for graceful shutdown | | Labels | org.opencontainers.image.source, revision, version, created for provenance | | Ports | Declare EXPOSE for documentation, but bind explicitly at runtime | | Health | No HEALTHCHECK curl hacks in minimal images; orchestration probes own health | | Time | Copy CA certificates and tzdata when TLS or local time matters |

• Never pass secrets as ARG or ENV; build args are visible in image history.
• Squashing layers hides history but does not remove leaked content from a shared registry — treat any leaked secret as compromised and rotate it.
• Keep images single-purpose. A container that runs an init system, a cron daemon, and the app is three failure modes.

Buildpacks

Buildpacks detect the language, build the app, and produce an OCI image without a Dockerfile. Choose them when:

• The runtime is standard and the team should not hand-maintain base images.
• You want automatic rebasing onto patched runtimes without rebuilding application layers.
• You need consistent images across many services with a platform team owning the builder.

Watch:

• The builder image is a supply-chain dependency: pin and update it deliberately.
• Build-time configuration is environment-variable driven; document and validate it.
• Custom system dependencies may still require a Dockerfile-based escape hatch.
• Verify what the buildpack injects by inspecting the final image; trust but verify.

Multi-Architecture

• Build for the architectures you deploy (linux/amd64, linux/arm64); do not publish what you cannot test.
• Native builders are faster and more reliable than QEMU emulation for cross-arch builds; use them when the CI provider offers them.
• For compiled languages, cross-compilation in a single builder is usually faster than emulation. Set the target platform explicitly.
• Publish multi-arch as an image index/manifest list with docker buildx imagetools or equivalent; consumers pull by digest. The digest of the index differs from the digest of each platform image — know which one your deploy pins.
• Test the arm64 variant at least once per release; most multi-arch failures are architecture-specific native dependencies.

Scanning, SBOM, and Provenance

Supply-chain artifacts are part of the deliverable, not a reporting afterthought.

• Generate an SBOM at build time (SPDX or CycloneDX) and attach it to the image or store it alongside the digest.
• Scan the image, not just the source: base-image and OS-package CVEs dominate findings.
• Gate CI on severity policy with documented exceptions and expiry dates; a gate that is always bypassed is worse than no gate.
• Sign images and attestations with a keyless signature bound to the pipeline identity (for example cosign with OIDC), and verify signatures in admission control.
• Record build provenance (builder identity, source commit, build parameters) in an attestation. Consumers verify before deploy.
• Re-scan registry contents continuously, not only at build time; new CVEs appear for old images.
• Keep a policy for how fast critical images are rebuilt and redeployed. An unredeployed patch is not a fix.

Reproducibility

• Pin toolchain and dependency versions; floating package ranges make byte-identical rebuilds impossible.
• Set deterministic build metadata where the ecosystem supports it; avoid embedding timestamps and hostnames in artifacts.
• Reproducible builds are a goal, not a precondition; the practical minimum is "same source + same lockfile + same builder version produces functionally equivalent artifacts".
• Keep the build definition in the repository next to the code it builds.

Anti-Patterns

• latest in any deployment manifest, Helm values file, or compose file.
• Installing build tools in the runtime image "just in case".
• Mounting the Docker socket into CI containers; it is root on the host.
• Baking environment-specific config or secrets into images.
• Rebuilding per environment, producing different artifacts under the same version.
• Ignoring base-image updates until a CVE forces an emergency rebuild.
• Running as root because the app "needs" it; fix permissions at build time.
• Huge build contexts because .dockerignore was never written.
• Scanning only the application dependencies and calling the image "clean".
• Trusting tags in production admission; verify digests and signatures.

Checklist

• [ ] Multi-stage build; runtime stage contains only the artifact and its runtime.
• [ ] Dependency install separated from source copy; cache mounts used.
• [ ] .dockerignore excludes VCS data, local state, and secrets.
• [ ] Base pinned by digest; automated update path exists.
• [ ] Non-root USER, exec-form entrypoint, SIGTERM handled.
• [ ] No secrets, no build toolchain, no package-manager caches in the final image.
• [ ] OCI labels for source, revision, and version present.
• [ ] Multi-arch index published and the deployed variant tested.
• [ ] SBOM generated and attached; scan gate enforced with exception expiry.
• [ ] Image signed; provenance attested; admission verifies.
• [ ] Deploy manifests reference digests, not tags.

CI

Scope: pipeline design and structure, GitHub Actions patterns, caching, matrix builds, required checks, artifact signing, and pipeline security with OIDC and least privilege.

Pipeline Design

A pipeline is a production system. It has an owner, an SLO for duration and reliability, and an error budget.

Principles:

• Fast feedback first: lint, format, type-check, and unit tests run in the first minutes. Slow suites run in parallel or after.
• Fail fast, but only on signal. Blocking on a flaky test trains people to rerun rather than read.
• Deterministic: same commit plus same inputs produces the same result. Pin tool versions and container images.
• Idempotent: re-running a job neither duplicates side effects nor corrupts state.
• Cached, not stale: caches are keyed on lockfiles and tool versions, with restore keys as a fallback.
• Observable: duration, queue time, failure rate, and flake rate are tracked per job. See ./07-observability-rollback.md.
• Hermetic where possible: no implicit network dependency on a developer's machine or a mutable external service.

Canonical stage order:

| Stage | Purpose | Gate | |---|---|---| | Static analysis | Format, lint, types, dependency policy | Blocking | | Unit tests | Fast, isolated, parallelized by shard | Blocking | | Build | Produce artifact once, sign, attest, SBOM | Blocking | | Integration tests | Real dependencies via containers or ephemeral infra | Blocking | | Security scans | SAST, dependency CVEs, secret scan, IaC policy | Blocking with documented exceptions | | Publish | Push image and versioned packages to registry | Blocking | | Deploy staging | Automated promotion of the same digest | Automatic | | Verify staging | Smoke, contract, migration checks | Automatic | | Deploy production | Progressive delivery per ./03-cd-strategies.md | Policy-driven | | Post-deploy | Release markers, SLO watch, auto-rollback | Automatic |

• Build artifacts exactly once. Everything downstream consumes the same digest. See ./01-containers.md.
• Keep deploy jobs thin: they should apply a known artifact, not compile code. Rollback then has no build step to fail.
• Separate orchestration from execution: a small number of reusable entry workflows call versioned reusable pipelines per service.

GitHub Actions Patterns

Workflow Structure

name: ci
on:
  pull_request:
  push:
    branches: [main]
permissions:
  contents: read
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true
jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@<full-sha> # pinned
        with: { persist-credentials: false }
      - uses: actions/setup-node@<full-sha>
        with: { node-version-file: .nvmrc, cache: npm }
      - run: npm ci
      - run: npm test

Rules:

• Pin third-party actions to a full commit SHA, not a tag. Dependabot or Renovate can bump SHAs; tags can be retargeted.
• Set permissions explicitly at workflow or job level. Default to contents: read; add only what a job needs.
• Use concurrency with cancel-in-progress on PRs to eliminate redundant runs.
• Use timeout-minutes on every job; a hung job holds a runner and a queue slot.
• persist-credentials: false on checkout unless the job pushes back.
• Prefer pull_request over pull_request_target. If you need pull_request_target, never run untrusted code with a write token or secrets.
• Reusable workflows (workflow_call) for shared pipelines; composite actions for shared step sequences. Version reusable workflows with a tag or SHA like any dependency.
• Use environments with protection rules for production deploys; required reviewers are a control, not a substitute for automated gates.
• Self-hosted runners are production infrastructure: isolate untrusted jobs, patch them, and never give them static cloud credentials.

Caching

| Cache | Key | Notes | |---|---|---| | Package manager | Lockfile hash plus OS and runtime version | Provider setup-* caches are usually best | | Compiled output | Source tree hash, tool version | Only safe when determinism is proven | | Container layers | Base digest plus lockfile | Registry-backed BuildKit cache | | Test results | Test file set plus commit range | Enables selective reruns | | Tool binaries | Version manifest hash | Avoids re-downloading CLIs every run |

• Scope caches: PR caches should not be writable from forks. Never cache credentials or .env files.
• Treat cache as an optimization only; a cold run must still pass. Flaky failures that vanish on cache hit usually mean undeclared inputs.
• Rotate caches when changing compilers or package managers, or restore stale artifacts.

Matrix Builds

• Use matrices for OS, runtime-version, and shard combinations; keep the full matrix meaningful, not decorative.
• fail-fast: false when you need the complete failure picture; fail-fast: true when a red leg makes the rest pointless.
• Keep matrix length bounded; a 40-leg matrix costs more than it informs. Use include/exclude to encode exceptions explicitly.
• Shard tests by file count or historical duration; unbalanced shards negate the parallelism.
• Aggregate results in a single required check so branch protection does not require knowing the matrix shape.

Required Checks and Merge Flow

• Branch protection requires the aggregate gate, not individual jobs, so adding a job does not break merges.
• Enable merge queue for busy repositories; it tests the actual merge result rather than the PR head.
• Required checks must be hermetic and reasonably fast (a target under ten minutes is healthy; verify expectations with the team).
• A required check that is skipped for some PRs (docs-only) should still report success explicitly, or protection will hang.
• Protect the workflow files themselves with CODEOWNERS; a PR that edits its own gate needs review.

Pipeline Security

Pipeline credentials are production credentials.

• Use OIDC federation to cloud providers and registries; delete long-lived access keys from CI. Bind the trust policy tightly: repository, ref or environment, and workflow.
• Scope tokens to the job and the environment: a test job must not assume the production deploy role.
• Never print secrets; mask values and review logs for accidental echoes. set -x in a job with secrets is an incident waiting for a trigger.
• Treat all PR input as hostile: branch names, commit messages, file contents. Avoid interpolating ${{ github.event.* }} directly into run: blocks; pass through env: and quote.
• Pin dependencies of the pipeline itself (actions, container images, CLIs) by digest or SHA.
• Sign artifacts and attest provenance in CI, and verify in deployment. See ./01-containers.md and ./09-release-management.md.
• Use ephemeral runners where feasible; persistent runners accumulate state and credentials.
• Audit who can edit workflows, and log secret access and OIDC token issuance.

jobs:
  deploy:
    permissions:
      id-token: write
      contents: read
    environment: production
    steps:
      - uses: aws-actions/configure-aws-credentials@<full-sha>
        with:
          role-to-assume: arn:aws:iam::<account>:role/ci-deploy
          aws-region: <region>

Test Selection

• Run everything on main and on merge queue candidates.
• On PRs, select affected tests by changed paths in a monorepo. Selection must fail open: if the tooling cannot determine impact, run the full set. See ../../monorepo/SKILL.md if the monorepo pack is installed.
• Track flake rate per test; quarantine with an owner and expiry, never silently retry forever.
• Upload test reports and coverage as artifacts; downstream tools should not re-run tests to render results.

Pipeline Observability

• Emit deploy events and build IDs into telemetry so a release can be traced from commit to production. See ./07-observability-rollback.md.
• Dashboard queue time, duration, success rate, and cost per pipeline.
• Alert on sustained failure of the default-branch pipeline: a broken main blocks everyone.
• Record who or what triggered a deploy (actor, workflow run, event) for audit. See ./09-release-management.md.

Anti-Patterns

• Secrets stored as repository or organization variables when OIDC is available.
• permissions: write-all or leaving default broad permissions.
• Unpinned @main actions in a workflow with deploy privileges.
• Long-lived service-account JSON keys in CI secrets.
• Rebuilding the artifact in the deploy job.
• Retrying failed jobs automatically to hide flakes.
• Matrix explosion for combinations nobody supports.
• Running privileged PR code on self-hosted runners with cloud access.
• Required checks that only run on some paths without explicit success reporting.
• Caches holding credentials or written by untrusted forks.

Checklist

• [ ] One artifact built once, signed, attested, and promoted by digest.
• [ ] Fast static and unit stage; slow suites parallelized or deferred.
• [ ] Actions pinned by SHA; permissions minimal per job.
• [ ] Concurrency and timeouts set on every workflow.
• [ ] Caches keyed correctly and never hold secrets.
• [ ] Matrix bounded, sharded, and aggregated into one required check.
• [ ] OIDC used for cloud and registry access; trust policy scoped to repo/ref/environment.
• [ ] Deploy jobs thin, environment-protected, and auditable.
• [ ] Flake tracking and quarantine process in place.
• [ ] Pipeline duration, failure, and cost visible on a dashboard.

CD Strategies

Scope: rolling, blue-green, canary, and progressive delivery with Argo Rollouts or Flagger, metric-driven analysis gates, and rollback mechanics.

Strategy Selection

| Strategy | Mechanics | Rollback | Cost | Use when | |---|---|---|---|---| | Recreate | Stop old, start new | Slow (full restart) | Lowest | Internal tools, batch workers with no availability requirement | | Rolling | Replace instances in batches within a surge budget | Medium (roll back batch by batch) | Low | Default Kubernetes strategy for stateless services | | Blue-green | Two full fleets, switch traffic atomically | Fast (switch back) | High (double capacity during cutover) | Instant cutover and instant rollback valued; schema changes are compatible | | Canary | Send a slice of traffic to the new version, expand on health | Fast (shift traffic back) | Low to medium | User-facing production; the default when automated analysis exists | | Progressive delivery | Canary or blue-green plus automated metric analysis and promotion steps | Fast and automated | Low plus observability investment | Mature platform with SLOs and dashboards | | A/B test | Route by user attributes, often with flags | Instant via flag | Low | Product experiments; not a substitute for canary safety |

Rules that apply to all strategies:

• Deploy by digest, never by tag. See ./01-containers.md.
• Readiness gates determine when an instance receives traffic; liveness gates restart it. Misconfigured probes cause more deploy incidents than bad code.
• maxSurge and maxUnavailable encode your risk tolerance; for critical services prefer surge over unavailability. Never set maxUnavailable to 100% and call it rolling.
• PodDisruptionBudgets protect against voluntary disruption, not deploy bugs.
• Graceful shutdown: handle SIGTERM, stop accepting new work, drain in-flight requests within the termination grace period. See ./07-observability-rollback.md.
• One deploy in flight per service per environment. Serialize with concurrency controls or an admission mechanism.

Rolling Updates

Kubernetes default. Mechanics:

• New ReplicaSet scales up by maxSurge; old scales down by maxUnavailable.
• The rollout completes when all new pods are ready and old pods are gone.
• progressDeadlineSeconds fails a stuck rollout; set it deliberately (large services need more than the default).

Requirements for zero downtime:

• Multiple replicas (at least two, three for meaningful availability).
• Correct readiness probes that fail before the process stops serving.
• preStop hooks or application drain logic when connection draining exceeds the platform default.
• Long-lived connections (websockets, gRPC streams) need connection draining or client-side retry with backoff.
• Stateful sets require ordered rollout awareness; database compatibility per ./06-database-migrations.md.

Rollback: kubectl rollout undo or redeploy the previous revision. Fast only if the replica history and image are still available; keep the previous revisions.

Blue-Green

Mechanics:

• Two identical fleets (blue is live, green is idle or scaled to a smoke level).
• Deploy to idle, run smoke and migration checks, then switch the router/load balancer/service selector.
• Keep the old fleet warm for the rollback window, then scale it down.

Considerations:

• Database migrations must be compatible with both colors simultaneously (expand-contract, ./06-database-migrations.md).
• Session state must be externalized; sticky sessions to a color defeat the cutover.
• DNS-based cutover is not instant (TTL and resolver caching); prefer a load balancer or service mesh switch.
• Double capacity is the cost; for large fleets, partial blue-green (a small green pool plus canary) is often better.
• Timer-based rollback reminder: forgetting to scale down the old fleet is a standing cost bug.

Canary

Mechanics:

• Deploy the new version to a small replica set or a subset of pods.
• Shift a small percentage of traffic, observe, then increase in steps (for example 1 to 5 to 25 to 50 to 100 percent).
• Abort on failure, shifting traffic back to the stable version.

Traffic splitting options:

| Mechanism | Granularity | Notes | |---|---|---| | Replica ratio (stable/canary) | Pod-level | Simple; crude at low traffic | | Ingress/load balancer weights | Request-level | Needs weighted routing support | | Service mesh | Request-level, header-aware | Strongest control; extra platform dependency | | Gateway API or provider-native | Request-level | Managed option; verify feature parity upstream |

Analysis inputs:

• Error rate and latency percentiles (p95/p99), by version.
• Saturation: CPU, memory, queue depth.
• Custom business KPIs where they are stable enough to gate on.
• Logs and traces for errors unique to the canary.

Requirements:

• Traffic volume sufficient for the analysis window to be statistically meaningful. A 1% canary on 10 requests per minute proves nothing.
• Version label on metrics and traces so the analysis can compare canary against baseline. See ./07-observability-rollback.md.
• Predefined thresholds and abort behavior, not human eyeballs on a dashboard.

Progressive Delivery Platforms

Argo Rollouts and Flagger are the common Kubernetes implementations. They provide:

• Rollout CRDs replacing Deployments, with steps, pauses, and traffic routing.
• Metric analysis (Prometheus, Datadog, New Relic, cloud monitoring) with success thresholds, failure limits, and intervals.
• Traffic shaping via ingress controllers or service meshes.
• Automated abort and rollback to the last healthy revision.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api
spec:
  replicas: 6
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - analysis:
            templates:
              - templateName: error-rate
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100

Analysis template essentials:

• Query compares canary to stable over the same window.
• Success condition and failure limit are explicit; one failing interval is not always a failure.
• startingStep or a warm-up delay avoids judging a cold canary.
• The template is versioned with the service and tested against historical data before first use.

Operational rules:

• Do not combine two progressive controllers on the same workload.
• Keep the analysis query cheap; a heavy query on the critical path delays deploys and can itself cause timeouts.
• Alert on rollouts that abort and on analysis that cannot evaluate (missing metrics is a failure mode, not a pass).
• Practice the abort path; an untested rollback is a hope, not a plan.

Automated Analysis

Designing gates:

| Signal | Good gate | Bad gate | |---|---|---| | Error rate | Ratio vs stable, sustained over the window | Absolute count with no baseline | | Latency | p95/p99 ratio canary vs stable | Average latency (hides tails) | | Saturation | Resource usage against limits | Raw CPU without limits context | | Business KPI | Conversion delta with sufficient traffic | Revenue per minute at low volume | | Availability | Synthetic probe success by version | One ping to the service root |

• Define the failure action before enabling the gate: abort, pause for human decision, or roll back automatically.
• Separate "analysis failed" from "analysis unavailable". The former blocks, the latter should page rather than silently promote.
• Keep the analysis window long enough to cover slow failures (leaks, connection exhaustion) and short enough to keep deploys moving. Ten to thirty minutes per step is a common range; verify against your traffic profile.

Rollback Mechanics

Rollback must be a routine operation, not an incident.

• Artifact immutability: the previous digest still exists in the registry and is deployable. Retention policies must protect the last N releases.
• Configuration rollback: config and secrets are versioned too; a bad config change needs the same rollback path.
• Database compatibility: the old code must run against the new schema for the rollback window. This is the hard constraint that expand-contract exists to satisfy (see ./06-database-migrations.md).
• Feature flags can decouple: disable the flag to remove the new behavior while leaving the artifact deployable. See ./08-feature-flags.md.
• One-command rollback with a tested runbook and an owner. Time it in a game day.
• Post-rollback: keep the evidence (analysis results, logs, traces) and open a follow-up instead of immediately redeploying the same artifact.

Anti-Patterns

• Deploying to 100% of instances at once because "it is just a small change".
• Canary without metrics comparison, judged by watching a dashboard manually.
• Rollback plans that require rebuilding the old artifact.
• Schema changes that break the N-1 version, making rollback impossible.
• Probes that always pass, or that call deep dependencies and can cascade failures.
• Multiple deploys racing for the same service.
• Canarying a change that only manifests at full traffic or after hours, with a ten-minute gate.
• Ignoring maxUnavailable defaults that can take down the whole service.
• Auto-promotion with no pause and no analysis, i.e. a rolling update with extra YAML.

Checklist

• [ ] Strategy chosen deliberately per service and recorded.
• [ ] Deploys reference immutable digests; previous releases retained.
• [ ] Readiness, liveness, and startup probes verified against real behavior.
• [ ] Graceful shutdown and connection draining tested.
• [ ] Traffic-splitting mechanism selected and capacity validated.
• [ ] Analysis queries compare canary to stable and are cheap to run.
• [ ] Failure action (abort, pause, rollback) is explicit and automated.
• [ ] Rollback rehearsed within the last quarter and timed.
• [ ] Schema changes compatible with rolling back one version.
• [ ] One deploy in flight per service; concurrency controlled.

IaC and Environments

Scope: environment topology and isolation, infrastructure as code structure per environment, ephemeral preview environments, state and drift management, and artifact promotion between environments.

Environment Topology

| Environment | Lifetime | Data | Purpose | Access | |---|---|---|---|---| | Local | Developer session | Synthetic | Fast iteration, unit and component tests | Developer | | Ephemeral preview | Per pull request, TTL hours to days | Synthetic or anonymized subset | Review UI and integration behavior of a change | Team plus reviewers | | Integration/QA | Long-lived | Synthetic, refreshed | Cross-service integration, migration rehearsal | Team | | Staging | Long-lived | Production-shaped, anonymized | Release rehearsal, performance smoke, final verification | Team, restrict writes | | Production | Permanent | Real | Serve users | Break-glass only, audited | | Sandbox | Long-lived | Synthetic | Experiments, demos, training | Broad, isolated from prod |

Rules:

• Every environment has an owner, a budget, and a purpose. Unused environments are cost and drift sources; reap them.
• Isolate by account/project/subscription or at minimum by network boundary and IAM. Production credentials must not be assumable from lower environments.
• Lower environments must not contain real customer PII. Use anonymized or synthetic data generation, or masked subsets with a documented policy.
• Keep environment parity where it matters: runtime version, deployment mechanism, IaC modules, and secret injection. Parity in every detail is a myth; parity in production-critical axes is mandatory.
• Fewer environments beats more. Each environment multiplies cost, drift, and maintenance. A team with six half-maintained environments would usually be better with three healthy ones.

IaC per Environment

Structure options:

| Pattern | Description | Use when | |---|---|---| | One root, workspaces | Same code, state per workspace | Small, uniform stacks | | Directory per environment | envs/dev, envs/stage, envs/prod composed from shared modules | Default for production systems; explicit and greppable | | Stack per component plus composition | Networking, data, app as separate states, wired by outputs | Large systems; blast-radius control | | Environment-agnostic modules plus tfvars | Modules in modules/, values per environment | Recommended baseline for any multi-env setup | | Generated composition (Terragrunt-style) | DRY wrappers over many states | Many similar stacks; adds a tool dependency |

Baseline recommendations:

• Shared modules own resource shape; environment directories own inputs (sizes, counts, domains, regions).
• Never fork a module per environment. Forks drift and diverge.
• Separate state per environment and per blast-radius boundary (network, data, cluster, app). A single state for everything makes one mistake global.
• Remote state with locking and versioning; state files contain secrets and must be access-controlled and encrypted at rest. See ./05-secrets-config.md.
• Reconcile providers and modules with an automated dependency updater and test upgrades in the lowest environment first.
• Changes to shared modules require a plan in every environment before merge; show the diffs in review.
• Policy as code (for example OPA/Rego, Cedar-style policies, or provider guardrails) enforces tagging, encryption, and public-access rules in CI and at deploy time.
• Tag everything with environment, service, owner, and cost center so cost and access reviews work.

State, Locking, and Drift

• Remote backend with locking is non-negotiable for shared environments. Local state is for throwaway experiments only.
• State is not a cache of truth; the provider is. Reconcile regularly.
• Drift detection runs on a schedule: plan (read-only) against each environment and alert on non-empty diffs. Some tools call this continuous reconciliation; verify feature names upstream.
• Drift triage: classify as expected (provider-managed fields, autoscaling), urgent (security-relevant changes), or accidental (console edits). Only accidental and urgent drift is auto-remediable.
• Guardrails: disable or restrict console mutation of IaC-managed resources where the provider allows it.
• Break-glass changes made in the console must be imported back into code or reverted within a fixed window, with a ticket. Untracked manual changes are how environments become snowflakes.
• Refresh before apply; a stale state plus a fast-moving provider is a recipe for destroying and recreating resources.

Ephemeral Preview Environments

Purpose: prove a change works end to end in a realistic topology before merge, and let reviewers see the UI.

Design:

• One environment per pull request, named by repository plus PR number, created on opened/synchronize, destroyed on closed.
• Provision from the same IaC modules and the same container image digest built by CI (./02-ci.md). A preview that builds different code proves nothing.
• TTL and reaping are mandatory. Orphaned previews are the most common cost incident in preview programs.
• Data: seed from a fixture set or an anonymized snapshot. Never a live replica of production.
• Budget caps and quotas per preview (namespace resource quotas, cloud budgets/alerts).
• Networking: unique hostname per PR (for example pr-123.preview.example.com) with automated DNS and TLS.
• Databases: use a per-preview schema or database on a shared instance for small teams; per-preview instances where isolation matters. Handle migrations automatically, including rollback of preview data.
• Secrets: scoped, non-production credentials or short-lived dynamic credentials (./05-secrets-config.md).
• Cleanup verification: a nightly job that finds and destroys unclaimed previews and reports leaks.
• Cost visibility: tag preview resources so their spend is attributable and capped.

Anti-patterns:

• Preview environments that require manual setup steps from the reviewer.
• Previews with production credentials or production data.
• No TTL, no owner, no quota.
• Previews that only build the frontend and stub the backend; they miss integration failures.
• One shared "preview" environment for all PRs; concurrent changes collide.

Promotion

The rule: build once, promote the artifact, change only configuration between environments.

Promotion models:

| Model | Description | Use when | |---|---|---| | Push-based | CI holds environment credentials and deploys sequentially | Small teams, simple topologies | | Pull-based / GitOps | A controller in-cluster reconciles from a config repository | Auditability and drift control required | | Hybrid | CI publishes an image and updates a version manifest; GitOps reconciles | Common production pattern |

• The promotion event is a version bump in an environment manifest or a parameter, not a rebuild.
• Config differences between environments are explicit inputs (instance sizes, replica counts, feature defaults), reviewed like code.
• Approval gates sit on the promotion step, not on the build. See ./09-release-management.md.
• Secrets are never in the promotion artifact; they are resolved by the target environment at runtime.
• Record what was promoted, by whom, when, and from which commit; this is the release ledger (./09-release-management.md).
• Rollback promotes the previous digest; the previous environment manifest is retained.
• Prefer promoting through a fixed order, with staging verification results attached to the production promotion.

Environment Configuration

• Configuration is data, not code branches. Environment checks in application logic (if env == "prod") should collapse into configuration (if features.x). See ./05-secrets-config.md.
• Feature defaults differ per environment deliberately: risky features default off in production and on in staging.
• Environment-specific endpoints (queue URLs, bucket names) are injected; no environment names in code.
• Keep a machine-readable inventory of environments and their properties; humans and automation both need it.

Naming and Inventory

• Fix a naming scheme before the second environment exists: <org>-<service>-<env>-<region>-<resource> is a workable default. Names appear in DNS, IAM policies, dashboards, and cost reports; renaming later is a migration.
• Environment names are stable identifiers: use production/staging/preview-<pr>, not prod2 or new-prod. Aliases age badly.
• Maintain a machine-readable inventory of environments (name, account, region, owner, tier, cost center) and generate documentation from it. Stale wiki pages are worse than no pages.
• Tag/label every resource consistently; policy-as-code rejects untagged resources.
• Keep a single glossary for environment tiers and their guarantees so that "staging" means the same thing to every team.

Cost Governance

• Give each environment a budget and an alert; preview and sandbox environments are the usual offenders.
• Right-size lower environments deliberately: they should be small, but large enough to surface performance pathologies where that is the purpose of the environment.
• Schedule non-production environments off outside working hours where the workload allows; keep staging data-safe but warm enough to be usable.
• Attribute cost by environment, service, and team so optimization work has an owner.
• Review environment spend quarterly; an environment nobody can justify is a candidate for deletion, not for downsizing.

Migration Notes

• Moving from console-managed infrastructure: import resources into IaC in a branch, prove a no-op plan, then enable drift detection. Do not start with drift auto-remediation; you will fight import gaps for weeks.
• Moving from one giant state to per-component states: use state moves and split incrementally, environment by environment, lowest risk first.
• Moving from copy-pasted environment directories to modules: extract one resource type at a time, verify no-op plans in every environment, then continue.
• Moving preview environments from a shared instance to per-PR: start with a TTL and a quota on the shared path before adding isolation, so the first failure mode is bounded.
• Moving from push-based to GitOps promotion: run both paths read-only in parallel until the controller's reconciliation matches the current state, then cut writes over.

Anti-Patterns

• Click-ops changes reconciled "later" that never get reconciled.
• One Terraform state for all environments and services.
• Copy-pasted environment directories that have silently diverged.
• Preview environments with no TTL or budget.
• Production data copied to lower environments without masking.
• Rebuilding artifacts per environment under the same version.
• Secrets stored in IaC variables or state without an encryption and access plan.
• Environment-specific branches (release/staging) instead of environment configuration.
• Promoting by "deploy the same branch again" rather than the same digest.
• Treating drift alerts as noise and muting them.

Checklist

• [ ] Every environment has an owner, purpose, budget, and reaping policy.
• [ ] Isolation boundary between production and non-production is enforced by identity, not convention.
• [ ] Modules shared; environment differences are inputs only.
• [ ] State is remote, encrypted, locked, and partitioned by blast radius.
• [ ] Scheduled drift detection with triage and a break-glass reconciliation process.
• [ ] Preview environments are per-PR, digest-accurate, TTL-bound, and budget-capped.
• [ ] Lower environments use synthetic or anonymized data only.
• [ ] Promotion moves the same artifact by digest with config as the only delta.
• [ ] Promotion events are recorded with actor, artifact, and source commit.
• [ ] Policy as code enforces tagging, encryption, and exposure rules.

Secrets and Config

Scope: externalized configuration, secret manager selection and injection patterns, workload-identity authentication, rotation, 12-factor configuration, and secret scanning.

Configuration Model

Treat configuration as a typed, validated interface of the service.

• Sources in precedence order, lowest to highest: built-in defaults, config file, environment variables, command-line flags. Document and test the precedence.
• Validate all required configuration at startup and fail fast with a clear message naming the missing key. A service that starts with defaults for a required value is a time bomb.
• Emit effective configuration (with secret values redacted) at startup or on a debug endpoint to make environment differences visible.
• Version the configuration schema alongside the code; renaming a key is a breaking change for deployments.
• 12-factor: store config in the environment (or a mounted source), not in the codebase. Environment variables are the portable interface; files are acceptable when the runtime prefers them.
• Separate configuration (non-sensitive, reviewable) from secrets (sensitive, access-controlled, audited). They have different lifecycles.

Typed config example:

const schema = z.object({
  PORT: z.coerce.number().int().positive().default(8080),
  DATABASE_URL: z.string().url(),
  FEATURE_NEW_BILLING: z.coerce.boolean().default(false),
  LOG_LEVEL: z.enum(["debug", "info", "warn", "error"]).default("info"),
});
const config = schema.parse(process.env);

• Defaults are for development only where possible; production should be explicit. A validation error beats a silent default in production.
• Never log full environment dumps; they leak secrets and tokens.

Secret Manager Selection

| Store | Strengths | Weaknesses | Use when | |---|---|---|---| | Cloud secret manager | Identity-integrated, managed rotation hooks, audit logs | Cloud-specific APIs and pricing | Default for cloud-native workloads | | Vault-compatible | Dynamic credentials, leases, transit encryption, multi-cloud | You operate it; unsealing and upgrades are real work | Dynamic secrets or multi-cloud requirements | | SOPS-encrypted files | GitOps-friendly, no runtime service, simple | Key distribution and rotation are manual-ish; no dynamic secrets | Small teams, Kubernetes GitOps, few secrets | | Sealed secrets | Cluster-scoped decryption, GitOps-native | Cluster-bound keys; controller is a trust anchor | Single-cluster Kubernetes without an external manager | | Cloud KMS plus custom tooling | Maximum control | High maintenance, easy to get wrong | Rare; usually a sign to use a manager | | Plain CI variables / repo files | Convenient | No rotation, broad exposure, hard to audit | Development-only placeholders, never production |

Selection rules:

• One system of record per environment. Duplicated secrets in two stores guarantee divergence.
• Every secret has an owner, a purpose, and a rotation schedule recorded in an inventory.
• Access is via identity (workload identity, IAM role, Kubernetes service account), and the trust policy is scoped to the workload, not the cluster.
• Audit reads; alert on unusual access patterns and on access from unexpected identities.

Injection Patterns

| Pattern | Mechanics | Notes | |---|---|---| | Runtime SDK fetch | App calls the manager at startup | Works anywhere; app depends on the manager library; cache briefly | | Init container / entrypoint fetch | Fetch, write to tmpfs, exec app | Language-agnostic; beware process environment exposure | | CSI secret store driver | Mount secrets as files in the pod | Files update on rotation; app must re-read or watch | | Operator sync to Kubernetes Secret | Controller materializes a Secret from the manager | Convenient; the Kubernetes Secret is now a copy to protect and refresh | | Sidecar/agent injection | Proxy or agent delivers secrets | Adds a component; strong when paired with dynamic credentials | | Encrypted files at deploy time | Decrypt with a key from KMS or an agent | GitOps-friendly; key management is the critical path |

Guidance:

• Prefer file mounts over environment variables for secrets: env vars leak into crash dumps, child processes, debug endpoints, and /proc.
• Do not bake secrets into container images. Build args and layer history are readable. See ./01-containers.md.
• Do not pass secrets through CI job outputs or logs; mask and avoid echoing.
• Inject the narrowest possible credential. A service that needs read access to one bucket should not get a database admin password.
• Cache secret values in memory with a TTL rather than fetching per request. Handle refresh failures without crashing.
• Treat the manager as a dependency: define behavior for startup when it is briefly unavailable (retry with bounded backoff, fail closed on missing required secrets).

Workload Identity and OIDC

Long-lived static credentials in CI or workloads are the primary cause of cloud credential leaks.

• CI assumes a cloud role via OIDC with a trust policy restricting repository, branch or environment, and workflow. See ./02-ci.md.
• Workloads use the platform identity: IAM roles for service accounts, workload identity federation, or managed identities. No key files on disk.
• Kubernetes: bind a service account to a cloud identity, and scope the identity per workload, not per namespace where possible.
• Database access uses short-lived credentials or IAM database authentication instead of long-lived passwords where the engine supports it.
• Tokens are audience-restricted and short-lived; a token minted for staging must not be accepted by production.
• Prefer mTLS plus identity documents over shared tokens for service-to-service authentication.

Rotation

• Automate rotation with a manager; manual rotation at scale does not happen.
• Rotation must be zero-downtime: support two valid credentials during the overlap window (dual-write or dual-read), then retire the old one.
• Pattern for static secrets: create new version, deploy consumers to read latest, verify usage of the new version, then revoke the old version.
• Prefer dynamic short-lived credentials where available (database users, cloud STS, broker credentials); rotation becomes lease renewal.
• Rotation triggers: schedule, personnel change, suspected exposure, and provider-recommended windows.
• Test rotation in staging on the same automation that production uses. An untested rotation script is not a plan.
• Rotation of certificates and signing keys follows the same discipline, with trust-store overlap.

Secret Scanning and Leak Response

• Scan in three places: pre-commit hooks (fast feedback), CI on every push, and historical/full-history scans. See ./02-ci.md.
• Enable provider push protection where available; it stops known-pattern secrets before they enter history.
• Scanning finds known formats; also scan for entropy and for internal patterns (connection strings, private keys).
• Treat any secret that entered git history, logs, or an image layer as compromised even if the commit was reverted. Reverting does not un-leak.
• Leak response runbook: revoke and rotate first, then assess blast radius from audit logs, then remove from history if required by policy, then add a detector to prevent recurrence.
• Prefer short-lived credentials so a leak has a short useful life; that is the only durable mitigation.

Kubernetes and Cloud Specifics

• Kubernetes Secrets are base64, not encryption: enable encryption at rest, restrict get/list on secrets with RBAC, and prefer a CSI driver or external manager over native Secrets where possible.
• Avoid mounting the same Secret into every container in a pod; sidecars inherit it. Scope service accounts and mounts per container where the platform permits.
• Use projected service account tokens with explicit audiences and short lifetimes for workload identity; do not reuse the default token.
• Cloud secret managers should be reached through the workload's identity, and the resource policy should name the identity, not the cluster or namespace broadly.
• Keep an escape hatch documented for the day the manager is unavailable: break-glass credentials stored offline, with dual control and automatic expiry, plus an alert if used.
• Consider regional failover for secret retrieval if the service is multi-region; an in-region outage should not prevent new instances from starting.

Multi-Replica and Multi-Tenant Considerations

• Secrets are fetched per process, not per request; refresh is asynchronous. Stagger refresh across replicas to avoid a thundering herd on the manager.
• Watch for rate limits: hundreds of pods fetching secrets at startup can exhaust manager quotas. Use caching, retries with jitter, and a shared agent where available.
• Tenant isolation in a shared cluster: a per-tenant secret path plus per-tenant identity; never a shared secret with tenant selection at the application layer when the security boundary matters.
• Config for multiple tenants should be validated the same as any other input; tenant-provided configuration is untrusted input.
• Rotation in a multi-tenant system is per tenant and scheduled; a global credential shared across tenants cannot be rotated safely and should be split first.

Audit and Detection

• Log every secret read and write with the requesting identity, resource, and decision; retain long enough to investigate an incident that is discovered late.
• Alert on reads from unexpected identities, from unexpected network locations, or at unusual rates; a single stolen credential often shows up as a burst of reads.
• Watch for secrets appearing in new places: image layers, CI artifacts, log streams, and issue trackers. Scanning coverage should grow with each incident or near miss.
• Correlate secret access with deploy events; a read by a service that was not deployed is worth investigating.
• Review access grants on a schedule and remove unused identities and permissions.

Migration Notes

• Moving from env-var secrets to file mounts: run both paths behind a config flag, verify the file path is read on refresh, then remove the env path. Remember child processes and crash dumps.
• Moving from CI-stored cloud keys to OIDC: create the trust policy first, prove it in a non-production role, then delete the static keys and monitor for failures.
• Moving from a shared database user to per-service users: create and validate the new grants in shadow mode, switch one service at a time, then revoke the shared user.
• Moving from plain Kubernetes Secrets to an external manager: sync first (operator), then migrate mounts to CSI, then remove the synced Secrets.
• Moving from manual rotation to automated: run the new automation in dry-run through two scheduled cycles before it performs the first real rotation.

Anti-Patterns

• Secrets in .env files committed to the repository, even in a private repo.
• One shared "app" credential across services and environments.
• Production credentials available to CI jobs that run untrusted PR code.
• Secrets in Helm values, Kubernetes manifests, or Terraform variables without encryption.
• Rotation performed by replacing a value in two systems by hand.
• Secrets in URLs or query strings, which end up in logs and traces.
• Logging configuration values without redaction.
• Dynamic secrets never revoked; lease sprawl.
• Relying on obscurity (private repo, internal network) as a control.
• A secret manager deployed but bypassed by "temporary" env vars.

Checklist

• [ ] All required config typed and validated at startup; failures are explicit.
• [ ] Precedence rules documented and tested.
• [ ] One secret store per environment, with an inventory of owner and rotation.
• [ ] Identity-based access only; no long-lived cloud or database credentials in CI or workloads.
• [ ] Injection uses file mounts or a CSI/agent pattern; no secrets in images or env where avoidable.
• [ ] Secrets cached with TTL; manager outage behavior is defined.
• [ ] Rotation automated, dual-credential, and rehearsed.
• [ ] Scanning in pre-commit, CI, and full history; push protection enabled.
• [ ] Leak runbook tested, with revocation before remediation.
• [ ] Audit logging on secret reads, with alerts for anomalies.

Database Migrations

Scope: zero-downtime schema evolution with expand-contract, backward-compatible changes, online DDL, batched backfills, verification, and rollback strategy.

The Compatibility Constraint

During any deploy, old and new application versions run simultaneously (rolling, canary, blue-green), and a rollback may return to the previous version. Therefore:

• Every schema state must be readable and writable by both the current and the previous application version.
• Migrations run before application rollout, and they must complete without assuming exclusive access to tables.
• Rollback of the application must not require a schema rollback. If it does, you have a maintenance window, not a zero-downtime deploy.

This is what expand-contract (also called parallel change) exists to guarantee.

Expand-Contract

Four phases, typically spread across two or three releases:

| Phase | Action | Application state | Reversible | |---|---|---|---| | Expand | Add new structure alongside the old; keep both consistent | Old version ignores it | Yes (drop new structure if unused) | | Dual-write / migrate | New version writes both; readers move to the new structure | Both versions compatible | Yes, with care | | Switch | All readers and writers use the new structure | Old code still compatible via old path | Yes | | Contract | Remove the old structure | Only new version remains | Hard; do not need to |

Worked example: rename users.name to users.full_name.

1. Expand: add full_name (nullable). Keep name untouched.
2. Dual-write: release app writing both columns and reading name. Backfill full_name from name in batches.
3. Switch: release app writing full_name and reading full_name; keep a write-through to name for rollback safety.
4. Contract: after the rollback window, drop name and stop writing it.

• Each phase is independently deployable and independently safe to abandon.
• Never combine expand and contract in one release. The gap is the safety margin.
• Record the target state and the cleanup ticket at expand time; contract work that is never scheduled becomes permanent debt.

Backward-Compatible Change Catalog

| Change | Safe without expand-contract | Pattern when not safe | |---|---|---| | Add nullable column | Yes | Keep default handling in the application | | Add table | Yes | None | | Add non-null column with volatile default | No | Add nullable, backfill, then add constraint; avoid table rewrites on large tables | | Add NOT NULL constraint | No | Validate existing rows first, then constrain in a later release | | Drop column or table | No | Contract only after all readers and writers migrated; verify with query logs | | Rename column | No | Expand-contract (add, dual-write, switch, drop) | | Change column type | No | New column, backfill with conversion, switch, drop | | Widen a varchar | Usually platform-dependent | Verify engine behavior; some widenings rewrite the table | | Add index | Depends | Use concurrent/online index build on hot tables; verify lock behavior upstream | | Drop index | Usually | Check query plans before dropping | | Add foreign key | No | Add NOT VALID, validate separately where the engine supports it | | Change enum values | Usually safe for additions | Removals and renames require expand-contract | | Reorder columns | No | Cosmetic; do it during a contract step if at all |

Rules:

• Prefer additive changes; they are almost always backward compatible.
• A NOT NULL with no default is the classic breaking change: old code inserts NULL and fails.
• Renames are dangerous because they break old code silently at runtime, not at deploy time.
• Test every migration against the previous application version, not just the new one. A migration test suite that applies the schema and runs the old code paths catches most breakage.

Migration Tooling

| Tool class | Examples | Notes | |---|---|---| | Versioned SQL migration runner | Flyway, Liquibase, golang-migrate, sqitch | Deterministic, language-agnostic; own the SQL | | Framework migrations | Rails, Django, Alembic, Prisma, Entity Framework | Convenient; review generated SQL for lock behavior | | Online schema change | gh-ost, pt-online-schema-change, provider-native online DDL | For large MySQL-family tables; copy is external to the transaction | | Declarative schema diffing | Atlas-style, provider tools | Diff-based; audit generated plans carefully | | Kubernetes Jobs / init hooks | Framework-specific | Ensure single execution and observability; avoid racing replicas |

Rules:

• One runner per service, one source of truth for migrations, checksummed and immutable once applied.
• Migrations are ordered, idempotent where possible, and recorded in a migration history table.
• Never edit an applied migration; add a new one.
• Run migrations as a dedicated job with its own identity, not from every application replica.
• Keep migration runtime visible: a migration that runs for an hour inside a deploy pipeline needs its own monitoring.
• Test migrations against a production-shaped dataset in staging; a migration that takes one second on an empty table can lock for minutes on a large one.

Online DDL and Locks

• DDL often takes an exclusive lock. Even "instant" operations matter when a long-running transaction holds a conflicting lock.
• Set a short lock_timeout (or engine equivalent) on DDL so the migration fails fast rather than blocking all queries behind it. Retry with backoff.
• Set statement_timeout for the migration session to bound runaway DDL.
• For large tables on engines without online DDL, use a shadow-copy tool; it copies and swaps, with throttling and pause controls.
• Index builds: use concurrent/online variants on production tables; monitor replication lag for replicas.
• Avoid running DDL during peak traffic; schedule migrations to start at low load and allow them to continue.
• Watch for lock queues: a blocked DDL can block every subsequent query on the table. Abort rather than wait.

Backfills

Backfills are data migrations and follow different rules from schema changes.

• Separate the backfill from the schema change: schema in one deploy, data in a background job.
• Batch by primary key or a stable order; never UPDATE ... WHERE ... over an unbounded table in one statement.
• Throttle: sleep between batches and monitor replica lag and primary load. Rate-limit by rows per second, not by arbitrary sleeps, and make it configurable.
• Idempotent and resumable: track progress, and restarting must not corrupt or double-apply. Use WHERE new_col IS NULL style predicates and a cursor.
• Backfill during dual-write so new writes are never lost; the backfill only handles historical rows.
• Validate with counts and checksums (or sampled comparisons) before switching readers.
• Long backfills can run across releases; deploy new code that tolerates partially filled columns (null handling) until the backfill completes.

Verification

| Check | Method | |---|---| | Schema state | Migration history table plus a schema diff against the expected definition | | Data completeness | Row counts and null counts for backfilled columns | | Data correctness | Checksums or sampled row-by-row comparison between old and new structures | | Application compatibility | Run the previous version's test suite against the new schema | | Performance | Query plans on hot queries using the new columns or indexes | | Replication health | Lag and error monitoring during DDL and backfill | | Rollback readiness | Confirm the previous version runs against the current schema |

• Verification is automated and blocks the promotion step. Manual spot checks are for exploration, not gating.
• Keep a read-only replica or snapshot of the pre-migration state for the rollback window when the platform supports it cheaply.

Rollback Strategy

• Rollback of code is preferred over rollback of schema. Design migrations so the previous code keeps working (that is the whole point of expand-contract).
• Prefer forward fixes over down migrations. Down migrations are often untested, destructive, and slower than the incident they are meant to resolve.
• If a migration must be reversible, write and test the down path in the same PR, and run it in staging.
• Irreversible operations (dropping a column, changing data in place) are contract-phase operations: run them after the rollback window has passed.
• Keep drops reversible by delaying them: rename to a quarantine name, keep for a defined period, then drop. Some teams keep a table backup for the retention window.
• Backfill errors: stop the job, fix, and resume from the cursor; never restart from zero on a live system without cause.
• Document, per migration, whether it is reversible, how long the rollback window is, and what the recovery action is.

Anti-Patterns

• Adding a NOT NULL column without default in one step on a large table.
• Dropping or renaming a column in the same release that stops using it.
• UPDATE without batching on a production table.
• Running DDL and backfill in the same transaction or the same deploy step.
• Down migrations that delete data to "restore" the previous shape.
• Long-running DDL with no lock timeout, blocking the entire service.
• Assuming the ORM generates safe DDL; ORMs frequently do not.
• Testing migrations only on an empty schema.
• One team member's laptop as the only place a manual data fix is recorded.
• Contract-phase cleanup with no owner, left forever.

Checklist

• [ ] Every change classified against the compatibility catalog.
• [ ] Expand and contract in separate releases with an owner for contract work.
• [ ] Migration tested against previous and new application versions.
• [ ] Lock and statement timeouts set; online DDL used for hot tables.
• [ ] Backfill separate, batched, throttled, idempotent, and resumable.
• [ ] Verification automated and gating: counts, checksums, compatibility, plans.
• [ ] Rollback window defined; previous code proven against new schema.
• [ ] Down path documented or explicitly waived as forward-fix-only.
• [ ] Migration runtime and replica lag monitored during execution.
• [ ] Cleanup ticket created at expand time and scheduled.

Observability and Rollback

Scope: release markers, health checks, SLO-based deploy gating, deploy dashboards, rollback automation, and disaster-recovery drills.

Release Markers

A release marker is a timestamped annotation in telemetry that says "version X was deployed at T to environment E". Without it, correlating a behavior change to a deploy is guesswork.

Emit into telemetry:

| Field | Example | Purpose | |---|---|---| | Version | semver or CalVer | Human-facing identity | | Commit | full SHA | Source of truth | | Artifact digest | image or package digest | What is actually running | | Deploy ID | pipeline run ID | Links to the pipeline and approval | | Deployed at | ISO timestamp | Overlay on dashboards | | Deployed by | actor or automation identity | Audit |

• Emit as a deployment event to the telemetry backend, as an annotation on dashboards, and as a resource attribute on traces/metrics/logs where practical.
• Kubernetes: annotate Deployments/pods, and let the collector or Operator enrich telemetry with the version. See the observability pack if installed.
• Make "which version is serving this request" answerable after the fact from a trace or log line.
• Keep a releases ledger queryable across environments (./09-release-management.md).
• Alert on version skew across replicas beyond a healthy rollout window; a partially rolled deployment is itself a signal.

Health Checks

| Probe | Question | Failure action | Must not | |---|---|---|---| | Startup | Has the process finished initializing? | Keep waiting, then restart | Run forever; block readiness of dependents | | Readiness | Can this instance serve traffic now? | Remove from load balancing | Depend on flaky deep dependencies | | Liveness | Is the process wedged beyond recovery? | Restart container | Be a cascading failure vector | | Synthetic | Can a user achieve a critical journey? | Page and possibly roll back | Replace unit and integration tests |

Rules:

• Readiness reflects local ability to serve; deep dependency checks belong in monitoring, not in the probe that removes capacity.
• Liveness must be conservative: an application under load responding slowly is alive. Aggressive liveness probes turn a latency spike into a restart loop.
• Shutdown: on SIGTERM, fail readiness first, drain, then exit within the grace period. See ./03-cd-strategies.md.
• Have a dedicated health endpoint that does not require authentication but does not leak internals, and is excluded from access logs' noise and from trace sampling budgets.
• Version the health contract: /healthz returns build metadata carefully (no secrets, no environment dumps).
• Probe timeouts and thresholds are tuned per service and tested; copy-pasted defaults are a common incident cause.

SLO-Based Deploy Gating

Deploy gating compares the candidate version against the stable baseline, not against an absolute ideal.

Inputs:

• Golden signals per version: error rate, latency percentiles, saturation, throughput.
• Traces with version attributes to localize new errors.
• Business KPIs where they are stable and high-volume enough.
• Log-derived error classes unique to the candidate.

Gate design:

| Gate | Baseline | Threshold style | Action | |---|---|---|---| | Error rate | Stable version, same window | Ratio-based, e.g. candidate no worse than baseline plus a margin | Abort or roll back | | Latency p95/p99 | Stable version, same window | Relative increase bound | Abort after sustained breach | | Saturation | Capacity limits | CPU/memory/queue headroom floor | Pause and inspect | | Availability | Synthetic probes | Success ratio floor | Immediate rollback | | Error budget | SLO target | Burn-rate based | Stop deploys when the budget is exhausted |

• Prefer burn-rate signals from SLOs where they exist: a deploy that consumes budget fast is failing, even if no alert has fired. See the observability pack for SLI/SLO mechanics.
• Require enough traffic before trusting a comparison. Document the minimum sample size; below it, pause and extend or fall back to manual review.
• Distinguish candidate-specific failure from environment-wide failure: if the whole cluster is sick, rolling back the deploy may not help. Check the baseline in the same window.
• Automate the abort. A gate that only notifies a human is a dashboard, not a gate.
• Treat "analysis could not run" as a failed gate by default; missing data during a deploy is itself suspicious.

Deploy Dashboards

A deploy dashboard answers: what changed, what is happening, should I continue or abort.

Panels:

• Current rollout state per environment: version, digest, progress, paused steps, analysis results.
• Golden signals split by version or revision, with the release marker drawn in.
• Error classes new in the candidate versus baseline.
• Saturation and queue depth for the affected service and its direct dependencies.
• SLO burn rate and remaining error budget.
• Links: pipeline run, release notes, runbook, rollback action.

Rules:

• Dashboards are code, versioned and provisioned automatically; no hand-built panels that drift from reality. See the observability pack for dashboard-as-code patterns.
• Keep one screen: if the deploy dashboard requires scrolling and tab switching, it will not be used during an incident.
• Every panel should answer a go/no-go question; decorative panels cost attention.
• Include the rollback link prominently and make sure it works from the dashboard.

Rollback

Rollback is the primary recovery mechanism and must be cheaper than debugging forward during an incident.

Triggers:

• Automated gate failure during progressive delivery (./03-cd-strategies.md).
• SLO burn or alert that starts within the deploy correlation window.
• Human judgment: rising error reports, support tickets, or a visible defect.

Mechanics:

• Redeploy the previous artifact digest with the previous configuration. No rebuild, no pipeline beyond the deploy job.
• For blue-green, switch traffic back. For canary, shift weight to stable. For rolling, redeploy the previous revision.
• Feature flags provide a faster, narrower lever for the offending behavior. See ./08-feature-flags.md.
• Database compatibility keeps rollback safe; if it does not exist, the rollback is a forward fix (./06-database-migrations.md).
• Configuration and secret rollback: version them with the release so config can be reverted atomically.

Runbook requirements:

• One-command or one-click rollback with documented preconditions.
• Named owner and escalation path.
• Estimated time to restore (measured, not guessed).
• Post-rollback checks: error rate returns to baseline, queues drain, data integrity verified.
• Evidence capture before recovery where safe: logs, traces, analysis output. Do not destroy the failing deployment until evidence is collected (or replicate it in a preserved environment).

Practice:

• Game day: perform a rollback in production-like conditions at least quarterly; time it and fix the friction found.
• Roll back one change, not everything: if three changes are in flight, identification matters more than speed.
• After rollback, the artifact is not automatically banned; fix, test against the failure mode, and redeploy with the marker in place.

Disaster Recovery

Deployment resilience and data resilience are different problems; both need drills.

• Define RTO (time to restore) and RPO (data loss tolerance) per service; derive backup frequency and restore tooling from them.
• Backups are unverified until restored: run scheduled restore tests into an isolated environment, and record results.
• Test regional or zone failover where the architecture claims multi-region.
• Exercise dependency loss: what happens when the secret manager, registry, DNS, or the primary database is unavailable at deploy time? Cache artifacts and credentials for just long enough.
• Document degraded modes: which features are disabled, what is still correct, and how users are informed.
• Keep the registry and artifact store as part of the DR plan; without the image, rollback is impossible.
• Drill outcomes become backlog items with owners; an untracked drill finding is theater.

Anti-Patterns

• No release markers; version inferred from deploy time approximations.
• Liveness probes that restart pods under load, causing an outage spiral.
• Deep dependency checks in readiness probes, so a downstream blip removes all capacity.
• Manual canary analysis by refreshing a dashboard.
• Rollback plans that require a rebuild or a schema reversal.
• Backups never restored until the real incident.
• Dashboards built by hand and never reviewed.
• Alerts with no owner and no runbook link.
• Deploying during an active incident without freezing other pipelines.
• Post-rollback immediate redeploy of the same broken artifact.

Checklist

• [ ] Release markers emitted to telemetry and dashboards with version and digest.
• [ ] "Which version served this request" answerable from logs or traces.
• [ ] Startup, readiness, liveness, and synthetic probes are distinct and tuned.
• [ ] Graceful shutdown drains readiness before exit.
• [ ] Deploy gates compare candidate versus baseline with defined thresholds and automated abort.
• [ ] Missing analysis data fails the gate by default.
• [ ] Deploy dashboard is one screen, version-split, and provisioned as code.
• [ ] Rollback is one command, rehearsed, and timed within the last quarter.
• [ ] Database and config compatibility preserved for the rollback window.
• [ ] Backups restored in a drill; RTO/RPO documented and met.

Feature Flags

Scope: flag categories and lifetimes, targeting and progressive exposure, evaluation architecture, lifecycle governance and cleanup, and testing flagged code.

Flag Categories

| Category | Purpose | Lifetime | Owner | Default when provider fails | |---|---|---|---|---| | Release | Hide incomplete or risky functionality during rollout | Days to weeks | Delivery team | Off | | Ops / kill switch | Disable a feature or dependency under stress | Permanent, exercised periodically | On-call | Fail-safe for the system, not for the feature | | Experiment | Compare variants on a metric | Weeks (duration of the experiment) | Product plus data | Control variant | | Permission / entitlement | Gate by plan, role, or contract | Long-lived | Product or billing | Plan-appropriate | | Ops tuning (migration/backfill control) | Slow or pause background work | Weeks | Data or platform team | Safe path |

• Classify every flag at creation. The category determines its lifetime, test requirements, and cleanup process.
• Release flags are temporary scaffolding. Kill switches and entitlements are configuration and are governed like code.
• Different categories belong in different namespaces or providers where possible; mixing them makes cleanup ambiguous.

Targeting and Progressive Exposure

Targeting dimensions:

• User or tenant ID (sticky bucketing), account attributes (plan, region), device or app version, and time window.
• Segments defined by stable attributes; avoid segments based on volatile behavior that changes mid-session.
• Percentage rollout: bucketing must be deterministic and sticky so a user does not flip variants between requests. Hash on the flag key plus a stable identifier.
• Ring-based exposure: internal users, then employees or dogfood tenants, then 1 percent, then 5, 25, 50, 100. Pause between rings and inspect metrics (./07-observability-rollback.md).
• Combined with canary: flags change behavior per cohort; canary changes which artifact serves traffic. They are complementary; do not use one to simulate the other (./03-cd-strategies.md).

Exposure design:

• Start with an allowlist of internal accounts; then named customers; then percentage; then general availability.
• Define the metric and the guardrail metrics before exposing: success metric, error rate, latency, and business guardrails.
• Predefine rollback criteria as numeric thresholds; "watch for a while" is not a criterion.
• Keep the rollout schedule in the flag system's audit trail, not only in a ticket.
• Consider dependencies between flags: killing flag A while flag B depends on it can leave a broken state. Document relationships.

Evaluation Architecture

| Pattern | Latency | Freshness | Complexity | Use when | |---|---|---|---|---| | Local SDK evaluation with streamed rules | Microseconds | Near-real-time updates | Provider coupling, SDK in every service | Default for performance-sensitive services | | Remote API evaluation per request | Network round trip | Instant | Availability dependency on the provider | Admin tools, low-QPS paths | | Server-side evaluation proxy | Low | Near-real-time | Extra component | Central policy and caching | | Edge/CDN evaluation | Very low | Near-real-time | Edge compute support | Frontend and edge routing | | Config baked at deploy | Zero | Deploy cycle | Simple, no provider | Slower cadence, fewer flags |

Rules:

• Flag evaluation should be local and fast for request paths; a network call per request couples availability to the flag provider.
• Bootstrap state: the SDK must start with a sensible snapshot (or defaults) when the provider is unreachable at startup.
• Provider outage behavior is per-flag policy, not a global default. A kill switch should fail closed for the feature (disabled) while a core feature should fail open (enabled) to preserve availability. Decide and document per flag.
• Cache rules with a TTL and refresh on stream updates; handle flapping.
• Never put secrets in flag values; flag rules are widely readable.
• Frontend flags: do not leak unreleased features as disabled-but-shipped code paths with discoverable payloads; server-gate sensitive functionality and treat client flags as UX toggles only.
• Track flag evaluation events if the provider supports it, for audit and for identifying unused flags.

Lifecycle Governance

Every flag has a lifecycle:

| Stage | Exit criterion | Gate | |---|---|---| | Proposed | Category, owner, expiry, and default defined | Flag review or template enforced | | Created | Flag exists in all environments with safe defaults | Default off in production unless justified | | Rolling out | Exposure plan executed with metrics | Guardrails green at each step | | Fully on | Flag permanently on in production | Cleanup ticket issued with a deadline | | Cleanup | Flag and dead code removed; evaluation removed | PR merged and flag archived |

Governance rules:

• Mandatory metadata: owner, category, creation date, expiry date, and a cleanup ticket. A creation form without these fields is rejected in the flag system or in review.
• Flag debt is tracked like dependency debt: a dashboard of flags past expiry, flags older than a threshold, flags with zero evaluations, and flags with a single state in production.
• Stale flags are removed on a schedule, owned by teams, with exceptions requiring justification.
• Removing a flag means removing the old branch of code, not just flipping it permanently on. Dead code behind an always-true flag is still dead code.
• Kill switches are the exception: they are reviewed periodically (tested in a game day), not removed. A kill switch that has never been exercised does not work.
• Entitlement flags are configuration; changes go through the same review and audit as pricing or permissions changes.
• Bulk operations (enable for all tenants) require the same care as a deploy: announce, measure, and keep a rollback path.

Testing with Flags

Flags multiply states; naive testing explodes.

Strategies:

• Test each branch as its own behavior: unit tests for on and off paths, not a full combinatorial matrix.
• Keep flag branches shallow and short-lived. Deep flag nesting is a signal to refactor.
• Provide a test configuration that sets explicit flag values per test; never depend on the live provider in tests.
• Integration tests should run critical journeys with the flags in their production-default state, plus targeted runs for flags being rolled out.
• Do not let flag evaluation make tests non-deterministic: inject an evaluator or use an in-memory provider.
• Test the default path: the behavior when the provider is unavailable and defaults apply. This path runs during incidents, so it must be correct.
• For frontend flags, test both visual states where the change is meaningful; snapshot the differences intentionally.
• Contract tests for the flag payload: if the SDK or rule schema changes, tests catch it before production.
• Add a test that fails when a flag reference has no matching definition, catching typos that would otherwise silently default.

Build versus Buy

| Option | Strengths | Costs | Use when | |---|---|---|---| | Hosted provider | Streaming updates, targeting UI, experiments, audit trail | Cost per seat or evaluation, vendor dependency, data egress | Product experiments and many flags | | Open-source self-hosted | Control, no per-seat cost, data stays in-house | You run the control plane and upgrades | Platform teams with strong operations | | Home-grown in config service | No new vendor; fits an existing config system | Targeting, streaming, audit, and experiments become your job | A small, stable number of flags | | Build-time/environment config | Zero runtime dependency | No dynamic rollout; change needs deploy | Few, coarse toggles only |

• Whatever the choice, keep an abstraction layer thin but present: flag keys and evaluation calls behind a small interface so a provider swap is a contained change. The observability pack's signal-abstraction reasoning applies here too.
• Do not let provider SDKs spread through the codebase; evaluation belongs in service entry points, not in domain logic.
• Budget the provider cost against the value of experiments; a provider bill larger than the experiment program is a signal to consolidate.

Operational Metrics for Flags

• Track evaluations per flag per version to prove rollout progress and to find unused flags.
• Track error and latency metrics split by flag variant for the flags under active rollout; variant-blind dashboards hide regressions. See ./07-observability-rollback.md.
• Alert on evaluation errors, provider lag, and SDK bootstrap failures: silent flag failures default to unknown behavior.
• Report flag age distribution and expired-flag count as a recurring engineering-health metric.
• Record flag changes (who flipped what, when) in the same audit path as deploys; a flag flip is a production change.

Rollout Runbook

1. Confirm the flag evaluates to the intended default in every environment before any exposure.
2. Enable for internal accounts; verify the success metric and guardrails for a defined window.
3. Expand to named customers, then percentage rings, pausing long enough for metrics to be meaningful.
4. At each step, compare variant metrics against the control, not against last week.
5. On guardrail breach, disable the flag first and investigate second; the flag is the fast lever. See ./07-observability-rollback.md.
6. At full exposure, start the cleanup clock: remove the losing path, then archive the flag.
7. Record each step with actor and timestamp so the rollout can be reconstructed later.

Migration Notes

• Moving from env-var toggles to a flag system: create the flag with the same default, switch reads one at a time, then remove the env variable. Never remove an env toggle and add a flag in the same release.
• Moving from one provider to another: run both SDKs in shadow mode and compare evaluations before cutting over; bulk-import targeting rules and audit the diff.
• Retiring an experiment flag: choose the winning variant, remove the losing code path, keep the flag record for history, then archive.
• Converting a release flag to permanent configuration: reclassify, move it to the config system with review, and document it in the service's config schema.
• Cleaning up after a provider migration: archive flags in the old system only after the new system has served production traffic for a full release cycle.

Anti-Patterns

• Flags as a permanent configuration system with no cleanup.
• Nested flags and combinatorial branches nobody can reason about.
• Provider dependency on the request path with no cache or fallback.
• Experimental flags used to gate safety-critical behavior without a fail-safe default.
• One flag controlling unrelated behaviors ("the new stuff" flag).
• Rolling out to 100 percent with no metric defined and no rollback criteria.
• Client-side flags hiding features whose payloads remain fully accessible.
• Flags without owners whose creator left the team.
• Testing only the on state, then discovering the off path is broken during an incident.
• Kill switches never exercised; the first use fails.

Checklist

• [ ] Every flag classified and carries owner, expiry, and cleanup ticket.
• [ ] Defaults chosen per flag for provider outage, explicitly documented.
• [ ] Evaluation is local and cached on request paths; bootstrap snapshot exists.
• [ ] Targeting uses stable, sticky bucketing.
• [ ] Exposure plan defined with metrics and numeric rollback criteria.
• [ ] Flag dependencies documented.
• [ ] Tests cover on and off paths plus provider-unavailable defaults.
• [ ] Flag debt dashboard exists; expired flags have owners and deadlines.
• [ ] Cleanup removes code, not just flips state.
• [ ] Kill switches exercised in a game day at least twice a year.

Release Management

Scope: versioning schemes, changelogs, artifact provenance and signing, release trains versus continuous delivery, approval flows, communication, and post-release review.

Versioning

| Scheme | Format | Use when | Notes | |---|---|---|---| | SemVer | MAJOR.MINOR.PATCH | Libraries, APIs, anything with compatibility promises | Breaking change means major; pre-1.0 still implies churn | | CalVer | YYYY.MM or YYYY.WW | Products and continuously delivered services | Less about compatibility, more about recency | | Build number / pipeline ID | Numeric | Internal services | Pair with commit SHA for traceability | | Date plus counter | YYYY.MM.DD.N | Daily releases | Readable and orderable | | Hash only | SHA prefix | Ephemeral artifacts | Poor human communication; acceptable as a secondary identifier |

Rules:

• One version per artifact, immutable. The same version is never rebuilt or republished; a fix gets a new version.
• Version the artifact and the release separately where useful: release = version plus environment plus deploy ID. See ./07-observability-rollback.md.
• Tag in version control exactly what was built; the tag points at the commit, and CI records the digest it produced from it.
• Never reuse a version after rollback; the rolled-back version remains a historical fact.
• Pre-release identifiers (-rc.1, -beta.2) are for artifacts that must be distinguishable without claiming stability.
• For APIs and packages, document the compatibility policy: what constitutes a breaking change and the deprecation window. See the api-design pack if installed.

Changelogs

Changelogs are for users; commit logs are for developers. Keep both and do not confuse them.

• Structure by release with date and version; group entries as Added, Changed, Fixed, Deprecated, Removed, Security.
• Write in terms of user-visible effects, not commit messages. "Reduce checkout latency" beats "refactor pricing service".
• Every release has changelog entries, including patch releases; "internal changes" is a valid entry when it is honest.
• Mark breaking changes prominently with the migration path.
• Link to the commit range or PRs for forensic use.
• Automate the skeleton from conventional commits or labels, and require human editing for user-facing clarity. Fully automated changelogs are noisy; fully manual changelogs rot.
• Keep a changelog file or release notes system in the repository as the source of truth; the release UI or announcement is derived from it.
• Publish per-release notes to the audience that consumes them: users, operators, or internal teams.
• Do not rewrite published changelog history; append corrections.

Artifact Provenance

Provenance answers: who built this, from what source, with what inputs, and can it be verified?

| Element | Purpose | |---|---| | Immutable digest | Identity and integrity of the artifact | | Signature | Authenticity of the producer | | SBOM | Dependency and license inventory | | Build attestation | Builder identity, source commit, parameters | | Release ledger | Mapping of version to digest, environment, time, actor | | Registry policy | Prevents overwrites and unsigned pushes |

Rules:

• Sign with a workload identity bound to the build system (keyless where supported) and verify at admission or before deploy. See ./02-ci.md.
• Store the mapping version -> digest -> commit in a queryable ledger; reconciliation between tag and digest must be automatic.
• Registries should reject tag mutation after publish and require signatures for production namespaces where the platform supports it.
• Provenance attests to the build, not to behavior: passing provenance does not mean the code is correct. It answers supply-chain questions only.
• Retain artifacts for the rollback window and any compliance period; retention policy must outlive the longest plausible rollback.
• Verify provenance at deploy time; a warning that can be ignored will be ignored. Make verification blocking for production.

Release Trains versus Continuous Delivery

| Model | Cadence | Merits | Costs | |---|---|---|---| | Continuous | On merge, gated automatically | Fast feedback, small batches, low risk per change | Requires strong tests, observability, rollback, and flags | | Release train | Fixed schedule (daily, weekly) | Predictable, bundles approvals, easier coordination | Larger batches, pressure to include late changes, slower feedback | | Windowed | Fixed maintenance windows | Regulatory fit, coordinated operations | Availability gaps, emergency processes dominate | | Feature-flag releases | Continuously deployed, deliberately exposed | Decouples deployment from release | Flag governance is load-bearing |

• Choose based on risk tolerance, regulatory context, and team capability, not fashion. A team without observability will not succeed at continuous delivery by announcing it.
• If using a train, freeze the train contents at a cutoff; late changes wait for the next train or ride an explicit emergency path.
• Continuous delivery still benefits from a release cadence for user communication; deploy continuously, announce deliberately.
• Keep release size bounded in every model. Large batches make rollback and diagnosis harder.

Approval Flows

Approval is a control, not a ritual. Design it to catch what automation cannot.

• Separate duties where required: the person who writes the change should not be the only approver for a production release in regulated contexts.
• Automated gates run first; human approval happens on top of green automated evidence, never instead of it.
• Approval scopes: PR review, environment promotion, emergency change. Keep them distinct; do not require a full change board for a two-line config fix.
• Emergency changes: a documented break-glass path with reduced ceremony, mandatory retrospective, and automatic follow-up. Speed comes from pre-authorized roles, not from skipping records.
• Record approvals with actor, scope, timestamp, and the artifact digest approved. An approval that names a branch rather than a digest is not an approval.
• Expiry: approvals for an artifact are invalidated when the artifact changes or after a defined window. Stale approvals accumulate risk.
• Keep the number of required approvers as low as the risk allows; high friction produces rubber stamps.

Communication and Deprecation

• Announce user-visible changes before they land: what, when, impact, and the rollback or mitigation path.
• Maintenance windows for disruptive operations are scheduled, announced, and rehearsed; preference is always zero-downtime operation.
• Deprecations have a timeline: announce, warn in logs or headers, provide the replacement, and only then remove. See the api-design pack if installed.
• Status communication during incidents follows the incident process; release communication and incident communication should agree on facts already published.

Post-Release Review

The review closes the loop and turns release practice into organizational learning.

Inputs:

• Did the release meet its SLOs and guardrails over the observation window? See ./07-observability-rollback.md.
• Rollbacks, aborted rollouts, flags flipped back, and emergency changes.
• Deploy frequency, lead time for changes, change failure rate, and time to restore: track the four delivery metrics, with care not to gamify them.
• Incidents correlated with releases; whether the release process or the code was at fault.
• Flag and migration cleanup status created by this release. See ./08-feature-flags.md and ./06-database-migrations.md.

Outputs:

• Action items with owners, especially for rollbacks and near misses.
• Process changes with a test for whether they helped (for example, a new gate must have a false-positive story).
• Updates to runbooks and checklists used during the release.
• Feed the next release plan: batch size, cadence, and gate tuning.

Cadence: lightweight per release, deeper monthly or quarterly. Avoid turning every deploy into a meeting; sample and deepen where the metrics say risk is concentrated.

Release Evidence and Audit

For regulated or high-trust contexts, assemble the evidence pack per release:

• Version, tag, commit, artifact digest, and SBOM.
• Signature and provenance attestation, plus the verification result at deploy.
• Approvers, approval scope, and timestamp bound to the digest.
• Migration list with compatibility classification and verification results (./06-database-migrations.md).
• Test and security scan results for the exact commit.
• Rollout plan, analysis results, and the final state (promoted or rolled back).
• Post-release metrics window and any follow-up actions.

Rules:

• Evidence is generated automatically as a byproduct of the pipeline; if assembling it takes a human afternoon, it will be wrong under deadline pressure.
• Store evidence immutably alongside the artifact or in a system with retention at least as long as the artifacts it describes.
• Audit asks "prove this version was tested and approved"; the answer must be a query, not a hunt through chat.

Migration Notes

• Moving from manual releases to a release train: start with a fixed weekly cutoff, automate the evidence, and only then shorten the cadence.
• Moving from a train to continuous delivery: prerequisites are progressive delivery with automated gates, SLO-based rollback, and flag governance (./03-cd-strategies.md, ./07-observability-rollback.md, ./08-feature-flags.md).
• Moving from branch-based releases to artifact promotion: tag the built commit, publish the digest, and make environments consume the digest; delete the per-environment build jobs afterward.
• Moving from human-only approvals to automated gates: run gates in advisory mode first, measure false negatives against real incidents, then make them blocking.

Anti-Patterns

• Rebuilding an artifact because the original pipeline is gone; version reuse after a rollback.
• Changelogs generated verbatim from commit messages with no user-facing editing.
• Signing images but never verifying signatures at deploy.
• Approvals on a branch name instead of an immutable digest.
• Trains that grow to a hundred changes and cannot be rolled back safely.
• Emergency changes with no retrospective and no follow-up.
• Provenance treated as a guarantee of correctness.
• Version numbers with no compatibility policy behind them.
• Release notes that omit breaking changes or migration steps.
• Post-release reviews that produce no owned actions.

Checklist

• [ ] Versioning scheme documented with a compatibility policy.
• [ ] Versions immutable; tags point at the exact built commit; digests recorded.
• [ ] Changelog written for users, grouped and linked to the release.
• [ ] SBOM, signature, and build attestation produced and stored per release.
• [ ] Provenance verified in the deploy path, blocking for production.
• [ ] Release model chosen and batch sizes kept intentional.
• [ ] Approval flow scoped by risk; approvals name a digest and expire.
• [ ] Emergency path documented with retroactive review.
• [ ] Deprecations announced with a timeline and replacement.
• [ ] Post-release review with owned actions and delivery metrics tracked.