Test Strategy

Choosing test layers, allocating budget by risk, deciding what not to test, assigning ownership, and balancing shift-left with shift-right.

The Strategic Question

A test strategy is an answer to four questions:

1. What can fail? (failure modes, not code coverage targets)
2. How would we know, and how fast? (detection layer and feedback time)
3. How much does that detection cost to build and keep? (runtime, setup, maintenance)
4. Who acts when it fires? (owner, runbook, severity)

If a test cannot be connected to a failure mode and an owner, it is ritual. Strategy is about deliberately accepting risk elsewhere, not pretending every risk can be tested.

Pyramid vs Trophy vs Honeycomb

The classic pyramid

| Layer | Share | Speed | Confidence in integration | |---|---|---|---| | Unit | ~70% | ms | None | | Integration / component | ~20% | 100 ms - 5 s | High for own boundaries | | E2E / UI | ~10% | 10 s - 5 min | Highest, slowest, flakiest |

The pyramid optimizes for fast feedback and cheap maintenance. It remains correct for logic-heavy backends, libraries, compilers, and data engines.

The testing trophy (frontend and CLI reality)

Kent C. Dodds' trophy is a better fit when behavior emerges from composition of many small units rather than from a few pure functions:

| Layer | Share | What it means in practice | |---|---|---| | Static analysis | "free" | Types and lint catch a large defect class before runtime | | Unit | ~20% | Pure logic, reducers, formatters, parsers | | Integration | ~50% | Render a component tree, hit a real router and fake server | | E2E | ~10% | Few critical journeys through a real browser |

For React/Vue/Svelte apps the component integration layer is where most regressions (simulated effects, routing, state wiring) actually surface; unit-testing every hook in isolation produces a broad but shallow suite.

Choosing explicitly

| Context | Model | Why | |---|---|---| | Domain/business logic library | Pyramid | Logic is pure, units are meaningful | | CRUD backend service | Pyramid with fat integration layer | Many defects live at SQL/HTTP boundaries | | SPA / design system | Trophy | Behavior emerges from composition | | Event-driven / microservices | Honeycomb (many contracts, thick service tests) | Correctness lives in interactions | | CLI / data pipeline | Pyramid + golden files | Deterministic I/O | | ML training code | Split: unit on transforms, slow integration on end-to-end training | Training is too slow to E2E often |

Do not argue about the metaphor. Agree on the number of tests per layer, their expected runtime, and what runs in which pipeline stage (08-flaky-ci).

Risk-Based Testing

Scoring model

Score each capability on likelihood (1-5) and impact (1-5); test investment follows the product.

| Factor | 1 | 3 | 5 | |---|---|---|---| | Change frequency | Stable for years | Monthly | Daily | | Business impact if wrong | Cosmetic | Recoverable | Revenue, safety, compliance | | Complexity / cyclomatic | Trivial | Moderate | Branch-heavy | | History of defects | None | Occasional | Repeated incidents | | Boundary count (I/O, network, DB) | None | One | Several | | Blast radius | One screen | One service | All tenants / all customers |

Risk score = likelihood x impact. High (>= 15) gets unit + integration + at least one E2E. Medium (6-14) gets unit + integration. Low (<= 5) gets a unit test or nothing but a smoke check.

Risk categories that are never "low"

• Authentication and authorization decisions.
• Money movement, pricing, and tax calculation.
• Personal data handling and consent.
• Schema migrations and data deletion paths.
• Idempotency and retry behavior of externally visible writes.
• Anything with a regulatory or contractual obligation.

For these, the expected artifact is an automated test at the lowest layer that can fail plus a monitoring signal in production (shift-right).

Test Budget

Budget along three axes: money (CI minutes, device farms), time (feedback latency for developers), and attention (how many failures a team will actually inspect).

Practical rules:

• Pull request gate target <= 15 minutes p95. Beyond that, developers batch changes and stop reading failures (08-flaky-ci).
• Keep the fast suite (unit + component) under 5 minutes on a laptop; this is the suite a developer runs while editing.
• Every E2E scenario must justify its minutes. If two journeys share 90% of steps, merge them into one journey with multiple assertions.
• Prefer one integration test over five unit tests that mock the same boundary, and one contract test over an E2E that exists only to catch field renames.
• Allocate roughly 10-20% of engineering time to test upkeep. If upkeep is invisible, the suite is decaying silently.

Cost of a test over its life

total cost = authoring + review + (runs per day x wall-clock cost) + maintenance + flake triage

A 200 ms unit test run 500 times a day costs ~100 seconds of machine time. A 4-minute E2E run 50 times a day costs 3.3 hours plus a person triaging flakes at ~10% rate.

What Not to Test

| Do not test | Reason | Do instead | |---|---|---| | Framework internals (routing library, ORM internals) | Upstream owns it | Test your configuration of it at one seam | | Generated code (protobuf, OpenAPI clients, migrations autogen) | Not hand-written | Test the generator invocation and the generator's own tests | | Trivial accessors and DTO pass-throughs | Zero logic, zero defect class | Rely on types and the first integration path | | Log message text | Non-contractual, drives churn | Assert on structured event fields if they are contractual | | Private methods | Implementation detail | Test via public behavior | | Exact UUIDs/timestamps | Non-deterministic inputs | Inject clock/ID generator; assert format and ordering | | Third-party behavior | You cannot change it | Contract test the boundary you consume (06-contract) | | Dead code, feature-flagged code past removal date | Waste | Delete the code, then the tests |

Counterpoint: "do not test" is not "do not verify". If unit tests are skipped for a boundary, some other artifact (contract test, monitor, canary) must cover it.

Ownership

• Every test file has an owner recorded in CODEOWNERS or an equivalent map. Unowned suites rot first.
• The team that owns the code owns its tests. A central QA team is a bottleneck and a knowledge island; platform teams own test infrastructure, not other teams' assertions.
• Quarantined tests have an owner and an expiry (issue link plus date). Expired quarantine auto-deletes or auto-fails the build, otherwise quarantine becomes a landfill (08-flaky-ci).
• Test infrastructure has an on-call. When the runner is down or the container registry is throttled, someone is paged.
• Failures page the code owner, not the test author. Ownership follows code, not the person who happened to write the assertion.

Shift-Left and Shift-Right

Shift-left (move detection earlier)

| Practice | Catches | Cost | |---|---|---| | Static analysis and strict types | Type errors, nullability, dead code | Minutes to adopt, ongoing discipline | | Pre-commit hooks (lint, format, fast tests) | Formatting, obvious breakage | Developer latency | | Contract tests in CI | Interface drift before deploy | Pact broker or schema registry | | Test-first / TDD where logic is unclear | Design problems, missing cases | Learning curve | | Property tests on pure functions | Edge cases humans miss | Strategy authoring | | Migration tests against a real DB | Schema breakage | Testcontainers runtime |

Shift-right (verify in production)

| Practice | Catches | Note | |---|---|---| | Canary / progressive delivery | Deployment faults, perf regressions | Requires rollback automation | | Synthetic monitoring of critical journeys | Outages, cert expiry, third-party failures | Independent of user traffic | | Structured error budgets and alerts | Unknown unknowns | Alert on user-visible symptoms | | Production traffic shadowing / replay | Behavioral drift under real inputs | PII scrubbing mandatory | | Feature-flag ramp with metric guardrails | Business-logic regressions | Flags carry expiry dates |

Pairs: shift-left catches what can be known before release; shift-right catches what can only be known after. A strategy with only one of the two is incomplete.

Anti-Patterns

• Coverage as goal — writing assertion-free tests to move a number (09-coverage-metrics).
• Ice cream cone — mostly E2E with weak unit/integration; slow, flaky, expensive.
• Test everything everywhere — same behavior asserted at three layers; triple maintenance, no extra detection.
• Set-and-forget suite — no owner, no flake budget, no runtime budget.
• QA as gatekeeper — manual verification at the end of the cycle; feedback too late.
• Copy-pasted test templates — 200 tests that all exercise the happy path with different names.
• Retry-as-policy — green build by rerunning red tests instead of fixing them.
• Testing the mock — assertions that only verify the test's own stubs (02-unit).

Strategy Review Checklist

• [ ] Risk register maps capabilities to test layers; high-risk items have E2E or contract coverage.
• [ ] Layer distribution is intentional and recorded (pyramid, trophy, or honeycomb).
• [ ] PR gate runtime p95 <= 15 minutes; fast suite <= 5 minutes.
• [ ] Every test file has an owner; quarantine has owner + expiry.
• [ ] What-not-to-test list exists and is enforced in review.
• [ ] Detection layers pair with production monitoring for non-deterministic risks.
• [ ] Flake rate and runtime are tracked as first-class quality metrics (09-coverage-metrics).
• [ ] Test budget is revisited quarterly or after each major incident.

Unit Testing

Structure, naming, determinism, isolation, test doubles, parameterization, edge cases, and assertion quality for fast, trustworthy unit tests.

Definition

A unit test exercises one unit of behavior in isolation: a function, a class method, a reducer, a small module. "Isolated" means no network, no filesystem surprises, no shared mutable state, and no dependence on test order or wall clock. A unit test should run in milliseconds and fail with a message that identifies the defect without debugging.

Arrange-Act-Assert

def test_discount_applies_loyalty_rate_before_tax():
    cart = Cart(items=[Item(price=100_00)], customer=Customer(loyalty_tier="gold"))

    total = price_cart(cart, tax_rate=Decimal("0.20"))

    assert total == Decimal("108.00")

• One logical behavior per test. Multiple asserts are fine when they describe the same outcome (e.g. fields of one returned object).
• Blank lines separate the three phases; if the act phase is more than a line or two, the unit is probably doing too much.
• Prefer pure functions with injected dependencies; they make AAA trivial.

Given-When-Then

The same structure in BDD vocabulary. Use whichever the team already speaks; mixing both in one repository is worse than either.

Naming

Names are documentation; a failing suite should read like a defect list.

| Style | Example | Good for | |---|---|---| | Behavior sentence | rejects_expired_coupon_at_checkout | Python, Ruby, Go | | Given/When/Then | given_expired_coupon_when_checkout_then_rejected | Java, Kotlin, explicit BDD | | should + verb | should_reject_expired_coupon | JS/TS, .NET | | Method + scenario + outcome | calculateTotal_expiredCoupon_throws | Legacy JVM conventions |

Rules:

• Name the behavior, not the method under test (test_parse_2 is noise).
• Avoid "test1", "works", "happy_path" without a subject.
• Keep names searchable: a stack trace line should be enough to find the test.
• Parameterized cases get explicit case IDs so CI output reads test_rounds_half_up[cash-fraction].

Determinism

| Non-deterministic input | Fix | |---|---| | Current time | Inject a clock (Clock interface, freezegun/time-machine, vi.setSystemTime) | | Random IDs / tokens | Inject an ID generator or seed; never assert on a specific random value | | Random order of collections | Sort before asserting or compare as sets/multisets | | Randomness in algorithms | Seed explicitly; property tests may be randomized with a recorded seed | | Locale, timezone, encoding | Pin in the test config (TZ=UTC, LC_ALL=C, explicit encoding="utf-8") | | Environment variables | Load from a test config object; do not read os.environ implicitly | | Network | Replace with a fake if the unit is not the network boundary | | Concurrency | Prefer deterministic scheduling (latch/barrier) over sleep |

Rule of thumb: if a test fails once every N runs without a code change, it is broken even if the current commit is "green" (08-flaky-ci).

Isolation

• Tests must pass when run alone, in any order, and in parallel.
• No shared mutable module-level state; reset singletons and caches between tests.
• No test may depend on another test's side effects. Interdependence is often introduced by session-scoped fixtures; make shared fixtures immutable or reset them explicitly.
• Each test owns its data (unique IDs per run allows parallel and repeated execution (07-test-data)).
• Randomize test order periodically (e.g. pytest-randomly, Vitest sequence.shuffle) to expose hidden coupling.
• Parallelism should be safe by construction, not by pinning tests to one worker.

Test Doubles Taxonomy

| Double | Definition | Use | Risk | |---|---|---|---| | Dummy | Placeholder passed but never used | Filling required parameters | Hides dead parameters | | Stub | Returns canned values | Force a code path | Drifts from real behavior | | Spy | Records interactions, delegates optionally | Assert a side effect happened | Over-specification | | Mock | Preprogrammed expectations, fails on unexpected calls | Strict interaction contracts | Very brittle | | Fake | Working, simplified implementation | Behavioral collaborator (in-memory repo) | Semantic divergence from production |

Rules for doubles

1. Mock boundaries you own. Define a port/interface, then substitute it. Do not patch library internals; wrap them first.
2. Prefer fakes for behavior, stubs for data, mocks sparingly. A fake in-memory repository exercises real branching; a mock only proves the test's own expectation.
3. Patch where it is looked up, not where it is defined. myapp.clients.payments, not stripe.
4. Never mock the unit under test. If you must, the unit is too large; split it.
5. Do not assert call order unless order is a documented contract.
6. Keep doubles in sync with reality through contract tests on the real boundary (06-contract); otherwise stubs become fiction.
7. One adapter per third-party keeps vendor types out of domain code and gives tests a single seam.

class InMemoryUserRepo:
    def __init__(self) -> None:
        self._by_id: dict[str, User] = {}

    def get(self, user_id: str) -> User | None:
        return self._by_id.get(user_id)

    def save(self, user: User) -> None:
        self._by_id[user.id] = user

Parameterized Tests

Use tables to cover input classes; one assertion per case keeps failures precise.

import pytest

@pytest.mark.parametrize(
    ("raw", "expected"),
    [
        pytest.param("10", 10, id="positive"),
        pytest.param("-3", -3, id="negative"),
        pytest.param("0", 0, id="zero"),
    ],
)
def test_parse_int(raw: str, expected: int) -> None:
    assert parse_int(raw) == expected

• Give every case a stable ID; CI output is the only documentation most readers see.
• Keep tables small and thematic; a 60-row boolean matrix is a sign the unit should be decomposed.
• Use indirect parameterization for fixture-level variation only when it stays readable.
• Do not parameterize when cases need different arrange logic; that is a new test.

Edge Cases and Boundary Analysis

For every input, ask: minimum, maximum, just inside/outside each boundary, empty, one, many, duplicates, malformed, and adversarial.

| Input class | Cases | |---|---| | Numbers | 0, -0, min, max, overflow, NaN, infinity, float precision | | Strings | empty, whitespace, unicode, combining chars, very long, embedded NUL | | Collections | empty, single, many, duplicates, unsorted, nested, null elements | | Dates/times | leap day, DST transition, month end, epoch, timezone boundaries | | Identifiers | empty, max length, invalid chars, case sensitivity, URL-encoded | | Concurrency | simultaneous writes, retries, cancellation mid-flight | | Money | rounding halves, currency precision, negative values, zero-decimal currencies |

State transitions: test valid transitions, invalid transitions, and idempotency where the domain claims it.

Assertion Quality

• Assert on the full meaningful result: a single assert result == expected beats five partial asserts that can all pass while the object is wrong.
• Include a context message for non-obvious comparisons.
• Prefer structural equality (dataclasses, records) so new fields are covered automatically.
• Use domain-specific assertions (assertThat(response).hasStatus(409)) only when the error message is materially better.
• Avoid boolean collapse: assert is_valid(x) is True hides which branch failed; assert the underlying property instead.
• For exceptions: assert the type and the message/error code that clients depend on, not the full stack text.
• Floating point: use tolerance (pytest.approx, expect(x).toBeCloseTo) and state why.
• Never assert on repr() of complex objects.

Weak vs strong assertions

| Weak | Strong | Why | |---|---|---| | assert result is not None | assert result == expected_order | Existence proves nothing | | assert mock.called | assert saved.status == "paid" | Interaction vs outcome | | assert "error" in text | assert response.error_code == "EXPIRED_COUPON" | Substring matching is accidental | | assert len(items) > 0 | assert [i.id for i in items] == ["a", "b"] | Order and content matter | | assert x != y | assert x == expected | Negative assertions miss most defects |

Snapshot Discipline

• Snapshots are useful for large serialized outputs (HTML, JSON, OpenAPI) when normalized.
• Normalize volatile fields (IDs, timestamps, sort order) before comparison.
• Snapshot updates are a reviewable change and must never happen automatically in CI.
• One snapshot per behavior; opaque 2000-line blobs hide regressions instead of revealing them.

Anti-Patterns

• Assertion-free tests that only check "no exception".
• Mock-heavy tests where the test verifies its own stubs.
• One giant test with ten phases and no isolation.
• time.sleep in unit tests; inject the clock or poll with a deadline.
• Copy-paste test bodies with one literal changed; use parameterization.
• Testing private helpers; refactor to expose behavior.
• Catching broad exceptions in the test to keep it green.
• @skip without a linked issue and an owner.

Unit Test Checklist

• [ ] AAA structure visible; one behavior per test.
• [ ] Name states scenario and expected outcome; parameterized cases have IDs.
• [ ] Time, randomness, locale, and I/O are injected or pinned.
• [ ] Tests pass alone, shuffled, and in parallel.
• [ ] Doubles sit at owned boundaries; fakes preferred for behavior.
• [ ] Boundary value analysis applied to numeric/string/date inputs.
• [ ] Assertions are outcome-based and produce readable failure messages.
• [ ] No assertion-free tests, no skipped tests without owner and expiry.

Integration Testing

Testing real component boundaries: databases, HTTP clients, queues, caches, and filesystems with production-like dependencies.

What Integration Tests Are For

An integration test exercises your code together with a real external dependency or a real infrastructure component. It catches the defect classes unit tests cannot:

• SQL correctness: joins, constraints, nullability, transaction behavior, migration drift.
• Serialization and protocol errors: JSON field names, headers, content types, encoding.
• Wiring bugs: dependency injection, configuration loading, middleware order.
• Concurrency at boundaries: connection pool exhaustion, lock contention, retry semantics.
• Third-party API assumptions: pagination, rate limits, error envelopes.

The unit/integration line is not "mock vs real" but "is the boundary under test". A test that spins a real Postgres but only exercises pure domain logic is an expensive unit test.

Component Boundaries

| Boundary | Real dependency | Substitute | Caught by real | |---|---|---|---| | Relational DB | Testcontainers (Postgres/MySQL) | H2/SQLite | Dialect, constraints, locking | | Document DB | Testcontainers (Mongo) | moto-style fake | Index and query semantics | | HTTP API (yours) | In-process test client | none | Middleware, auth, serialization | | HTTP API (third-party) | Contract-tested stub / WireMock | respx / msw | Headers, retries, timeouts | | Message broker | Testcontainers (Kafka/Rabbit/NATS) | in-memory bus | Ordering, ack, redelivery | | Object storage | Testcontainers (MinIO) | fake client | Multipart, presign, permissions | | Cache | Testcontainer (Valkey/Redis) | in-memory dict | TTL, eviction, atomicity | | Clock/locale | injected clock | n/a | Timezone, DST, formatting |

Two speeds of integration test:

• Narrow integration — one component plus one real dependency (repository + real DB). Keep these in the PR gate; aim < 5 s each.
• Wide integration — several components wired together (API + DB + queue) without a browser. Useful for smoke coverage; keep them in a separate job or nightly.

Database Tests

Testcontainers pattern

import pytest
from sqlalchemy import create_engine
from testcontainers.postgres import PostgresContainer

@pytest.fixture(scope="session")
def engine():
    with PostgresContainer("postgres:17-alpine") as pg:
        eng = create_engine(pg.get_connection_url(), future=True)
        run_migrations(eng)
        yield eng
        eng.dispose()

@pytest.fixture
def session(engine):
    with engine.begin() as conn:
        yield conn
        conn.rollback()

• Pin image tags. latest breaks reproducibility without warning; bump tags deliberately.
• Run migrations once per session, not per test. Migration speed dominates otherwise.
• Use the same database engine and major version as production. SQLite in place of Postgres systematically hides type, constraint, and concurrency bugs.
• Keep an embedded option (e.g. SQLite) only for tests that genuinely do not touch SQL semantics; do not maintain two schemas.

Transaction strategies

| Strategy | How | Pros | Cons | |---|---|---|---| | Rollback per test | Wrap each test in a transaction and roll back | Fastest, perfect isolation | Cannot test code that commits internally | | Truncate per test | TRUNCATE ... RESTART IDENTITY CASCADE | Tests commit; simple | Slower; sequence restarts matter | | Schema per worker | Create a fresh schema/DB per worker | Parallel-safe | Setup overhead; migration run per worker | | Snapshot restore | Load a prepared DB dump | Fast for large fixtures | Snapshot drift when schema changes | | Reset by delete | Delete rows by tenant key | Good for shared DB | Ordering and FK complexity |

Decision rule: default to rollback. Switch to truncate or schema-per-worker when the code under test manages its own transactions; use snapshots when seeding dominates runtime.

@pytest.fixture(autouse=True)
def _clean(db):
    yield
    with db.cursor() as cur:
        cur.execute("TRUNCATE orders, order_items RESTART IDENTITY CASCADE")

• Beware of autocommit/pooling interplay: a connection returned to the pool with an open transaction poisons the next test. Ensure teardown closes or rolls back explicitly.
• Never truncate shared reference data; seed it in migrations or a session fixture.

What to assert

• Round-trip persistence: write then read through different code paths.
• Constraints: unique, foreign key, check, NOT NULL — assert the error, not just failure.
• Transactions: rollback on exception, isolation level behavior where it matters.
• Migrations: forward migration on a populated DB and, where supported, rollback.
• Query correctness: ordering, pagination cursors, aggregates under multi-row data.

HTTP Stubs and Fakes

| Tool class | Examples | When | |---|---|---| | In-process interceptor | respx (httpx), responses (requests), nock (Node) | Client library tests, fast, no ports | | Standalone stub server | WireMock, MockServer, Prism | Language-agnostic, cross-process | | MSW (browser/node) | msw | Frontend tests and Storybook | | Contract stub server | Pact stub / broker | Provider behavior pinned by consumers (06-contract) | | Record and replay | VCR-style cassettes | Legacy APIs; scrub secrets and expire recordings |

Rules:

• Stub at the HTTP boundary, never at the SDK method level. Stubbing stripe.Charges.create teaches you nothing about auth headers or payload shape.
• Assert the outgoing request: URL, method, auth header presence, idempotency key. Most integration defects are in what you send, not what you receive.
• Model error responses explicitly: 4xx with body, 5xx, timeouts, malformed JSON, partial reads. Happy-path-only stubs hide the retry and circuit-breaker bugs.
• Record credentials in fixtures only as placeholders. Never commit real tokens.
• Set timeouts in tests lower than production defaults so a hung stub fails fast.

Queues and Event Streams

• Publish, then consume with a bounded poll (deadline + backoff), never a fixed sleep.
• Assert at-least-once semantics: consumers must be idempotent; test a duplicate delivery.
• Test poison messages: malformed payload goes to DLQ and does not block the partition.
• For Kafka, prefer a dedicated topic per test run with unique names; cleanup via topic deletion or short retention.
• Pin consumer group IDs per test to avoid cross-run interference.

Cleanup and Isolation

• Total isolation is the goal: a test that passes only when run after another test is a defect. Prefer per-test unique keys (tenant_id = uuid4()) over global cleanup scripts (07-test-data).
• Cleanup must be exception-safe. Use fixtures with teardown guarantees (yield fixtures, t.Cleanup, afterEach registration).
• Leftovers are a signal: monitor container logs and database size in CI to catch leaks.
• External state that cannot be reset (third-party sandboxes) should be handled through idempotent test data and unique identifiers rather than assumptions of a clean slate.
• Run integration tests in parallel with per-worker resources: database name from worker ID, unique topic names, unique bucket prefixes.

Speed

| Technique | Gain | Caveat | |---|---|---| | Session-scoped container and engine | Seconds per suite | State leaks; clean between tests | | Transaction rollback | Milliseconds per test | Cannot test commits | | Parallel workers with isolated DBs | Near-linear to 4-8x | More memory, container startup burst | | Truncate over drop/create | Big for small schemas | Sequence resets | | Prepared snapshot / template DB | Big when seed data is heavy | Must be rebuilt on migration change | | Reuse containers locally only | Seconds per run | Hidden state; never in CI | | Move slow cases to nightly | PR stays fast | Coverage shifts; keep smoke subset |

Targets: narrow integration case <= 2 s, full PR integration job <= 8 minutes. If a single test takes longer than 10 seconds, question the scope, not the runner.

Anti-Patterns

• Testing against a shared staging database; cross-team interference and data drift.
• Mocking the ORM/driver instead of the database; you are testing the mock's SQL dialect.
• Hidden ordering dependencies between tests through leftover rows.
• One "integration" suite that covers everything and takes 45 minutes, run on every push.
• Retrying DB connection failures inside tests instead of failing fast.
• Asserting only HTTP 200; response bodies and side effects are where bugs live.
• Using SQLite for Postgres code paths and calling it integration coverage.
• Leaving containers running after crashes; no --rm/testcontainers reaper.

Integration Test Checklist

• [ ] Real database engine matching production, pinned image tag.
• [ ] Migrations run once; schema version asserted in a test.
• [ ] Isolation strategy chosen deliberately (rollback/truncate/schema-per-worker).
• [ ] HTTP dependencies stubbed at HTTP layer with error cases covered.
• [ ] Outgoing requests asserted (method, path, auth, idempotency keys).
• [ ] Cleanup exception-safe; no cross-test leakage; parallel-safe by unique keys.
• [ ] Narrow integration in PR gate <= 8 minutes; wide integration nightly.
• [ ] No staging environment dependency, no production credentials, no committed secrets.

End-to-End Testing

Selecting user-critical journeys, writing maintainable Playwright suites, managing auth and test data, capturing artifacts, and controlling flake.

What E2E Is For

An E2E test drives the system through its real user-facing entry point (browser, mobile app, CLI) against a deployed or locally composed environment. Its unique value is proving that the whole chain works together: frontend build, API, auth, database, third parties, and configuration.

E2E is the most expensive and most brittle layer. Its scope should be journeys, not pages or endpoints:

• Sign up, verify email, first login.
• Checkout with a saved payment method.
• Password reset and session expiry.
• Permission boundary: the action a user must not be able to do.
• Deployment smoke: app loads, health endpoint green, login works.

If a journey can be verified by an API-level or component test, do that instead (01-strategy).

Critical-Path Selection

Score journeys by revenue, safety, and frequency. Select 5-15 for the main suite; more for regulated products, fewer for early-stage ones.

| Candidate | Keep as E2E | Better layer | |---|---|---| | Login (happy path + failure) | Yes | - | | Payment/checkout | Yes | - | | Signup with email verification | Yes | - | | Authorization denial | Yes | - | | Form field validation | No | Component | | Pagination/sorting | No | Integration/component | | API schema correctness | No | Contract (06-contract) | | Every settings toggle | No | Component + one E2E smoke | | Mobile/responsive variants | One per breakpoint | Visual regression |

Rules:

• One journey per test; no "mega journeys" that chain ten features.
• Tag tests (@smoke, @critical, @nightly) so pipelines can select subsets.
• A failed critical journey must name the user-visible capability, not the selector.
• Re-evaluate quarterly: delete E2E tests that have not caught a bug in a year or that duplicate lower-layer coverage.

Playwright Patterns

import { test, expect } from "@playwright/test";

test("checkout with saved card", async ({ page }) => {
  await page.goto("/cart");
  await page.getByRole("button", { name: "Checkout" }).click();
  await page.getByLabel("Card ending in 4242").click();
  await page.getByRole("button", { name: "Pay now" }).click();
  await expect(page.getByRole("heading", { name: "Order confirmed" })).toBeVisible();
  await expect(page.getByTestId("order-total")).toHaveText(/\$\d+\.\d{2}/);
});

• Prefer role and label selectors (getByRole, getByLabel, getByText) — they track accessibility and survive refactors.
• data-testid for elements without a stable accessible name. Name test IDs by domain concept, not by layout.
• Never use CSS chains tied to implementation (div > .MuiBox-root:nth-child(3)) or XPath.
• Assert on user-visible outcomes (heading, URL, toast, network response) — not on internal component state.
• Use auto-waiting assertions (toBeVisible, toHaveText) instead of explicit sleeps.
• Keep helpers thin: page objects are fine for repeated flows, but over-abstracted page objects hide the user story.
• Use test.step for readable traces on long journeys.
• Parallelism is per-file by default; make tests independent so this stays safe.

Playwright project layout

e2e/
  playwright.config.ts
  fixtures/
    auth.ts            # storageState setup
    data.ts            # unique data builders
  tests/
    checkout.spec.ts
    auth.spec.ts
  .auth/               # gitignored storage states

Selectors: Priority Order

| Priority | Locator | Notes | |---|---|---| | 1 | getByRole | Best proxy for user intent; enforces a11y | | 2 | getByLabel / getByPlaceholder | Form fields | | 3 | getByText | Visible copy; keep assertions tolerant of case/wrapping | | 4 | getByTestId | Explicit contract with tests; stable | | 5 | CSS attributes ([name=email]) | Acceptable when semantic | | 6 | CSS structure, XPath, nth-child | Forbidden; refactor-brittle |

If an element has no queryable role or label, that is often an accessibility defect worth fixing in the app itself.

Authentication State

Authenticating through the UI in every test multiplies runtime and flake. Use storage state or API login:

// global setup: log in once, save storage state
import { test as setup, expect } from "@playwright/test";

setup("authenticate", async ({ page }) => {
  await page.goto("/login");
  await page.getByLabel("Email").fill(process.env.E2E_USER!);
  await page.getByLabel("Password").fill(process.env.E2E_PASS!);
  await page.getByRole("button", { name: "Sign in" }).click();
  await expect(page).toHaveURL(/\/dashboard/);
  await page.context().storageState({ path: ".auth/user.json" });
});

• Keep one authenticated context per role, not per test. For role-based tests, save separate storage states.
• Test the login journey itself in a dedicated spec, unauthenticated.
• Verify session expiry/login-required behavior in at least one negative test.
• Never store real user credentials; create ephemeral users via API in global setup (07-test-data).
• Storage state contains tokens — gitignore it and treat CI artifacts accordingly.

Test Data for E2E

• Generate unique data per run (timestamp/UUID suffix) so parallel runs and retries do not collide.
• Prefer API-based setup (seed via endpoints or direct DB) over clicking through ten pages; keep only the flow under test in the UI.
• Delete or expire created data when the environment is shared; tag rows with a run ID for bulk cleanup.
• Use dedicated sandbox/test accounts for third parties (payments, email, SMS). Never point E2E at production services.
• For email/SMS verification, use catch-all mailboxes or provider sandboxes with polling helpers; do not assert on external delivery.
• Keep a documented test-user matrix for roles and entitlements.

Artifacts: Trace, Video, Screenshot

Configure artifacts to fire on failure, not on every passing test:

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  use: {
    trace: "on-first-retry",
    video: "retain-on-failure",
    screenshot: "only-on-failure",
    baseURL: process.env.E2E_BASE_URL ?? "http://localhost:3000",
  },
  reporter: [["html", { open: "never" }], ["junit", { outputFile: "results.xml" }]],
});

• Trace is the primary debugging tool. npx playwright show-trace trace.zip replays DOM, network, console, and screenshots.
• Upload artifacts to CI for failures only; retention windows keep storage bounded.
• Add timestamps and run IDs to artifact names when artifacts are aggregated.
• On flake, read the trace before rerunning. Retrying without reading the trace discards the only evidence (08-flaky-ci).

Flake Control in Browser Tests

| Symptom | Likely cause | Fix | |---|---|---| | Passes locally, fails in CI | Animations/CPU speed, fonts | Disable animations, increase assertion timeout, pin viewport | | Passes alone, fails in suite | Shared user/data, storage state races | Unique data per test, independent contexts | | Intermittent "element not visible" | Explicit sleep or missing wait | Auto-waiting locator assertions | | Failed on retry only | Real race or external dependency | Diagnose from trace; do not keep retrying | | Time-based assertions | Locale/timezone differences | Pin timezoneId, locale, freeze test clock | | Network flake | Third-party sandbox hiccups | Stub non-critical third parties; retry only at boundary | | Hover/animation flake | Moving targets | page.mouse.move sequences or disable motion |

Determinism levers:

use: {
  viewport: { width: 1280, height: 720 },
  deviceScaleFactor: 1,
  timezoneId: "UTC",
  locale: "en-US",
  colorScheme: "light",
  reducedMotion: "reduce",
},

• Seed random data from a per-test seed recorded in the report.
• Block or stub non-essential third parties (analytics, chat widgets, ads) to cut noise.
• Prefer one browser (Chromium) in the PR gate; run WebKit and Firefox nightly or before release.
• Budget: fail the build if the suite's flake rate exceeds the threshold, not just if tests fail (09-coverage-metrics).

Environment Strategy

| Environment | Purpose | Data | |---|---|---| | Local compose (app + DB + stubs) | Developer inner loop | Ephemeral, generated | | CI ephemeral | PR gate | Ephemeral per run, containers | | Shared staging | Nightly E2E, integration | Reset or namespaced per run | | Production | Synthetic monitoring only | Read-only journeys or dedicated tenant |

• Prefer ephemeral environments for PRs: no shared state, no queueing behind other teams.
• If staging is shared, namespace all data by run ID and enforce cleanup TTLs.
• Production synthetic tests use dedicated accounts and read paths; never mutate real customer data.
• Keep environment configuration explicit and versioned; no hand-edited staging state.

Anti-Patterns

• E2E for everything; suites of hundreds of browser tests in the PR gate.
• Sleep-based waits; waitForTimeout(5000) as a fix.
• Selectors on CSS classes or DOM structure.
• UI login in every test.
• Asserting on exact pixel positions or unmasked timestamps.
• Running against production data or real payment credentials.
• Retries set to 3+ with no flake tracking; the suite becomes a noise generator.
• Copy-pasted journeys that differ only in test data; parameterize.
• No artifact retention: failures cannot be diagnosed after the runner is gone.

E2E Checklist

• [ ] 5-15 critical journeys selected by risk, not by page count.
• [ ] Role/label selectors dominate; no structural CSS or XPath.
• [ ] Auth via storage state or API; login journey tested separately.
• [ ] Test data unique per run; cleanup owned and automated.
• [ ] Trace/video/screenshot on failure, uploaded with retention.
• [ ] Deterministic viewport, timezone, locale, and motion settings.
• [ ] Flake rate measured and gated; quarantine with owner and expiry.
• [ ] Ephemeral or namespaced environments; no production data.
• [ ] Browser matrix: Chromium per PR, cross-browser nightly/release.

Property-Based and Mutation Testing

Proving invariants over input classes with generated data, and measuring whether assertions actually constrain behavior.

Why Examples Are Not Enough

Example-based tests verify the cases the author thought of. They systematically miss:

• Boundary values the author forgot (empty, max, unicode, overflow).
• Combinations of features (e.g. discount + tax + rounding together).
• Round-trip and ordering invariants that should hold for all inputs.
• Code paths that only fail for rare structures.

Property-based testing (PBT) generates hundreds of inputs per test and validates a statement that must hold for all of them. It is the cheapest way to find the edge cases you did not enumerate.

Property-Based Testing Model

for all x satisfying preconditions:
    assert invariant(f(x))

A property is one of:

| Property kind | Statement | Example | |---|---|---| | Round-trip | decode(encode(x)) == x | JSON, protobuf, compression | | Invariant | predicate holds after operation | balance never negative | | Idempotence | f(f(x)) == f(x) | normalization, deduplication | | Commutativity | f(a, b) == f(b, a) | set union, merge | | Associativity | f(f(a, b), c) == f(a, f(b, c)) | aggregation | | Oracle | compare with a simple reference implementation | fast path vs slow path | | Metamorphic | relationship between inputs and outputs | sorting preserves multiset | | Safety/liveness | bad states unreachable; good states eventually reached | state machines |

Tooling by language

| Language | Library | Notes | |---|---|---| | Python | Hypothesis | Best-in-class shrinking and stateful testing; works with pytest | | TypeScript/JS | fast-check | Integrates with Jest/Vitest; model-based fc.commands | | Rust | proptest, quickcheck | proptest has composable strategies and shrinking | | Go | rapid (native testing integration), gopter | rapid by the Go team, fast | | Java/Kotlin | jqwik | JUnit 5 platform; strong shrinking; stateful ActionSequence | | C# | FsCheck, CsCheck | FsCheck mature; CsCheck high performance | | Elixir | StreamData | Pairs with ExUnit | | Scala | ScalaCheck | The original inspiration |

All of these are stable but release at different cadences; pin versions and verify current APIs upstream.

Writing Properties (Python / Hypothesis)

from decimal import Decimal

from hypothesis import given, strategies as st

money = st.decimals(min_value=Decimal("0"), max_value=Decimal("999999.99"), places=2)

@given(money, money)
def test_addition_is_commutative(a: Decimal, b: Decimal) -> None:
    assert add_amounts(a, b) == add_amounts(b, a)

@given(st.lists(st.integers()))
def test_sort_preserves_multiset(xs: list[int]) -> None:
    assert sorted(xs) == sorted(sorted(xs))
    assert len(sorted(xs)) == len(xs)

Practical guidance:

• Pair every property test with at least one example test for the documented cases and a regression @example for bugs found in production.
• Constrain strategies to the real domain (min_value, places, custom composite strategies) instead of filtering everything with assume — filters make generation slow and hide the domain model.
• Use @example(...) to keep permanent regressions deterministic.
• Record the seed from failures: Hypothesis prints Falsifying example and the seed; reproduce with --hypothesis-seed=<seed>.
• In CI, allow a larger max_examples, but keep PR runs bounded (for example 100-200 examples) to protect the time budget.
• derandomize=True gives reproducible runs at the cost of exploration; use it when debugging, not as the default in nightly.
• Hypothesis profiles make this explicit: ci vs dev vs debug.

Strategy design

from hypothesis import strategies as st

emails = st.builds(
    lambda user, domain: f"{user}@{domain}",
    user=st.text(alphabet="abcdefghijklmnopqrstuvwxyz0123456789._-", min_size=1, max_size=32),
    domain=st.sampled_from(["example.com", "test.example.org"]),
)

• Build domain objects with st.builds/composite so failures look like real inputs.
• Model optionality with st.none() | st.text() rather than filtering.
• Keep collections bounded (max_size) to prevent memory blowups.
• Avoid st.floats() without bounds for domain values that are never NaN or infinite.

Shrinking

Shrinking is the process of reducing a failing input to its minimal form. It is what makes a PBT failure actionable instead of a 3000-character blob.

How to make shrinking effective:

• Do not transform inputs destructively; libraries shrink the original, so derived values must be deterministic functions of it.
• Write preconditions as strategy constraints, not assume, so the search space is valid by construction.
• Never swallow exceptions in the property; a try/except: pass defeats both detection and shrinking.
• When a custom strategy generates opaque IDs, shrink the components (strings, ints) that compose them.
• If a failure does not shrink well, add a dedicated @example with the minimal case and fix the strategy separately.

Stateful / Model-Based Testing

For objects with lifecycle (caches, sessions, retries, state machines), generate sequences of operations and check invariants after each step.

from hypothesis.stateful import RuleBasedStateMachine, rule, invariant

class CartMachine(RuleBasedStateMachine):
    @rule(price=st.integers(min_value=0, max_value=10_000))
    def add_item(self, price: int) -> None:
        self.cart.add(price)

    @rule()
    def checkout(self) -> None:
        self.cart.checkout()

    @invariant()
    def total_never_negative(self) -> None:
        assert self.cart.total >= 0

• Keep a simple reference model in the test and compare after each operation (oracle pattern).
• Use stateful testing for idempotent consumers, retry logic, and any state machine whose transition table is non-trivial.
• Bound the sequence length so runs stay fast; the shrinker handles the rest.

Mutation Testing

Mutation testing answers: "if this code were wrong, would any test fail?" It modifies the program (flips >= to >, replaces + with -, negates conditions, removes statements) and re-runs the suite. A mutant that survives means a real defect class you do not detect.

Tools

| Ecosystem | Tool | Notes | |---|---|---| | JS/TS | StrykerJS | Supports Jest, Vitest, Mocha; incremental mode | | Java/Kotlin | PIT (pitest) | Fast, JUnit 5 integration, Maven/Gradle plugins | | .NET | Stryker.NET | Similar model to StrykerJS | | Rust | cargo-mutants | Crate-aware; respects workspaces; emits mutants report | | Python | mutmut, cosmic-ray | Slower; scope to hot modules | | Go | go-mutesting, gremlins | gremlins is actively maintained; verify upstream |

Metrics

| Metric | Definition | Interpretation | |---|---|---| | Mutation score | killed / (killed + survived) | Overall assertion strength | | Killed | Test failed because of the mutation | Good | | Survived | All tests passed with mutated code | Missing/weak assertion | | No coverage | No test executed the mutated line | Coverage gap | | Timeout | Tests hung on the mutant | Usually counted as killed; investigate | | Equivalent mutant | Mutation preserves semantics | False positive; ignore with reason |

Making mutation testing practical

Mutation testing is 10-100x the cost of the test suite, so treat it as a scoped gate:

• Run it on changed files only in PRs (incremental/diff mode), and on the full critical modules nightly.
• Exclude generated code, migrations, DTOs, and configuration.
• Set a threshold per module: for example >= 80% on domain modules, report-only elsewhere. Never set a repo-wide percentage that forces people to write trivia tests.
• Surviving mutants in domain logic are action items with owners; surviving mutants in logging/config are noise — suppress with a reason.
• Use it to find missing assertions, not to reach 100%. The interesting output is the list of surviving mutants, not the score.
• Combine with PBT: property tests kill many mutants that example tests miss, especially boundary mutants.

Interpreting survivors

| Survivor pattern | Meaning | Fix | |---|---|---| | Boundary flipped and tests still pass | No tests at the exact edge | Add boundary example or property | | Return value ignored | Assertion only checks "no exception" | Assert the value | | Condition removal survives | Dead branch or untested error path | Delete dead code or test the path | | String literal changed | Snapshot/assertion too loose | Pin exact contract values | | Off-by-one in loop | Untested collection sizes | Property test with variable lengths | | Short-circuit reordering | Test lacks side-effect assertion | Assert side effects explicitly |

Quality Gates

| Gate | Where | Threshold guidance | |---|---|---| | Property tests on core invariants | PR | Required for pure domain functions with stated invariants | | Regression example for each escaped bug | PR | Always; cheapest possible regression | | Diff mutation score on touched domain files | PR | >= 80% on domain modules, report elsewhere | | Full mutation run | Nightly | Trend over time; investigate regressions, not absolute | | Equivalent-mutant suppressions | Review | Each suppression has a comment and reviewer approval |

Anti-Patterns

• Property tests with no properties ("re-run the implementation and compare").
• Using assume for the majority of generated inputs; generation becomes slow and shallow.
• Ignoring the falsifying example and re-running until green.
• Mutation score as a vanity metric with repo-wide 100% targets.
• Testing getters for mutation coverage; spend the budget on branching logic.
• Suppressing mutants without a reason or owner.
• Letting PBT run unbounded in the PR gate; cap examples and time.
• Copying a property from a tutorial without adapting the strategy to the domain.

Checklist

• [ ] Core domain invariants have property tests (round-trip, idempotence, invariants, oracle).
• [ ] Strategies model the domain (bounds, composites) rather than filtering broadly.
• [ ] Every production bug adds a deterministic regression example.
• [ ] Shrinking works: failures report minimal inputs.
• [ ] Stateful testing covers at least one lifecycle object in the codebase.
• [ ] Mutation testing runs incrementally on changed domain files in PRs.
• [ ] Surviving mutants are triaged with owners; suppressions are justified.
• [ ] PR property runs are time-bounded; nightly runs explore more.

Contract Testing

Verifying that services can talk to each other without running a full end-to-end environment: consumer-driven contracts, schema compatibility, provider verification, and versioning.

The Problem Contracts Solve

Integration between two services breaks when the provider changes the shape or semantics of an interface in a way the consumer did not expect. Full E2E tests catch this only if every consumer-provider pair is deployed together, which does not scale. Contract tests pin the interface from both sides, independently, in each service's own pipeline.

• Consumer-driven contract (CDC) — the consumer declares what it needs; the provider verifies it still satisfies every consumer's expectations.
• Schema-based contract — a shared schema (OpenAPI, protobuf, Avro, JSON Schema) is the source of truth and both sides verify against it.

Choose CDC when the interface evolves with consumer needs and teams negotiate changes. Choose schema-based when the interface is a published protocol with many consumers and a governance process (Kafka topics, public APIs).

Contract Layers

| Layer | Artifact | Tool examples | Verifies | |---|---|---|---| | Synchronous HTTP | Pact file, OpenAPI diff | Pact, schemathesis, openapi-diff | Requests/responses, status codes | | Async messaging | Message pact, Avro schema | Pact message, Schema Registry | Payload shape, key, headers | | RPC | Protobuf descriptor | buf breaking, grpcurl | Field evolution, reserved tags | | GraphQL | Schema + operations | GraphQL Inspector | Operation validity, nullability | | Events/database CDC | Schema registry compatibility | Confluent/Redpanda registry | Backward/forward compatibility |

Consumer-Driven Contracts with Pact

Workflow

1. Consumer test drives the consumer against a mock provider, recording interactions (request matchers, response examples) and asserting the consumer handles them.
2. The test publishes the pact to a broker (Pact Broker, PactFlow, or an artifact store) tagged with the consumer version and branch.
3. Provider verification replays each pact against the real provider, with provider states setting up data, and publishes verification results.
4. The broker computes the can-i-deploy decision: may this version ship together with the versions currently in the target environment.

# consumer side (pytest + pact-python)
def test_get_order(pact):
    (
        pact.upon_receiving("a request for order 42")
        .given("order 42 exists")
        .with_request("GET", "/orders/42")
        .will_respond_with(200)
        .with_body({"id": 42, "status": "paid", "total": {"amount": 1999, "currency": "USD"}})
    )
    with pact.serve() as srv:
        client = OrderClient(str(srv.url))
        order = client.get(42)
        assert order.status == "paid"

# provider side (JVM example)
./gradlew pactVerify -Ppactbroker.url=... -Ppactbroker.consumers=web,partner-api

Key rules:

• Contracts are consumer expectations, not provider documentation. Only assert fields the consumer actually reads and depends on.
• Use matchers, not exact values, for volatile fields (IDs, timestamps, amounts).
• Use exact values for enums and status codes the consumer branches on.
• One pact per consumer-provider pair (or per interaction group). Do not bundle unrelated consumers into one contract.
• Provider states (given(...)) are named, documented, and implemented as test fixtures on the provider, not ad-hoc scripts.

What a contract should assert

| Assert | Example | Why | |---|---|---| | Path and method | GET /orders/42 | Routing contract | | Required headers | Accept, auth scheme, content type | Middleware assumptions | | Status codes per case | 200, 404, 409 | Branching behavior | | Body fields used by consumer | id, status, total | Field presence and types | | Nullability | coupon may be null | Prevents null-pointer incidents | | Error envelope | {code, message} shape | Error handling paths | | Pagination shape | cursor + items array | Loop termination |

Do not assert fields the consumer ignores; that makes the provider's internal evolution needlessly constrained.

Provider Verification

Provider verification runs the recorded pacts against the real provider:

• Data setup through provider states, ideally reusing integration fixtures (03-integration).
• Run against an in-process or containerized provider with a real or isolated database; never against production.
• Publish results to the broker with the provider version and branch/tag.
• A provider that fails verification must not deploy; this is the enforcement point.

Common pitfalls:

• Provider state names drift from implementation and silently stop setting up data.
• Verification runs against a provider configured differently from production (feature flags, serialization options), giving false confidence.
• Pacts accumulate stale interactions from deleted consumer code; expire them via broker policies.

Schema Compatibility

When a schema is the contract, enforce evolution rules mechanically.

| Change | Backward compatible | Forward compatible | Full | |---|---|---|---| | Add optional field | Yes | Yes | Yes | | Add required field | No | Yes | No | | Remove optional field | Yes | No | No | | Remove required field | No | No | No | | Widen a type (int -> long/string) | Yes | Yes | Yes | | Narrow a type (string -> int) | No | No | No | | Rename a field without aliasing | No | No | No | | Change enum member meaning | No | No | No | | Add enum member | Yes (consumers must tolerate) | Yes | Yes | | Reorder fields in JSON | Yes | Yes | Yes (unless positional) | | Reorder fields in Avro binary | No | No | No |

Rules:

• Treat compatibility as a CI check, not a review convention: protobuf buf breaking, Avro registry compatibility levels, OpenAPI diff with fail-on-breaking.
• Additive changes still require consumers to tolerate unknown fields; document the tolerant-reader rule.
• Deprecate before removing: mark deprecated, expose usage telemetry, announce, then remove after the agreed window.
• Version the interface only when compatibility cannot be preserved: URL version (/v2), media type, topic name, or protobuf package. Versioning is a last resort, not the default evolution path.

Contracts vs E2E

| Aspect | Contract tests | E2E tests | |---|---|---| | Scope | Interface semantics | Whole system behavior | | Runtime | Seconds | Minutes | | Environment | Per-service, no shared env | Full composed environment | | Catches | Interface drift, nullability, routing | Cross-system timing, auth config, UI | | Misses | Data semantics beyond stated cases, flows across 3+ services | Rare interactions not in the journey set | | Failure locality | Points at the pair | Points at the system |

Use both, but with different jobs: contracts guard interfaces on every PR; a handful of E2E journeys guard the assembled product (04-e2e). Do not use contracts as a substitute for E2E or vice versa.

Versioning and Deployment Safety

• Tag consumer/provider versions with both application version and environment/branch so the broker can answer "can version A deploy with version B?" per environment.
• Gate deployments on can-i-deploy in the pipeline, and record record-deployment after success.
• Contracts live beside the code that defines them; breaking a contract is a reviewed change with a migration note.
• When multiple versions run concurrently (blue/green, canary), contracts must hold for the deployed pair; use the broker's environment matrix rather than assuming one version.
• Keep a compatibility test for the oldest supported consumer (N-2 or the fleet's actual floor).

Anti-Patterns

• Provider-side "contract tests" that assert the provider's own mock expectations.
• Recording every field in the response; the provider becomes frozen.
• Exact-match assertions on timestamps/UUIDs; unstable pacts that fail randomly.
• Contract tests that depend on a shared staging environment.
• Treating a green OpenAPI diff as proof of semantic compatibility; meaning changes are invisible to schemas.
• Publishing pacts from only one consumer and assuming full coverage.
• Running provider verification manually, after deploy, or never.
• Using contracts to test business rules that belong in integration tests.

Contract Testing Checklist

• [ ] Every consumer-provider pair that changes independently has a contract.
• [ ] Consumer tests publish pacts with version and branch tags.
• [ ] Provider verification runs in the provider pipeline against a real instance with provider states.
• [ ] can-i-deploy gates deployment; record-deployment runs after success.
• [ ] Pacts expire; stale interactions are removed.
• [ ] Schema evolution rules are enforced by CI tooling, not convention.
• [ ] Nullability and error envelopes are explicitly contracted.
• [ ] Contracts target interface semantics; E2E covers assembled behavior.

Test Data Management

Factories, builders, fixtures, seeding, PII handling, per-run uniqueness, and database snapshots.

Principles

1. Tests own their data. No test depends on rows it did not create. Shared mutable fixtures are the root of half of all cross-test flake.
2. Generate, do not copy. Production data in tests is a compliance incident waiting to happen. Generate synthetic data that is structurally faithful.
3. Unique per run. Every run gets a namespace (run ID) so parallel CI jobs, retries, and repeated local runs never collide.
4. Minimal but sufficient. Data should contain exactly the fields the test reads. Wide sparse objects hide which fields matter.
5. Deterministic where asserted, random where identity is irrelevant. Fix values that the test asserts on; randomize values that only need to be unique.

Factories and Builders

Factory pattern

Factories produce valid objects with sane defaults and per-field overrides. Prefer them over shared fixture rows.

from dataclasses import dataclass, replace

@dataclass(frozen=True)
class User:
    id: str
    email: str
    plan: str = "free"
    created_at: str = "2026-01-01T00:00:00Z"

def make_user(**overrides) -> User:
    base = User(id=f"usr_{next(counter)}", email=f"user{next(counter)}@example.test")
    return replace(base, **overrides)

Rules:

• Defaults must be valid; a factory that produces invalid objects by default forces every caller to fix it.
• Overrides are explicit in the test body, making the relevant variation visible.
• Factories are plain functions or typed builders; avoid magic global registries that make data sources invisible.
• Never use a factory to hide the setup that is the point of the test.

Builder pattern

Use builders when construction is sequenced or when many optional fields must be varied:

order = (
    OrderBuilder()
    .with_item("sku-1", qty=2, price="9.99")
    .with_coupon("SAVE10")
    .shipping_to("DE")
    .build()
)

Builders shine for HTTP request bodies, protobuf/JSON payloads, and complex aggregates. Keep the fluent chain shallow (<= 5 steps) before splitting into named scenarios.

Factory libraries by ecosystem

| Ecosystem | Library | Notes | |---|---|---| | Python | factory_boy, polyfactory, faker + plain functions | factory_boy with SQLAlchemy/Django; polyfactory for pydantic models | | TypeScript | fishery, @faker-js/faker, zod fixtures | fishery for typed factories | | Ruby | FactoryBot | Mature; beware association graph explosion | | Java | Instancio, Easy Random | Instancio supports object graph generation with overrides | | .NET | AutoFixture, Bogus | AutoFixture for anonymous values; Bogus for realistic fakes | | Go | gofakeit, plain builders | Idiomatic Go favors explicit builders | | Rust | fake, proptest strategies | fake for rand-derived values |

Verify current versions upstream; the library ecosystems move faster than this table.

Fixtures: Scope and Discipline

| Fixture kind | Example | Lifetime | Rule | |---|---|---|---| | Ephemeral value | record, DTO | Per test | Default; no cleanup needed | | Local resource | temp dir, container | Per test/module | Teardown guaranteed, exception-safe | | Session resource | DB engine, browser | Per suite | Immutable or explicitly reset | | Reference data | countries, currencies, permissions | Per environment | Seed via migrations; never truncate | | Golden data | expected report output | Per repository | Reviewed changes only |

• Fixtures provide lifecycle, factories provide data. Do not encode business data in fixtures; that couples unrelated tests.
• Hidden autouse fixtures that create data are an anti-pattern: tests must declare what they need.
• Session-scoped mutable state must have a documented reset path or the tests using it must be read-only.

Seeding Reference Data

• Seed reference/lookup tables (currencies, countries, plans, permission catalogs) through migrations or an idempotent seed script shipped with the application.
• Tests must not depend on seed ordering or auto-increment IDs for business meaning; look up by natural key.
• Keep seed data versioned with the schema; drift between seeds and migrations breaks integration suites in confusing ways.
• For multi-tenant products, seed shared catalogs once and tenant-specific data per test.

PII and Anonymization

Never in tests

• Real names, emails, phone numbers, addresses, government IDs, payment instruments, or health data.
• Production tokens, cookies, or API keys; even expired ones leak infrastructure details.
• Screenshots or traces containing real customer data.

Anonymization pipeline (when production-derived data is unavoidable)

1. Inventory columns and events that can contain PII; tag them in the schema or catalog.
2. Apply deterministic pseudonymization (HMAC with a rotated key) so joins still work.
3. Generalize or bucket quasi-identifiers (birth date -> age band, postal code -> region).
4. Validate with automated PII scanning (regex plus a detector service) before load.
5. Restrict access to the anonymized dataset; log who reads it.
6. Re-anonymize from scratch rather than mutating a prior copy, so removals and consent status flow through.

Rules:

• Prefer synthetic generation over anonymization; it removes the leak surface entirely.
• Synthetic data must preserve distributions and referential integrity, or tests pass for unrealistic reasons.
• Use reserved domains for generated emails (@example.com, @example.test) and documentation phone ranges so accidental sends fail safely.
• Never send test emails/SMS/payments through production providers; use sandboxes or local sinks.

Unique Data per Run

Patterns for collision-free parallel execution:

| Target | Strategy | |---|---| | Database rows | Unique keys prefixed with run ID: tenant = f"t-{run_id}-{n}" | | Emails | user+{run_id}-{n}@example.test | | Object storage | Bucket/prefix per run with lifecycle expiry | | Message topics | Topic per run, auto-deleted on teardown | | External sandboxes | Idempotency keys containing run ID | | Feature flags | Unique flag key per run; delete on teardown |

• Generate run_id once (CI job ID or UUID) and thread it through fixtures.
• Deterministic within a run, unique across runs: derive values from (run_id, logical_name) so retries of the same test reuse the same data and do not accumulate garbage.
• Cleanup strategy: delete by run prefix at the end of the job, plus a TTL sweep in the environment for crashed jobs.
• Assert-uniqueness: a test that inserts the same natural key twice should be an explicit intended scenario, not an accident.

Database Snapshots and Templates

When seeding is expensive (large catalogs, complex graphs), use a prepared template:

| Technique | How | When | |---|---|---| | Template database | Create + migrate + seed once; CREATE DATABASE x TEMPLATE y per worker | Postgres, per-worker isolation | | Dump/restore | pg_dump/pg_restore or schema snapshot | Cross-engine, versioned fixtures | | Migration + seed script | Run migrations, then idempotent seeds | Always; the source of truth | | Masked prod clone | One-time masked extract | Manual exploratory testing only, not CI |

Rules:

• Templates must be rebuilt whenever migrations change; stale templates cause test suites to test the wrong schema. Hash the migration directory and rebuild on mismatch.
• Never check binary dumps into the repository without a documented regeneration process.
• Prefer schema-level creation (migrations) plus programmatic seeds over opaque dumps; they diff, review, and version.
• Snapshot restore needs to reset sequences and ownership to avoid privilege drift between environments.

Anti-Patterns

• Shared "seed once" global data mutated by tests; hidden ordering dependencies.
• Real production extracts in CI (even "just for debugging").
• Random data with seeds that change every run, breaking reproducibility and assertions.
• Factories with dozens of fixed magic values; tests that vary one field but assert none.
• Massive YAML fixtures containing every possible field; changes ripple everywhere.
• Cleanup that runs only on success; crashed runs leave garbage that poisons later runs.
• Using auto-increment IDs in assertions; they vary with execution order.
• Emails sent to real addresses from test suites.

Test Data Checklist

• [ ] Factories/builders with valid defaults; tests override only what they assert.
• [ ] No production PII, tokens, or screenshots anywhere in the repo or CI artifacts.
• [ ] Run-scoped uniqueness for rows, topics, buckets, tenants, and sandboxes.
• [ ] Reference data seeded via migrations or idempotent script, keyed by natural keys.
• [ ] Cleanup by run prefix plus TTL sweep; exception-safe teardown.
• [ ] Snapshot/template rebuild is automated and triggered by migration hash changes.
• [ ] External services use sandbox credentials or local sinks only (03-integration).
• [ ] Data volume is bounded per test; suite runtime stable as factories grow.

Flaky Tests and CI Test Infrastructure

Detecting and quarantining flake, rerun policy, parallelism and sharding, timeouts, environment isolation, and ownership.

Why Flake Is the Priority

A suite is trusted only if red means broken. Every flaky test erodes that trust: teams rerun instead of reading, real regressions hide behind noise, and CI minutes burn. Flake rate is a first-class quality metric, tracked with the same seriousness as coverage (09-coverage-metrics).

Definitions used throughout:

• Flaky — a test that produces different results across runs of the same commit.
• Flake rate — flaky test executions / total executions over a window (per test and per suite).
• Quarantine — moving a flaky test out of the blocking suite into a tracked, owned, non-blocking suite with an expiry date.

Detecting Flake

| Technique | What it catches | Notes | |---|---|---| | Rerun failures once and record | Intermittent tests | The record is the point; parsing JUnit XML required | | Same-commit reruns in nightly | Order/timing dependence | Repeat the full suite 2-3 times | | Test-level metrics over time | Gradual decay | Store per-test pass rate, duration, retries | | Random order / seed sweep | Hidden coupling | pytest-randomly, Vitest shuffle, JUnit ordering | | Parallelism | Shared resource collisions | Each worker must have isolated DB/ports | | Nightly long runs (property, E2E) | Rare-input failures | Higher example counts, more browsers | | Static heuristics | Known smells | sleep, fixed ports, shared temp paths, wall clock |

Tooling: JUnit XML is the common denominator; ingestion into the CI provider's test analytics or a small database gives per-test history. Datadog CI Visibility, BuildPulse, GitHub test reports, and codecov's flake detection all consume the same format. Pick one; the analysis matters more than the brand.

Quarantine Policy

1. Detect — CI marks a test flaky when it fails then passes on a same-commit rerun, or when its historical pass rate drops below a threshold (for example < 99%).
2. Quarantine immediately — an automated bot moves the test to a quarantine suite, opens an issue, and assigns the CODEOWNER. Blocking CI returns to green-fast.
3. Budget — quarantine has a hard expiry (for example 14 days) and a maximum size (for example 2% of tests). Exceeding the budget fails the quality metrics review, not the per-PR build.
4. Fix or delete — at expiry: fix with a root-cause note, or delete. Deleted flaky tests that covered real behavior must be replaced by a lower-layer test (01-strategy).
5. Never re-block the gate with an unfixed flaky test — that trains reruns.

# quarantine metadata example
quarantine:
  - test: "e2e/checkout.spec.ts::applies gift card"
    owner: payments-team
    issue: PAY-1234
    quarantined_at: 2026-09-01
    expires_at: 2026-09-15
    reason: "race on inventory reservation"
    replacement: "unit: inventory reservation is atomic"

• Quarantined tests still run (non-blocking) so signal is not lost and fixes can be verified.
• Report quarantine size and age in the weekly quality dashboard.
• No silent @skip; skips without issue and owner fail a lint check.

Rerun Policy

Reruns are diagnostic, not a fix.

| Context | Allowed | Policy | |---|---|---| | PR gate | 1 automatic retry for infrastructure-classified failures only | Every retry logged; flake counted | | Nightly E2E | 1-2 retries with artifact capture per attempt | Failure after retries becomes an issue | | Local dev | Manual, unlimited | Developer investigates before pushing | | Test-flight branch | Full reruns | Used to reproduce and confirm fixes |

• Classify failures: assertion (deterministic, no retry), timeout (retry once), infra (runner/network, retry once), unknown (capture artifacts, no retry until triaged).
• Count every retry as a data point; a test that retries frequently is flaky even if it never finally fails.
• Do not let retries mask a real intermittent bug in production code. Many "flaky tests" are actually race conditions in the system (04-e2e flake table).
• Playwright/Pytest/Vitest all support retries; configure them centrally so a single policy applies.

Parallelism and Sharding

Parallelism within a run

• Ensure per-worker isolation: database schema/name per worker, unique ports (ephemeral port assignment, not fixed 5432 collisions), unique temp dirs, unique topic/bucket names (07-test-data).
• Worker-aware fixtures: derive names from PYTEST_XDIST_WORKER, Playwright's parallelIndex, or a UUID per worker.
• Cap concurrency by resource, not CPU count alone: DB connection limits and container memory are the real bottlenecks.
• Keep tests independent: no shared login files written concurrently, no global mutable cache.

Sharding across machines

| Sharding strategy | How | When | |---|---|---| | By duration (dynamic) | Historical timings split into equal buckets | Default for large suites | | By file/package | Round-robin files | When tests are file-scoped | | By tag | Critical vs extended | Separate the PR gate from nightly | | By historical failure | Group known-slow suites separately | Avoid stragglers |

• Record durations from JUnit XML; a sharder without timings produces unbalanced runs.
• Keep a fallback: if a shard fails to report, fail the job rather than silently dropping tests.
• Verify total coverage: a shard plan must cover every test exactly once; add a meta-test for the collection step.
• Deterministic ordering within a shard helps debugging; cross-shard ordering is never guaranteed.

Timeouts

Timeouts must exist at every layer, from smallest to largest:

| Layer | Example | Guidance | |---|---|---| | Unit test | 5 s default | pytest-timeout, Vitest testTimeout | | Integration test | 30-60 s | Container startup excluded or measured separately | | E2E test | 60-120 s | Playwright per-test timeout; action timeout shorter | | Job | 15 min PR, 2 h nightly | CI-level; kills hung runs | | Network call in test | 2-5 s | Lower than production defaults to fail fast | | Global suite guard | Job timeout - 10% | Prevent zombie runners |

Rules:

• Never use a timeout as the correctness mechanism for waiting; poll with a deadline for the condition (expect(...).toPass, custom retry helper).
• Turn "hang forever" into an explicit failure with a diagnostic: which test, which step.
• Server-side timeouts under test (HTTP clients, queues) should be asserted, not just configured.
• Increase timeouts only with a recorded reason; a timeout bump is often masking a resource problem.

Environment Isolation

• Ephemeral > shared. Per-PR environments avoid cross-team interference entirely. If shared staging is unavoidable, namespace every resource by run ID and enforce cleanup TTLs.
• Pin external dependency versions: container images by digest or tag, browser versions, database versions, locale/timezone, seeds.
• Pin locale and timezone explicitly (TZ=UTC, locale=en-US) — a runner default change breaks date assertions overnight.
• Freeze or inject clocks rather than relying on runner time (02-unit).
• Avoid port collisions: bind to port 0 and read the assigned port; never hardcode.
• Cache carefully: dependency caches are safe; build caches keyed on environment can hide drift. Include lockfile hashes in cache keys.
• Treat runner images as mutable infrastructure and version them; record the image digest in test reports.

Ownership and Process

• Owner per suite in CODEOWNERS; the code owner receives failure notifications.
• On-call for test infrastructure — the platform team owns runners, containers, and sharding; product teams own their assertions.
• Flake review in the weekly quality meeting: top flaky tests, quarantine size, mean time to fix, flake rate trend.
• Post-incident rule: if a production incident would have been caught by a test that was quarantined or deleted, the quarantine policy gets reviewed.
• No blame: flake is a system property, tracked and fixed, not an individual failing.

Anti-Patterns

• Retry counts of 3+ applied globally, with no recording of which tests retried.
• Quarantine without owner or expiry; a growing graveyard of skipped tests.
• Sharding without duration data; one shard runs 20 minutes while others run 3.
• Fixed ports, fixed temp paths, fixed tenant IDs across parallel workers.
• Increasing timeout after timeout instead of investigating the slow/hung path.
• Running E2E against shared staging with mutable global fixtures.
• Treating a "flaky infrastructure" excuse as resolution; infrastructure flake must be measured too.
• Hiding collection errors by allowing "no tests found" to pass.

CI Health Checklist

• [ ] Same-commit reruns detect flake; flake rate is tracked per test and suite.
• [ ] Quarantine bot moves and tracks tests; owner, issue, and expiry required.
• [ ] Retry policy is explicit and excludes deterministic assertion failures.
• [ ] Workers are isolated (DB, ports, temp, topics) and concurrency is resource-bounded.
• [ ] Sharding uses recorded durations; shard completeness is verified.
• [ ] Timeouts exist at test, action, and job layers; no unbounded waits.
• [ ] Environments are ephemeral or run-namespaced; images, locale, TZ, and seeds pinned.
• [ ] CODEOWNERS covers test suites; infra on-call exists.
• [ ] "No tests found" and collection failures fail the build.

Coverage and Quality Metrics

Measuring what tests actually verify, setting honest gates, and building dashboards that drive improvement rather than gaming.

Coverage Types

| Metric | Measures | Blind spot | |---|---|---| | Line/statement | Executed lines | Branches, assertions | | Branch | Both outcomes of conditionals | Assertion strength, data interactions | | Function | Called functions | Untested inputs within function | | Path | Combinations through a function | Grows exponentially; rarely usable | | MC/DC | Condition independence (safety-critical) | Expensive tooling; overkill for most apps | | Diff/patch | Coverage of changed lines | Untested old code on the same path | | Requirement | Mapped tests to requirements | Manual mapping cost |

Coverage tells you what code ran, never whether the result was checked. A suite with 95% line coverage and no assertions has the same defect-detection power as no tests at all (05-property-mutation measures the missing piece).

Why Coverage Is Easy to Game

| Behavior | Metric effect | Actual quality | |---|---|---| | Assertion-free "smoke" tests | Coverage up | Unchanged | | Testing trivial getters | Coverage up | Negligible | | # pragma: no cover on hard code | Coverage up | Hidden risk | | Excluding modules from the tool config | Coverage number up | Hidden risk | | One giant test calling everything | Coverage up | Failures ambiguous | | Snapshot tests with auto-update in CI | Coverage up | Tautological verification | | Duplicating production logic in tests | Branches covered | Independent bugs cancel out |

Because targets are easy to game, coverage should be a diagnostic (find untested critical code) with a low, honest gate, not a primary goal.

Setting Coverage Gates

Guidance that survives contact with real teams:

• Use branch coverage with a modest global floor (for example 70-80%) and strict review of exclusions.
• Prefer diff coverage on the PR (for example >= 80% of changed lines) plus component- level floors. Global numbers move too slowly to guide daily work and are dominated by legacy code.
• Set component-specific targets by risk: domain and payment logic 90%+, glue and adapters 60%, generated code excluded.
• Never allow coverage to drop on touched modules: ratchet per directory over time.
• Exclude only with reasons: generated files, platform shims, if TYPE_CHECKING, vendored code. Every exclusion is reviewed in the config file's own PR.
• Treat "coverage down 0.1%" PR failures as a false-positive generator; use rounded thresholds and allow documented exceptions.

# coverage.py / pytest-cov style
[tool.coverage.run]
branch = true
source = ["src"]

[tool.coverage.report]
fail_under = 75
show_missing = true
exclude_also = [
  "if TYPE_CHECKING:",
  "raise NotImplementedError",
  "class .*Protocol.*:",
]

# diff coverage gate in CI (concept; use your coverage tool's diff mode)
quality-gate:
  diff-coverage-min: 80
  component-floors:
    src/domain: 90
    src/adapters: 60

Diff Coverage in Practice

Diff coverage answers: "of the lines this PR changes or adds, how many are executed by tests in the repo?" It is the most actionable coverage gate because it is local to the change and cannot be diluted by legacy code.

Workflow:

1. Run the suite with coverage and emit machine-readable output (coverage json, cobertura, lcov).
2. Compute coverage restricted to the PR diff (tools: diff-cover, codecov patch status, Sonar new-code mode).
3. Fail the check when below the threshold; allow explicit waivers for pure refactors where lines moved without change.
4. Require an explanation when new error-handling branches are uncovered — those are exactly where defects hide.

Caveats:

• Renames and moves inflate or deflate the diff; restrict to added/modified lines.
• Generated or vendored diffs must be excluded before computing.
• Do not gate on deleted lines; deletion decreases code and should not require tests.

Mutation Score as a Stronger Signal

Mutation score measures the fraction of seeded faults the suite detects, so it approximates assertion strength.

| Signal | Weak suite | Strong suite | |---|---|---| | Line coverage | 90% | 90% | | Mutation score | 40% | 85% | | Flake rate | 5% | < 0.5% |

How to use it without drowning:

• Run incrementally on changed files in PRs; run full on core modules nightly (05-property-mutation).
• Gate on domain modules with a meaningful threshold (for example >= 80%), report-only elsewhere.
• Track survivors as a backlog of specific missing assertions, not a global percentage.
• Suppress equivalent mutants with a written reason; review suppressions like code.

Runtime and Flake Metrics

Test quality is not only coverage:

| Metric | Target guidance | Purpose | |---|---|---| | PR gate duration p95 | <= 15 min | Developer feedback loop | | Fast suite duration | <= 5 min | Inner loop viability | | Flake rate per suite | < 1% | Trust in red | | Quarantine size | < 2% of tests, all with expiry | Avoid graveyards | | Mean time to fix flake | <= 14 days | Enforcement of ownership | | Retry rate | < 2% of executions | Hidden instability | | Test maintenance ratio | ~10-20% of eng time | Sustainability |

A suite that is fast and reliable but shallow (low mutation score) and a suite that is deep but flaky are both unhealthy; dashboards should show both axes.

DORA and Delivery Metrics Alongside Tests

Test metrics answer "is our verification good?"; DORA answers "is our delivery system good?" Track both, understanding the causal links.

| Metric | Definition | Test-suite relationship | |---|---|---| | Deployment frequency | Deploys to production per period | Fast CI enables smaller, more frequent releases | | Lead time for changes | Commit to production | Test runtime is a leading component | | Change failure rate | Deploys causing degradation | Low change failure rate with high coverage/mutation suggests tests work | | Failed deployment recovery time | Time to restore service | Fast, trusted suites shorten recovery and enable reverts |

Use DORA trends to calibrate test investment: if lead time is dominated by CI waits, the fix is test speed and sharding, not more tests. If change failure rate is high while coverage is high, the tests are shallow — raise assertion strength with mutation and property testing, not coverage.

Dashboards That Do Not Mislead

Rules for quality dashboards:

1. Show trends, not just snapshots. A 72% coverage that rose from 60% is healthy; a flat 85% with rising flake is not.
2. Show distributions. Per-module coverage or mutation, not one global average that hides a 10% critical module.
3. Pair every headline with a guardrail. Coverage next to flake rate and gate duration; DORA next to coverage and mutation.
4. Make ownership visible. Color or group by owner so the right team sees their row.
5. Never show individual rankings. Per-person test metrics create gaming and blame; aggregate by team or component.
6. Define each metric on the dashboard with its formula and data source; undefined metrics get re-litigated monthly.
7. Alert on regressions, not absolute values (coverage dropped > 2 points after a merge; flake rate doubled week over week).
8. Retire metrics nobody acts on. A dashboard with 40 panels is wallpaper; keep fewer than 10 actionable signals.

Anti-Patterns

• 100% coverage mandates; teams write assertion-free tests, and the number is meaningless.
• Comparing coverage across languages or repositories as a team performance metric.
• Ignoring flake while raising coverage; the suite becomes both slow and untrusted.
• Using coverage as a performance review input.
• Hiding critical modules behind exclude lists to pass the gate.
• Dashboards that count test cases rather than risk covered; 5000 trivial tests is not better than 300 meaningful ones.
• Treating DORA metrics as individual performance scores rather than system signals.
• Chasing path coverage or MC/DC outside safety-critical or regulated contexts.

Metrics Checklist

• [ ] Coverage collected in branch mode for source code only.
• [ ] Global floor modest; per-component floors risk-weighted.
• [ ] Diff coverage gates every PR; waivers are explicit.
• [ ] Exclusions are reviewed, reasoned, and few.
• [ ] Mutation score gates domain modules incrementally; survivors triaged.
• [ ] Flake rate, retry rate, and quarantine size tracked with owners and expiries (08-flaky-ci).
• [ ] PR gate duration tracked against the 15-minute budget.
• [ ] DORA metrics reported alongside test metrics to explain causal links.
• [ ] Dashboard shows trends, distributions, and owner groupings; under 10 actionable panels.
• [ ] Every gate has a stated purpose and an owner; useless gates are removed.

AI-Assisted Test Generation

Using LLMs to draft tests safely: review requirements, adversarial coverage, prompt patterns, and the failure modes that make generated tests dangerous.

What LLMs Are Good and Bad At

| Task | LLM strength | Risk | |---|---|---| | Drafting happy-path example tests | High productivity | Shallow coverage, over-fitting to visible code | | Enumerating edge cases | Good with explicit prompting | Invents domain rules that do not exist | | Writing factories/builders | Strong, mechanical | Overly permissive or invalid defaults | | Translating existing tests between frameworks | Strong | Silent behavior differences between frameworks | | Property identification | Moderate | Proposes tautologies, re-implements the code | | Finding missing branches | Useful as a reviewer | Hallucinated APIs and assertions | | Diagnosing flaky tests from artifacts | Moderate | Confident wrong conclusions; verify evidence | | Writing adversarial/security tests | Good when primed | May generate unsafe payloads without sandboxing |

Use LLMs to accelerate drafting and enumeration, never to replace human ownership of what a test asserts and why.

Non-Negotiable Review Requirements

1. Every generated test is reviewed like production code. Same PR, same checklist, same CODEOWNERS.
2. Run it. Do not merge untested tests. Verify they fail when the behavior is wrong: mutate the code or temporarily break the assertion to confirm the test can go red (05-property-mutation).
3. Verify every API and import against the real codebase and the installed library version. Hallucinated methods are the most common defect.
4. Human sign-off on assertions. The reviewer must be able to explain what defect each assertion catches.
5. No generated test may touch production data, real credentials, or live third-party services.
6. Label provenance when the process requires it (a commit trailer or PR note), so audits can trace AI-generated code.

Failure Modes of Generated Tests

| Failure mode | Symptom | Detection | |---|---|---| | Hallucinated API | Test references methods/classes that do not exist | Compiler/type checker, test run | | Tautological assertion | assert result == result, asserts on the mock | Mutation testing; careful reading | | Re-implementation | Test re-computes expected value with the same logic as production | Independent expected value or fixed example | | Over-mocking | Test passes regardless of production bugs | Remove mocks where possible; integration layer | | Happy-path bias | No error, boundary, or permission cases | Coverage of error branches; mutation survivors | | Non-deterministic data | Uses datetime.now(), random without seed | Determinism review, run in CI repeatedly | | Wrong framework idioms | Jest patterns in Vitest, pytest fixtures misused | Lint/runner failures; framework review | | Silent assertion weakening | assert x is not None where equality mattered | Mutation score drop | | Copy-paste naming | Ten tests with near-identical bodies | Deduplicate via parameterization | | Sensitive data invention | Realistic PII or tokens in fixtures | PII scanning in CI (07-test-data) |

The cheapest defense is the existing pipeline: types, linters, the test run itself, and mutation testing. The human review catches the semantic failures the pipeline cannot.

Prompt Patterns That Work

Enumerate first, write second

Given this function and its docstring:
1. List the input classes and boundaries.
2. List the error conditions and how they surface.
3. Only then write one parameterized test per class.
Do not invent domain rules not present in the code or docstring.

Separating enumeration from code generation makes wrong assumptions visible before they are buried in test bodies.

Anchor on the specification

Here is the OpenAPI schema / RFC section / ADR for this behavior.
Write tests only for behavior specified there.
For anything not specified, list open questions instead of guessing.

Adversarial pass

Review this existing test file as a skeptical reviewer.
For each test, name the defect it would catch.
Then list behaviors with no test: error paths, boundaries, permissions,
concurrency, idempotency. Output a prioritized list; do not write code yet.

Property extraction

Identify properties that must hold for all inputs (round-trip, idempotence,
invariants, ordering). For each, state it formally and propose a strategy.
Flag any property that merely restates the implementation.

Regression from a bug report

Here is the bug report with reproduction steps and the fix diff.
Write a test that fails on the pre-fix code and passes on the fixed code.
The test must assert the user-visible outcome, not the implementation detail
that changed.

This is one of the highest-value uses of LLMs: converting incident descriptions into deterministic regression tests, then verifying the red/green transition.

Flake diagnosis

Here is the failing log, the trace summary, and the test source.
List plausible causes ranked by evidence. For each, state what evidence would
confirm or refute it. Do not propose a fix until the cause is identified.

The constraint "do not fix until diagnosed" prevents the common AI-suggested sleep/retry patch that hides the race.

Adversarial and Security Cases

LLMs are effective at enumerating hostile inputs; treat their output as candidates, not production payloads.

• Generate boundary and injection candidates (long strings, unicode edge cases, mixed encodings, nested structures, negative numbers, overflow) for validation logic.
• Keep security tests in a quarantined fixture set: no live exploits, no payloads against systems you do not own.
• For authz, generate the full role x action x resource matrix and assert denials explicitly; this is where models find gaps humans forget.
• Fuzz-like generation belongs in property-based testing with a seed, not in ad-hoc generated strings (05-property-mutation).

Workflow and Governance

Recommended pipeline:

1. Draft — LLM proposes test skeletons and case lists, human picks the important ones.
2. Constrain — project conventions injected via prompt or rules file: framework, fixtures/factories, naming, no sleeps, no real network, unique per-run data.
3. Verify locally — run, type-check, lint. Delete anything that cannot run.
4. Prove sensitivity — break the implementation intentionally and confirm the test fails; restore.
5. Review — human reviewer checks assertions, determinism, layering, and data hygiene.
6. Merge — same gate as handwritten tests; gate coverage/mutation applies.
7. Monitor — track AI-drafted tests for flake and mutation survival separately at first; a spike indicates a prompting or review gap.

Governance decisions to make explicitly:

• Whether AI-generated tests may be committed and under what provenance labeling.
• Which repositories are off-limits (regulated, or containing sensitive logic).
• Whether prompt/context may include proprietary source (check provider data policies).
• Who owns an AI-drafted test once merged — the code owner, same as any test.

Anti-Patterns

• Merging generated tests without running them.
• Accepting assertions you cannot explain.
• Generating hundreds of shallow tests to raise coverage (09-coverage-metrics).
• Letting the model write the expected value by re-running the implementation.
• Prompting with production code containing secrets or customer data.
• Using generated security payloads against live systems.
• Trusting a model's "these tests pass" without executing them.
• Replacing human test design with prompting; ownership and judgment stay human.

Checklist

• [ ] Generated tests run, type-check, and lint in the same pipeline as handwritten tests.
• [ ] Every assertion maps to a stated defect class; reviewer can explain each one.
• [ ] APIs, imports, and fixtures verified against the real codebase and installed versions.
• [ ] Sensitivity verified: at least one intentional break makes each new test fail.
• [ ] Determinism verified: no wall clock, unseeded randomness, real network, or shared state.
• [ ] Data is synthetic and unique per run; no PII, tokens, or production extracts.
• [ ] Duplicate/copy-paste tests consolidated via parameterization.
• [ ] Provenance policy applied; off-limits repositories respected.
• [ ] Flake and mutation outcomes tracked for AI-drafted tests until the process matures.