REST

Resource modeling, method and status-code semantics, representations, idempotency, concurrency control, caching, and bulk operations for HTTP/JSON APIs.

Resource Modeling

Model the domain, not the database. A resource is a named thing with state and identity; the database schema is an implementation detail that may expose several resources.

• Nouns, plural, lower-case, hyphenated: /orders, /shipping-labels, /users/{id}/sessions.
• Identifiers stay opaque: never expose incrementing integers as the only id (enumeration); use a UUIDv7, ULID, or a prefixed opaque id such as ord_01J.... Clients must treat ids as strings.
• Nesting encodes containment, not convenience: at most one or two levels where the child cannot exist without the parent (/users/{id}/sessions). For deep graphs, use top-level resources with filters (/orders?customer_id=...) and document the relation.
• Singleton resources for things that exist once per scope: /me, /settings, /cart.
• State transitions that are not CRUD should be sub-resources that model the event: POST /orders/{id}/cancellation, POST /orders/{id}/refunds, or an explicit state patch PATCH /orders/{id} {"status": "cancelled"} when transitions are simple and documented. Custom action verbs (POST /orders/{id}/cancel) are an accepted escape hatch, but make them idempotent-safe and document return semantics.
• Collections are queried, not designed ad hoc: every collection supports pagination (see pagination and filtering) and has a documented default and maximum page size.
• Do not mirror tables: order_items exposed as a top-level collection invites clients to build joins the API never intended to support.
• Name the URL for the resource the response represents. If POST /payments returns a payment, the response is the payment, not an envelope with payment plus receipt.

Representation conventions

| Decision | Recommendation | Notes | |---|---|---| | Field naming | One casing everywhere (snake_case or camelCase) | Document it; do not translate per endpoint | | Dates/times | RFC 3339 UTC strings (2026-01-02T03:04:05Z) | Never local time; accept +00:00 | | Money | Minor units integer + ISO 4217 code ({"amount": 1099, "currency": "USD"}) | Never floats | | Enums | Upper SNAKE_CASE strings, additive only | Unknown values must not break clients | | Empty vs missing | Omit unset fields unless the client asked for them | null means explicit absence | | Booleans | Positive names (is_active, not is_not_active) | Avoid tri-state confusion | | IDs in responses | Always include the canonical resource id and a self link if linking is used | HATEOAS optional |

Methods and Status Codes

| Method | Meaning | Safe | Idempotent | Typical success | |---|---|---|---|---| | GET | Read resource/collection | yes | yes | 200 | | HEAD | Same as GET without body | yes | yes | 200 | | OPTIONS | Capabilities / CORS preflight | yes | yes | 204 | | POST | Create or submit; non-idempotent by default | no | no | 201 (+ Location) | | PUT | Replace the resource at a known URI | no | yes | 200/201/204 | | PATCH | Partial update | no | no (unless conditional) | 200/204 | | DELETE | Remove | no | yes | 204/202 |

Status code rules that prevent most review cycles:

• 201 + Location on create. If the server does not know the final id yet, return 202 with a status resource instead.
• 202 Accepted for operations that will complete asynchronously; return a job resource URL and let clients poll or subscribe (see events and webhooks).
• 204 No Content for successful mutations with nothing to say. Do not return 200 with {}.
• 400 vs 422: 400 for malformed syntax (unparseable JSON, bad query string), 422 for well-formed but semantically invalid input (unknown enum value, failing business rule).
• 401 vs 403: 401 means the credential is missing, invalid, or expired; 403 means the credential is valid but lacks access. Never use 404 to hide 403 unless the resource existence itself is sensitive.
• 404 vs 410: 410 Gone for resources that existed and were intentionally removed; include a link to the replacement when possible.
• 405 with an Allow header for wrong method; 415 for unsupported media type; 406 when acceptable representation cannot be produced.
• 409 Conflict for state conflicts (duplicate unique key, illegal transition). 412 Precondition Failed for failed conditional requests (If-Match).
• 428 Precondition Required when the server refuses unconditional writes, forcing If-Match.
• 429 for rate limits with Retry-After (see errors).
• 503 with Retry-After for planned maintenance or dependency exhaustion; do not return 500 for something you know is temporary.

Anti-patterns

• Returning 200 for errors or a different 200 shape per endpoint.
• POST /getOrder or other RPC verbs in paths when a resource model exists.
• 500 for validation failures because an exception escaped.
• Different status codes for the same failure across endpoints.

Idempotency

Retries are inevitable (network, timeouts, load balancers). Safe methods and PUT/DELETE are idempotent by definition; POST needs an explicit key.

• Clients send Idempotency-Key: <uuid> (a header name now also standardized in the idempotency-key IETF draft; verify the current RFC status). Server behavior:
  1. Look up the key scoped to the authenticated principal + route.
  2. If seen with an identical request fingerprint, replay the stored response (status, headers, body) and mark it as a replay.
  3. If seen with a different payload, return 422/409 — never silently overwrite.
  4. If a concurrent request with the same key is in flight, return 409 or 425 Too Early and let the client retry; some APIs block briefly and return the same result.
• Store keys with a TTL (commonly 24 hours to 30 days depending on domain) and prune aggressively. Never store full request bodies unless required; store a hash plus response.
• Generate the key before the first attempt and reuse it across retries with the same body.
• Document which endpoints accept keys and whether keys are scoped per account or per route.

POST /payments
Idempotency-Key: 8f14e45f-ea0d-4d2b-9f2e-1c9a7b3d5e10
Content-Type: application/json

{"amount": 1099, "currency": "USD", "source": "pm_123"}

Anti-patterns

• Treating POST as idempotent "because it usually is".
• Replaying with a 200 when the original returned 500, hiding the failure.
• Keying only on the key value, so two tenants can collide.

Concurrency Control (Optimistic Locking)

Use ETags for lost-update prevention on read-modify-write flows.

1. GET /orders/123 returns ETag: "7" (or a hash of the representation).
2. Client sends the update with If-Match: "7".
3. Server compares; mismatch returns 412 Precondition Failed with the current resource so the client can reapply.
4. If-None-Match: * on PUT creates only if absent (201) and returns 412 if it exists.

• Keep ETag generation cheap and deterministic for a representation; weak ETags (W/"...") are acceptable when byte equality is not guaranteed.
• Do not rely on Last-Modified for concurrency: second granularity causes lost updates.
• For collections, ETags help caching but do not protect list-level edits; per-item ETags do.
• Document conflict payloads: return the current state plus the fields that conflict when the domain allows a merge.

Anti-patterns

• If-Match checked against the new body instead of the stored version.
• Returning 409 and 412 interchangeably; pick 412 for conditional-header failures.
• ETags that change on every response due to timestamps or request ids.

Caching Semantics

| Header | Purpose | Notes | |---|---|---| | Cache-Control: public, max-age=60, stale-while-revalidate=30 | Freshness and revalidation window | Use private for per-user data | | Cache-Control: no-store | Sensitive responses | Tokens, PII | | ETag + If-None-Match | Validation | 304 saves bandwidth | | Vary | Cache key dimensions | Vary: Authorization, Accept-Language; missing Vary is a data-leak bug | | Age | Time in shared cache | Useful in debugging | | Surrogate-Control | CDN-specific TTL | Keep origin and CDN TTLs aligned |

• GETs should be cacheable unless they are per-user and marked private, no-store. Authenticated responses served through shared caches must vary on credentials.
• 304 Not Modified on validation with If-None-Match/If-Modified-Since; never return 200 with the full body just because you did not implement validation.
• Invalidation: prefer short TTLs plus versioned URLs (/v1/...?v=hash) over aggressive purge logic in the origin.
• POST responses are cacheable only with explicit Cache-Control and a Content-Location.

Anti-patterns

• Caching Authorization-dependent responses publicly.
• Vary: * (kills cache efficiency) or no Vary with content negotiation.
• Returning Set-Cookie on cacheable responses.

Bulk and Batch Operations

| Pattern | Shape | Use when | |---|---|---| | Multi-id read | GET /orders?ids=a,b,c or repeated id params | Small, bounded sets (cap the count) | | Batch endpoint | POST /batch with an array of sub-requests | Heterogeneous operations, one round trip | | Bulk create/update | POST /orders:batchCreate or an array body | Homogeneous high-volume writes | | Async job | POST /exports -> 202 + job resource | Large or slow work; result URL when done | | Long-running mutation | POST returns operation resource (Operation pattern) | Anything over a few seconds |

• Bounded sizes: cap items per request (for example 100-1000) and reject oversize requests with 400 rather than truncating silently.
• Partial success: choose and document one semantic. Either all-or-nothing (single transaction, fail fast) or per-item results with 200 and a mixed status array, or a batch status of 207 Multi-Status where clients understand it. Do not mix semantics by endpoint.
• Ordering: state whether batch items are processed in order and whether earlier failures stop the batch. Consumers depend on this during retries.
• Atomicity: if all-or-nothing, say so; if not, return enough information to resume (index, id, error per item).
• Idempotency: batch endpoints need an idempotency key for the whole request and per-item keys when items can be retried independently.

{
  "results": [
    {"index": 0, "status": 201, "id": "ord_01J"},
    {"index": 1, "status": 422, "error": {"code": "INVALID_CURRENCY", "detail": "XYZ unsupported"}}
  ]
}

Anti-patterns

• Unbounded batch endpoints that let one request exhaust the database.
• Returning only the failures without indexes, forcing consumers to diff.
• Accepting batches over GET when the URL exceeds practical limits (8 KB+); use POST with a query-like body only when semantics are read-only, and document it.

Review Checklist

• [ ] Resources are nouns, ids are opaque, nesting is shallow and justified.
• [ ] Methods and status codes follow the table above; errors never return 200.
• [ ] Create returns 201 + Location; async work returns 202 + job resource.
• [ ] POST mutations accept Idempotency-Key with documented replay semantics.
• [ ] Read-modify-write supports If-Match (412 on conflict) and 428 where required.
• [ ] Cache headers are correct per resource; Vary covers auth and negotiation.
• [ ] List endpoints are paginated with bounded sizes and a stable order.
• [ ] Bulk endpoints are size-capped, transactional semantics are documented, and results are addressable.
• [ ] Every error body conforms to the shared problem-details shape (see errors).
• [ ] The published OpenAPI spec matches the implementation (see contracts).

GraphQL

Schema design, nullability, Relay-style pagination, federation, persisted queries, query cost limits, authorization, and resolver performance.

Schema Design

The schema is the contract; SDL is the source of truth. Generate types and clients from it, and review SDL changes like code.

• Schema-first, single style. Use one casing (camelCase fields, PascalCase types), consistent suffixes (*Input, *Payload, *Connection, *Edge), and documented descriptions on every type and field.
• Separate input and output types. Never reuse a domain type as an input: outputs evolve under output compatibility rules, inputs under input rules.
• Prefer object types plus explicit relations over generic JSON scalars. Only use custom scalars (DateTime, URL, JSON) with a documented serialization and validation.
• Interfaces and unions model polymorphism (Node, SearchResult = Order | Customer); always provide __typename-driven client fragments.
• Enums are additive. Adding a value is breaking for exhaustive switch clients only if they do not handle unknown values; document that clients must tolerate unknown enum values (see versioning).
• Mutation shape: one field per action, verb-led (createOrder, cancelOrder), returning a payload object with the mutated entity plus any side-effect fields. Never return the bare entity from a mutation.
• Errors inside mutations: distinguish expected domain failures (return in the payload as a union or typed error list) from unexpected failures (transport/GraphQL errors). Do not throw for validation feedback; clients then have to parse error strings.
• Deprecate, do not delete: @deprecated(reason: "..."), keep the field resolvable, and remove only after usage telemetry shows zero traffic and a version/sunset has passed.
• Descriptions are documentation; include units, ranges, and null semantics. Lint them (see the GraphQL style guidance in contracts).

type Query {
  order(id: ID!): Order
  orders(first: Int = 20, after: String, filter: OrderFilter): OrderConnection!
}

type Mutation {
  createOrder(input: CreateOrderInput!): CreateOrderPayload!
}

type CreateOrderPayload {
  order: Order
  userErrors: [UserError!]!
}

type UserError {
  field: [String!]
  code: OrderErrorCode!
  message: String!
}

Anti-patterns

• A single Query { everything } namespace with hundreds of root fields and no grouping.
• Reusing Order as OrderInput or vice versa.
• Nullable everything "to be safe"; see nullability below.
• Exposing database ids as the only identifier while node(id:) expects global ids.

Nullability

Nullability is the hardest schema decision: it is expensive to change in both directions (making nullable fields non-null is breaking; making non-null fields nullable is also breaking for clients with non-null expectations).

• Errors are not null by default. Use non-null for fields your resolver can always satisfy, including errors arrays from mutations.
• Null propagates: if a non-null field resolves to null, null bubbles to the nearest nullable parent, potentially nulling a whole object. A single failing field can kill a large response; design for this rather than discovering it in production.
• Prefer non-null for lists and list items when the domain guarantees them ([User!]!), and make the list item nullable only when "missing item" is meaningful.
• External/legacy data boundaries: keep fields nullable where a dependency may fail, and return partial data; pair with error handling that records the failure rather than throwing.
• Changing direction is a breaking change. Treat both directions as breaking in CI diffing unless you control every client (see versioning).
• Some servers expose a @semanticNonNull-style directive to mean "null only on error" while keeping schema-level nullability; check whether your framework/client stack supports it before adopting (verify upstream).

Pagination (Connections)

Use the Relay connection pattern for every list field. It gives clients a stable cursor, cheap "next page" navigation, and a uniform shape.

type OrderConnection {
  edges: [OrderEdge!]!
  pageInfo: PageInfo!
  totalCount: Int
}

type OrderEdge {
  node: Order!
  cursor: String!
}

type PageInfo {
  hasNextPage: Boolean!
  hasPreviousPage: Boolean!
  startCursor: String
  endCursor: String
}

type Query {
  orders(first: Int, after: String, last: Int, before: String): OrderConnection!
}

• Cursors are opaque (base64 of a sort key plus a tiebreaker). Never expose offsets as cursors; they drift under concurrent writes.
• One direction per query: forbid first with last and after with before together to keep ordering unambiguous.
• Cap first server-side (for example 100) and return a clear error or clamp with a documented rule. Do not silently paginate differently than requested.
• totalCount is optional and expensive; expose it only when the backing store can answer cheaply, or provide an approximate/hasMore alternative.
• Ordering must be total: append id as a tiebreaker so cursors never skip/duplicate items (same rules as HTTP pagination).
• Nested connections multiply cost; require arguments on nested list fields that could return unbounded data.

Federation

Federation lets multiple teams own types in subgraphs and compose one supergraph served by a router. Moving to federation is an organizational decision; a single graph service is simpler.

• Only federate when ownership is real: multiple teams deploying independently, or a gateway over pre-existing services. Federating a small team's graph adds a composition and operations burden for no benefit.
• Entity keys: subgraphs that contribute fields to a type declare @key(fields: "id") (Apollo Federation 2.x; verify the exact directive set for your router and composition tooling). Keys must be stable and resolvable; prefer globally unique ids.
• Referencing without ownership: use @external + @requires carefully; move logic to the owning subgraph where possible. @shareable (Federation 2) marks value fields multiple subgraphs may resolve; use it deliberately, not to silence composition errors.
• Composition is a CI gate. Run composition in CI and reject PRs that break it; keep a checked-in supergraph/composition result.
• Router vs gateway: a router (Apollo Router, Cosmo, Hive Gateway, or an equivalent Rust/Go router) is the current generation; legacy gateway implementations are on a deprecation path. Verify the supported federation spec version before depending on new directives.
• Entity resolution performance: batching entity fetches (DataLoader or the framework's equivalent) is mandatory; a router will fan out entity requests per field otherwise.
• Alternatives: schema stitching, GraphQL Mesh, or BFF-per-client. Choose one approach and document why; mixing them creates duplicate type systems.

Anti-patterns

• Subgraphs sharing a database and calling it federation.
• A "god" subgraph owning everything while others only extend.
• Deploying a subgraph without composition checks, so the supergraph silently loses fields.

Persisted Queries and Allow-lists

Production graphs should not accept arbitrary client documents.

• Trusted documents / allow-lists: build a manifest of every operation used in production (extracted from clients at build time), and reject unknown operations at the edge. This is the strongest protection against malicious queries and schema scraping.
• Automatic Persisted Queries (APQ): clients send a hash and the server returns PersistedQueryNotFound so the client retries with the document; convenient but only a bandwidth optimization, not a security control, unless combined with an allow-list.
• Persisted query TTL and version pinning: invalidate manifests on schema changes and keep per-client versions during rollout.
• Schema introspection: disabling introspection in production is defense in depth, not a security boundary. With an allow-list, arbitrary introspection is already blocked; without one, attackers can reconstruct the schema from error messages.

Depth, Cost, and Rate Limits

| Control | What it stops | Implementation notes | |---|---|---| | Max depth | Deeply nested relation walks | Reject over a configured depth (commonly 10-15); account for fragments | | Max complexity/cost | Wide sibling fan-out, expensive fields | Assign per-field cost, multiply by list size arguments | | Max aliases/batch size | Aliasing attacks (same field N times) | Count aliases per operation | | Query timeout | Long-running resolution | Server-side deadline; return partial data or error | | Pagination caps | Unbounded lists | Enforced first/last maximums | | Rate limit by cost | Expensive clients | Deduct computed cost from a token bucket per subject | | Operation allow-list | Arbitrary documents | Strongest; see persisted queries above | | Disable batching arrays | Batched-query amplification | Reject JSON array batches unless explicitly supported |

• Compute cost before execution where possible, and always enforce a hard timeout during execution.
• Count fragmented/aliased fields once per occurrence in the response, not once per definition.
• Publish the cost model (field weights) to heavy clients so they can self-regulate.

Authorization

• Enforce per field, not only per operation: a type reachable from many paths needs checks at the resolver or via a policy layer, not only at the root query.
• Pass identity through context, never through arguments. A userId argument is an authorization bypass waiting to happen; derive the subject from the token.
• Return 200 with a partial result plus errors for field-level authorization failures where the schema allows nullability; otherwise fail the whole operation. Do not return null for forbidden fields without an error entry.
• Depth/cost limits are also authorization: restrict expensive admin-only fields by scope before computing cost.
• Subscription auth on connect (token in the connection init payload or headers), plus per-message re-checks for long-lived sockets.
• Cross-tenant safety is identical to REST: scope every resolver by tenant from the token (see auth).

Resolver Performance

• DataLoader (or equivalent) for every relation. N+1 queries are the default GraphQL failure mode; batch by key per request, with per-request cache only.
• Persist the parsed/validated document (query plan cache keyed by document hash) and reuse it across requests.
• Avoid per-field round trips to remote services; batch entity resolution in federation and cache read-heavy reference data with explicit invalidation.
• Watch resolver fan-out in lists: a connection of N items times a relation of M each is N*M calls unless batched.
• Measure with tracing (OpenTelemetry spans per resolver) rather than guessing which field dominates latency.

Anti-patterns

• DataLoaders cached across requests, leaking data between users.
• Throwing for expected validation errors in mutations.
• Returning arrays where a connection is required.
• Disabling depth limits because "our clients are trusted".

Checklist

• [ ] SDL is the checked-in source of truth; clients/codegen come from it.
• [ ] Naming, input/output separation, and mutation payload shapes are consistent.
• [ ] Nullability is deliberate; both nullability directions are treated as breaking.
• [ ] Every list field is a connection with opaque cursors and capped page sizes.
• [ ] Federation (if used) has composition in CI and batched entity resolution.
• [ ] Production uses trusted documents/allow-lists; arbitrary queries are not accepted.
• [ ] Depth, cost, alias, timeout, and rate limits are configured and tested with an attack fixture.
• [ ] Field-level authorization; identity from context; tenant scoping.
• [ ] DataLoaders are per-request; tracing covers resolver latency.
• [ ] Deprecations carry reasons and removal is gated on usage telemetry.

gRPC

Protobuf service and message style, wire compatibility, streaming patterns, deadlines and cancellation, the canonical error model, browser access via Connect/gRPC-Web, and HTTP gateways.

Proto Style Guide

• One service per file for public APIs; group messages with the service that owns them.
• Package naming: reverse-DNS plus version, e.g. package acme.orders.v1;. The version lives in the package, so files can coexist during migration.
• File names: lower_snake_case.proto, one top-level entity family each (order_service.proto, order_types.proto).
• Message names: PascalCase; request/response types named after the RPC (CreateOrderRequest, CreateOrderResponse) and defined in the same file.
• Field names: lower_snake_case; field numbers are never reused.
• Enums: PascalCase type, SCREAMING_SNAKE_CASE values, a zero value of <ENUM_NAME>_UNSPECIFIED, and the type name prefix on every value to avoid C++ scope clashes.
• Reserve for breaking changes: reserved 3, 5 to 8; reserved "old_field"; immediately when removing a field. Never remove without reserving.
• Optional presence: prefer explicit optional (proto3 field presence) for scalar fields where absence differs from zero. Wrap messages/oneofs only when needed.
• googleapis types: google.protobuf.Timestamp for instants, Duration for spans, google.type.Money for currency, google.type.Date for dates, FieldMask for partial updates, google.rpc.Status for rich errors. Do not hand-roll equivalents.
• Editions: protobuf editions (2023+) replace the syntax toggle with explicit features; adopt it when your toolchain supports it end to end, otherwise stay on proto3 (verify the current edition and tooling support upstream).
• Comments become docs: every service, RPC, message, field, and enum value gets a comment // that generators can surface to clients.

// Orders service (v1).
package acme.orders.v1;

import "google/protobuf/timestamp.proto";
import "google/type/money.proto";

service OrderService {
  // Creates an order; idempotent when idempotency_key is set.
  rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse);
  // Streams orders as they are created.
  rpc WatchOrders(WatchOrdersRequest) returns (stream Order);
}

message Order {
  string id = 1;
  string customer_id = 2;
  google.type.Money total = 3;
  OrderStatus status = 4;
  google.protobuf.Timestamp created_at = 5;
}

enum OrderStatus {
  ORDER_STATUS_UNSPECIFIED = 0;
  ORDER_STATUS_PENDING = 1;
  ORDER_STATUS_PAID = 2;
}

Anti-patterns

• Changing a field's type or number ("it is only used internally") — both break the wire.
• Reusing a deleted field number for a new meaning.
• A single api.proto with every service and a Common package of unrelated messages.
• Using int32 for money or string for timestamps.

Compatibility Rules

The wire format is forever once shipped. Treat every published v1 as immutable except for additive changes.

| Change | Breaking? | Notes | |---|---|---| | Add a new field | No | Old readers ignore it; new readers see default | | Add a new RPC | No | Servers may return UNIMPLEMENTED for old deployments | | Add an enum value | Usually no | Old clients must tolerate unknown values; document it | | Remove/rename a field | Yes | Reserve number and name | | Change field type/number | Yes | Wire-incompatible | | Move a field into/out of oneof | Yes | Changes presence and wire layout | | Change optional to repeated (or vice versa) | Yes | Different wire types | | Rename a package/message | Yes | Full name is part of the type URL for Any | | Add to a oneof | Careful | Old clients treat unknown variant as unset; usually safe, verify | | Delete an RPC/method | Yes | Deprecate first, monitor, then remove |

• buf breaking (or equivalent) in CI compares against the published baseline and fails on breaking changes. Store the baseline (Buf Schema Registry or a git tag) and pin it.
• Unknown fields are preserved: do not rely on them for security decisions, but do not strip them in proxies (gRPC/HTTP transcoding handles this).
• JSON mapping is part of the contract for gateway users: field names snake_case by default, json_name overrides, enums as strings, int64/uint64 as strings. Changing a field breaks generated JSON clients too.

Streaming Patterns

| Pattern | Signature | Use when | Beware | |---|---|---|---| | Unary | rpc F(Req) returns (Resp) | CRUD, short operations | Default; prefer it | | Server streaming | returns stream Resp | Subscriptions, large result sets, tailing | Client must drain or cancel; flow control | | Client streaming | rpc F(stream Req) returns (Resp) | Uploads, batched ingestion | Server must bound memory; partial failure semantics | | Bidirectional | rpc F(stream Req) returns (stream Resp) | Sessions, real-time negotiation | Ordering, heartbeats, half-close handling |

• Message size: keep individual messages small (well under the 4 MB default receive limit; many deployments configure 1-4 MB). Stream large payloads in chunks rather than raising limits blindly.
• Flow control is per-stream (HTTP/2 windows). A slow consumer applies backpressure to the sender; do not buffer unboundedly in the handler.
• Client-streaming semantics: define what happens on malformed mid-stream messages (abort the stream vs skip) and how the server acknowledges progress.
• Server-streaming lists: streaming is not a substitute for pagination. For finite result sets, a paginated unary RPC is more retryable and cacheable; use streaming for live feeds.
• Bidirectional protocols: define a message envelope with a oneof for types, a client-initiated heartbeat or keepalive, and explicit half-close rules.
• Idempotency: streams are retried from the beginning at the connection level; include a resume token or client sequence numbers if resumability matters (see errors).

Deadlines and Cancellation

• Every call sets a deadline. Servers should reject or clamp absurd deadlines; clients should never rely on infinite waits. Propagate the remaining deadline to downstream calls so the whole chain fails at the same time.
• Cancellation propagates through context: when a client disconnects, the server context is cancelled; handlers must stop work and release resources (database queries, locks, upstream calls).
• Do not swallow cancellation: treat it as control flow and clean up in deferred/finally handlers (see the language packs for idioms).
• Retry budgets: retries multiply load during incidents. Configure retries with exponential backoff plus jitter, cap attempts, and only for idempotent methods or explicitly retryable codes. In client-side config, use the service config retry policy or retryOn equivalents; hedging is rarely appropriate and needs idempotency.
• Keepalives: configure server keepalive enforcement and client ping intervals deliberately; aggressive pings get connections closed by GOAWAY (too_many_pings).
• Lame-duck/graceful shutdown: send GOAWAY, stop accepting new streams, drain existing ones up to a grace period.

Error Model

Use the canonical google.rpc.Status shape with a code, message, and typed details. Do not invent per-service error codes as the only signal; attach them as ErrorInfo details.

| Code | Number | Retryable? | Maps to HTTP | Meaning | |---|---|---|---|---| | OK | 0 | — | 200 | Success | | CANCELLED | 1 | No | 499 | Caller cancelled | | UNKNOWN | 2 | No | 500 | Unknown server error | | INVALID_ARGUMENT | 3 | No | 400 | Bad request fields | | DEADLINE_EXCEEDED | 4 | Maybe | 504 | Deadline passed; safe only if idempotent | | NOT_FOUND | 5 | No | 404 | Missing resource | | ALREADY_EXISTS | 6 | No | 409 | Create conflict | | PERMISSION_DENIED | 7 | No | 403 | Authenticated, not allowed | | RESOURCE_EXHAUSTED | 8 | Maybe | 429 | Quota/rate limit; RetryInfo detail | | FAILED_PRECONDITION | 9 | No | 400/412 | State prevents the operation | | ABORTED | 10 | Yes | 409 | Concurrency conflict; retry the transaction | | OUT_OF_RANGE | 11 | No | 400 | Value outside valid range | | UNIMPLEMENTED | 12 | No | 501 | Method not supported by this server | | INTERNAL | 13 | Maybe | 500 | Invariant broken | | UNAVAILABLE | 14 | Yes | 503 | Transient; retry with backoff | | DATA_LOSS | 15 | No | 500 | Corruption | | UNAUTHENTICATED | 16 | No | 401 | Missing/invalid credentials |

• Attach google.rpc.ErrorInfo (reason, domain, metadata) for machine-readable causes, BadRequest for field violations, QuotaFailure/RetryInfo for limits, DebugInfo for trace correlation, PreconditionFailure for state conflicts.
• Never leak internals in message; use it for a human-readable summary and put structured data in details.
• Idempotency: provide an idempotency key field on mutating requests; server deduplicates and returns the original result (see errors).
• Rich error coverage in tests: assert codes, not just failures; clients switch on codes.

service ConfigService {
  rpc SetConfig(SetConfigRequest) returns (SetConfigResponse);
}

message SetConfigRequest {
  string key = 1;
  string value = 2;
  // Optional: deduplicate retries of this request.
  string idempotency_key = 3;
}

Browser Access: Connect, gRPC-Web, and Gateways

Browsers cannot speak raw gRPC over HTTP/2 with trailers. Choose one:

| Option | How it works | Trade-offs | |---|---|---| | Connect (Buf) | Connect protocol client/server, also speaks gRPC and gRPC-Web | No proxy needed for same-origin; verify ecosystem support in your stack | | gRPC-Web | Envoy or similar proxy translates to gRPC | Requires edge proxy; limited streaming (server-streaming only) | | HTTP/JSON gateway | grpc-gateway / Envoy transcoding from google.api.http annotations | Broadest compatibility; JSON semantics differ (int64 as string) | | REST facade | Separate hand-written REST service | Duplicate surface; usually avoid |

• CORS must allow the gRPC-Web/Connect headers: content-type, x-grpc-web, x-user-agent, connect-protocol-version, grpc-timeout, plus grpc-status/grpc-message in Access-Control-Expose-Headers.
• Auth from browsers: short-lived bearer tokens via headers; cookies need credentialed CORS and CSRF protection; do not put API keys in browser code.
• grpc-gateway annotations keep one contract for both HTTP and gRPC; generate the gateway from the same .proto and include it in compatibility checks. Custom HTTP methods/verbs need explicit annotations and a documented mapping.

Anti-patterns

• Exposing raw gRPC to the internet without a TLS-terminating proxy and per-RPC auth.
• Hand-maintaining a separate REST API that shadows the gRPC contract.
• Relying on client-side retries without server-side idempotency.

Checklist

• [ ] Style: versioned package, reserved fields, googleapis types, comments everywhere.
• [ ] Breaking-change detection (buf breaking or equivalent) runs in CI against a pinned baseline.
• [ ] Streaming shapes match the workload; sizes bounded; backpressure understood.
• [ ] Deadlines set and propagated; cancellation cleans up; retries have budgets and backoff.
• [ ] Errors use canonical codes plus ErrorInfo/BadRequest/RetryInfo details.
• [ ] Mutating RPCs accept an idempotency key.
• [ ] Browser access uses Connect/gRPC-Web/gateway with CORS and auth configured.
• [ ] Graceful shutdown and keepalive settings are tested under load.

Versioning and Compatibility

Versioning strategies, additive change rules per surface, deprecation and Sunset headers, the sunset process, breaking-change detection, and compatibility testing.

Strategy Selection

Default to additive evolution with no version number. Version only when a breaking change is unavoidable and consumers cannot be updated in time.

| Strategy | Identifier | Use when | Costs | |---|---|---|---| | Additive, unversioned | none | Default: private APIs, most public APIs with disciplined reviews | Accidental breaks are possible; needs diffing in CI | | URL path | /v2/orders | Public breaking change; discoverability and routing simplicity | Duplicate docs and routes; long dual-serving period | | Media type | Accept: application/vnd.acme.v2+json | Content negotiation already used; few clients | Invisible in access logs; harder to debug | | Custom header | API-Version: 2026-01-01 | Many small breaking changes; date-based pinning (Stripe-style) | Clients must pin; default-version shifts are breaking by definition | | Query parameter | ?api_version=2 | Quick fixes, rarely a good long-term contract | Caching/CDN keys; easy to forget in clients | | Protobuf package | acme.orders.v2 | gRPC breaking change | New wire service; old still hosted | | Event schema version | type: acme.order.created.v2 | Async payload breaking change | Consumers subscribe to both during migration |

Rules that keep this honest:

• One version axis per API. Do not also version individual endpoints with a second scheme.
• Never remove a version without a sunset date and telemetry. Publishing v2 is half the work; migrating v1 clients is the other half.
• A version is a contract, not a copy of the codebase. Keep shared domain logic; version the boundary (serialization, validation, route mapping).
• Date-based versions: the default version must change only with a new version label; never change the meaning of an existing pinned version.

Additive Change Rules

A change is non-breaking if every valid existing client request keeps working with the same meaning and every existing response remains parseable.

| Change | REST | GraphQL | gRPC | Events | |---|---|---|---|---| | Add response field | Safe | Safe | Safe | Safe if optional | | Add request field (optional) | Safe | Safe | Safe | Safe | | Add required request field | Breaking | Breaking (non-null arg) | Breaking | Breaking | | Add enum value | Safe if clients tolerate unknown | Safe if clients handle unknown | Safe (document) | Depends on consumer validation | | Remove field | Breaking | Breaking (unless deprecated long) | Breaking | Breaking | | Rename field | Breaking | Breaking | Breaking (reserve) | Breaking | | Change type/format | Breaking | Breaking | Breaking | Breaking | | Tighten validation (new max length, stricter enum, new required) | Breaking | Breaking | Breaking | Breaking | | Loosen validation | Safe (usually) | Safe | Safe | Safe | | Change ordering/defaults | Breaking if clients rely on it | Breaking | Breaking | Depends | | Add new endpoint/method/operation | Safe | Safe | Safe | n/a | | Change error codes/status | Breaking if clients branch on them | Breaking | Breaking | n/a | | Change auth requirements | Breaking | Breaking | Breaking | n/a | | Change rate limits downward | Breaking operationally | Same | Same | Same |

Practical guidance:

• Unknown enum/union values must be tolerated by clients. Document it in your client guidance; validate it in generated SDKs where possible.
• Tightening validation is the most-missed break: adding maxLength, making an optional field required, or rejecting values previously accepted breaks real clients. Treat as breaking.
• Defaults are part of the contract: if an omitted field changes behavior, that is a break.
• Deprecate before deleting with at least one full client release cycle plus the sunset period (see below).
• Error taxonomy changes are breaking when consumers switch on stable code values. Keep codes stable; add new ones for new cases (see errors).

Deprecation Headers

Signal deprecation at the HTTP layer so monitoring and tooling can detect it automatically.

| Header | Meaning | Example | |---|---|---| | Deprecation | When the endpoint/resource became deprecated (RFC 9745; verify current status) | Deprecation: @1772323200 (IMF-fixdate or structured date) | | Sunset | When it will stop working (RFC 8594) | Sunset: Wed, 31 Dec 2026 23:59:59 GMT | | Link | Documentation and successor | Link: <https://docs.acme.com/migrate-v2>; rel="deprecation"; type="text/html" | | Warning | Legacy; do not use for new deprecations | Superseded by Deprecation/Sunset | | Custom X-API-Deprecated | Avoid | Non-standard and easy to miss |

Deprecation: @1772323200
Sunset: Wed, 31 Dec 2026 23:59:59 GMT
Link: <https://api.acme.com/v2/orders>; rel="successor-version",
      <https://docs.acme.com/migrate-v2>; rel="deprecation"

• Emit on every call, not just once; caches and SDKs must see it.
• Also document deprecation in the spec: OpenAPI deprecated: true on operations and schemas, plus descriptions with the sunset date; GraphQL @deprecated; proto deprecated = true options.
• GraphQL @deprecated reasons should name the replacement and the removal date.
• Changelogs and docs: a dated changelog entry, migration guide, and a machine-readable feed if partners automate. Keep the deprecation list in the API docs index.

Sunset Process

A repeatable process beats an ad-hoc announcement. Typical public-API timeline is 6-12 months; internal APIs can be weeks if all consumers are known and instrumented.

1. Decide and publish. Identify the break, choose the replacement, and post the sunset date (RFC 8594) with a migration guide. Announce in the changelog and to known consumers.
2. Instrument usage. Emit per-consumer metrics keyed by API key/client id/token (client_id), endpoint, and version. You cannot manage migration without usage data.
3. Communicate on a schedule. 90/30/7-day reminders; include per-tenant usage summaries so consumers know their exposure.
4. Degrade progressively. Warnings first (headers and non-fatal logs), then brownouts (short windows returning 410) to surface clients that ignore messages, then final shutdown.
5. Return 410 Gone, not 404, for retired endpoints, with a problem-details body linking the replacement and migration guide.
6. Kill-switch plan. Keep the ability to extend the date; migrations slip.
7. Remove code and docs only after the shutdown date plus a safety window.

| Timeline | Action | |---|---| | T-0 (deprecation) | Headers, spec flags, changelog, migration guide, usage metric | | T+30% of window | First reminder with per-consumer usage | | T+75% | Brownout schedule published and tested (staging first) | | T-30 days | Final reminder; escalate to account owners | | T-0 (sunset) | Brownouts, then 410 responses | | T+30 days | Remove code paths; archive docs |

Anti-patterns

• Announcing a sunset with no replacement documented.
• Deprecating without usage telemetry, so nobody knows who is affected.
• Brownouts without warning or a published schedule.
• Quietly changing the default version (a breaking change in disguise).

Compatibility Testing in CI

Automate the detection of breaks; human review misses them.

| Surface | Tooling category | Gate | |---|---|---| | OpenAPI | oasdiff, openapi-diff, Optic-class tools | Fail on breaking diff vs the released spec (see contracts) | | GraphQL | GraphQL Inspector, schema diff in the registry, composition checks | Fail on breaking changes and composition errors | | Protobuf | buf breaking against a pinned baseline | Fail on wire-incompatible changes | | Events | Schema registry compatibility mode (backward/forward/full), AsyncAPI diff | Fail on incompatible subject registration | | Runtime | Consumer-driven contract tests (Pact-class) plus provider verification | Fail when a published contract no longer verifies | | Responses | Golden/snapshot tests of representative payloads per version | Catch serialization drift |

• Pin the baseline: compare against the last released artifact, not main. Store the released spec/schema next to the release tag or in a registry.
• Run diffing on every PR and require an explicit approval label for intentional breaks (with a linked sunset plan).
• Test old clients against new servers: keep a compatibility suite with previous SDK versions, or at minimum representative recorded requests, and run it against staging.
• Contract tests are two-sided: providers must run consumer contract verification in CI, and consumers must pull the published contract. Contract tests complement, not replace, integration tests.
• Version-aware integration tests: for URL-versioned APIs, run the same suite against every supported major version until its sunset.

Review Checklist

• [ ] Versioning strategy is documented and applied consistently; one axis per API.
• [ ] Additive-change rules are codified; tightening validation is treated as breaking.
• [ ] Deprecation and Sunset headers (plus Link) are emitted on deprecated endpoints.
• [ ] Specs mark deprecations (deprecated: true, @deprecated, proto options).
• [ ] A sunset process exists with telemetry, reminders, brownouts, and 410 responses.
• [ ] CI detects breaking changes for every surface (OpenAPI, GraphQL, proto, events).
• [ ] Baseline artifacts are pinned to the last release.
• [ ] Consumer-driven contract tests run on both sides.
• [ ] Changelog and migration guides are part of the release definition of done.

Pagination, Sorting, and Filtering

Cursor vs offset, cursor design and keyset queries, sorting contracts, sparse fieldsets, filtering grammars, batching, consistency, and limits.

Choosing a Pagination Model

| Model | Request | Stability under writes | Random access | Performance | Use when | |---|---|---|---|---|---| | Offset/limit | ?offset=100&limit=50 | Poor: items shift, duplicates/skips | Yes | Degrades with depth (OFFSET scans) | Small admin tables, static data, page numbers required | | Page/per_page | ?page=3&per_page=50 | Poor (offset in disguise) | Yes | Same as offset | UI with page numbers, bounded totals | | Cursor | ?cursor=eyJ... | Good: anchored to last item | No | Index-friendly keyset | Feeds, exports, API-to-API, social timelines | | Keyset (explicit) | ?after_id=123&limit=50 | Good | No | Best with composite index | Public APIs where opaque cursor annoys debuggers | | Snapshot | ?snapshot=ts&cursor=... | Perfect (frozen view) | No | Needs MVCC/PITR support | Reconciliation, billing close, audit exports |

Default recommendation: cursor/keyset for anything ordered and growing; offset only for small admin surfaces with stable ordering and capped depth.

Anti-patterns

• Offset pagination on a hot table with millions of rows and a UI "next" button.
• Returning an unbounded list when the client does not ask for pagination.
• Inconsistent page sizes across endpoints, or no documented maximum.

Cursor Design

A cursor encodes the position in a total order. It must be opaque, stable, and tamper-evident if it influences filtering.

• Payload: the sort key values of the last item plus a tiebreaker (created_at + id), base64url-encoded. Optionally include a query fingerprint and expiry.
• Sign or encrypt cursors when clients could otherwise skip authorization scopes or forge filters (for example a cursor scoped to a tenant). HMAC is sufficient for integrity; encrypt if the payload leaks sensitive ordering data.
• Opaque to clients: document "treat as a token; do not parse or construct".
• Single direction per request: forbid after + before together; document that reversing direction requires re-sorting.
• Cursor expiry: optional, but if cursors expire, return a clear 400 INVALID_CURSOR with a code the client can branch on, and document restart semantics.
• Never put offsets inside cursors; that reintroduces drift.

{
  "data": [{"id": "ord_01J", "created_at": "2026-01-02T03:04:05Z"}],
  "pagination": {
    "next_cursor": "eyJjcmVhdGVkX2F0IjoiMjAyNi0wMS0wMlQwMzowNDowNVoiLCJpZCI6Im9yZF8wMUoifQ",
    "has_more": true
  }
}

Keyset query shape for ORDER BY created_at DESC, id DESC:

SELECT id, created_at, total
FROM orders
WHERE tenant_id = $1
  AND (created_at, id) < ($2, $3)
ORDER BY created_at DESC, id DESC
LIMIT $4;

• The composite index must match the sort: (tenant_id, created_at DESC, id DESC).
• Row-value comparison (a, b) < (?, ?) works in modern PostgreSQL/MySQL; verify support and plan shape in your engine. For engines without row values, expand to a < ? OR (a = ? AND b < ?).
• Keyset requires a total order. Add the primary key as the final tiebreaker everywhere, including when clients sort by a non-unique field.

Sorting

• Parameter style: ?sort=-created_at,name (leading - for descending) or repeated ?sort=created_at:desc. Pick one and document it; expose it in OpenAPI as an array or a pattern-validated string.
• Whitelist sortable fields. Never interpolate client input into ORDER BY; map public names to columns and reject unknown fields with a 400 problem-details error.
• Append a deterministic tiebreaker (id) to every sort so pages do not overlap.
• Default sort is part of the contract. Document it, and keep it stable; changing the default order is a breaking change for pagination cursors and for consumers assuming recency.
• Collation and null ordering: document case sensitivity and where nulls sort; these differ across databases and are observable client behavior.
• Expensive sorts (unindexed columns) should be rejected or documented as slow; consider materialized views or search indexes for relevance ordering.

Anti-patterns

• ORDER BY built from req.query.sort without a whitelist.
• Non-deterministic order (ORDER BY created_at alone) with cursor pagination.
• Sorting by relevance without documenting the scoring or how ties break.

Sparse Fieldsets and Expansion

Let clients fetch what they need without a combinatorial set of endpoints.

| Feature | Parameter | Notes | |---|---|---| | Sparse fieldsets | ?fields=id,total,status or fields[orders]=... | Whitelist per resource; always include id | | Expansion | ?expand=customer,items | Cap depth and count; document allowed expansions | | Exclusion | ?exclude=internal_notes | Simpler for admin surfaces; still whitelist | | Embedded vs linked | ?include=items (JSON:API style) | Prefer explicit include over magic auto-expansion |

• Nested field selection (fields=id,customer(id,name)) is powerful but adds parser and authorization complexity; only adopt if your framework supports it, and validate depth.
• GraphQL solves this natively with field selection; do not bolt field selection onto a GraphQL endpoint via query parameters.
• Default representations must stay modest: unbounded nested includes per list item are a performance trap (N+1 over rows, giant payloads). Cap expansions and paginate sub-collections.

Filtering Grammar

Complexity grows fast; choose the simplest grammar that covers the contract and document it precisely.

| Grammar | Example | Strengths | Weaknesses | |---|---|---|---| | Equality params | ?status=paid&customer_id=42 | Simple, cacheable, index-friendly | No ranges/OR | | Operator suffix | ?created_at_gte=...&total_lt=... | Readable, easy to validate | Many params, naming conventions | | Repeated with operator | ?status=paid&status=pending (implicit OR) | Simple OR for enums | Ambiguous with AND across fields | | Structured/bracketed | ?filter[status][in]=paid,pending | Expressive, parseable | Verbose, framework support varies | | Expression grammar | ?filter=status eq "paid" and total gt 100 | Very expressive | Parsing, injection risk, index-unfriendly | | RSQL/FIQL | ?filter=status==paid;total>100 | Compact, tooling exists | Niche, needs escaping rules | | OData $filter | ?$filter=Status eq 'paid' | Standardized | Heavy; partial support; SQL injection risk if translated naively |

• Semantics must be explicit: AND across different fields; for repeated values on one field, state whether it is OR or last-wins. Never leave it to the framework default.
• Translate to parameterized queries only. An expression grammar that concatenates SQL is the classic injection path; parse to an AST and map to SQLAlchemy/Kysely/query builders.
• Index alignment: only allow filtering on indexed/allowlisted fields, or implement full-text/search-engine-backed filtering for free text. LIKE '%term%' on a large table is a denial-of-service.
• Unknown filter fields should be rejected (400 with a machine-readable code listing allowed fields) rather than silently ignored — silent ignores produce wrong data with no signal.
• filter on nested relations is usually best served by dedicated endpoints or search indices; deep joins through list endpoints blow up query plans.
• Response metadata: include the applied filters and sort in the response envelope if clients need to confirm interpretation.

Batching and Prefetch

| Need | Pattern | Notes | |---|---|---| | Fetch N known resources | ?ids=a,b,c (cap count) | Returns found items; document missing-id behavior | | Heterogeneous reads | POST /batch sub-requests | One round trip; per-item statuses | | GraphQL batching | DataLoader per request | Never batch across requests/users | | gRPC | Batch RPC with repeated ids | Cap size; same semantics as REST | | Client cache priming | ETag-based conditional multi-get | See REST caching |

• Document missing ids: return a map keyed by requested id, a list with nulls, or 404 if any are missing. Pick one and keep it consistent.
• Cap batch size and return 400 (not truncation) when exceeded.
• Batching is not a substitute for pagination; it is for known-id lookups only.

Consistency and Limits

• Read-your-writes: after a mutation, a subsequent list read may lag on replicas. Accept a consistency token (write LSN/timestamp) or route the session to the primary for a window. Document the guarantee per endpoint.
• Cursor consistency: cursors anchor to a sort key, so concurrent inserts do not shift pages. Deletes can still remove an item you have not yet read; that is expected, document it.
• Snapshot exports: for "give me everything as of T", run an async job against a snapshot (repeatable-read transaction, read replica with PITR, or warehouse export) and hand back a result URL (see events/webhooks for completion notification).
• Limits table (publish it):

| Limit | Typical | Enforcement | |---|---|---| | Default page size | 20-50 | Applied when client omits pagination | | Max page size | 100-1000 | 400 or clamp; document which | | Max offset | 1k-10k | Reject deeper offsets with a cursor hint | | Max ids per batch | 100-1000 | 400 | | Max expansions | 3-5 | 400 | | Max filter predicates | 10-20 | 400 | | Query timeout | 1-30 s | Abort and return 503/timeout error |

Anti-patterns

• Returning total_count with an expensive unbounded count on every list request.
• Using has_more computed by fetching limit + 1 but then exposing the extra item.
• Mixing cursor and offset parameters on one endpoint.
• Ignoring unknown query parameters so typos silently change result sets.

Checklist

• [ ] Pagination model chosen per endpoint and documented (cursor default for ordered lists).
• [ ] Sort keys deterministic, whitelisted, index-aligned; default sort documented.
• [ ] Cursors opaque, signed/encrypted when they carry scope, and single-direction.
• [ ] Sparse fieldsets/expansions whitelisted, capped, and default-modest.
• [ ] Filter grammar documented with AND/OR semantics and unknown-field rejection.
• [ ] Filters translate to parameterized queries; no string concatenation.
• [ ] Batch size, page size, offset depth, and timeout limits are published and enforced.
• [ ] Missing-id semantics for multi-get are defined.
• [ ] Consistency guarantees (read-your-writes, snapshot exports) are stated.
• [ ] has_more/next_cursor semantics tested at boundaries (empty page, last page, exact multiple of page size).

Errors, Retries, Idempotency, and Rate Limits

RFC 9457 problem details, error taxonomy, validation error shapes, retry semantics and budgets, idempotency keys, and rate-limit signaling.

Problem Details (RFC 9457)

Use application/problem+json (or application/problem+xml when applicable) for every error response. RFC 9457 obsoletes RFC 7807 and keeps the same five members.

| Member | Type | Meaning | |---|---|---| | type | URI | Stable identifier for the error class; documentation link by convention | | title | string | Short, human-readable summary; stable per type | | status | integer | HTTP status code (repeated in the body for logs/queues that lose headers) | | detail | string | Human-readable explanation of this occurrence | | instance | URI | Identifier for this occurrence (often the request id path) | | extensions | any | code, errors[], request_id, retry_after, docs links |

{
  "type": "https://api.acme.com/problems/validation-error",
  "title": "Request validation failed",
  "status": 422,
  "detail": "One or more fields are invalid.",
  "instance": "/requests/req_01J8Z",
  "code": "VALIDATION_ERROR",
  "request_id": "req_01J8Z",
  "errors": [
    {
      "pointer": "/items/0/quantity",
      "code": "OUT_OF_RANGE",
      "detail": "Must be between 1 and 999."
    }
  ]
}

Rules:

• type is stable and dereferenceable. Serve a short HTML page at that URI explaining the error, causes, and remedies. For generic cases, about:blank is allowed, but a real URI is better for consumers.
• title is constant per type; detail varies per occurrence. Do not put dynamic data in title, and do not leak internals in detail.
• Always include a machine-readable code. Clients should switch on code, not on detail strings or HTTP status alone.
• Correlation: include request_id and, when tracing, a traceparent-compatible id. Return the same id in the X-Request-Id response header.
• Content negotiation: honor Accept; fall back to JSON problem details when the client sends Accept: */*. Do not return HTML errors to API clients.
• Errors are part of the contract: document every type/code per operation in OpenAPI (see contracts) and keep codes stable (see versioning).
• Never include stack traces, SQL, internal hostnames, or secret material. Log those server-side with the same request_id.

Error Taxonomy

Keep a small, stable set. Most domains need fewer than 30 codes.

| Category | HTTP | Example codes | Client action | |---|---|---|---| | Malformed request | 400 | MALFORMED_JSON, INVALID_QUERY_PARAM | Fix request; do not retry | | Validation | 422 | VALIDATION_ERROR, OUT_OF_RANGE, INVALID_ENUM | Fix fields; do not retry | | Authentication | 401 | AUTHENTICATION_REQUIRED, TOKEN_EXPIRED, TOKEN_INVALID | Refresh or re-auth once | | Authorization | 403 | PERMISSION_DENIED, SCOPE_REQUIRED, TENANT_MISMATCH | Do not retry; show operator | | Missing resource | 404/410 | RESOURCE_NOT_FOUND, RESOURCE_GONE | Do not retry; fix identifier | | Conflict | 409/412 | DUPLICATE_RESOURCE, STATE_CONFLICT, PRECONDITION_FAILED | Re-read, merge, retry conditionally | | Not allowed method/type | 405/415/406 | METHOD_NOT_ALLOWED, UNSUPPORTED_MEDIA_TYPE | Fix client | | Rate/quota | 429 | RATE_LIMITED, QUOTA_EXCEEDED | Backoff with jitter; respect Retry-After | | Dependency failure | 502/503/504 | UPSTREAM_UNAVAILABLE, TIMEOUT, MAINTENANCE | Retry idempotent requests with budget | | Internal | 500 | INTERNAL_ERROR | Retry cautiously; report with request id |

Validation error detail array (choose one shape and keep it):

{
  "errors": [
    {"pointer": "/email", "code": "FORMAT", "detail": "Must be a valid email address."},
    {"pointer": "/items/2/sku", "code": "UNKNOWN_SKU", "detail": "SKU not found."}
  ]
}

• Use JSON Pointer (RFC 6901) for request-body locations and document whether query parameter errors use a different convention.
• Report all field errors at once, not only the first; form UX and client retries depend on it.
• Cap the number of reported errors (for example 50) to bound response size.
• Do not use detail to carry structured data; add a dedicated extension when clients must parse it.

Anti-patterns

• Per-endpoint error shapes, or {"error": "..."} strings.
• Exposing internal exception class names.
• Returning 200 with {"success": false}; use the status code.
• Changing code values between releases; consumers branch on them.

Retry Semantics

Retries are a client responsibility, but the server must publish enough information to make them safe.

• Classify failures: retry only on transient conditions (429, 502, 503, 504, connection resets, deadline exceeded on idempotent calls). Never auto-retry 400/401/403/404/409/422/500 (500 may be retried once with a budget, but treat repeated 500s as a bug).
• Idempotency first: only retry POSTs/PATCHes when the request carries an idempotency key or is naturally idempotent. See below.
• Exponential backoff with full jitter:

delay = min(cap, base * 2^attempt)
sleep = random_uniform(0, delay)        # full jitter

Typical: base 100-500 ms, cap 10-30 s, 3-5 attempts, total budget under the caller's own timeout. Respect Retry-After when present; it overrides computed delays.

• Retry budgets: cap retries as a fraction of total traffic (for example 10%) to prevent retry storms during incidents. Prefer client-side rate limiters that shed retries when the budget is exhausted.
• Circuit breakers per dependency with half-open probes, and load shedding at the edge before queues saturate.
• The server can ask for retries safely with 503 + Retry-After + problem details; do not return 200 with a "try again" body.
• Never retry non-idempotent operations without a key, including internal service calls; the gRPC ABORTED/UNAVAILABLE guidance is the same pattern (see gRPC).
• Hedging (sending a second request before the first returns) is only safe for idempotent reads and needs strict per-attempt budgets; otherwise it doubles write load.

Idempotency Keys

For unsafe methods, the server, not the client, guarantees "at most one side effect per key".

• Header: Idempotency-Key: <client-generated UUID> (an idempotency-key header is being standardized; verify the current IETF draft/RFC status before claiming conformance).
• Scope: key is unique per (principal, route, key). The same key from two accounts must not collide. Hash the key before storage if it may contain sensitive data.
• Request fingerprint: hash the canonical body plus relevant headers. On replay with a different fingerprint, return 422 or 409 with code: IDEMPOTENCY_KEY_REUSED — never execute the new body under the old key.
• Concurrency: two in-flight requests with the same key should yield a deterministic outcome: either one executes and the other waits and replays, or the second gets 409/425 Too Early with retry guidance. Document which.
• Replay response: return the original status and body (minus volatile data), and mark it (for example Idempotency-Replayed: true) so clients and logs can distinguish.
• TTL: store records for a documented window (common: 24 h to 30 days). After expiry, the same key may execute again; document that and choose a TTL longer than your retry horizon.
• Storage: a unique constraint on (principal, route, key) plus a row lock or advisory lock beats read-then-write races. The record should store status, response body (bounded), and headers needed for replay.
• Failure handling: only store a completed result. If the first attempt fails mid-flight, delete or mark the record so a retry can proceed, or store a terminal failure and replay it consistently.

POST /payments
Idempotency-Key: 0f8fad5b-d9cb-469f-a165-70867728950e

-> 201 Created (first call)
-> 201 Created, Idempotency-Replayed: true (retry with same body)
-> 409 Conflict, code: IDEMPOTENCY_KEY_REUSED (same key, different body)

Rate Limiting and Quota Signaling

Distinguish rate limits (requests per time window) from quotas (usage allowances per period) and concurrency limits (in-flight work). Signal all three clearly.

| Signal | Where | Notes | |---|---|---| | 429 Too Many Requests | Response status | With problem details body (type = rate-limited) | | Retry-After | Header | Seconds or HTTP date; required on 429 and useful on 503 | | RateLimit-Limit / RateLimit-Remaining / RateLimit-Reset | Headers | IETF rate-limit header fields (verify current RFC status); include on success and failure | | RateLimit (structured) | Header | Single structured field with policies; check current spec before adopting | | X-RateLimit-* | Headers | Legacy/de facto; do not use for new APIs | | 503 + Retry-After | Response | Quota exhausted for the billing period, or planned maintenance |

• Limit by the right subject: API key, OAuth client, user, tenant, and IP (edge only). Per-IP alone breaks shared NATs and punishes proxies; combine with identity limits.
• Multiple windows: burst (per second) plus sustained (per hour/day). Document both.
• Return 429 before shedding silently; a dropped connection is impossible to debug.
• Include enough context in the body: which limit, the window, the reset time, and a docs link. Extensions such as retry_after and limit/remaining/reset in problem details.
• Counters must be shared across instances (Redis/Valkey or a gateway) or limits multiply by replica count.
• Header consistency: always send remaining/reset on 2xx so well-behaved clients can slow down proactively; do not only signal at failure.
• Billing quotas deserve a distinct code (QUOTA_EXCEEDED) and a different remediation (upgrade plan) than a transient burst limit.
• Webhooks and async jobs carry the same limits; document per-event delivery limits (see events).

Anti-patterns

• 503 with no Retry-After for a known maintenance window.
• Returning X-RateLimit-* on some endpoints only.
• Applying a global IP limit that punishes an entire customer's NAT egress.
• Silently dropping requests at the load balancer without a response body.

Checklist

• [ ] All errors use RFC 9457 problem details with the same media type.
• [ ] Every error has a stable code, a stable type URI, and a request_id.
• [ ] Validation errors report all fields with JSON Pointer locations.
• [ ] Error taxonomy is documented per operation and versioned with change control.
• [ ] No stack traces or internals leak; logs correlate by request id.
• [ ] Retries use exponential backoff with jitter, budgets, and Retry-After respect.
• [ ] Idempotency keys are implemented for unsafe methods with fingerprint and TTL semantics.
• [ ] Rate/quota limits are published and signaled on success and failure.
• [ ] Distinguish rate limit, quota, and maintenance in status and codes.

API Authentication and Authorization

OAuth 2.1, OIDC, scopes and audiences, token lifetimes and revocation, DPoP, mTLS, API keys, rate limiting as an access control, and multi-tenancy.

Choosing a Mechanism

| Caller | Mechanism | Notes | |---|---|---| | Browser SPA, mobile app (user present) | OAuth 2.1 authorization code + PKCE; OIDC for identity | BFF pattern preferred for browsers; never keep refresh tokens in JS | | Service-to-service (no user) | OAuth 2.1 client credentials, or mTLS/private_key_jwt | Prefer sender-constrained tokens | | Third-party integration (public client) | OAuth 2.1 code + PKCE, consent, scopes | Redirect URI allow-list, exact match | | Partner server | OAuth 2.1 + scopes, optional mTLS/DPoP | Resource indicators for audience binding | | Simple internal tooling | API key, per-key scoped and rate limited | Hashed at rest, rotatable, never in URLs | | Legacy client with no OAuth | Edge session or signed requests | Record as migration debt with a sunset date |

Never mix: one endpoint should not accept a bearer token and an API key and a session cookie. Declare the accepted schemes in OpenAPI securitySchemes and enforce exactly those.

OAuth 2.1

OAuth 2.1 consolidates the OAuth 2.0 security best current practice into the core spec. Design against 2.1, not 2.0.

• PKCE is mandatory for all clients, including confidential ones (S256, never plain).
• Implicit and password grants are removed. Use authorization code + PKCE; use client credentials for machine identities.
• Exact redirect URI matching, no wildcard subdomains, no fragments in redirect URIs.
• state is required for browser-based flows (CSRF binding) even with PKCE.
• Public clients (SPAs, mobile) cannot hold secrets; do not issue them one. Use the BFF pattern (server-side session) for browser apps as the default recommendation.
• Refresh token rotation with reuse detection: a reused rotated refresh token invalidates the family and forces re-authentication.
• Resource indicators (RFC 8707): clients pass the target resource so the authorization server can mint audience-bound tokens and prevent token replay across APIs.
• Sender-constrained tokens: prefer DPoP (RFC 9449) or mTLS-bound tokens (RFC 8705) over plain bearer tokens for high-value scopes.
• Token storage: server-side sessions or secure storage; never localStorage for refresh tokens. Mobile apps use the OS keychain/keystore.
• Redirect handling in native apps: custom URI schemes or claimed HTTPS links, with PKCE; no embedded webviews for the authorization endpoint.

Anti-patterns

• Long-lived access tokens "to reduce latency".
• Refresh tokens that never rotate or are not invalidated on reuse.
• Accepting tokens from any issuer the resource server happens to trust.
• Passing tokens in query strings (they leak in logs and referrers).

OpenID Connect

Use OIDC for user identity; keep OAuth for authorization.

• Discovery (/.well-known/openid-configuration) and JWKS (jwks_uri) drive validation. Cache JWKS with a TTL and refresh on unknown kid; never fetch JWKS per request.
• Validate every ID token: signature (RS256/ES256 as advertised), iss exact match, aud contains the client id, exp/iat/nbf with a small clock skew (for example 60 s), and nonce for browser flows. azp must match when multiple audiences exist.
• Do not use ID tokens as API access tokens. The ID token is for the client; the access token is for the resource server, with its own audience.
• Claims are hints until verified: do not authorize on email alone; use the stable sub subject and map it to a local principal.
• Logout: implement RP-initiated logout for browser apps; access tokens may remain valid until expiry, so pair with short lifetimes and revocation where it matters.
• Account linking: key on issuer + subject (iss+sub), never email, for identity.

Scopes and Audiences

• Scopes express capabilities, not roles. Prefer orders:read, orders:write, refunds:create over admin, user.
• Declare scopes per operation in the contract (OpenAPI security per operation, proto service config or annotations, GraphQL field policy). Reviewers can then see required permissions in the diff.
• Least privilege: default-deny; a token with no matching scope gets 403 with code: SCOPE_REQUIRED (see errors).
• Audience every token: aud must identify this API (single resource). Resource servers reject tokens issued for another service, even in the same trust domain. Use resource indicators to request narrow audiences.
• Do not put authorization data in scopes that changes often (tenant ids, resource ids); scopes are coarse and rarely revoked mid-token. Check dynamic permissions at request time.
• Group/role scopes (roles:admin) are a bottleneck; map IdP groups to local roles at token exchange or session creation instead.
• Scope explosion: keep the catalog under a few dozen; if you need more, your resources need a policy layer (OpenFGA/SpiceDB/Cedar-style) with relationship-based checks.

Token Lifetimes and Revocation

| Token | Recommended lifetime | Notes | |---|---|---| | Access token | 5-60 minutes | Shorter for sensitive scopes; supports fast revocation via expiry | | Refresh token | Days to months, rotated each use | Rotate + reuse detection; bind to client and device | | ID token | Minutes | Usually unused after login; do not call APIs with it | | API key | 90-365 days with rotation | Per-key revocation; scan for leaked keys | | mTLS certificate | 1-90 days (automate renewal) | Short-lived certs beat CRL/OCSP complexity | | DPoP proof | 1-5 minutes iat window | Replay-protected by jti cache |

• JWT vs introspection: self-contained JWTs scale statelessly but cannot be revoked before expiry. introspection (RFC 7662) allows instant revocation at the cost of per-request I/O. Hybrid: short JWTs plus a revocation list for high-value scopes.
• Revocation events: publish token revocation to resource servers (pub/sub or a shared cache) so "logout everywhere" and "compromised key" take effect in seconds.
• Clock skew: allow a small leeway; reject tokens far in the future.
• Key rotation: publish overlapping kids, rotate signing keys regularly, and keep the previous key verifiable during the max token lifetime.

Sender-Constrained Tokens: DPoP and mTLS

Bearer tokens are replayable; bind them to a key when the threat model requires it.

DPoP (RFC 9449):

• Client generates a key pair per device/session; the access token carries cnf.jkt (JWK thumbprint).
• Every request includes a DPoP header: a JWS over htm, htu, iat, jti, and optionally ath (access token hash), signed with the client key.
• Server verifies: proof signature, htm/htu match the actual request, iat within window, jti unseen (short replay cache), and thumbprint equals the token's cnf.jkt.
• DPoP-Nonce challenges (use_dpop_nonce + DPoP-Nonce header) raise security at the cost of an extra round trip; use for high-value APIs.
• DPoP is a strong fit for public clients (SPAs via BFF, mobile) and service-to-service where mTLS is impractical.

mTLS (RFC 8705):

• Client certificates on every connection; the token may carry cnf.x5t#S256 to bind it.
• The edge (mesh sidecar, ingress) terminates TLS and forwards verified identity in a header your app trusts only from the edge (strip client-supplied versions of that header).
• Short-lived certificates with automated issuance (SPIFFE/SPIRE, cert-manager) remove the need for CRL/OCSP in practice.
• Good for internal service meshes and high-security partner links; operationally heavier for arbitrary public clients.

API Keys

API keys are simple, not automatically insecure — but they are bearer credentials.

• Format: recognizable prefix (sk_live_, pk_test_), high entropy (256 bits), base62 or base64url. The prefix enables scanners and log redaction.
• Storage: store only a hash (SHA-256 with a server-side pepper, or Argon2 if you must support low-entropy keys). Show the plaintext once at creation.
• Scope and rate limit per key: keys belong to a principal, carry explicit scopes, and have their own quota. Do not create one universal key.
• Rotation: support two active keys per principal with overlapping validity, plus last-used timestamps so you can revoke unused keys.
• Transport: header only (Authorization: Bearer ... or a dedicated header), never query strings. Reject over cleartext; HSTS on the edge.
• Leak detection: integrate secret scanning (push protection in the forge, runtime scanning of logs), and support instant revocation from the console/API.
• Do not expose API keys in browser code or mobile apps; those are public clients.

Authorization Model

• Deny by default; every route declares its required scopes/permissions in the contract.
• Object-level authorization: check that the subject may act on this resource, not just that it holds a scope. Scope says "can create orders"; policy says "can refund this order".
• Do not trust client-supplied identity or tenant fields. Derive sub, tenant, and roles from the token; ignore body/query overrides.
• Centralize policy for complex rules (RBAC/ABAC/ReBAC engine or a policy-as-code module) and test it like code; scattered if user.role == checks drift.
• Admin surfaces need stronger mechanisms: step-up authentication, hardware-backed MFA, short sessions, and audit logging on every mutation.
• Return 403, not 404, when hiding existence is not a requirement; when it is, return 404 consistently for both missing and unauthorized resources and document it.

Rate Limiting and Abuse Controls as Auth

• Limit per token/client/user/tenant, not just per IP (see errors for signaling).
• Credential stuffing defense: rate limit auth endpoints aggressively, lock or delay after failures, and consider proof-of-work or bot detection at the edge.
• Tiered limits by plan are authorization-adjacent: enforce quota from the same principal model so billing and access agree.
• Denial-of-wallet: cap expensive operations per principal and per tenant; concurrency limits protect shared dependencies.

Multi-Tenancy

• Tenant identity is a first-class claim (tenant_id/org_id) minted by the authorization server, never chosen by the client at call time.
• Every data access is tenant-scoped: row-level security in the database or a mandatory repository predicate; a missing WHERE tenant_id = ? is the classic cross-tenant breach.
• Key management: per-tenant encryption keys (KMS) for data isolation where required; document the tenancy model (shared schema, schema-per-tenant, database-per-tenant) and its isolation guarantees.
• Cross-tenant operations (support tools, migrations) go through explicit, audited privileged paths, not a bypass flag.
• Token audience per tenant where regulations require isolation; otherwise enforce at the data layer.
• Testing: automated cross-tenant access tests that assert 403/404 for every resource type; include them in CI.
• Noisy neighbor: per-tenant concurrency and quota limits at the gateway.

Checklist

• [ ] Each caller class has an approved mechanism; no endpoint mixes schemes.
• [ ] OAuth 2.1: PKCE everywhere, exact redirect URIs, rotation with reuse detection.
• [ ] OIDC tokens fully validated (issuer, audience, signature via JWKS cache, nonce).
• [ ] Scopes are capability-oriented, declared per operation, and default-deny.
• [ ] Tokens are audience-bound; resource indicators used where applicable.
• [ ] Access tokens are short-lived; refresh rotation and revocation paths are implemented.
• [ ] DPoP or mTLS binding for high-value scopes and public clients.
• [ ] API keys are hashed, prefixed, scoped, rotated, and revocable.
• [ ] Object-level authorization and tenant scoping are enforced and tested.
• [ ] Auth endpoints are rate limited and abuse controls are in place.
• [ ] Edge-trusted identity headers are stripped from client requests.

Contracts and OpenAPI

OpenAPI 3.1 authoring, spec style and linting, design-first and code-first workflows, SDK generation, contract testing, mocks, and spec governance.

OpenAPI 3.1 Authoring

OpenAPI 3.1 aligns with JSON Schema 2020-12, adds full-document examples, and supports webhooks. Treat the spec as the source of truth and review it like code. (OpenAPI 3.2 exists in the 3.x line; verify upstream which minor your tooling supports before adopting new keywords such as additional methods on path items.)

openapi: 3.1.0
info:
  title: Orders API
  version: 2.3.0
  description: |
    Order management. All errors use RFC 9457 problem details.
servers:
  - url: https://api.acme.com/v2
components:
  securitySchemes:
    oauth:
      type: oauth2
      flows:
        authorizationCode:
          authorizationUrl: https://auth.acme.com/authorize
          tokenUrl: https://auth.acme.com/token
          scopes:
            orders:read: Read orders
            orders:write: Create and update orders
  schemas:
    Order:
      type: object
      required: [id, status, total]
      properties:
        id: {type: string, examples: [ord_01J8Z]}
        status: {type: string, enum: [pending, paid, cancelled]}
        total: {$ref: "#/components/schemas/Money"}
paths:
  /orders:
    get:
      operationId: listOrders
      summary: List orders
      security: [{oauth: [orders:read]}]
      parameters:
        - $ref: "#/components/parameters/Cursor"
      responses:
        "200":
          description: A page of orders
          content:
            application/json:
              schema: {$ref: "#/components/schemas/OrderList"}
        "429": {$ref: "#/components/responses/RateLimited"}
    post:
      operationId: createOrder
      security: [{oauth: [orders:write]}]
      requestBody:
        required: true
        content:
          application/json:
            schema: {$ref: "#/components/schemas/CreateOrder"}
      responses:
        "201":
          description: Created
          headers:
            Location: {schema: {type: string, format: uri}}
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Order"}
        "422": {$ref: "#/components/responses/Problem"}

Authoring rules:

• operationId is unique, stable, camelCase, and verb-led (listOrders, createOrder). SDK generators depend on it; renaming it is a breaking change for generated clients.
• Reuse via components for schemas, parameters, responses, and headers. Reference them; never copy-paste.
• Errors are first-class: declare problem-details responses for 400/401/403/404/409/422/429 per operation (or in shared components) with the specific code enum where practical.
• Examples over prose: put request and response examples on operations (3.1 supports examples at the operation level) so mocks and docs are useful without hand-authoring.
• Descriptions: units, ranges, idempotency, authentication, and rate limits; lint for missing descriptions on operations and parameters.
• Use format and constraints (format: uuid, maxLength, pattern) so generated clients validate rather than passing strings through.
• Keep info.version meaningful and sync it with the release; do not leave it at 1.0.0 forever.
• Deprecate in the spec: deprecated: true plus a description with the sunset date (see versioning).

Anti-patterns

• additionalProperties: true everywhere, erasing the contract.
• Inline schemas repeated across operations instead of components.
• A spec that documents only 200 responses.
• Descriptions that say "see wiki".

Style and Linting

• Split large specs by tag (paths/orders.yaml, paths/users.yaml) and compose with $ref; validate that every $ref resolves before merge.
• One public style guide, enforced by a linter rather than review comments. Spectral-class rulesets typically encode: operation ids present and unique, descriptions present, tags sorted, error responses documented, no trailing slashes, casing conventions, and security on every operation.
• Lint severity: errors block CI, warnings are reviewed. Keep the ruleset versioned in the repo so rule changes are visible.
• Naming: paths kebab-case nouns, schema names PascalCase, parameters camelCase, enum values UPPER_SNAKE or lower_snake (pick one).
• Tie the linter to the contract test suite: a rule should exist only if a human agreed it matters; avoid unmaintainable rule sprawl.

Design-First vs Code-First

| Approach | Flow | Strengths | Risks | |---|---|---|---| | Design-first | Spec -> review -> codegen server stubs/client -> implement | Contract reviewed before code; parallel work; SDKs early | Drift if handlers are not validated against the spec | | Code-first | Annotations/decorators -> generated spec | Low friction, single source in code | Spec is an afterthought; reviews happen too late; SDK churn | | Hybrid | Code-first for internals, design-first for the public edge | Pragmatic | Requires a clear boundary and CI diffing |

Default for public and cross-team APIs: design-first. For internal services owned by one team, code-first is acceptable if the generated spec is checked in, linted, diffed in CI, and served at /openapi.json in all environments.

• Round-trip discipline: if code generates the spec, commit the generated artifact and review its diff; never publish only at runtime.
• Serve the spec: /openapi.json (and optionally a docs UI) from the deployed service, with the same auth rules as the API.
• Version the artifact: attach the spec to releases/tags so breaking-change detection has a stable baseline (see versioning).

SDK and Codegen Workflows

• Generate, do not hand-maintain, clients from the spec. Choose one generator per language and pin its version (openapi-generator, orval, kiota, and framework-native generators; verify current support for your OpenAPI version).
• Stable operation ids and tags determine generated method and module names; treat them as API surface (renames break consumers even if HTTP did not change).
• Generated clients should: parse problem details into typed errors, expose cursor pagination helpers, handle retries with backoff for idempotent calls, and support Idempotency-Key injection.
• Automate publishing SDKs on spec release, with changelogs and semver tied to the API version. Do not let generated clients drift from the spec.
• Test the generated client against a mock server in CI (see below); an SDK that does not compile or round-trip is worse than none.
• Server codegen: generating stubs/routes from the spec prevents drift but can fight the framework; either commit to the generated layer or add request/response validation middleware at runtime.

Anti-patterns

• Hand-writing a client and calling it the official SDK.
• Publishing generated clients without versioning or changelogs.
• Different operation ids across environments, which changes generated class names.

Contract Testing

Contract tests verify that the implementation still satisfies the published contract, and that consumers and providers agree.

• Schema conformance on every response: validate real responses against the spec in integration tests (middleware or test client). This catches undocumented fields, wrong types, and missing problem details.
• Consumer-driven contracts (Pact-class): consumers publish expectations, providers verify them in CI. Best when provider changes could silently break many consumers.
• Provider-driven contracts: the published OpenAPI/AsyncAPI artifact is the contract; CI validates the server against it. Lighter, sufficient when one team owns both sides.
• Breaking-change detection: diff the candidate spec against the released baseline with oasdiff-class tooling and fail on breaking changes; require an explicit override with a linked deprecation plan.
• Example-driven tests: use spec examples as fixtures so docs stay true and request validation is exercised with realistic payloads.
• Validate requests, not only responses: reject undocumented/unknown fields where the contract says they are invalid; do not silently accept drift.
• Run the suite against every supported version (/v1, /v2) until its sunset.

Sample CI pipeline order:

1. Lint the spec (style ruleset).
2. Diff against the released baseline (breaking-change gate).
3. Generate server interfaces and clients.
4. Run contract tests (schema conformance, consumer contracts, examples).
5. Publish SDKs and attach the spec to the release.

Mocking and Sandboxing

• Spec-based mock servers (Prism, WireMock, MSW, or framework equivalents) serve examples from the spec; they unblock consumers before the implementation exists.
• Contract mocks in consumer tests keep integration tests fast and deterministic without reaching the real provider.
• Stateful mocks (a sandbox environment with real auth and data) are needed for onboarding flows that depend on sequencing; provide one when the API is a product.
• Mock fidelity: generate mocks from the same spec used by CI and refresh on every spec release; a stale mock is a source of false confidence.
• Never mock in production paths: test doubles belong in tests; production code paths must hit the real service or an explicitly documented fallback.
• Recorded fixtures/prerecorded responses help regression tests but must be scrubbed of PII and tokens before commit.

Spec Governance

• One registry/index per organization listing APIs, versions, owners, lifecycle state, and spec URLs; treat it as the catalog for discovery and deprecation tracking.
• Ownership: every spec has a CODEOWNERS entry; schema changes require the owning team's review.
• Style guide as code (ruleset in the repo) and a review checklist for new operations.
• Version the spec separately from the service only if they genuinely diverge; otherwise release them together (spec artifact + service deploy) and record both in the changelog.
• Audit gaps: a scheduled job checks that every deployed service serves a linted spec and that every spec is in the registry.
• Deprecation tracking: registry fields for deprecation date and sunset date; alert when sunset passes with traffic still recorded (see versioning).

Checklist

• [ ] OpenAPI 3.1 (or the 3.x your tooling supports) is the committed source of truth.
• [ ] Every operation has a unique stable operationId, summary, and error responses.
• [ ] Schemas live in components; no inline copy-paste.
• [ ] Examples (not just schemas) exist for requests and responses.
• [ ] Linter ruleset is versioned, enforced in CI, and blocking on errors.
• [ ] Generated clients are published, versioned, tested, and derived from operation ids.
• [ ] Contract tests validate responses against the spec and run in CI.
• [ ] Breaking-change diff runs against the released baseline.
• [ ] Mock servers are generated from the spec and refreshed on release.
• [ ] Registry lists owner, version, lifecycle state, and spec URL for every API.

Events and Webhooks

Event design, CloudEvents envelope, webhook delivery, signatures and replay protection, retries and dead letters, AsyncAPI, schema registries, and the transactional outbox pattern.

Event Design

Events describe facts that happened, not commands ("order.created", not "create.order").

• Name events noun.verb_past (order.created, payment.refunded), versioned when the payload breaks (order.created.v2).
• Include a unique event id (ULID/UUIDv7) and an occurred_at timestamp. Consumers deduplicate on the id; producers must never reuse it.
• Producer and subject: include source (producer identity) and subject (the entity id, e.g. orders/ord_01J).
• Payload is self-contained for the consumer's use case, but avoid leaking internal structure. Prefer referencing entities (order_id) over embedding entire aggregates.
• Sequence/ordering metadata: sequence per subject when ordering matters; otherwise document that ordering is not guaranteed.
• Correlation and causation ids link an event to the request and to the event that caused it, which is essential for tracing workflows.
• No PII beyond need, and never secrets. If consumers need sensitive data, send a reference they can fetch with their own credentials.
• Event vs command vs state snapshot: events are immutable facts. A consumer that needs current state should read the API; events may arrive out of order.
• Ownership: each event type has one producer. Multiple producers of the same type break ordering and schema guarantees.

{
  "id": "evt_01J8Z6H2",
  "type": "order.created.v1",
  "source": "orders-service",
  "subject": "orders/ord_01J",
  "occurred_at": "2026-01-02T03:04:05Z",
  "correlation_id": "req_01J8Z",
  "data": {"order_id": "ord_01J", "customer_id": "cus_9", "total": {"amount": 1099, "currency": "USD"}}
}

CloudEvents

CloudEvents 1.0 is the interoperable envelope; use it instead of inventing one. (Check the CloudEvents project for the current spec minor version and any newer release before pinning.)

Required attributes: id, source, specversion, type. Common optional: datacontenttype, dataschema, subject, time, data.

| Binding mode | How | Use when | |---|---|---| | Structured | Event attributes in the JSON envelope (application/cloudevents+json) | HTTP webhooks, brokers with JSON payloads, readability | | Binary | Attributes as HTTP headers (ce-id, ce-type, ...), data as the body | Existing HTTP pipelines, no body wrapping, proxies preserve headers | | Batch | Array of structured events | High-volume delivery, fewer HTTP round trips |

• dataschema points to the JSON Schema for data; serve it at a stable, versioned URI.
• time equals occurred_at; do not overwrite it at dispatch time (add a separate delivery timestamp if needed).
• Extensions (traceparent, tenant, sequence) are lower-case and must not collide with required names.
• Broker mappings: Kafka/NATS/SQS transports have CloudEvents protocol bindings; follow them so consumers can use generic SDKs.

Anti-patterns

• Wrapping CloudEvents inside another payload envelope "just in case".
• Using time as the delivery time so retries change the event.
• Custom attributes in PascalCase, breaking generic tooling.

Webhook Delivery

Webhooks are HTTP callbacks from producer to consumer. Assume hostile networks and flaky consumers.

• Endpoint registration: consumers supply an HTTPS URL; verify ownership (challenge response or a signed ping) and allow per-endpoint secret rotation and event-type filters.
• Attempt semantics: at-least-once. Duplicates are expected; consumers deduplicate on the event id.
• Delivery timeout: aggressive (for example 5-15 s); a slow consumer should not stall the pipeline. Deliver asynchronously with a queue and workers.
• Retries: exponential backoff with jitter, capped (for example up to 24-72 h), with documented schedule. Honor consumer 429/503 + Retry-After where present.
• 2xx means accepted. Document whether 202 is acceptable and whether body content is read. Do not require a response body.
• Dead-letter after retries exhausted; keep the event retrievable for manual replay and notify the consumer (dashboard, email, or API for listing failed deliveries).
• Ordering: HTTP delivery is unordered across retries. Include sequence and document that consumers must handle out-of-order or reorder locally. If ordering is mandatory, use a broker with partitioned ordering, not webhooks.
• Fan-out: one event to many endpoints, isolated per endpoint; one failing endpoint must not delay others.
• Manual replay by id/time range with the same signature scheme, plus audit logging.
• Payload size: keep webhooks small; link to the resource for large data.
• Observability: per-endpoint delivery success rate, latency, and queue depth; alert on sustained failures.

Anti-patterns

• Synchronous webhook calls inside a request transaction (latency and rollback coupling).
• Treating a non-2xx as "consumer is broken" without retries or visibility.
• Retrying forever with no dead letter; a poison event blocks progress.
• Requiring consumers to fetch a token from the sender API just to receive webhooks.

Signatures and Replay Protection

Sign every webhook. A shared secret plus HMAC over timestamp and body is the baseline (Standard Webhooks-style); asymmetric signatures (ed25519) are preferable when many parties verify.

• Sign timestamp + "." + raw_body with HMAC-SHA256 using a per-endpoint secret. Include the timestamp in a header (Webhook-Timestamp) and one or more signatures (Webhook-Signature: v1,<base64url>), so secrets can rotate without breaking consumers.
• Verify over the raw body bytes, before JSON parsing; any reserialization changes the signature.
• Constant-time comparison; never == on digests.
• Replay window: reject timestamps outside a small tolerance (for example 5 minutes) and optionally cache event ids/jti for the window.
• Key rotation: support two active secrets (v1a, v1b) and publish a rotation schedule; give consumers overlap to update.
• Asymmetric option: publish a JWKS and sign the envelope (JWS) so consumers verify with a public key; avoids secret distribution at scale.
• Never send the secret in the payload or URL, and never log it.

import hashlib, hmac, base64, time

def sign(secret: bytes, body: bytes, ts: int | None = None) -> str:
    ts = ts or int(time.time())
    msg = str(ts).encode() + b"." + body
    mac = hmac.new(secret, msg, hashlib.sha256).digest()
    return f"t={ts},v1={base64.urlsafe_b64encode(mac).decode().rstrip('=')}"

AsyncAPI and Schema Registries

• AsyncAPI (2.x and 3.x; confirm which major your tooling supports) documents channels or operations, message schemas, bindings, and security for brokers and webhooks. Use it as the contract for event surfaces the same way OpenAPI is used for HTTP.
• AsyncAPI 3 reorganized around operations, channels, and messages; treat a migration from 2.x as a tooling project, not a mechanical rewrite.
• Generate types and docs from AsyncAPI; validate published messages against schemas in CI.
• Schema registry (Confluent-compatible, Apicurio, or cloud-native) is the source of truth for message schemas on Kafka-class transports. Subject naming: topic-value, or domain.event.type-value; document it.
• Compatibility mode per subject: BACKWARD (default for consumers-first), FORWARD, FULL, or NONE (discouraged). Adding optional fields is backward-compatible; removing or retyping is not (see versioning).
• Serialization: Avro or Protobuf with a registry is compact and enforces schemas; JSON Schema with dataschema in CloudEvents is more interoperable but larger and looser. Pick per transport and document.
• Envelope vs payload schema: version the event type in the envelope; schemas evolve under registry compatibility rules.

Transactional Outbox

Never publish an event directly from a request handler between the database commit and the broker call; a crash or broker outage loses or duplicates events. Use an outbox.

1. In the same transaction as the state change, insert the event into an outbox table (id, type, payload, headers, created_at, published_at null).
2. A relay (poller or CDC) reads unpublished rows and publishes them, marking them published atomically (or via CDC without a mark).
3. Consumers deduplicate on the event id; the outbox gives at-least-once.
4. Prune published rows after a retention window.

| Relay | Mechanism | Notes | |---|---|---| | Polling publisher | SELECT ... FOR UPDATE SKIP LOCKED + publish + mark | Simple, no new infra; adds small latency | | CDC (Debezium-class) | Tail the WAL/binlog of the outbox table | Lowest latency, no polling; requires CDC infrastructure and schema discipline | | Transaction log tailing | Database logical replication to a relay | Similar to CDC; couples to DB features |

• Ordering: publish per aggregate key (partition by subject) and include sequence; global ordering is not achievable at scale.
• Inbox pattern on the consumer side stores processed event ids to make consumption idempotent.
• Exactly-once is a trade-off, not a guarantee. Even with Kafka transactions, external side effects (email, payment) need idempotency keys; design consumers to be idempotent by construction.
• Schema changes to the outbox payload follow the registry compatibility mode.
• Failure visibility: alert on relay lag, outbox depth, and publish errors; a stuck relay silently freezes the event stream.

Anti-patterns

• Publishing after commit without a fallback (dual-write problem).
• Marking published before the broker acknowledges.
• Treating the outbox as a permanent event archive; prune or move to a store.
• Consumers that assume exactly-once and mutate state non-idempotently.

Checklist

• [ ] Events are past-tense facts with unique ids, source, subject, and occurred_at.
• [ ] CloudEvents 1.0 envelope with structured or binary binding; dataschema published.
• [ ] Webhook endpoints verified, filterable, and independently retried.
• [ ] Every webhook is signed over the raw body with a timestamp and rotation plan.
• [ ] Consumers deduplicate; replay window enforced; constant-time comparison.
• [ ] Retry schedule, timeouts, dead-letter, and manual replay are documented and observable.
• [ ] Delivery is at-least-once, ordering is either not promised or explicitly sequenced.
• [ ] AsyncAPI (or registry) documents channels, messages, and compatibility mode.
• [ ] Producers use a transactional outbox with lag alerting; consumers use an inbox/idempotency.
• [ ] Event schema changes go through compatibility checks and the sunset rules for versions.