Measurement Methodology

Scope: measurement discipline for performance work — USE and RED, percentiles and tails, Little's Law, Amdahl's law, performance budgets, and benchmarking hygiene.

Measurement Discipline

• Start from a question, not a tool: "Is p95 checkout under 400 ms at 2x peak traffic?" is measurable; "make it faster" is not.
• Capture a baseline before touching anything, under conditions you can recreate: same build type, same data volume, same hardware class, same load model.
• Change one variable at a time. Bundled changes make attribution impossible.
• Record the method next to the number: build, commit, environment, load level, sample window, and percentile.
• Prefer production evidence when available; production is the only environment with real data shape, cache state, and traffic mix.
• Treat every performance number as a distribution, never a scalar. A single "average" has hidden the last decade of performance incidents.

The Investigation Loop

1. Define the metric and the budget (latency percentile, throughput, memory, cost).
2. Measure the current state and quantify the gap against the budget.
3. Classify where time or capacity goes: CPU, memory, I/O, locks, queueing, downstream dependency.
4. Form one hypothesis and predict its effect if true.
5. Test the hypothesis with the smallest safe experiment (profile, query plan, canary, load test slice).
6. Fix, re-measure under the same conditions, and store the before/after with the change.
7. Guard the result with a CI gate, dashboard, or alert so it cannot silently regress.

USE and RED

Two complementary checklists. Use them before reaching for a profiler.

• USE — for every resource (CPU, memory, disk, network, connection pools, thread pools, queues): Utilization, Saturation, Errors.
• RED — for every service: Rate, Errors, Duration (as a distribution).

| Question | Method | Typical signals | |---|---|---| | Is a resource near its limit? | USE | CPU %, run-queue length, memory used, pool in-use, disk await, retransmits | | Is a service healthy? | RED | requests/s, error ratio, latency histogram | | Is the system queueing? | USE saturation plus Little's Law | queue depth, wait time, in-flight requests | | Is the tail caused by the resource or the code? | Histogram plus profile | p99 trend vs saturation, flamegraph | | Did a deploy regress it? | RED split by version | latency per service.version |

• Utilization alone lies. A pool at 60 percent utilization can still produce multi-second queue waits if service-time variance is high.
• Saturation is the leading indicator; errors are the lagging one. Alert on saturation and latency before errors spike.
• For every resource, write down its unit, ceiling, and current headroom. Unknown ceilings cause surprise outages.

Percentiles and Tails

• Report p50, p95, p99 for most APIs; add p99.9 for payment, auth, and control-plane paths. State the window and load level with every number.
• Never average percentiles across instances or time buckets. Average the underlying histograms, then compute the percentile.
• Percentile accuracy needs samples: you cannot resolve p99.9 from 1,000 requests. Rule of thumb: collect at least 100x the sample size of the percentile you claim.
• Use exponential or log-linear histogram buckets so tail resolution is not lost; the oldest fixed linear buckets are the most common cause of useless p99 dashboards.
• Heatmaps over time show bimodality and shifts that a single percentile hides. A rising p99.9 with flat p50 usually means a growing slice of slow paths, not a uniform slowdown.

Common tail sources, in rough order of frequency:

| Source | Signature | Where to look | |---|---|---| | Queueing at a saturated resource | latency rises non-linearly with load | pool, queue, CPU, DB | | GC pauses or collector work | periodic spikes, CPU frames in collector | ./07-memory-gc.md | | Lock contention | spikes under concurrency, flat when single-threaded | ./04-backend-profiling.md | | Cache misses or cold starts | spikes after deploys or expiry windows | ./02-caching.md | | Fan-out to N dependencies | tail is max of N, grows with N | ./06-concurrency-backpressure.md | | Retries and timeouts | clusters at timeout boundaries | ./06-concurrency-backpressure.md | | Noisy neighbors or throttling | host-correlated, sporadic | ./09-capacity-cost.md |

• Tail latency interacts with fan-out: at 50 ms p99 per call, a request touching 20 calls in parallel has an expected worst-call tail far above 50 ms. Reduce fan-out or hedge deliberately.
• Percentiles from a closed-loop benchmark understate real tails because clients wait (coordinated omission). Use arrival-rate models. See ./08-load-testing.md.

Little's Law

The most useful back-of-envelope formula in performance work:

L = lambda * W
concurrency = arrival rate * time in system

Where L is the number of items in the system, lambda the arrival rate, and W the average time each item spends.

• 1,000 req/s at 80 ms end-to-end means about 80 requests are in flight. Size pools, workers, and instance limits to cover that number with headroom.
• Required DB connections: arrival rate * average query time. 500 queries/s at 10 ms is about 5 busy connections; with headroom and variance, size for a multiple, not the minimum.
• Queue wait is a function of utilization. An M/M/1 approximation: Wq = rho / (1 - rho) * service time. The same service time at 50 percent utilization queues about 1x service time; at 90 percent it queues about 9x.

| Utilization | Approx. queue factor | Practical reading | |---|---|---| | 50% | ~1x | Comfortable; room for bursts | | 70% | ~2.3x | Normal operating target | | 80% | ~4x | Tails become visible | | 90% | ~9x | Fragile; small spikes hurt | | 95% | ~19x | Incident waiting to happen |

• Convert every latency SLO into a concurrency limit: max in-flight = throughput target * latency budget. Enforce it rather than hoping load balancers distribute perfectly.
• Little's Law applies to any stable queue: message consumers, thread pools, CDN requests, database sessions. Use it to reconcile "we have capacity" claims with observed queue depth.

Amdahl and the Scalability Law

• Amdahl: if a fraction s of the work is serial, speedup is capped at 1/s no matter how many workers you add. S(n) = 1 / (s + (1-s)/n).
• A 10 percent serial fraction caps total speedup at 10x. Optimization effort spent parallelizing the 90 percent has a hard ceiling until the serial part shrinks.
• Serial sources are often hidden: a shared lock, a single writer, a global counter, one database primary, a synchronous audit log, a global rate limiter.
• The Universal Scalability Law adds coherency cost: past a point, adding workers can reduce throughput because coordination grows. Retrogression is real; measure scaling curves by traffic level, not just at one point.
• Practical rule: find the serial fraction first. Parallelizing an already-parallel section is often wasted work.

Performance Budgets

A budget turns performance into an engineering constraint that can be reviewed, tested, and alerted on.

| Budget type | Example | Enforced by | |---|---|---| | Latency | API p95 < 300 ms server time | Alert on histogram, CI threshold on benchmark | | Frontend | LCP p75 < 2.5 s, INP p75 < 200 ms | RUM alert, Lighthouse CI budget | | Throughput | 2x peak with p99 < 1 s | Load test gate, capacity plan | | Resource | < 100 MB RSS per pod, < 2 CPU cores per 1k rps | Limits, saturation alerts, profiling | | Connections | DB pool never > 80 percent busy at peak | Pool metrics alert | | Cost | < $0.002 per 1k requests | Cost dashboard and unit review | | Regression | no benchmark slower by more than noise band | CI benchmark comparison |

Latency budget example for a 400 ms p95 API endpoint:

| Segment | Budget | |---|---| | Client network | 40 ms | | Edge / gateway | 20 ms | | Auth / middleware | 25 ms | | Application logic | 80 ms | | Database | 150 ms | | Serialization and payload | 25 ms | | Slack for variance | 60 ms |

• Allocate budgets before optimizing; when a segment exceeds its share, that is the target.
• Budgets must have an owner and a consequence (build failure, alert, ticket). A budget nobody enforces is documentation.
• Set budgets from user impact: research on abandonment and conversion thresholds, not from what the current system happens to do.

Benchmarking Hygiene

• Warm up before measuring: JIT compilation, lazy initialization, connection pools, and caches all change behavior in the first seconds.
• Measure steady state, then report variance. If run-to-run spread exceeds the effect you are claiming, the claim is noise.
• Use open models (arrival-rate) for latency claims; closed-loop models hide coordinated omission and can overstate capacity by large factors. See ./08-load-testing.md.
• Pin and document the environment: CPU model and frequency, memory, container limits, kernel and runtime versions, database volume. Compare like with like.
• Disable or account for background work (autovacuum, backups, cron, log shipping) during benchmark windows.
• Run at least 3-5 repetitions and report median plus range; never pick the best run.
• For microbenchmarks, use a real harness (JMH, criterion, pytest-benchmark, hyperfine) to avoid dead-code elimination, constant folding, and loop-invariant hoisting.
• Validate that the benchmark measures what you think: instrument it, count calls, and confirm the work actually happens.

Microbenchmark pitfalls worth stating explicitly:

| Pitfall | Effect | Mitigation | |---|---|---| | Dead-code elimination | Optimizer removes the work | Consume results via side effect or sink | | Constant folding | Same inputs optimized away | Vary inputs from an array or runtime source | | Loop-invariant hoisting | Computation moves out of the loop | Time inside the loop body explicitly | | Timer resolution | Tiny operations measure as zero | Batch iterations, use monotonic clocks | | Warmup mixing | Mean skews high or low | Discard warmup runs, report medians |

Reading Results Without Fooling Yourself

• Compare against the budget first; ignore improvements that do not move the target metric.
• Check the load level alongside the number. "p99 improved" is meaningless if the new test ran at half the traffic.
• Look for discontinuities: a knee in the throughput/latency curve marks the real operating ceiling, not the peak number observed.
• Distinguish throughput-limited from latency-limited results; the fix is different.
• Treat single-run wins under 2x the noise band as unproven and repeat the measurement.

Anti-Patterns

• Optimizing from intuition with no baseline, then declaring victory without a re-measurement.
• Reporting an average latency and calling it the latency.
• Averaging percentiles across hosts or minutes.
• Comparing benchmarks run on different hardware, builds, or data volumes.
• Closed-loop load testing and then quoting the resulting percentile as the production tail.
• Tuning the benchmark rather than the system (shorter payloads, warm caches, seed data).
• Chasing a 5 percent win while a 10x pathological path remains on the critical route.
• Measuring only with empty caches and then promising cache-friendly numbers.
• Letting performance numbers live in a chat thread instead of next to the change.

Checklist

• [ ] The question, metric, percentile, load level, and budget are written down before work starts.
• [ ] A reproducible baseline exists in a comparable environment.
• [ ] USE and RED checked for every component on the path.
• [ ] Latency reported as percentiles with sample size and window; no averaged percentiles.
• [ ] Little's Law used to sanity-check concurrency, pool sizes, and queue depths.
• [ ] Serial fraction identified before parallelization work.
• [ ] Budgets have owners and enforcement (CI gate or alert).
• [ ] Benchmarks use open models, warmup, repetitions, and documented environments.
• [ ] Before/after numbers recorded with the change and re-measured after release.

Caching

Scope: cache layers from browser to database, HTTP caching semantics, application and Redis/Valkey patterns, invalidation, and stampede protection.

Cache Layer Map

Order layers by distance from the user; each layer removes load from everything behind it.

| Layer | Typical latency | Scope | Eviction | Typical use | |---|---|---|---|---| | Browser cache | 0 ms (local) | Per user/device | TTL, storage pressure | Static assets, immutable bundles | | Service worker / app cache | ~0 ms | Per user origin | Explicit, versioned | Offline assets, API read cache | | CDN / edge | 5-40 ms | Shared, global | TTL plus purge | Static assets, public API responses | | Reverse proxy / gateway | ~1 ms | Shared, per region | TTL, LRU | Authenticated API reads, fragments | | In-process (L1) | microseconds | Per instance | LRU/TTL, size-bounded | Hot config, dictionaries, computed values | | Shared cache (Redis/Valkey) | 0.5-3 ms | Fleet-wide | TTL, LRU/LFU | Sessions, rate limits, shared reads | | Database cache / materialized view | query-dependent | Shared | Engine-managed | Precomputed aggregates, denormalized reads |

• Every layer is a consistency decision: each hop you add increases the set of things that can be stale.
• The fastest cache hit is one that never has to traverse the network; prefer L1 for hot immutable data with a memory ceiling.
• Do not use a cache to hide a query you could fix; indexed reads beat cached reads on correctness and cost. See ./03-databases.md.

Correctness Before Speed

• Define the cache key identity explicitly. What inputs change the response? User, tenant, locale, role, feature flag, device class, API version.
• Private data must be scoped by user or tenant in the key, and must not be stored in shared layers without that scope. Cross-tenant leakage from a cache is a security incident.
• Be deliberate about negative caching: cache "not found" briefly to absorb abuse, but never cache transient errors as if they were data.
• Distinguish client-visible validation from server-side reuse. A 304 Not Modified can save bandwidth without saving origin compute.
• Decide freshness tolerance per data class: prices and balances may be seconds; permissions and feature flags often need explicit invalidation.
• Write down the answer to "how does this entry become correct again after a mutation?" before the cache ships.

HTTP Caching

HTTP caching is the highest-leverage, lowest-cost layer because it is implemented by browsers and CDNs.

| Directive | Meaning | |---|---| | max-age=N | Fresh for N seconds in any cache | | s-maxage=N | Fresh for N seconds in shared caches; overrides max-age there | | public / private | Shared caches may / may not store | | no-store | Do not store at all (sensitive responses) | | no-cache | May store, but must revalidate before each use | | must-revalidate | Cannot serve stale once expired | | immutable | Do not revalidate during freshness (fingerprinted assets) | | stale-while-revalidate=N | Serve stale for N seconds while refreshing in background | | stale-if-error=N | Serve stale for N seconds if origin errors | | Vary | Key on request headers (for example Accept-Encoding) |

• Fingerprinted static assets: Cache-Control: public, max-age=31536000, immutable, with a new filename per release.
• HTML and API reads: short max-age, longer s-maxage, plus stale-while-revalidate to absorb bursts without origin load.
• Use ETag or Last-Modified validators for mutable resources so conditional requests return 304 cheaply. Weak validators are acceptable when byte equality is not required.
• Vary is part of the key: a careless Vary: * or Vary: Cookie makes the cache useless or unsafe. Prefer explicit Vary: Accept-Encoding and route identity through the URL or explicit cache keys.
• CDN purge is an operational dependency: know the purge latency, API limits, and failure behavior before promising "instant" invalidation.
• Cache poisoning: never let user-controlled headers or parameters enter shared cache keys in a way that lets one user poison responses for others. Normalize inputs and strip unknown keys.
• Cookie-bearing responses default to private; verify that sessions are not accidentally shared by an over-broad CDN rule.

Application Patterns

| Pattern | Flow | Strengths | Weaknesses | Use when | |---|---|---|---|---| | Cache-aside | App reads cache, on miss reads source and populates | Simple, resilient, lazy | Race on fill, first-miss slow | General purpose | | Read-through | Cache library loads on miss | Uniform code path | Needs library support | Repeated reads of same shape | | Write-through | Write cache and source together | Fresh, predictable | Write latency includes cache | Strong read-your-write needs | | Write-behind | Write cache, flush source asynchronously | Fast writes | Data-loss window, complexity | High write throughput, loss-tolerant | | Refresh-ahead | Refresh before expiry | Few cold misses | Wasted refresh on cold keys | Hot keys with strict latency | | Negative caching | Cache misses briefly | Protects against abuse | Can hide new data | Scanned or probed identifiers |

• TTLs plus jitter: add random +/- 10-20 percent to TTLs so entries do not expire in synchronized waves.
• Key design: stable prefix, tenant or user segment, entity id, schema or version suffix. Include a version component so a data-shape change invalidates implicitly.
• Serialization costs count: JSON is portable but expensive at volume; compact binary formats reduce CPU and bytes. Measure before choosing.
• Compress large values, but do not compress tiny ones; compression has fixed overhead and CPU cost.
• Bound the in-process cache by entry size and count; unbounded maps are memory leaks that happen to have a TTL. See ./07-memory-gc.md.
• Cache the result of expensive deterministic computation by input hash, never by a mutable object identity.

Redis and Valkey Patterns

• Redis 7.x and Valkey 7.x/8.x era APIs are broadly compatible; clustering, scripting, and eviction semantics differ in details. Verify the exact feature set upstream for your distribution and version.
• Choose the data structure for the operation: strings for counters and blobs, hashes for objects, sorted sets for rankings and sliding windows, sets for membership, streams for event fan-out.
• Set maxmemory and an eviction policy deliberately: noeviction for authoritative data (fail writes rather than silently drop), allkeys-lru or allkeys-lfu for pure caches. volatile-* policies only evict keys with TTLs.
• Use pipelining or multi-key commands to cut round trips; latency is often RTT-bound, not CPU-bound.
• Lua scripts and MULTI/EXEC give atomicity for read-modify-write; prefer scripts for idempotent compound operations, and keep them short.
• Keyspace design: use hash tags to colocate related keys in cluster mode; avoid hot keys that land on one shard. Replicate or shard hot keys deliberately.
• Separate logical databases or instances by use: sessions, cache, rate limits, and queues have different failure and persistence needs.
• Persistence and replication affect latency: AOF fsync policies and replica syncs can pause writes. Decide whether the cache is allowed to be lost entirely.
• Use cache timeouts and a circuit breaker. If the cache is down, the application must degrade to the source of truth, not hang.

Invalidation Strategies

| Strategy | Mechanism | Latency to correct | Notes | |---|---|---|---| | TTL expiry | Time-based | Up to TTL | Simplest; tune per data class | | Versioned keys | Key includes content or schema version | Immediate on version bump | Old keys age out; doubles storage briefly | | Write-invalidate | Delete or update keys on mutation | Near-immediate | Must cover all mutation paths | | Write-through | Update cache in the same transaction path | Immediate | Write cost; partial failure handling | | Event-driven | CDC or pub/sub triggers invalidation | Seconds | Decoupled but adds moving parts | | Purge API | CDN or gateway purge by key/tag | Seconds to minutes | Vendor limits and propagation delays |

• The classic race: request A reads stale value, request B updates the source and evicts, request A then writes the stale value into the cache. Mitigations: versioned writes, delete-after-write with a short delay, or write-through with version checks.
• Update paths that bypass the invalidation hook are the most common source of stale data. Centralize writes or enforce invalidation in one layer.
• Prefer invalidation by version or generation over enumerating keys; pattern deletes are slow and dangerous at scale.
• For search or list caches, cache the page identity and fetch entities from a fresher store so entity updates do not require purging every list.

Stampede and Failure Protection

• A cold key under high concurrency triggers many identical origin requests at once. Single-flight or request coalescing collapses them into one.
• Implementation options: per-key mutex or promise map in-process, distributed lock (with a timeout), or a cache library with load coalescing.
• Probabilistic early expiration (XFetch): refresh a key slightly before expiry with a probability proportional to remaining time, spreading refreshes.
• Serve stale on origin error where the business tolerates it (stale-if-error); fail fast where it does not.
• Warm critical caches on deploy or after eviction events so the first users do not pay for everyone.
• Hot keys: a single key exceeding one shard's capacity needs key splitting or a local L1 in front of the shared cache.
• Decide what happens when the cache dies: fail open (hit the database and risk overload) or fail closed (reject traffic). Most read paths should fail open with concurrency limits. See ./06-concurrency-backpressure.md.

Metrics and Operations

| Metric | Why | Watch for | |---|---|---| | Hit ratio (overall and per key class) | Direct input to origin load | Aggregate ratios hide cold classes | | Miss cost | Origin time per miss | A 90 percent hit ratio can still be bad if misses are expensive | | Evictions and expirations | Working-set fit | Rising evictions mean undersized cache | | Memory used and fragmentation | Capacity planning | Fragment ratio and allocator overhead | | Cache call latency p99 | User-facing impact | Tail latency from network or hot shards | | Error and timeout rate | Failure mode | Must trigger degradation, not hangs | | Stale-serve count | Correctness exposure | Should be visible and bounded |

• Test invalidation explicitly: mutate, then assert the cache and origin agree within the stated window.
• Load-test cold-cache and cache-down scenarios; both are common post-deploy incidents. See ./08-load-testing.md.

Anti-Patterns

• Caching without a defined invalidation path, hoping TTL will be short enough.
• Unbounded in-process caches with no size limit.
• Sharing one cache namespace across environments or tenants.
• Caching errors or empty results indefinitely.
• Using cache to mask an unindexed query.
• Synchronized TTLs that expire a whole working set at once.
• No single-flight protection on an expensive key.
• Treating a cache outage as impossible; no timeout, no fallback, no breaker.
• Invalidating by scanning or pattern-deleting production keys at scale.
• Ignoring serialization and compression CPU cost in the cache's own budget.

Checklist

• [ ] Cache key identity and scope (user, tenant, locale, version) are explicit.
• [ ] Invalidation story documented: TTL, version, write-invalidate, or purge, with expected correction window.
• [ ] TTLs jittered; critical keys warmed or refreshed ahead.
• [ ] Single-flight or distributed lock protects expensive misses.
• [ ] Cache-down behavior defined (fail open with limits, or fail closed) and tested.
• [ ] maxmemory and eviction policy set deliberately; not silently dropping authoritative data.
• [ ] Hit ratio, miss cost, evictions, and p99 cache latency are monitored.
• [ ] Private data never stored in shared layers without key scoping.
• [ ] CDN purge latency and limits verified upstream; release process updates fingerprinted assets.
• [ ] Cold-cache and cache-failure load tests exist.

Databases

Scope: indexing strategy, query plan reading, N+1 detection, connection pool sizing, partitioning, read replicas, and the slow query workflow.

Indexing Strategy

An index is a write-time tax paid for read-time speed. Add one only when a measured query needs it.

| Index type | Best for | Cost / caveat | |---|---|---| | B-tree | Equality, range, ordering, LIKE 'prefix%' | Default; can bloat and need maintenance | | Composite B-tree | Multi-column predicates and sort order | Column order decides usability | | Covering (INCLUDE) | Avoiding heap or table fetches | Wider index, more write cost | | Partial | Hot subsets (WHERE status = 'open') | Only usable when the predicate is implied | | Expression / functional | Queries on lower(email) or computed values | Must match the expression exactly | | Hash | Equality only (rarely better than B-tree) | No range or ordering; crash-safety history | | GIN | JSONB containment, arrays, full-text search | Slow writes, fast containment | | GiST / SP-GiST | Geometry, ranges, nearest-neighbor | Specialized operators and selectivity | | BRIN | Very large append-only tables, naturally ordered | Small and cheap; coarser filtering |

• Column order in a composite index follows the query: equality predicates first, then range or sort columns. An index on (a, b) serves a and a, b but not b alone.
• Selectivity matters: an index on a boolean or gender column rarely helps alone, but works as the leading column of a filtered composite or partial index.
• Covering indexes trade storage for eliminating random heap access on hot queries; verify the plan actually stops fetching.
• Every index slows inserts, updates that touch indexed columns, and vacuum. Find unused indexes (pg_stat_user_indexes) and drop them; verify upstream before trusting any single view.
• Watch index bloat: bulk deletes and updates leave dead entries. Monitor size versus live rows and plan REINDEX CONCURRENTLY where supported.
• Statistics drive plans. After large data changes, run ANALYZE; for correlated columns, consider extended statistics. Raise statistics targets on columns used in selective predicates.
• New indexes on large production tables: build concurrently or online; know the lock level of the exact operation before running it.

Reading Query Plans

• Use EXPLAIN (ANALYZE, BUFFERS) for real timings, row counts, and cache behavior. Plain EXPLAIN shows estimates only.
• Compare estimated to actual rows. A misestimate by 100x or more is the usual root cause of a bad join strategy or a spill to disk.
• Scan nodes: sequential scan (fine for large result fractions), index scan, index-only scan (covering), bitmap heap scan (moderate selectivity), parallel variants.
• Join nodes: nested loop (good for small outer sets, dangerous when the outer is large), hash join (good for large unsorted sets, memory-sensitive), merge join (pre-sorted inputs).
• Red flags: nested loop with a huge outer side, sorts with external merge Disk in the plan, Rows Removed by Filter far above returned rows, hash batches exceeding work_mem, repeated inner scans.
• Set work_mem deliberately per session or per workload, not globally huge: it is per operation and multiplied by concurrency.
• MySQL: EXPLAIN ANALYZE (8.0.18+) and EXPLAIN FORMAT=JSON expose actual timings and rows_examined_per_scan. Watch Using filesort, Using temporary, and high rows_examined relative to returned rows.
• Non-sargable predicates defeat indexes: functions on the indexed column, implicit casts, LIKE '%term', arithmetic on the column, and OR across different columns. Rewrite, add an expression index, or split the query.
• Avoid deep OFFSET pagination; it reads and discards rows. Use keyset (seek) pagination on an indexed, ordered key.

Rewrites That Pay

| Anti-pattern | Rewrite | |---|---| | Function on an indexed column | Precompute the value or add an expression index | | LIKE '%term%' | Full-text or trigram index; prefix search where acceptable | | OR across different columns | UNION ALL of indexed branches, or one composite index | | Deep OFFSET | Keyset pagination on (sort_key, id) | | Repeated aggregate in a loop | GROUP BY in one query or a materialized rollup | | SELECT * on wide rows | Select needed columns; enable index-only access | | Row-by-row upserts | Set-based upsert, bulk INSERT ... ON CONFLICT, or COPY |

• Verify every rewrite with the same plan inspection; some help only at certain data distributions.
• Keep query shapes stable across parameter values so plans do not flip between fast and slow.

N+1 Detection and Fixes

N+1 is the most common application-level database performance bug: one query for the list, then one per element.

| Signal | Where to see it | |---|---| | Query count scales linearly with result count | ORM logs, pg_stat_statements call counts, APM span counts | | Same statement shape repeated with different IDs | Slow query log, statement fingerprints | | Request latency proportional to collection length | Traces with hundreds of sibling database spans | | Database CPU high while unique queries are few | Fingerprint aggregation |

• Detect in tests: assert a maximum query count for key endpoints. A list endpoint that performs 500 queries with 500 rows should fail the suite.
• Fix with eager loading: JOIN or IN (...) prefetch, ORM select_related/joinedload/include, or a dataloader that batches lookups per request tick.
• Batch size has limits: chunk IN lists (hundreds to low thousands of ids) and keep parameter counts inside protocol and plan-cache limits.
• Enforce the policy: configure ORM lazy-loading to raise in test or production profiles where an accidental lazy load is a defect.
• GraphQL and RPC layers need the same discipline: dataloaders per request scope, deduplicated by key.
• Watch fan-out amplification: a serializer that lazily touches associations can reintroduce N+1 behind an otherwise fixed endpoint.

Connection Pool Sizing

Every application instance opens a pool; the fleet total must fit the database.

pool_size * app_instances + admin_reserve <= server_max_connections

• Server-side memory per connection is real (work memory, session state, TLS). Hundreds of idle connections cost memory and scheduler overhead for no throughput.
• A widely used starting heuristic (HikariCP and similar): connections = (cpu_cores * 2) + effective_spindles, then adjust from measured wait time. Small pools with fast queries often beat large pools; the database is a shared resource, not a queue to enlarge at will.
• Little's Law for pools: minimum busy connections equals queries_per_second * average_query_time, adjusted upward for variance and headroom. See ./01-methodology.md.
• Set an acquire timeout. A request that waits forever for a connection turns pool exhaustion into a total outage; fail fast and shed instead.
• Monitor: pool in-use ratio, wait time (p95/p99), acquisition timeouts, and connection churn. Churn means connections are being opened and closed too often.
• Pool exhaustion has three common causes: a slow query holding connections, a leak (missing release on an error path), and a burst exceeding the pool's throughput ceiling.

| Symptom | Likely cause | Action | |---|---|---| | Wait time rises with traffic, DB CPU modest | Pool too small or queries too slow | Size from Little's Law, profile slow queries | | Connections exhausted during errors | Leak on exception path | Add release guarantees, test failure paths | | DB CPU saturated, pool queued | Too much concurrency at the database | Reduce pool, batch, cache, scale reads | | High connect/disconnect rate | No pooling or aggressive idle eviction | Use a pool; tune idle timeout and lifetime | | Prepared statements break through PgBouncer | Transaction pooling mode resets session state | Use statement-safe mode or named prepared statement support; verify upstream |

• External poolers (PgBouncer, pgpool, ProxySQL) add another queue. Sizing now spans app pool, pooler pool, and server limit; document all three.
• Serverless and bursty workloads need pooled access paths; opening a connection per invocation does not scale. Verify managed pooler limits upstream.

Partitioning

• Partition when a table is too large to vacuum, back up, or query efficiently, or when retention is a regular operation.
• Range partitioning on time is the classic fit: periodic partitions plus DROP PARTITION for retention instead of mass DELETE.
• List partitioning isolates tenants or regions; hash partitioning spreads write-heavy tables evenly when no natural range exists.
• Partition pruning only works when queries filter on the partition key. A predicate wrapped in a function or cast may disable pruning.
• Global uniqueness and foreign keys behave differently across engines; verify constraints per engine version before relying on them.
• Do not over-partition: thousands of empty partitions slow planning and catalog work. Pre-create only the horizon you need and automate creation.
• Keep a default partition only if you have a monitor for rows landing there; it silently defeats pruning and can block attaching new partitions.
• Indexes are per-partition on many engines; plan index count and maintenance time with partition count in mind.
• Rebalance cautiously: detach, copy, and attach with minimal locks; test on a clone first.

Read Replicas

• Replicas scale read throughput and offload reporting, but introduce lag. Define the lag budget per use case (for example, under one second for product pages, minutes for analytics).
• Read-your-writes: after a user writes, their subsequent reads must see it. Route that session to the primary for a bounded window, or track a write timestamp and require a replica at least that fresh.
• Monotonic reads: avoid bouncing a user between replicas with different lags so they see data move backwards.
• Replica lag grows under write bursts, long-running queries on the replica, and network issues. Monitor and alarm on lag; stop routing reads when it exceeds budget.
• Async replication loses recent writes on failover; synchronous replication trades write latency for durability. Choose per data class.
• Route explicitly: separate connection pools and clear naming (primary, replica) so a read accidentally hitting the primary is visible in code review.
• Long analytical queries on replicas can conflict with vacuum or cause lag; isolate heavy analytics from user-facing replicas where possible.

Slow Query Workflow

1. Capture: enable the slow query log with a threshold, aggregate with pg_stat_statements or the MySQL performance schema, and correlate with APM traces.
2. Rank by total time (calls * mean_time), not mean alone. A 5 ms query called a million times beats a rare 2 s query.
3. Reproduce on a realistic clone: production-sized tables, similar statistics, same indexes. Seed data lies.
4. Inspect the plan with EXPLAIN (ANALYZE, BUFFERS); find the first node where actual rows diverge from estimates or where time concentrates.
5. Fix in order of leverage: add or fix an index, rewrite the predicate or join, reduce returned columns and rows, batch calls, then consider caching or materialization. See ./02-caching.md.
6. Apply schema changes online: concurrent index builds, no long exclusive locks, batched backfills. Know lock levels before executing.
7. Re-measure the same query on the same data and record the before/after. Watch for plan changes under different parameter values.
8. Guard: query-count assertions, latency dashboards by fingerprint, and review of new query shapes.

• Parameter sniffing and generic plan selection can make the same query fast for one value and slow for another. Test with both common and skewed values.
• Beware accidental full-table reads from unbounded endpoints: pagination and time filters are correctness and performance controls.

Anti-Patterns

• Adding indexes by reflex without checking selectivity, write cost, or whether the query is even hot.
• Reading plans without ANALYZE and trusting estimates.
• Ignoring N+1 because "the ORM handles it".
• One global work_mem or buffer pool sized without concurrency in mind.
• Pool sizes set by copy-paste, not by Little's Law and measured wait time.
• Deep OFFSET pagination on large tables.
• Mass DELETE for retention where partition drop exists.
• Sending read-your-write traffic to a lagging replica.
• Reclaiming from a long analytics query on a user-facing replica.
• Treating the database as infinitely scalable because replicas exist.

Checklist

• [ ] Every hot query has an index whose column order matches its predicates and sort.
• [ ] Plans checked with actual rows and buffers; misestimates investigated.
• [ ] Query-count assertions exist for list and detail endpoints to prevent N+1.
• [ ] Pool sized from Little's Law, with acquire timeout and wait-time alerts.
• [ ] Fleet connection math documented: app pools + poolers + reserves <= server limit.
• [ ] Partition key is present in queries; retention plans use partition drop.
• [ ] Replica lag budget defined; read-your-writes routing implemented and tested.
• [ ] Slow query review ranks by total time and validates on production-like data.
• [ ] Schema changes are online-safe and reversible.
• [ ] Before/after query performance recorded with the change.

Backend Profiling

Scope: CPU and memory profiling, flamegraph reading, continuous profiling, eBPF, lock contention, and tracing slow paths in backend services.

Choosing the Signal

| Question | Signal | Cost | |---|---|---| | Which endpoint or dependency is slow? | Traces, latency histograms | Low | | Which function burns CPU? | On-CPU profile | Low with sampling | | Why is latency high while CPU is low? | Off-CPU or wall-clock profile, traces | Low to moderate | | What allocates and drives GC? | Allocation profile | Moderate | | What retains memory? | Heap profile or snapshot | Moderate to high | | What serializes work? | Block/mutex/lock profiles, thread dumps | Low | | Did this version regress? | Continuous profile diff by version | Low |

• Profiles explain code-level cost; traces explain request-level cost; metrics explain resource-level cost. Use all three together. Cross-pack: the sibling observability pack covers telemetry plumbing and SLO alerting.
• If the CPU profile shows the runtime mostly idle, stop profiling CPU and look at waits, queues, and downstream latency instead.

Profiling Workflow

1. State the question: CPU, allocation, retention, lock, or off-CPU latency.
2. Establish a baseline from the same build type and load level; debug builds and seed data produce misleading profiles.
3. Pick the least invasive method that answers the question: continuous profiler for production, snapshot profiler for staging, system profiler for native issues.
4. Capture with symbols and useful labels (service, version, environment, and span when available).
5. Read top-down for the call path and bottom-up for self-time; ignore cheap frames.
6. Validate against an independent signal (metric, trace, or benchmark) before concluding.
7. Fix one hot path, redeploy, recapture the identical view, and compare.
8. Guard with a benchmark or budget so the regression cannot return silently.

CPU Profiling

• Sampling profilers interrupt at a fixed frequency and aggregate stacks; they have low overhead and are the default for production. Instrumentation profilers count every call and distort hot loops; use them for call counts, not timing.
• Sampling frequency trades overhead for detail. Raise it briefly for an investigation; leave production at defaults.
• Symbols and unwinding decide profile quality. Build with debug info or publish symbol mappings keyed by build ID. Inlined and optimized frames are approximate.
• CPU time and wall-clock time are different profiles. CPU profiles omit blocked threads; a service waiting on a lock or network shows low CPU and high wall time.
• Container note: profile inside the container's namespaces so host noise does not contaminate results, and account for CPU limits and throttling in the labels.

Reading Flamegraphs

• Width is cost or time in the aggregate, not call order. A wide frame is expensive; a deep stack can be cheap.
• Top-down view: follow the widest branch from the root to find the dominant subsystem. Bottom-up view: sort by self-time to find the function actually burning samples.
• Self-time versus total time: dispatchers, frameworks, and middleware look expensive in total time and cheap in self-time. Optimize self-time first, then reconsider architecture.
• Look for landmarks: serialization (JSON encode/decode), regex compilation, TLS handshakes, syscalls, allocator paths, collector/GC frames, lock waits, and retry loops.
• Inverted (icicle) graphs read better for deep stacks; differential graphs color added versus removed cost between two captures.
• Compare version to version or before to after; a single graph proves very little. A new frame at the top of a broken stack usually means missing symbols, not a new hotspot.
• Aggregate across hosts but watch for one host dominating; check per-host profiles before drawing fleet-wide conclusions.

Memory Profiling

• Allocation profiles find churn (what allocates the most bytes/objects), heap profiles find retention (what is still live). GC pressure and leaks require both.
• Analyze allocation rates, not just totals: high allocation rate with short lifetimes causes CPU-heavy collection even without a leak.
• Heap snapshots: capture, then diff two snapshots taken minutes apart under steady traffic. Sort by delta and then find the dominant retainer path of the biggest growth.
• Common retainers: unbounded maps and caches, request-scoped data captured in long-lived closures, event listeners never removed, thread-locals, connection pools holding large buffers, and static registries.
• Managed runtimes can force a GC before a heap snapshot for a cleaner reading; expect a pause, and avoid doing it in latency-critical production windows.
• Correlate memory profiles with runtime GC metrics and container memory to distinguish leak from working-set growth from fragmentation.

Continuous Profiling

• Sample production continuously at low overhead (typically low single-digit percent CPU with sane settings) and aggregate over time; this turns "SSH and profile" into a query.
• Labels make profiles useful: service.name, service.version, deployment.environment.name, host or pod, and ideally span_id during span-correlated sampling.
• Version labels attribute regressions directly: compare the profile of the new version against the previous release.
• Span profiles attach sampled stacks to active traces and pinpoints which span consumed the time; verify upstream support for your profiler and SDK versions.
• Keep label cardinality controlled for long retention; per-span samples are high-cardinality and should be short-lived.
• Budget the agent: measure its own CPU, memory, and egress, and keep ingestion off the request path.

eBPF Profiling

• eBPF profiles sample stacks in the kernel without linking a language agent, covering every process on a node including legacy binaries and JIT runtimes.
• Typical coverage: on-CPU stacks, off-CPU and I/O waits, scheduler delays, and some allocation and network signals. Language-aware unwinding exists for major runtimes but quality varies.
• Constraints: kernel version and build flags, container privilege model (CAP_BPF or privileged), available symbols, stripped binaries, and JIT runtimes that need runtime-specific unwinders. Verify the matrix upstream for your kernel and runtime versions.
• Security: eBPF agents observe process memory and syscalls. Run with least privilege, restrict the loaded programs, and treat profile data as sensitive.
• Cluster deployment: one agent per node (DaemonSet) exporting to a central backend is the common shape; per-pod sidecars rarely justify the overhead.
• Use eBPF when you need coverage without redeploying or when per-language agents conflict; use language agents when you need allocation and runtime-internal detail.

Syscall and I/O Visibility

• System-level views answer what language profilers cannot: syscall mix, socket and file waits, page faults, and scheduler delay.
• strace -c -f summarizes syscalls; per-call latency modes show tail waits. Trace for short windows only; tracing itself is expensive.
• perf stat reports cycles, instructions, cache misses, and context switches, which distinguish CPU-bound from stalled.
• iostat and /proc/pressure/io expose disk saturation; ss exposes socket queue and retransmit behavior.
• cgroup metrics (cpu.stat, memory.events, PSI files) reveal throttling and pressure that in-process profilers never see.
• Correlate I/O waits with traces: a slow span usually ends at a socket, disk, or lock rather than in application code.
• CPU throttling looks like latency spikes with CPU "not busy"; check throttled periods before blaming the code.

Lock Contention

| Tool / signal | Runtime | Reveals | |---|---|---| | Mutex / block profiles | Go | Time blocked on mutexes and channels | | perf lock / off-CPU analysis | Native, any | Kernel lock waits and scheduler delay | | JFR lock events, thread dumps | JVM | Monitor contention, deadlocks, thread states | | Thread dump stacks | Any | Who waits on what; repeated dumps show stalling | | strace -c -f | Any | Syscall mix and time, futex waits | | Tracing around the critical section | Any | Time inside versus outside the lock |

• Lock contention signature: latency grows with concurrency while CPU stays partial; throughput plateaus and then degrades.
• Find the lock: block/mutex profiles, lock-wait traces, or repeated thread dumps showing many threads parked on the same object.
• Shrink critical sections: move I/O, serialization, and callbacks outside the lock; compute under the lock only what must be atomic.
• Reduce lock scope with per-shard or striped locks, read-write locks for read-dominated data, or copy-on-write snapshots for readers.
• Immutable data and message passing remove locks structurally and are usually more maintainable than clever lock-free code. Use lock-free structures only with evidence and careful review.
• Watch coarse locks hidden in libraries: global buffers, connection acquisition, logging appenders, metrics registries, and lazy initialization.

Tracing Slow Paths

• Distributed traces show where a request spent time across services; use them to localize before profiling. Span granularity is a design choice: instrument database calls, HTTP clients, queue publishes, and expensive local steps.
• The critical path is what matters; parallel sibling spans mean total time is the max, not the sum.
• Async and fire-and-forget work needs links or explicit child spans, or the trace will hide the real queueing.
• Exemplars connect histogram buckets to concrete traces, so a p99 latency alert can open the exact slow request.
• Fan-out amplifies tails: a request calling 20 services inherits the slowest of 20 tails. Reduce fan-out, hedge deliberately, or cache aggressively. See ./06-concurrency-backpressure.md and ./02-caching.md.
• Sample smartly: keep all errors and slow requests, sample the rest. Cross-pack: the observability pack covers sampling strategy and collector configuration.

Tooling by Runtime

| Runtime | CPU | Memory / heap | Locks / off-CPU | |---|---|---|---| | Go | pprof CPU, continuous agents | heap, allocs profiles | block, mutex profiles | | JVM | async-profiler, JFR | JFR allocation, heap dumps | JFR monitor events, thread dumps | | Python | py-spy, sampling agents | tracemalloc, memray | py-spy wall mode, faulthandler | | Node.js | --cpu-prof, inspector | --heap-prof, heap snapshots | async hooks, --prof, event-loop lag | | .NET | dotnet-trace, EventPipe | dotnet-gcdump, dotnet-counters | dotnet-dump, contention events | | Native / Rust | perf, eBPF | valgrind, heaptrack, jemalloc stats | perf lock, off-CPU profiling | | Any (kernel) | eBPF profilers | eBPF allocation probes | off-CPU eBPF |

• Check whether a profile is CPU-time or wall-clock before interpreting it; I/O-bound services look innocent under CPU profiles.
• For managed runtimes, separate GC time from application time. A CPU profile dominated by collector work points to allocation reduction, not algorithm changes. See ./07-memory-gc.md.

Overhead and Safety

• Profiling endpoints (/debug/pprof, JMX, inspector sockets, diagnostics ports) expose internals. Bind them to localhost or an internal interface, require authentication, and never expose them publicly.
• Sampling agents add CPU, memory, and egress; budget them like telemetry and measure their cost.
• Heap snapshots and forced GCs pause the process; schedule them outside peak windows or in staging.
• Stack data can contain PII: file paths, function names, and sometimes captured strings. Treat profiles under the same privacy policy as logs.
• Test profilers after runtime, kernel, and orchestrator upgrades; unwinding and privilege changes are the usual breakage points.

Anti-Patterns

• Profiling only in development with debug builds and toy data.
• Chasing the widest bar without checking self-time; wrappers are not hotspots.
• Comparing profiles from different builds, flags, or environments.
• Leaving pprof or diagnostics endpoints publicly reachable.
• Treating a single heap snapshot as leak proof; leaks are trends.
• Ignoring off-CPU time because the CPU profile looks clean.
• Running full-fidelity heap profiling continuously on p99-sensitive services.
• Concluding "the profile is useless" because symbols were missing.
• Fixing micro-hotspots before checking the database, lock, or downstream waits that dominate wall-clock time.

Checklist

• [ ] The question determined the profile type: CPU, wall/off-CPU, allocation, heap, lock, or goroutine/thread.
• [ ] Baseline captured on a production-like build and data shape.
• [ ] Symbols or a symbol mapping exist and build IDs are recorded.
• [ ] Flamegraphs read both top-down and bottom-up; self-time verified before optimizing.
• [ ] Continuous profiling runs in production with documented overhead and owner.
• [ ] Labels consistent with traces and metrics; span correlation verified where supported.
• [ ] eBPF agent privilege and kernel/runtime compatibility reviewed on upgrades.
• [ ] Lock contention checked when latency grows with concurrency and CPU stays low.
• [ ] Profiling endpoints private and authenticated; profile data treated as sensitive.
• [ ] Before/after profile comparison recorded with the fix.

Frontend Performance

Scope: bundles, Core Web Vitals and INP, images and fonts, hydration cost, rendering performance, and measurement tooling for web frontends.

Field Data First

• Core Web Vitals are assessed on field data at the 75th percentile, segmented by mobile and desktop. Lab scores are diagnostic, not the goal.
• Track real-user monitoring (RUM) with the web-vitals library or equivalent; align metric definitions with CrUX so dashboards and search tooling agree.
• Lab tools (Lighthouse, WebPageTest, DevTools) explain causes; field data decides whether users are actually affected.
• Keep a lab budget in CI for regressions and field alerts for reality. Neither replaces the other.

| Metric | Good | Needs improvement | Poor | Measures | |---|---|---|---|---| | LCP | <= 2.5 s | 2.5-4.0 s | > 4.0 s | Loading: largest visible element | | INP | <= 200 ms | 200-500 ms | > 500 ms | Responsiveness across interactions | | CLS | <= 0.1 | 0.1-0.25 | > 0.25 | Visual stability | | TTFB | <= 0.8 s | 0.8-1.8 s | > 1.8 s | Server and network latency | | FCP | <= 1.8 s | 1.8-3.0 s | > 3.0 s | First content painted |

• INP replaced FID as a Core Web Vital in 2024; FID is retired. Use INP plus TTFB as the working pair.
• Report distributions and the worst interactions, not just a page-level average; INP is scored per interaction and reported as a high percentile.

LCP: Loading

LCP decomposes into four phases; optimize the largest one you control:

| Phase | Levers | |---|---| | Time to first byte | Server rendering, caching, CDN, edge, database. See ./02-caching.md and ./03-databases.md | | Resource load delay | Discoverability: preload, early hints, no render-blocking chains | | Resource load time | Compression, format, smaller payloads, faster origin | | Element render delay | Unblock the main thread; avoid lazy-loading the LCP element |

• Identify the LCP element per template (hero image, heading, video poster) and optimize that specific resource.
• Preload the LCP image or font with fetchpriority="high"; never lazy-load an above-the-fold LCP element.
• Serve modern formats (AVIF, WebP) with fallbacks and correct srcset/sizes; let a CDN transform and cache derivatives.
• Avoid render-blocking third-party scripts before the LCP; defer, async, or move them behind interaction facades.
• Streaming SSR and server components can improve TTFB and LCP together; measure per template rather than trusting a framework default.

INP: Responsiveness

INP measures the latency from user input to the next paint across the interaction.

• Break long tasks into smaller chunks; keep any single task well under 50 ms to stay responsive. Use scheduler.yield() where available, and a setTimeout/MessageChannel fallback otherwise; verify browser support upstream.
• Move expensive work off the main thread: Web Workers for parsing, computation, and search indexes.
• Keep event handlers thin: capture the input, update state minimally, and schedule the heavy work. Avoid synchronous layout reads in handlers.
• Debounce expensive work but never the visual response; the interface should acknowledge input immediately.
• Third-party scripts are a leading cause of long tasks: load analytics and tag managers late, with size and CPU budgets, and prefer server-side or first-party wrappers.
• Measure with the Long Animation Frames (LoAF) API and PerformanceObserver where supported; attribute the worst interactions to scripts and frames.

CLS: Stability

• Reserve space for images and media with explicit width/height or aspect-ratio.
• Reserve space for ads, embeds, banners, and consent dialogs; late-inserted content is the common cause of CLS.
• Avoid inserting content above existing content unless triggered by user interaction.
• Prefer transform and opacity animations; they stay on the compositor and do not shift layout.
• Use font fallback metric tuning (size-adjust, ascent-override) to reduce reflow when the web font swaps in.

Bundles

| Technique | Typical effect | Notes | |---|---|---| | Dependency audit and dedupe | Removes duplicate versions and dead weight | Check package manager aliases and lockfile duplicates | | Tree shaking | Drops unused exports | Requires ESM and side-effect-free modules | | Route-level code splitting | Loads only what the route needs | Lazy boundaries per route and heavy component | | Dynamic import for rare UI | Defers modals, editors, charts | Prefetch on intent where possible | | Compression (Brotli preferred) | Large transfer reduction | Content-Encoding with correct Vary | | HTTP/2 or HTTP/3 | Multiplexing, fewer round trips | Keep request count sane; no domain sharding | | Long-term caching | Repeat visits near zero transfer | Content-hash filenames plus immutable caching | | Polyfill trimming | Drops legacy JavaScript | Ship modern bundles to modern browsers |

• Analyze the bundle before optimizing: a visualizer, source-map-explorer-style tools, or framework bundle analysis. React to the biggest rectangles, not to raw total size.
• Watch for duplicate framework copies and full-library imports (lodash-style) where per-function imports exist.
• Budget per route and per entry: initial JavaScript (compressed), total fonts, total images, third-party count.
• Do not ship source maps publicly unless intended; use hidden maps for error reporting.
• Evaluate dependencies by cost, not popularity: a date library or icon set can outweigh application code.

Resource Loading Strategy

| Hint | Use | |---|---| | preload | Critical late-discovered resources (LCP image, hero font, critical CSS) | | preconnect | Origins needed early (font, API, CDN) | | dns-prefetch | Cheap fallback for origins likely needed | | prefetch | Next navigation resources for likely paths | | modulepreload | ESM entry chunks and their dependencies | | fetchpriority="high" | The LCP image or critical script; use sparingly | | Lazy loading | Below-the-fold images and iframes only |

• Every hint is a resource claim: too many preloads compete for bandwidth and delay the critical path.
• Inline critical CSS and defer the rest; avoid shipping a large stylesheet that blocks first paint.
• Use early hints (103) only when the origin can support them correctly; verify behavior at the CDN upstream.

Images and Fonts

• Images usually dominate transfer weight. Set explicit dimensions, use responsive srcset/sizes, and serve formats matched to device capability.
• Lazy-load below-the-fold images with native loading="lazy"; keep above-the-fold and LCP images eager with high priority.
• Use a CDN image pipeline for resize, format negotiation, and caching instead of runtime transforms.
• Fonts: prefer woff2, self-host for control and privacy, subset to the scripts actually used, and preload only the critical face.
• font-display: swap trades a flash of unstyled text for no invisible text; tune with fallback metrics so the swap does not shift layout.
• Variable fonts reduce file count but can carry large glyph sets; subset aggressively.
• Avoid icon fonts; inline SVG or sprite sheets render faster and are more accessible.

Rendering Performance

• Avoid layout thrash: batch DOM reads before writes; never interleave getBoundingClientRect-style reads with style mutations in a loop.
• Reduce DOM size and depth; huge trees slow style recalculation, layout, and reconciliation. Virtualize long lists.
• Use CSS containment (contain, content-visibility: auto) for large offscreen sections, and verify paint behavior.
• Animate only compositor-friendly properties (transform, opacity); avoid animating layout properties (width, top, margin).
• Keep will-change off long-lived elements; promote layers only around active animations.
• In component frameworks, memoize expensive subtrees deliberately, keep state local to avoid broad re-renders, and profile renders rather than guessing. Framework-specific devtools exist; use them.

Hydration and SSR Cost

• Server rendering improves first paint but hydration re-executes component code on the client, which costs main-thread time and hurts INP.
• Choose the rendering model per route: static, SSR, streaming, islands, partial hydration, or client-only. Mixed models are normal in 2026-era frameworks; verify what your framework version supports upstream.
• Keep server-only data out of client bundles; serialized payloads are easy to duplicate and expensive to parse.
• Hydration mismatches cause re-renders and layout shifts; keep server and client output deterministic (avoid Date.now(), random ids, locale-dependent formatting on first render).
• Islands and partial hydration restrict interactivity to the components that need it; restructure components to make that boundary possible.
• Consider progressive enhancement: the page should be usable before hydration completes, at least for reading and navigation.

Third-Party Scripts

• Treat every third party as a performance dependency with a budget: count, bytes, main-thread time, and failure mode.
• Load non-critical tags behind interaction facades (click-to-load maps, video players, chat widgets) or after load, idle, or first interaction.
• Prefer first-party or server-side integrations over client-side tags where the data flow allows.
• Sandbox third-party scripts in iframes where possible; verify supply-chain and consent requirements. Cross-pack: the security pack covers third-party risk.
• Audit tags quarterly; stale marketing tags are a common source of silent regressions.

Measurement Tools

| Tool | Mode | Use for | |---|---|---| | Chrome DevTools Performance / Insights | Lab, per session | Long tasks, layout, INP attribution on a reproduction | | Lighthouse / PageSpeed Insights | Lab | Category diagnosis and CI regression checks | | WebPageTest | Lab, controlled | Filmstrips, network shaping, multi-run variance | | web-vitals + RUM endpoint | Field | Real Core Web Vitals with attribution | | CrUX / CrUX History | Field | Competitor and origin-level baseline | | LoAF plus PerformanceObserver | Field | Script and frame attribution for INP | | Bundle analyzers | Build | Bundle composition and duplication |

• Instrument with real devices and throttling that matches your audience; fast desktops hide mobile INP problems.
• Segment RUM by route, device, connection, and release; aggregate scores hide the templates that hurt.
• Keep synthetic and field definitions aligned to avoid chasing phantom regressions.

Budgets and CI

• Set budgets per route and entry: JavaScript compressed, CSS, images, fonts, third-party requests, and metric thresholds.
• Enforce with Lighthouse CI assertions, bundle-size checks, and field alerts. Fail builds on budget breaches before merge.
• Track budgets over releases to catch slow creep; small regressions compound and are harder to attribute later.
• Re-baseline deliberately after intentional redesigns, with a recorded decision.

Anti-Patterns

• Optimizing to a green Lighthouse score while field 75th percentile INP or LCP stays poor.
• Lazy-loading the LCP element.
• Shipping a full component library and a tag manager before any content renders.
• Synchronous, layout-reading event handlers.
• Hydrating the entire page for one interactive widget.
• Unbounded third-party scripts with no budget or owner.
• Animating layout properties and shifting content.
• Measuring on a fast machine with warm caches and calling it done.
• Adding preload hints to everything until the critical path is congested.

Checklist

• [ ] Field RUM tracks LCP, INP, and CLS with 75th-percentile reporting by segment.
• [ ] The LCP element per template is identified and explicitly prioritized.
• [ ] INP long tasks are attributed to scripts; third-party work moved off the critical path.
• [ ] Images have dimensions, responsive sources, and modern formats above the fold.
• [ ] Fonts are subset, self-hosted where practical, and styled to avoid layout shift.
• [ ] Bundles are split by route; duplicate and unused dependencies removed.
• [ ] Hydration model chosen per route; server data not duplicated into client bundles.
• [ ] Budgets enforced in CI with field alerts in production.
• [ ] Third-party scripts have owners, budgets, and facades.
• [ ] Release process re-measures field metrics after deploy.

Concurrency and Backpressure

Scope: pools, bounded queues, rate limiting, batching, bulkheads, backpressure, timeouts, retries, and load shedding under overload.

Why Concurrency Control

• Latency is a queueing function. Past roughly 70-80 percent utilization, waiting dominates service time and tails explode. See ./01-methodology.md.
• Unbounded concurrency converts overload into an outage: every new request adds work, queues grow, timeouts fire, and retries multiply the load.
• The goal is not maximum parallelism; it is the highest sustainable throughput that keeps the latency budget. That requires explicit limits on every shared resource.
• Limits belong in code and configuration, not in hope: connection pools, worker pools, request concurrency, queue lengths, retry counts, and cache sizes.

Pools and Sizing

| Pool | Bounds | Sizing basis | |---|---|---| | CPU thread/worker pool | Concurrent CPU work | CPU cores, with slight oversubscription for mixed workloads | | Blocking I/O pool | Threads blocked on I/O | Runtime limits plus downstream concurrency budget | | Async task concurrency | In-flight coroutines/tasks | Semaphore from Little's Law and downstream capacity | | Database connections | DB sessions | Little's Law plus headroom; fleet total under server max | | HTTP client connections | Sockets per host | Per-host limit to protect dependencies | | Queue workers | Parallel consumers | Downstream throughput and queue drain SLO |

• Little's Law sizing: concurrency = arrival_rate * latency. If a service handles 2,000 rps at 40 ms downstream, about 80 calls are in flight; a pool of 8 will queue forever. See ./01-methodology.md.
• CPU-bound work should roughly match available cores; oversubscribing threads adds context switching and worsens tails.
• Blocking runtimes (traditional servlet, sync Python) need pools sized by downstream capacity, not by thread-count slogans. Async runtimes need semaphores because the event loop will happily schedule unlimited concurrent operations.
• Pools are per dependency: one slow dependency must not consume the capacity reserved for others. Give each downstream its own bound.
• GIL and event-loop notes: Python threads do not parallelize CPU work; use processes or native extensions. Node.js and single-threaded event loops need bounded async concurrency and worker threads for CPU tasks.
• Tune from measured wait time: if pool usage is high and wait time is non-trivial, either raise the pool (if the downstream can absorb it) or reduce work per item.

Queues and Queueing

• Prefer bounded queues everywhere: worker queues, message consumption, request admission, log buffers, and retry queues. Unbounded queues are delayed outages with good initial metrics.
• Queue depth is both a buffer and a signal: use it for autoscaling and alerts, but cap it by expected drain time. A queue 10 minutes long cannot meet a 1-second latency SLO.
• Choose queue semantics deliberately: FIFO for fairness, priority for incident-critical traffic, LIFO only when old work is worthless (freshness-first feeds).
• Backpressure propagation: when a queue fills, the producer must slow down, block, or be rejected. A full queue that silently drops is a data-loss decision; make it explicit.
• Dead-letter queues need an owner, a review cadence, and replay tooling. An unmonitored DLQ is a silent failure store.

Backpressure Mechanisms

| Mechanism | How | Where it fits | |---|---|---| | Bounded blocking | Producer blocks until space is available | In-process pipelines, streaming | | Credit-based flow control | Consumer grants permits | gRPC, message protocols, custom pipelines | | 429 Too Many Requests | Reject with Retry-After | Public APIs, per-tenant limits | | 503 Service Unavailable | Reject when shedding, with retry guidance | Overloaded services, maintenance | | TCP flow control | Kernel receive window closes | Network-level, automatic | | Reactive streams / demand | Consumer requests N items | Reactive stacks, stream processing | | Client-side adaptive limiting | Clients reduce concurrency on errors/latency | SDKs, service meshes |

• Backpressure should propagate end to end. If the innermost dependency is slow and every caller buffers unboundedly, the pressure never reaches the source; the system just accumulates latency and memory.
• Prefer explicit rejection over unbounded waiting: a fast 429 with retry guidance preserves the capacity to serve other users.
• Include a machine-readable reason and a Retry-After where supported; clients that ignore it must be rate-limited or blocked.

Rate Limiting

| Algorithm | Burst behavior | Notes | |---|---|---| | Token bucket | Allows configured bursts | Default choice; simple and intuitive | | Leaky bucket | Smooths output | Good for shaping downstream calls | | Fixed window | Burst at window edges | Simplest; double-rate edge effects | | Sliding window log | Exact but memory-heavy | Small keys or low volumes | | Sliding window counter | Approximate and smooth | Common in distributed limits | | Concurrency limit | Caps in-flight, not rate | Better for long-lived or streaming calls |

• Distinguish rate (requests per second) from concurrency (requests in flight). Long-running calls need concurrency limits even at low rates.
• Distributed limits need shared state; a Redis/Valkey script or equivalent provides atomic counters. Verify your implementation handles clock skew and partition failure upstream.
• Decide failure mode when the limiter store is unavailable: fail open (allow) can overload the backend; fail closed (reject) can take down the API. Most systems choose fail open with a local fallback limit.
• Per-tenant limits protect the fleet from a single customer; global limits protect shared dependencies. Implement both, with different thresholds.
• Return limit information (RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset) where the standard is supported, and document the policy.

Timeouts and Deadlines

• Every network call needs a timeout: connect, read, and total. Defaults that wait forever are latent outages.
• Propagate deadlines: downstream calls get less time than the caller has remaining, leaving time to serialize and respond.
• Do not set all timeouts equal across layers; the budget shrinks as requests move inward. A common shape: client 5 s, gateway 3 s, service 2 s, database 1.5 s.
• On timeout, cancel remaining work (context cancellation or equivalent); canceled work still consumes real CPU and connections if nothing stops it.
• Timeout values must exceed the observed healthy p99.9 with margin, or healthy requests will fail under normal variance.
• Streaming and long-lived connections need idle timeouts in addition to total duration limits.

Retries and Circuit Breakers

• Retry only transient errors, only idempotent operations, and only within a budget. Blind retries turn a brief dependency failure into a self-inflicted DDoS.
• Cap retry attempts (commonly 1-2 beyond the original), use exponential backoff with jitter, and honor Retry-After.
• A retry budget caps retries as a fraction of successful requests (for example, 10 percent of traffic), so healthy systems can retry more while overloaded ones retry less.
• Use idempotency keys for writes that may be retried; otherwise a retried payment or order becomes a duplicate.
• Circuit breakers protect callers from slow dependencies: half-open probes, failure thresholds, and a short-circuit response are the common shape. Return a degraded response rather than hanging.
• Hedging (sending the same request twice after a delay) improves tails only with strict budgets and idempotency; otherwise it doubles load.
• Track retry rate as a first-class metric; a rising retry rate is an early warning even when success rates look fine.

Batching

• Batching trades latency for throughput. Use it where the downstream supports bulk operations and the extra wait fits the latency budget.
• Micro-batching: accumulate requests for a few milliseconds, then send one bulk call. Pair a size limit with a time limit (for example, 100 items or 10 ms, whichever first).
• Coalesce identical concurrent reads into one call (single-flight); this is batching by another name. See ./02-caching.md.
• Dynamic batch sizes that grow under load and shrink on latency pressure are effective but need guardrails; never let a batch exceed the downstream's maximum payload.
• Batch writes need partial-failure semantics: one bad item must not fail the whole batch, and callers must know which items succeeded.
• Batch at the boundary where it pays: database round trips, object storage, message publishes, and external APIs. Do not batch when the latency budget has no room.

Bulkheads

• Partition capacity by failure domain: separate pools, queues, or even clusters per dependency, tenant tier, or route class.
• A bulkhead means one tenant or dependency exhausting its share cannot consume everyone else's. Without it, shared pools convert any single slow caller into a site-wide incident.
• Cell-based architecture extends the idea to infrastructure: independent cells limit blast radius and allow incremental rollout. Costlier to operate; use for high-consequence systems.
• Resource isolation is not free: more pools mean more overhead and more configuration. Reserve bulkheads for proven shared-fate risks.

Load Shedding and Admission Control

• Shed before saturation, not after: reject early with 503 or queue with a deadline long before every worker is stuck.
• Admission control checks capacity at the edge: concurrency limits, queue depth thresholds, and priority classes decide what enters.
• Priority classes: interactive traffic above batch, paying tenants above free tiers, health checks above product traffic. Prefer explicit classes over implicit arrival order.
• Degrade features before failing requests: serve a cached or partial response, disable recommendations, skip analytics writes, shorten results. State the degradation in logs and metrics.
• Emergency limits are configuration, not deploys: a runbook switch to tighten limits must not require a release.
• Keep a static floor of capacity reserved for health checks and critical operations; otherwise the service cannot even report that it is overloaded.

Autoscaling Interplay

• Scale on demand signals, not just CPU: queue depth, request concurrency, and arrival rate respond before CPU saturates. See ./09-capacity-cost.md.
• Targets should leave headroom: scaling to 60-70 percent utilization keeps queueing in the linear region.
• Scale-out is not instant: pods take seconds to minutes to become ready. Shedding and queueing cover the gap; that is why limits are not optional with autoscaling.
• Scale-in must respect in-flight work; terminating a pod mid-request adds latency and errors. Use graceful shutdown with drain windows and connection draining.

Observability

| Signal | Why it matters | |---|---| | In-flight requests / tasks | Concurrency versus limit | | Queue depth and oldest item age | Latency already accumulated | | Wait time for pool/queue | Saturation before errors | | Rejection and shed rate | How much user demand is being refused | | Timeout and cancellation rate | Downstream health | | Retry rate per dependency | Amplification risk | | Per-tenant concurrency and rate | Noisy-neighbor detection |

• Alert on saturation and wait time; errors and timeouts are the late signals.
• Track limits as configuration with values visible in dashboards; unknown limits cannot be reasoned about during incidents.

Anti-Patterns

• Unbounded queues, unbounded pools, unlimited retries, or unlimited fan-out.
• One shared pool for all dependencies, so the slowest consumes everything.
• Timeouts missing, equal across layers, or longer than the caller's own budget.
• Retry without jitter, budget, or idempotency.
• Rate limiting only at the edge while internal callers bypass it.
• Batching without a latency cap or partial-failure handling.
• Shedding only after CPU is pegged and memory is exhausted.
• Buffering in every layer so backpressure never reaches the producer.
• Treating autoscaling as a substitute for admission control.

Checklist

• [ ] Every shared resource has an explicit, documented limit.
• [ ] Pool sizes derived from Little's Law and measured wait time, not defaults.
• [ ] Queues bounded with a drain-time rationale; DLQ owned and monitored.
• [ ] Backpressure propagates to producers; rejection reasons and Retry-After returned.
• [ ] Rate limits exist per tenant and globally, with a defined failure mode.
• [ ] Timeouts per hop with deadline propagation; cancellation honored.
• [ ] Retries bounded, jittered, budgeted, and idempotent; circuit breakers configured.
• [ ] Batching has size and time caps plus partial-failure semantics.
• [ ] Bulkheads isolate high-risk dependencies and top tenants.
• [ ] Load shedding with priority classes and a static capacity floor.
• [ ] Saturation metrics (in-flight, queue depth, wait time, shed rate) alerted before errors.

Memory and GC

Scope: leak detection, per-runtime GC tuning, allocation reduction, pools and arenas, object lifetimes, and OOM diagnosis.

Memory Model Basics

• Distinguish virtual memory, resident set size (RSS), heap, and off-heap. Only RSS and container/cgroup accounting decide whether the OOM killer fires.
• Page cache and memory-mapped files count against the cgroup in container environments; a service can be OOM-killed while its own heap looks small.
• Managed heaps need headroom beyond live data; collectors need room to run concurrently, and allocators fragment. A heap 95 percent full is already unhealthy.
• Distinguish leak (live set grows forever), burst (temporary spike), working-set growth (legitimate new data), and fragmentation (high RSS with low live bytes). The fixes are different.
• Set memory requests and limits thoughtfully: too low causes OOM kills; too high wastes capacity. See ./09-capacity-cost.md.

Object Lifetimes

• Most objects die young in most workloads (the generational hypothesis). Nursery/young collections are cheap; promotion of surviving objects into long-lived space is what causes major collections.
• Long-lived object graphs are expensive: they get marked on every major collection and retain everything they reference. Keep caches and registries bounded and evictable.
• Escape analysis lets compilers allocate on the stack when references do not escape; returning pointers to locals or capturing them in closures forces heap allocation.
• Allocation rate is the primary GC lever: halving allocation rate often cuts collection CPU and pause frequency more than any collector flag.
• Watch object graph depth: a long linked structure or deeply nested tree keeps many objects alive transitively and slows marking.
• Ephemeral but large allocations (big buffers, parsed documents, response payloads) dominate peak RSS even if they are short-lived; reuse or stream them.

Allocation Reduction

| Pattern | Problem it fixes | Notes | |---|---|---| | Reusable buffers (pool or explicit) | Per-request buffer churn | Reset length, never reuse across security domains | | Streaming instead of buffering | Large payload memory spikes | Parse and emit incrementally | | Slice/array preallocation | Repeated growth copies | Size hints from expected cardinality | | String building (builder/rope) | Quadratic concatenation | Avoid += in loops on immutable strings | | Struct/value types | Boxing and heap escapes | Language-specific; measure first | | Zero-copy views | Copies between layers | Watch lifetime and aliasing hazards | | Batching of small objects | Allocator overhead per object | Better locality and fewer collections |

• Profile allocations before optimizing; the hottest allocator call site is often not the largest consumer. See ./04-backend-profiling.md.
• Reduce temporary objects in serialization paths: encoders that allocate per field dominate CPU profiles in some services. Reuse encoders and write into buffers.
• Avoid accidental retention: closure captures, slices of large arrays, and substring implementations that keep the parent buffer alive.
• For JSON and binary formats, consider zero-copy or precompiled codecs; verify latency and memory tradeoffs with a benchmark.
• Beware copy amplification: a payload copied through several layers can multiply memory and CPU. Pass references or move ownership.

Pools and Arenas

• Object and buffer pools amortize allocation and collection cost for frequently created, similarly sized, short-lived objects. They are most effective under high churn and predictable sizes.
• Pools require discipline: reset state on get/put, bound total size, and account for pooled memory in limits. A pool that only grows is a leak with extra steps.
• Buffer pools (sync.Pool-style, bytebufferpool-style) reduce GC pressure in network services; bytes must be cleared or truncated before reuse when they held sensitive data.
• Arena and region allocators allocate many objects in one block and free together; ideal for request-scoped or batch-scoped lifetimes.
• Slab allocators reduce fragmentation for same-size allocations at the system-allocator level; mostly an infrastructure concern unless you write a custom allocator.
• Pools can hurt: cache misses, retention of rarely used objects, and cross-thread contention. Benchmark with and without.

Allocators and Fragmentation

• General-purpose allocators trade speed, overhead, and fragmentation. Under churn with mixed size classes, RSS can exceed live bytes by a large factor.
• Size-class fragmentation comes from many distinct allocation sizes; aligning buffer sizes to common classes improves reuse.
• Arena retention: freed memory may stay in the process instead of returning to the OS. Release policies exist per allocator and runtime; verify upstream.
• Switch allocators only with benchmarks: jemalloc, mimalloc, and tcmalloc often help multicore scaling and fragmentation, but change CPU behavior too.
• Distinguish allocator fragmentation from collector heap fragmentation; the metrics and fixes differ.
• Huge-page and transparent huge-page settings change both performance and memory accounting on modern kernels and cloud hosts.

Leak Detection

• Leaks are trends, not snapshots. Monitor RSS and heap live-set over days, normalized by traffic; a slow upward slope under steady load is the signal.
• Heap snapshot diff workflow: capture under steady traffic, wait or force GC, capture again, then rank by delta and inspect the dominant retainer path of the largest growers.
• Compare live size after full collections, not total heap size; a large heap can be mostly garbage waiting to be collected.
• Common leak sources:

| Source | Signature | Fix | |---|---|---| | Unbounded cache / map | Live set tracks key cardinality | Bound, TTL, evict. See ./02-caching.md | | Event listeners never removed | Listeners equal subscriptions over time | Deregister on teardown and scope lifetimes | | Captured request context in long-lived closure | Heap grows with requests processed | Do not hold request-scoped objects in singletons | | Static or global registry | Unique keys accumulate | Weak references or explicit expiry | | Thread-locals holding payloads | Per-thread growth that never shrinks | Clear on release; sized pools | | Queue or DLQ backlog | Depth grows, not heap shape | Bound and drain; fix consumer. See ./06-concurrency-backpressure.md | | Goroutines/threads blocked | Stack memory plus retained references | Find blocking call; add timeouts | | Subprocesses or file handles | OS memory outside the heap | Close/reap; watch handle counts |

• In managed runtimes, weak and soft references plus reference queues help caches hold objects only while memory permits; understand which reference type the runtime treats as evictable.
• Native and off-heap memory leaks do not appear in heap profiles: watch metrics for direct buffers, JNI/native areas, mmap counts, and RSS. Tools differ per runtime.
• Add a soak test to CI or staging that asserts a stable memory ceiling under sustained load. See ./08-load-testing.md.

OOM Diagnosis

1. Confirm the mechanism: kernel OOM killer (cgroup limit), runtime out-of-memory error (heap), or native allocation failure. They need different fixes.
2. Read the evidence: cgroup memory.events/memory.peak on v2 or memory.usage_in_bytes on v1, kernel logs for OOM kills, and the runtime's own logs.
3. Check limits against actual usage: a working set that grew past a stale limit looks like a leak but may be legitimate growth. Reconcile with traffic growth.
4. Capture before death: configure heap dump on OOM where supported, and core dumps only if you have the tooling and storage to use them.
5. Distinguish scenarios:

| Scenario | Evidence | Typical fix | |---|---|---| | Leak | Live set rises steadily under steady load | Fix retainer; see leak table above | | Traffic growth | Memory tracks load, resets on scale-in | Right-size limits and instance count | | Burst / large request | Spike around specific payloads | Stream, bound request size, pool buffers | | Fragmentation | High RSS, low live heap | Allocator tuning, size classes, pools | | GC headroom too small | GC thrash before OOM | Raise heap/limit or cut allocation rate | | Off-heap growth | Heap stable, RSS grows | Direct buffers, JNI, mmap, native libs |

6. After the fix, add a memory ceiling alert and a soak test so the failure mode stays visible.

GC Fundamentals

• Every collector trades throughput, latency (pause), and footprint. You cannot maximize all three; tune for the service's SLO.
• Stop-the-world pauses are what users feel; concurrent phases cost CPU and need headroom to complete before the heap fills.
• Generational collectors assume most objects die young; surviving data is promoted, which is why long-lived caches eventually make every major collection expensive.
• Allocation stalls and "GC thrash" happen when the live set approaches the heap limit; the collector spends most of its CPU reclaiming little memory. The fix is less live data or a larger ceiling, not more aggressive collection.
• Compact or evacuate based collectors move objects, which limits fragmentation but adds copy cost; mark-sweep leaves fragmentation behind.
• GC logging is nearly free and should be enabled in production where supported; you cannot tune what you cannot see.

Per-Runtime Tuning

| Runtime | Collector(s) | Primary knobs | Notes | |---|---|---|---| | JVM | G1 (default), ZGC, Shenandoah | -Xmx/-Xms, MaxRAMPercentage, region size, GC threads | Generational ZGC and Shenandoah are the low-pause options in the JDK 21+ era; verify upstream | | Go | Concurrent mark-sweep with pacing | GOGC, GOMEMLIMIT, GOMAXPROCS, GODEBUG | GOMEMLIMIT is a soft limit; set it under container limits. Verify upstream for your Go release | | Python | Reference counting plus cyclic GC | gc.freeze, generation thresholds, PYTHONMALLOC | Cyclic GC handles reference cycles only; pymalloc fragmentation is a separate problem | | Node.js | Scavenger plus mark-compact | --max-old-space-size, --max-semi-space-size, GC traces | External buffers live outside the V8 heap; track them separately | | .NET | Workstation and Server GC, background GC | Server GC mode, heap hard limit, DATAS | LOH and POH have special allocation behavior | | Rust | None (ownership) | Allocator choice, arena crates | Reference cycles need Weak or arenas; native memory is still yours to manage |

• JVM in containers: use MaxRAMPercentage or an explicit -Xmx derived from the cgroup limit, leaving headroom for metaspace, threads, JIT code, direct buffers, and the collector's own structures.
• Go: GOGC sets the heap growth target; GOMEMLIMIT complements it for containerized services. Set the limit below the container limit and watch for GC thrash if live data approaches it.
• Python: long-running services with many immutable objects can disable or tune cyclic GC only after proving no cycles leak; gc.freeze helps fork-heavy prefork servers.
• Node: raise --max-old-space-size only with matching container limits; most "Node memory leaks" are unbounded caches, listeners, or async queues.
• GC tuning changes should be measured with the same load model; small pause improvements that cut throughput are usually a bad trade.

Sizing and Limits

• Container limit = runtime heap + off-heap + native + page-cache share + collector headroom. Undersizing headroom is the most common OOM cause after actual leaks.
• Leave at least 20-30 percent headroom beyond the expected live set so concurrent collection and bursts have room; verify with a soak test and a worst-case workload.
• Prefer live-set monitoring over heap-size monitoring: heap utilization near 100 percent with a low live set is healthy; a high live set is the risk.
• Scale out when per-instance memory cannot grow further, and revisit object lifetimes before scaling up. See ./09-capacity-cost.md.

Anti-Patterns

• Treating any memory growth as a leak without normalizing by traffic and time.
• Tuning GC flags before reducing allocation rate or bounding caches.
• Setting container limits without heap/off-heap headroom analysis.
• Using RSS or heap size as the sole alert when live-set is the actionable metric.
• Pools without size bounds or reset discipline.
• Heap snapshots taken without a GC, then interpreted as live data.
• Ignoring off-heap and native allocations because the managed heap looks clean.
• Disabling GC logging in production, then guessing during an incident.
• Copying GC flags between services with different allocation profiles.

Checklist

• [ ] Memory alerts use live-set and RSS trends normalized by traffic, not raw heap size.
• [ ] Caches, registries, and queues are bounded and evictable.
• [ ] Allocation hot spots reviewed before/after GC tuning.
• [ ] Collector and heap settings derived from container limits with documented headroom.
• [ ] GC logging enabled where supported; pause and CPU metrics collected.
• [ ] Leak workflow documented: trend, snapshot diff, dominant retainer, fix, verify.
• [ ] OOM response distinguishes leak, growth, burst, fragmentation, and headroom.
• [ ] Off-heap and native memory tracked separately from the managed heap.
• [ ] Soak test asserts a stable memory ceiling under sustained load.
• [ ] Tuning changes are measured under the same load model and recorded.

Load Testing

Scope: scenario design (smoke, load, stress, spike, soak), open versus closed models, k6 and locust patterns, result interpretation, and noise control.

Purpose

• Load testing answers capacity questions before production does: what breaks first, at what load, and how the system degrades.
• A test is only meaningful against a budget or SLO: "no errors, p95 under 300 ms at 2x peak for 30 minutes" is a pass criterion.
• Tests are maintained assets: they are versioned with the service, run on a schedule (or before releases), and updated when traffic shape changes.
• Load tests validate the system, not the tool. The most common failure is a well-run test against an unrealistic workload.

Test Types

| Type | Question | Load profile | Duration | Pass criteria | |---|---|---|---|---| | Smoke | Does it work at all? | 1-5 users, core journey | Minutes | No errors, sane latency | | Load | Does it meet the SLO at expected peak? | Expected peak with headroom | 30-60+ minutes | SLO met, resources stable | | Stress | Where does it break and how? | Ramp past capacity | Until failure or ceiling | Graceful degradation, no data loss | | Spike | Can it absorb sudden bursts? | Step or pulse to many times baseline | Seconds to minutes | Recovers without collapse | | Soak | Does it stay healthy over time? | Sustained moderate load | Hours to days | No leaks, no drift, stable latency | | Breakpoint | What is the actual ceiling? | Gradual ramp | Until knee/errors | Knee identified with evidence |

• Run a smoke test on every meaningful configuration change; full load tests on releases and architecture changes; soak tests before long campaigns or seasonal peaks.
• Stress and spike tests are how you validate shedding, retries, recovery, and autoscaling. Breaking the system on purpose is the point.

Scenario Design

1. Model the workload from evidence: production traffic mix (rps by endpoint or journey), peak factor, tenant mix, payload sizes, and cache hit ratios. See ./01-methodology.md.
2. Write user journeys, not just endpoint hits: browse, search, add to cart, checkout. Weight journeys by real proportions.
3. Choose the load model deliberately: open (arrival rate) or closed (fixed users with think time). They produce different latency results.
4. Set thresholds up front: error rate, percentiles per endpoint, and resource ceilings.
5. Use production-like data volumes and distributions; empty tables and uniform ids hide N+1 and index problems. See ./03-databases.md.
6. Handle authentication realistically: token caches for synthetic users, not a login per request unless that is the journey being tested.
7. Define cache state: cold-start test versus warm steady-state test. Both matter; they measure different risks.
8. Plan cleanup and isolation: test data must not pollute production or shared staging consumers.

• Think time and pacing: real users pause. Arrival-rate models encode pacing directly; closed models need realistic think time or they behave like a stampede.
• Session model: some journeys are stateful and sequential; replaying them requires sticky sessions or a shared state store.
• Record a versioned test plan: scope, workload model, environment, thresholds, and known limitations.

Open versus Closed Models

| Model | Behavior | Latency meaning | Use for | |---|---|---|---| | Open (arrival rate) | New requests arrive regardless of completion | True system latency under offered load | Capacity, SLO validation | | Closed (fixed users) | Each virtual user waits for its response | Response time seen by a specific user pool | Throughput ceiling, per-user experience | | Semi-open | Arrival rate with client-side concurrency caps | Mixed | Approximating real client pools |

• Closed-loop tests suffer coordinated omission: when the system stalls, the generator also stalls, and the measured tail misses the requests that would have arrived. The result overstates capacity. Prefer open models for latency claims.
• If you must use a closed model, correct or model the missed arrivals, or explicitly state the limitation.
• Reconciliation with Little's Law: for closed tests, throughput = users / latency. If the numbers do not reconcile, the scenario definition is wrong. See ./01-methodology.md.

Tooling Patterns

k6 is the common scriptable default for developer-driven tests. Minimal arrival-rate scenario:

import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  scenarios: {
    steady: {
      executor: "constant-arrival-rate",
      rate: 200, timeUnit: "1s", duration: "30m",
      preAllocatedVUs: 200, maxVUs: 1000,
    },
  },
  thresholds: {
    http_req_failed: ["rate<0.01"],
    http_req_duration: ["p(95)<300", "p(99)<800"],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/api/items?limit=20`);
  check(res, { "status 200": (r) => r.status === 200 });
  sleep(0.5 + Math.random());
}

• Thresholds fail the test run, which makes k6 usable as a CI gate. Use ramping-arrival-rate for ramp and spike profiles.
• Do not confuse pre-allocated VUs with load: with arrival-rate executors, VUs only need to cover rate * latency; undersized max VUs cause dropped iterations rather than backpressure.
• Locust is the Python-side common choice. Minimal shape:

from locust import HttpUser, task, between

class Shopper(HttpUser):
    wait_time = between(0.5, 2.0)

    @task(5)
    def browse(self):
        self.client.get("/api/items?limit=20")

    @task(1)
    def checkout(self):
        self.client.post("/api/checkout", json={"cart_id": "c-1"})

• Locust loads can be shaped with custom LoadTestShape classes for ramp, spike, and step profiles; run distributed workers for high rates and aggregate stats centrally.
• Other tools: JMeter for protocol breadth and legacy suites, Gatling for Scala/Java shops, Vegeta and wrk for simple HTTP saturation, cloud load generators when you need geographic distribution or very high rates.
• Generate load from outside the target network path you are measuring; a load generator sharing CPU with the service invalidates results.

Interpreting Results

1. Plot throughput and latency percentiles against time and against load level; inspect the shape, not just the final number.
2. Find the knee: the load level where latency departs from linear and errors or queue depth begin to grow. The knee is the real capacity, not the highest rate that completed.
3. Correlate client metrics with server metrics (CPU, memory, pool wait, queue depth, GC, DB) captured on the same clock. The test without server telemetry is only half the data.
4. Check per-endpoint and per-journey breakdowns; aggregate success can hide a failed journey.
5. Verify recovery: after a stress or spike test, does the system return to baseline latency without a restart? Lingering queue or connection growth is a finding.
6. Reconcile with Little's Law: observed concurrency should match throughput times latency; mismatch points to measurement or scenario defects.
7. Look for coordinated omission artifacts: suspiciously flat latency at high load in a closed test.
8. Record findings as defects with evidence, not as a single summary number.

| Result shape | Reading | |---|---| | Flat latency, linear throughput | Headroom exists; find the next resource ceiling | | Latency rises before CPU saturates | Queueing or lock contention; find the bottleneck. See ./04-backend-profiling.md | | Throughput plateaus, latency rises | Saturated resource; the knee is reached | | Throughput falls as load rises | Contention, retries, or thrashing; inspect error mix | | Errors at a threshold with stable latency | Hard limit or rate limiter; confirm intended behavior | | Latency recovers slowly after spike | Backlog drainage issue; check queues and pools |

Noise Control

• Run tests in a dedicated, isolated environment with the same topology, build, configuration, and data volume as production. Shared staging invalidates results.
• Freeze deployments, schema changes, and background jobs for the test window; record any exceptions.
• Warm up before the measurement window; report both cold-start and steady-state results separately.
• Repeat runs (at least three) and report median and spread; a single run is an anecdote. See ./01-methodology.md.
• Control data state: reset to a known snapshot or account for stateful drift between runs.
• Synchronize clocks (NTP) and capture server-side telemetry; correlation depends on alignment.
• Record the environment fingerprint with every result: versions, instance types, limits, replicas, and data snapshot id.
• Beware the observer effect: monitoring agents, tracing at 100 percent, and verbose logging change performance; keep production sampling rates.

What Load Tests Do Not Catch

• Real production data distribution, cache warmth, and multi-tenant skew, unless the test data reproduces them.
• Third-party dependencies and their limits; mock them or test their rate limits explicitly.
• Multi-region, CDN, and edge behavior; test where the traffic actually flows.
• Real client diversity (devices, network conditions, retry behavior).
• Security controls and WAF behavior under load; validate separately.
• Slow-building failures (disk fill, log growth, certificate expiry, connection table exhaustion) unless soak coverage is long enough.

Complement load tests with canary releases, shadow traffic, and production traffic slices. Progressive delivery catches what pre-production models miss.

Failure-Mode Testing

• Dependency latency and failure injection: make a downstream slow, return errors, or disappear; assert timeouts, breakers, and shedding behave as designed. See ./06-concurrency-backpressure.md.
• Node and zone loss: kill instances mid-test and verify graceful drain and recovery.
• Database failover and replica lag: force a failover or pause replication and measure user impact. See ./03-databases.md.
• Resource exhaustion: fill queues, exhaust a pool, hit memory ceilings, and verify alerts and shedding rather than collapse.
• Chaos experiments are load tests with an attack: run them in a controlled window with a rollback plan and clear stop conditions.

Reporting

• State the question, pass criteria, environment fingerprint, workload model, and duration.
• Present throughput and percentile curves, the knee, error mix, and resource saturation evidence.
• List defects with reproduction steps and severity; list explicit non-findings and coverage gaps.
• Record the versioned test plan and results in the repository so the next run is comparable.

Anti-Patterns

• One big test without a question or pass criteria.
• Closed-loop tests quoted as production tail latency.
• Seed data and empty caches that hide N+1 and index problems.
• Load generators co-located with the system under test.
• A single run, no warmup, no repetitions.
• No server-side telemetry during the test.
• Testing only the happy path; no dependency failure, failover, or recovery checks.
• Treating a passed load test as proof of production behavior; ignoring canary and field evidence.
• Soak tests too short to surface leaks or resource drift.

Checklist

• [ ] Test type and pass criteria chosen from the SLO before running.
• [ ] Workload model derived from production traffic mix and peak factor.
• [ ] Open or closed model selected deliberately; coordinated omission addressed.
• [ ] Data volume, auth, cache state, and journey weights are production-like.
• [ ] Thresholds enforced in CI or release gates for critical paths.
• [ ] Environment isolated, frozen, warmed, and fingerprinted.
• [ ] Server telemetry captured on synchronized clocks and correlated.
• [ ] Knee identified; recovery after stress verified.
• [ ] Failure-mode and failover tests included.
• [ ] Results reported with limitations, defects, and coverage gaps.

Capacity and Cost

Scope: capacity planning, autoscaling signals, SLO-aware scaling, cost-per-request optimization, and the tradeoff checklist.

Capacity Planning Basics

• Define the unit of capacity: requests per second per instance for stateless services, messages per second per consumer, queries per second per database, or concurrent connections per gateway.
• Capacity planning is a demand forecast plus a headroom policy. Without both, it is either guesswork or over-provisioning.
• Plan to the peak, not the average. Peak factors of 2-5x daily average are common for consumer services; seasonal and campaign peaks can be far higher.
• Plan for failure, not just demand: losing one zone, one region, or one large dependency should not breach the SLO. Determine the required redundancy level explicitly.
• Re-verify capacity after every architecture change, runtime upgrade, dependency change, or major data growth. Capacity plans expire.

Building a Capacity Model

1. Measure the per-instance ceiling with a load test at the target latency: the load level at the knee, not the maximum observed. See ./08-load-testing.md.
2. Identify the bottleneck order: what saturates first (CPU, memory, DB, connections, network, locks), second, and third.
3. Model demand: instances = peak_rps / per_instance_capacity * safety_factor, then round up and add failure headroom.
4. Include startup time: an instance that takes minutes to become useful does not help with a spike unless pre-warmed or over-provisioned.
5. Validate the model against production saturation metrics; adjust the per-instance capacity to observed reality.
6. Stress-test the model by removing capacity (kill a zone in staging or during an exercise) and confirming the SLO holds.

required_instances = ceil( peak_rps / per_instance_rps * (1 + safety_margin) )
required_instances += failure_headroom   # e.g. one zone worth of capacity

cost_per_1k_requests = hourly_cost_per_instance * instances / (rps * 3600) * 1000

• Keep the model in the repository next to the service, with its assumptions and measurement dates.
• Track the model's error: predicted versus actual saturation after each peak. A persistent overestimate wastes money; an underestimate pages people.
• Databases and caches rarely autoscale horizontally like stateless services; plan their capacity separately with explicit upgrade steps.

Planning Inputs

| Input | Source | Use | |---|---|---| | Traffic history | Metrics by endpoint and tenant | Baseline and growth rate | | Seasonality and campaigns | Product and marketing calendar | Peak windows and multipliers | | Growth rate | Month-over-month trend | When new capacity is needed | | Launch events | Roadmap | Step changes in load | | Failure scenarios | SLO and dependency map | Redundancy headroom | | Cost ceiling | Finance and unit economics | Bound on instance class and count | | Lead times | Procurement and quota | When to start scaling work |

• Keep a rolling forecast at least one quarter ahead and update it monthly.
• For campaigns, pre-scale and pre-warm; autoscaling reacts after the spike has already arrived.
• Capacity review should be a scheduled activity, not a reaction to an incident.

Autoscaling Signals

| Signal | Responds to | Notes | |---|---|---| | CPU utilization | Compute-bound services | Lagging; use as a coarse fallback | | Memory utilization | Memory-heavy services | Poor for leak-prone services; scale on load instead | | Requests per second per target | Stateless request services | Simple, but ignores latency and request cost variance | | Concurrent connections or in-flight requests | Connection-heavy services | Closer to queueing reality; pairs with Little's Law | | Queue depth or age | Worker and async pipelines | Best leading signal for consumers; the queue is the backlog | | Custom metric (for example, active jobs, tenant count) | Specialized workloads | Requires a reliable metric pipeline and scaling policy | | Scheduled or predictive | Known diurnal and campaign patterns | Pre-warms before the peak; combine with reactive signals |

• Scale on a leading signal. CPU lags because it rises only after requests are already queued; concurrency and queue depth lead.
• Set targets with headroom: scaling to 70 percent utilization keeps queueing near linear and absorbs bursts while new capacity arrives. See ./01-methodology.md.
• Configure scale-up aggressively and scale-down cautiously: fast add, slow remove, with stabilization windows long enough to avoid oscillation (flapping).
• Scale-down must respect in-flight work and connection draining; a pod that dies mid-request turns a cost optimization into an error budget burn.
• Beware metrics with low resolution or delayed ingestion; autoscaling decisions based on stale data overshoot and thrash.
• Verify that the scaling policy cannot exceed quotas and that hitting a quota produces an alert, not a silent cap.
• Combine autoscaling with load shedding: capacity arrives in tens of seconds at best, shedding works immediately. See ./06-concurrency-backpressure.md.

SLO-Aware Scaling

• Translate the latency SLO into a concurrency target: max_in_flight = target_rps * latency_budget. Scale when measured in-flight concurrency exceeds a fraction of that, before latency degrades.
• Watch the saturation metric that correlates with the SLO. If p95 latency is your SLO, scale on the resource that queues first, not on the one that is easiest to measure.
• Prefer scaling on latency-adjacent signals (queue wait, pool wait, concurrency) over raw CPU; they move before user-visible degradation.
• During incidents, override autoscaling if the signal is misleading (for example, low CPU because of a lock or downstream wait). Manual floors and ceilings are valuable.
• For predictable peaks, use scheduled scaling to reach target capacity before the demand arrives, then let reactive policies handle the variance.
• Test scaling policies with spike tests and confirm containers reach ready state and receive traffic before the spike passes.

Cost per Request and Unit Economics

• Track cost per meaningful unit: per 1,000 requests, per active user, per job, or per GB processed. A falling bill with faster-growing traffic is not efficiency if cost per request rose.
• Build the cost model from components:

| Component | Drivers | Typical levers | |---|---|---| | Compute | Instance hours, vCPU, memory | Rightsizing, ARM instances, spot/preemptible, autoscaling targets | | Database | Instance class, storage, IOPS, replicas | Query tuning, index fixes, pooling, read replicas, tiering | | Cache | Memory GB-hours, bandwidth | Hit ratio, TTL design, value sizes. See ./02-caching.md | | Storage | GB stored, operations, retrieval | Lifecycle tiering, compression, retention policy | | Network egress | Cross-zone, cross-region, internet bytes | Caching, compression, locality, CDN | | Third-party APIs | Calls, tokens | Batching, caching, contract review | | Observability | Ingested bytes, retained samples | Sampling, filtering, retention tiers |

• Attribute cost to features and tenants where possible; shared-cost allocation is approximate, but directionally useful for prioritization.
• Compare cost per request across releases: a performance regression often shows up as a cost regression before a latency regression.
• Set a cost budget per service with an alert, the same way latency has a budget.

Efficiency Levers

| Lever | Typical effect | Risk / caveat | |---|---|---| | Rightsizing instances | 10-40% compute savings | Needs real utilization data, not averages | | Raising autoscaling utilization target | Fewer idle instances | Tails grow; validate against SLO | | Cache hit ratio improvement | Large downstream savings | Correctness and invalidation risk. See ./02-caching.md | | Query and index tuning | Large DB savings | Requires plan analysis and maintenance | | Batching and compression | Fewer calls, fewer bytes | Latency tradeoff; batch caps | | Storage tiering and lifecycle | Large storage savings | Retrieval latency and cost; legal holds | | ARM-based instances | Often meaningful price/performance gain | Build compatibility; test performance, do not assume | | Spot / preemptible capacity | Deep discounts | Interruption handling; fit for batch and stateless tiers | | Reserved or committed capacity | Discount for steady baseline | Lock-in; forecast must be sound | | Serverless pay-per-use | Zero idle cost | Cold starts, per-request pricing cliffs at high volume | | Removing unused resources | Immediate savings | Orphaned volumes, old snapshots, idle environments |

• Apply levers in order of value and reversibility: rightsizing and query fixes first, commitments last.
• Re-measure after every efficiency change; savings that cost latency usually get reverted, and it is better to know before rollout.
• Consolidate environments (preview, staging, dev) with schedules and quotas; idle non-production is a common hidden cost.

Tradeoff Checklist

| Decision | Cost side | Risk side | Question to ask | |---|---|---|---| | Longer cache TTL | Cheaper origin | Staler data | What is the maximum acceptable staleness? | | Higher utilization target | Fewer instances | Tail latency | Which percentile does the SLO protect? | | More replicas | Higher DB cost | Lag and complexity | Is the read budget actually the bottleneck? | | Bigger instances vs more small | Simpler ops vs elasticity | Blast radius, scaling granularity | Which scales with traffic shape? | | Spot capacity | Cheaper compute | Interruptions | Can work be checkpointed or drained? | | More observability retention | Better investigations | Ingest and storage cost | Which signals are queried after 30 days? | | Aggressive load shedding | Protects core users | Some rejected traffic | Are rejections visible and budgeted? | | Denormalized or precomputed data | Faster reads, cheaper joins | Consistency and write cost | Who owns reconciliation? |

• Record tradeoff decisions with their context; the same choice may be wrong under a different SLO or traffic mix.
• Revisit decisions when the input changes (new SLO, new traffic pattern, price change), not on a fixed calendar alone.

Review Cadence

• Monthly: unit cost review (cost per request or active user) and saturation trend against the capacity model.
• Per release: performance and cost regression check on canary; scale settings unchanged unless evidence says otherwise.
• Quarterly: capacity and failure-mode test (load test, zone-loss exercise), model refresh, and commitment review.
• Before major launches: forecast update, pre-scaling plan, load test at the projected peak, and a rollback plan.

Anti-Patterns

• Planning on averages and discovering the peak in production.
• Scaling on CPU for a service whose bottleneck is database connections or queue depth.
• Autoscaling without load shedding and assuming capacity arrives instantly.
• Cutting headroom to hit a cost target, then blaming the collector, the database, or "unexpected" spikes.
• Optimizing the bill without tracking cost per request; total cost can fall while unit cost rises.
• Buying commitments before rightsizing and query tuning.
• Copying instance classes and scaling policies between services with different profiles.
• Ignoring egress, third-party, and observability costs in unit economics.
• Stale capacity plans from a previous architecture.

Checklist

• [ ] Capacity model exists with per-instance ceiling, peak factor, and failure headroom.
• [ ] Bottleneck order documented and validated by load test at the knee.
• [ ] Autoscaling uses leading signals with headroom targets and anti-flapping windows.
• [ ] SLO translated into concurrency and saturation targets; shedding covers the scale-up gap.
• [ ] Cost per request (or per active user) tracked over time with an owner.
• [ ] Cost model covers compute, database, cache, storage, egress, third parties, and telemetry.
• [ ] Efficiency levers applied in value/reversibility order; each change re-measured.
• [ ] Tradeoff decisions recorded with context, including caching and utilization choices.
• [ ] Forecast updated monthly; capacity and failure reviewed quarterly.
• [ ] Quotas and limits monitored with alerts, not discovered via silent caps.