Language Core

Scope: modern Go language mechanics — generics, stdlib collection helpers, iterators, errors, defer/panic, struct tags, embedding, zero values, and the footguns that survive into production.

Generics

Generics (1.18+) are for algorithms and containers over homogeneous types; interfaces remain the tool for polymorphic behavior.

func Map[S ~[]E, E, R any](in S, f func(E) R) []R {
	out := make([]R, len(in))
	for i, v := range in {
		out[i] = f(v)
	}
	return out
}

func Sum[T cmp.Ordered](vals ...T) T {
	var zero T
	for _, v := range vals {
		zero += v
	}
	return zero
}

Rules that matter in review:

• Constraints use type sets: ~int accepts defined types whose underlying type is int. Use cmp.Ordered for <-comparable types; comparable only permits ==/!=.
• Methods cannot declare their own type parameters. If a method needs one, write a generic function or a generic type.
• T may be a pointer type, so if v == nil only compiles when the constraint allows it (for example interface{ ~*S } or a concrete pointer); a bare any cannot be compared structurally.
• Inference fails on return types; annotate explicitly at call sites, e.g. Map[[]int, int, string](...) when the function argument is not enough.
• Generic type aliases require Go 1.24+; before that, aliases could not be parameterized. Verify the floor in go.mod.
• Do not genericize for one caller. A concrete function plus duplication beats an unreadable constraint lattice. See ./09-ecosystem-2026.md for library alternatives.

Slices, maps, cmp helpers

The slices, maps, and cmp packages (1.21+) replace most hand-rolled loops and sort.Slice (which is still fine for one-off Less closures).

| Need | Use | Instead of | |---|---|---| | Membership | slices.Contains | manual loop | | Sort by key | slices.SortFunc(xs, func(a, b T) int) | sort.Slice | | Sort ascending | slices.Sort | sort.Ints/Strings | | Binary search | slices.BinarySearch | sort.Search | | Remove dupes (sorted) | slices.Compact | map round-trip | | Remove by predicate | slices.DeleteFunc | filter loop | | Copy before mutation | slices.Clone | append([]T(nil), s...) | | Preallocate | slices.Grow(s, n) | make + copy gymnastics | | Map key/value snapshots | maps.Keys/maps.Values + slices.Collect | map loops | | Compare maps | maps.Equal | reflect.DeepEqual | | First non-zero | cmp.Or(a, b, c) | nested if |

Builtins min, max, clear (1.21+) apply to ordered types, and clear empties maps/slices. For strings, prefer strings.Cut over SplitN+indexing and strings.Builder over += in loops. bytes.Clone avoids retaining large backing arrays when slicing.

Allocation traps: maps.Keys is lazy (an iterator) and must be collected; slices.DeleteFunc zeroes removed slots to release pointers; slices.Clip releases spare capacity when retaining a small prefix of a large slice.

Iterators and range-over-func

Go 1.23 added iter.Seq[V] and iter.Seq2[K,V] with for range support. A Seq is func(yield func(V) bool); return false from yield to stop. Standard producers: slices.Values, slices.All, maps.All, strings.Lines/SplitSeq (1.24+).

func Count[T comparable](seq iter.Seq[T], want T) int {
	n := 0
	for v := range seq {
		if v == want {
			n++
		}
	}
	return n
}

func ReadAll(r io.Reader) iter.Seq2[[]byte, error] { /* batched reads */ }

Patterns:

• Pull-style consumption from a push iterator: next, stop := iter.Pull(seq); defer stop().
• Materialize with slices.Collect(seq), slices.Sorted(maps.Keys(m)), or slices.AppendSeq(dst, seq).
• Return errors per element with Seq2[V, error]; the caller stops on the first error.
• Convert a channel to an iterator only if the channel is closed; a nil channel blocks forever.
• Iterators are lazy and single-use; documenting ownership matters as much as for channels.
• for range 10 (range over int) requires 1.22+ and is the idiomatic "do n times".

Loop-variable semantics: since 1.22 each iteration has fresh variables, so closures and &v are safe. Modules declaring an older go version keep the old behavior unless you set GODEBUG=loopvar=1; check go.mod before "fixing" captures. See ./02-concurrency.md for how this interacts with goroutines in loops.

Errors

Three mechanisms, used deliberately: sentinels for conditions callers branch on, typed errors for structured data, wrapping for context.

var ErrNotFound = errors.New("not found")

type ValidationError struct {
	Field string
	Err   error
}

func (e *ValidationError) Error() string { return "invalid " + e.Field + ": " + e.Err.Error() }
func (e *ValidationError) Unwrap() error { return e.Err }

func Load(ctx context.Context, id string) (*Record, error) {
	rec, err := db.Get(ctx, id)
	if err != nil {
		if errors.Is(err, sql.ErrNoRows) {
			return nil, fmt.Errorf("load record %q: %w", id, ErrNotFound)
		}
		return nil, fmt.Errorf("load record %q: %w", id, err)
	}
	return rec, nil
}

• Wrap with %w when the caller may need errors.Is/errors.As; use %v when you deliberately break the chain.
• Multiple %w verbs (1.20+) and errors.Join create multi-error trees that errors.Is traverses.
• Implement Is/As methods on custom error types to compare by logical identity while retaining wrapping.
• errors.As requires a pointer to the target type: var ve *ValidationError; errors.As(err, &ve).
• errors.Is(err, nil) is false; check err != nil first, and never == compare wrapped errors.
• Error strings are lowercase, no trailing punctuation, and carry the failing operation ("decode config: unexpected EOF"). They must not contain secrets or user payloads.
• Do not log and return; let the owner of the request decide severity. Libraries never call log.Fatal/os.Exit.
• panic is for programmer errors and unrecoverable invariants. recover only works in a directly deferred function; use it at process boundaries (HTTP middleware, goroutine roots), never as control flow. Panics from goroutines cannot be recovered by another goroutine's recover.
• github.com/pkg/errors is obsolete: fmt.Errorf with %w covers its use cases. Verify in ./09-ecosystem-2026.md.

Defer semantics

• Deferred calls run LIFO when the surrounding function returns, panics, or runtime.Goexits — but not on os.Exit.
• Arguments (including method receivers) are evaluated at the defer statement, not at execution:

for i := 0; i < 3; i++ {
	defer fmt.Print(i) // prints 210
}

m := &Metrics{}
defer m.Close() // receiver captured now; later m = nil does not matter

• Named results can be observed and modified by deferred functions, which enables defer func() { err = errors.Join(err, tx.Rollback()) }().
• Modern compilers open-code defers in straight-line functions; a defer inside a loop still allocates and fires only at function exit. To release per-iteration resources, wrap the body in a closure/function call.
• recover must be called directly in the deferred function; defer recover() works but discards the panic info.

Struct tags and JSON

Tags are metadata conventions, not a type system. Validate them with go vet's structtag analyzer.

type Event struct {
	ID        string    `json:"id"`
	Type      string    `json:"type"`
	CreatedAt time.Time `json:"created_at"`
	Payload   []byte    `json:"-"`
	Score     float64   `json:"score,omitempty"`
	Updated   time.Time `json:"updated,omitzero"` // 1.24+
}

• omitempty omits zero values (empty string, 0, empty slice/map, nil pointer); omitzero (1.24+) uses the IsZero method and covers time.Time.
• Only exported fields are marshaled; embedded structs flatten by default.
• encoding/json matches field names case-insensitively when no tag exists, which hides contract drift — always tag wire types.
• Unknown fields are ignored by default; use Decoder.DisallowUnknownFields when consuming first-party JSON.
• For databases use db/pgx tags, validation uses validate, config uses koanf/env, and YAML uses yaml. One struct per boundary; do not overload one struct with five tag dialects.
• Custom MarshalJSON/UnmarshalJSON must handle nil receivers and avoid infinite recursion by aliasing the type.

Embedding

• Embedding promotes fields and methods; the outer type satisfies embedded interfaces.
• Conflicts: if two embedded types promote the same name at the same depth, selection is ambiguous and fails to compile; qualify explicitly.
• Embedding is not inheritance: the outer value does not become the inner type for assignment or errors.As unless you implement Unwrap/As.
• Embedding a concrete type in a test double silently inherits real behavior. Embed the interface you are faking instead.
• Never embed sync.Mutex in an exported struct: Lock() becomes part of the public API and the struct becomes non-copyable. Use a named field.
• Embedding sync.Mutex also trips copylocks vet checks when the struct is copied.

Zero values

Design for usable zero values where the type allows it: bytes.Buffer, sync.Mutex, sync.Once, slog.Logger values, and most slices/maps.

| Zero value | Behavior | |---|---| | nil slice | len 0; append allocates; ranging is fine | | nil map | reads return zero, delete is fine, write panics | | nil channel | send/receive block forever; close panics; use in select to disable a case | | nil func | calling panics | | nil interface | holds no type; a typed nil pointer inside an interface is not nil | | zero time.Time | IsZero; not the Unix epoch |

The typed-nil-in-interface trap is the most common production bite: return nil explicitly, not a typed nil pointer, or check with reflection/IsZero.

Footguns

• append aliasing: two slices can share a backing array after s1 = append(s1, x); clone at API boundaries when ownership transfers.
• range copies each element; mutating for _, v := range s loops silently edits copies. Use index assignment.
• Map iteration order is randomized; never rely on it for output stability. Sort keys.
• Integer conversions truncate and overflow silently; bound before converting, and use math/strconv helpers.
• == on floats, time.Time (monotonic + location), and structs containing uncomparable fields (slices, maps, funcs) fails or misbehaves; use Equal/cmp.Compare.
• Shadowing with := inside if/for hides outer variables; enable shadow checks in review for critical paths.
• String concatenation in loops allocates quadratically; use strings.Builder with Grow.
• Taking the address of a range variable: safe since 1.22, but still surprising when the pointer outlives the loop and the module targets an older language version.
• init() with I/O, goroutines, or flag registration makes tests order-dependent; prefer explicit constructors.
• Comparing errors with == where wrapping is possible; always errors.Is.

Review checklist

• [ ] Constraint is minimal (comparable, cmp.Ordered, exact method set) and the generic actually has two or more call sites.
• [ ] Collection operations use slices/maps/cmp rather than hand loops where clearer.
• [ ] Iterator functions stop on yield == false and document ownership/single-use.
• [ ] Every error is wrapped with operation context, and callers branch with errors.Is/errors.As.
• [ ] defer argument-evaluation semantics are understood where mutation is involved.
• [ ] Wire structs are tagged; unknown fields rejected on inbound JSON where appropriate.
• [ ] Embedded types are intentional; no embedded mutexes or inherited test doubles.
• [ ] No typed-nil interface returns; zero values documented when unusual.

Concurrency

Scope: goroutine lifecycle, channels, select, context, errgroup/semaphore, worker pools, the memory model, race detection, leak patterns, and how to test concurrent code deterministically.

Ownership rules

Every goroutine needs three things before it is written:

1. Owner — a function/struct that starts and joins it.
2. Stop condition — context cancellation, channel close, or finite input.
3. Join point — WaitGroup.Wait, errgroup.Wait, or a done channel the caller reads.

If you cannot name all three, you are writing a leak. go func() inside a request handler, loop, or init without a join is a bug.

var wg sync.WaitGroup
for _, job := range jobs {
	wg.Add(1)
	go func(job Job) {
		defer wg.Done()
		process(job)
	}(job)
}
wg.Wait()

Since Go 1.22 loop variables are per-iteration, so the explicit job parameter is no longer required for correctness; keep it when the module declares an older language version. Never capture the loop variable in a goroutine you do not join before the next iteration mutates state — see ./01-language-core.md.

Channels

• Channels are for transferring ownership or signaling; mutexes are for protecting state. Do not build shared-state containers out of channels.
• Closing signals "no more values", never "a value was consumed". Only the sender closes; closing from the receiver side panics on the next send.
• Receiving from a closed channel yields the zero value immediately; use the comma-ok form or range to distinguish.
• Sends on a closed channel panic; sends/receives on a nil channel block forever. Nil channels are useful only to disable a select case.
• Buffered channels decouple producer and consumer; they are not a queueing strategy. If buffer size encodes a policy (backpressure, batching), document it and bound it.
• Synchronous handoff (unbuffered) guarantees the receiver observed the value before the sender proceeds; use it when that guarantee matters.
• Do not use len(ch) for control flow; it is a snapshot with no happens-before guarantees.

func generate(ctx context.Context, n int) <-chan int {
	out := make(chan int)
	go func() {
		defer close(out)
		for i := 0; i < n; i++ {
			select {
			case out <- i:
			case <-ctx.Done():
				return
			}
		}
	}()
	return out
}

select

select {
case v := <-in:
	handle(v)
case <-ticker.C:
	flush()
case <-ctx.Done():
	return ctx.Err()
default:
	// non-blocking: only when dropping is correct
}

• If several cases are ready, one is chosen uniformly at random; never rely on priority.
• A default turns the select into a poll; in a hot loop it becomes a busy-wait. Prefer blocking with cancellation.
• select {} blocks forever (deadlock detector fires); for { select { case <-ctx.Done(): return } } is the standard cancellable loop.
• time.After inside a loop allocates a timer per iteration that cannot be collected until it fires; use a time.Timer with Reset, or context.WithTimeout per operation.

Context

func (s *Service) Fetch(ctx context.Context, id string) (*Item, error) {
	ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
	defer cancel()

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
	if err != nil {
		return nil, fmt.Errorf("build request: %w", err)
	}
	resp, err := s.client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("fetch %s: %w", id, err)
	}
	defer resp.Body.Close()
	...
}

• First parameter, named ctx; never stored in a struct, never nil (context.TODO() at uncertain boundaries).
• Cancel functions must always be called, including on the error path; defer cancel() immediately after creation.
• context.Cause(ctx) (1.20+) returns the cancellation cause set by WithCancelCause; use errors.Is against it rather than guessing.
• context.WithoutCancel (1.21+) detaches values while keeping them (for fire-and-forget audit writes). context.AfterFunc runs cleanup when a context ends.
• Context values are for request-scoped data (request ID, auth principal, logger) crossing API boundaries — never for optional parameters or config. Typed, unexported keys only.
• Do not pass a context with a deadline that outlives the caller's; child deadlines shorten, never extend.
• Shutdown contexts and request contexts are different; the process-wide root context is canceled by signal handling in main, not by handlers. See ./04-web-services.md.

errgroup, semaphore, singleflight

golang.org/x/sync is part of the extended standard library and the default answer for structured concurrency.

g, ctx := errgroup.WithContext(ctx)
g.SetLimit(8)
for _, u := range urls {
	g.Go(func() error {
		return fetch(ctx, u)
	})
}
if err := g.Wait(); err != nil {
	return fmt.Errorf("fetch batch: %w", err)
}

• WithContext cancels the derived context when any goroutine returns a non-nil error; Wait returns the first error. Returning early does not stop running goroutines — they must observe ctx.
• SetLimit bounds concurrency; TryGo returns false instead of blocking. Prefer these to hand-built worker pools unless you need pipelining.
• semaphore.Weighted is for weighted resources (memory, connections). Always Release with the same weight, normally via defer after a successful Acquire.
• singleflight.Group collapses concurrent identical work (cache stampedes, token refreshes). Use DoChan when callers need cancellation; forget entries on failure if retries must be possible.
• sync.OnceFunc, sync.OnceValue, sync.OnceValues (1.21+) memoize initialization with panic propagation. They are safe to copy after first use only if never copied before; treat like Once.
• sync.Map is for append-mostly caches with disjoint key sets, not a general map replacement. atomic.Pointer[T] and typed atomics (1.19+) beat Mutex for single-word state.

Worker pools, pipelines, fan-in/out

A worker pool bounds resource use; a pipeline composes stages. Cancellation must travel downstream, and every stage must close its output.

func runPool(ctx context.Context, in <-chan Job, workers int) <-chan Result {
	out := make(chan Result)
	var wg sync.WaitGroup
	for range workers {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for job := range in {
				select {
				case out <- process(job):
				case <-ctx.Done():
					return
				}
			}
		}()
	}
	go func() {
		wg.Wait()
		close(out)
	}()
	return out
}

Fan-in: a single collector goroutine plus a WaitGroup closing the merged channel. Fan-out: N goroutines reading the same input channel (channels distribute each value to exactly one reader, which is a load-balancing primitive).

Anti-patterns: unbounded go per item; result channels with no reader after early cancellation; pipelines that never close intermediate channels; WaitGroup.Add called after Wait can start.

Memory model and race detection

Guarantees you can rely on:

• Goroutine start happens-before the first statement of the new goroutine.
• A channel send happens-before the corresponding receive completes; close happens-before a receive of the zero value.
• Mutex.Unlock happens-before a subsequent Lock; RWMutex read locks provide the same for writers.
• Atomic operations on the same location are sequentially consistent per the Go memory model.
• Anything else shared needs synchronization. "It worked locally" is not a happens-before edge.

Race detector:

go test -race ./...
go build -race -o bin/app-race ./cmd/app

• Detects racy accesses that actually occur in the run; it is not a proof of absence.
• Treat every report as a real bug: add synchronization or redesign ownership.
• CI should run -race for unit tests and at least one integration path; the detector needs cgo enabled on most platforms, so keep a race-enabled CI job separate from pure static builds.
• GORACE="halt_on_error=1" makes CI fail fast.

Leak patterns

| Pattern | Symptom | Fix | |---|---|---| | Send with no guaranteed receiver | goroutine parked on channel forever | select on ctx.Done(); buffered result when the contract allows drop | | context.WithCancel cancel not called | context tree and child goroutines persist | always defer cancel() | | Ticker/time.After in a loop | timer heap growth, delayed wakeups | time.NewTicker + defer Stop; reuse timers | | Worker pool never signaled to stop | workers idle forever after work | close input or cancel context; join workers | | HTTP response body not closed | connection stuck in pool, leak report | defer resp.Body.Close() plus drain | | WaitGroup.Add inside spawned goroutine | Wait returns early or panics | Add before go | | Blocked logger/metrics sink | whole pipeline stalls | bounded async sinks with drop policy | | Context value carrying a client | request lifetime pins transport/resources | keep clients in structs with explicit lifecycle |

Diagnostics: go tool pprof goroutine profile, runtime.NumGoroutine deltas in tests, and GOEXPERIMENT/runtime leak detection when available (verify upstream for the current toolchain). See ./07-performance-profiling.md.

Testing concurrent code

• testing/synctest (stable in 1.25+) runs a goroutine inside a "bubble" with a fake clock: synctest.Test(t, func(t *testing.T) { ... synctest.Wait() ... }). Use it to test timeouts, debounce, retries, and leak behavior deterministically.
• go.uber.org/goleak catches leaked goroutines: goleak.VerifyTestMain(m) in TestMain, or defer goleak.VerifyNone(t)() in focused tests.
• Use t.Context() (1.24+) for per-test cancellation and context.WithTimeout in tests to fail fast instead of hanging CI.
• Replace channels with synctest.Wait() where possible; when not, synchronize with channels rather than sleeps.
• Run go test -race -count=100 on concurrency-sensitive packages; add -shuffle=on to catch interleaving assumptions.
• Assert on observable outcomes (counts, errors) rather than internal scheduling.
• See ./06-testing.md for fuzz/benchmark hygiene and Testcontainers integration.

Anti-patterns

• Goroutines started in init() or package-level variables.
• A sync.Mutex protecting an entire struct while methods call each other (reentrancy deadlock).
• RWMutex used for write-heavy state; prefer Mutex or atomics.
• Channels used to signal completion where WaitGroup/errgroup expresses it directly.
• Background goroutine writing to a map read by request handlers.
• time.Sleep as synchronization; polling loops without backoff.
• Unbounded fan-out from request handlers ("goroutine per item").
• Forgetting resp.Body.Close() or rows.Close() in concurrent code paths.
• Canceling a context to "stop" work while downstream code ignores ctx.
• Using runtime.GOMAXPROCS(1) or LockOSThread as a substitute for correct synchronization.

Review checklist

• [ ] Every goroutine has owner, stop condition, and join.
• [ ] Channels are closed exactly once, by the sender, and only when no more sends are possible.
• [ ] select cases include cancellation and never use default to avoid handling backpressure.
• [ ] Context is first argument, canceled on all paths, never stored, never nil.
• [ ] Bounded concurrency via errgroup.SetLimit/semaphore for external calls.
• [ ] go test -race passes; leak checks run on packages that spawn long-lived goroutines.
• [ ] Shared state has an explicit happens-before edge (channel, mutex, or atomic).
• [ ] Timers/tickers stopped; no time.After in loops.
• [ ] Concurrent tests are deterministic, not sleep-based.

Project Layout and Tooling

Scope: repository structure, modules and workspaces, toolchain and tool directives, linters, code generation, build tags, cross-compilation, and release mechanics for Go 1.22-1.26.

Layout

Start with three directories; add the rest only when a concrete need appears.

service/
  cmd/
    api/main.go          # thin: flags, wiring, signal handling
    worker/main.go
  internal/
    httpapi/             # transport: handlers, DTOs, middleware
    order/               # domain: service, rules, errors
    storage/             # adapters: postgres, valkey
  migrations/
  testdata/
  go.mod
  go.sum
  Makefile

• cmd/<binary>/ holds only main packages. main parses config, constructs dependencies, starts the server, and blocks on signals — no business logic.
• internal/... is enforced by the toolchain: importable only within the module subtree rooted at the internal directory. If you do not intend to share code, internal is the default.
• pkg/ is not required by Go and buys nothing for applications; use it only when publishing a deliberate public library surface. Many teams skip it entirely.
• Name packages by what they provide (order, storage), never util, common, helpers, or models. A directory name and its package name should match; avoid stutter (order.Order, not order.OrderModel).
• Prefer package-by-feature for domains and split by layer only inside small packages. Cycles are a design smell: domain must not import storage.
• testdata/ is ignored by the toolchain; keep fixtures there. Golden files live next to tests per ./06-testing.md.
• api/ for OpenAPI/proto specs, migrations/ for SQL, docs/ for ADRs. Architecture decision records beat tribal knowledge.
• One module per independently versioned deliverable. Over-splitting into dozens of modules multiplies go.work, release, and tooling cost; under-splitting couples unrelated services.

Modules

go.mod is the source of truth for the dependency graph; go.sum records hashes. Commit both.

module github.com/acme/service

go 1.24

toolchain go1.25.4

require (
	github.com/jackc/pgx/v5 v5.x.y
	golang.org/x/sync v0.x.y
)

require (
	github.com/jackc/pgpassfile v1.0.0 // indirect
	...
)

• Minimal Version Selection (MVS) picks the maximum of the minimum versions each module requires. go mod tidy computes the needed set; run it after every dependency change and commit the result.
• The go directive is a language version and a strict minimum toolchain requirement since 1.21. Do not bump it casually: it changes language semantics (for example loop variables) and blocks older toolchains.
• toolchain line records the preferred toolchain; GOTOOLCHAIN=auto (default) downloads it. GOTOOLCHAIN=local pins CI to the installed toolchain. Verify the current policy upstream.
• replace is for local development and forks. Replacements are ignored in downstream modules, so never rely on them for published libraries; use go mod edit -replace and document.
• Semantic import versioning: after v1, major versions live in the path (/v2) and are separate modules. Publish tags v2.x.y, never move tags.
• retract in go.mod marks bad versions so consumers stop selecting them.
• go mod tidy -diff (1.23+) checks cleanliness in CI without writing.
• Private modules: set GOPRIVATE=github.com/acme/* (implies GONOSUMDB/GONOSUMCHECK behavior) and configure credentials in ~/.netrc or git. Keep GOSUMDB=sum.golang.org on for public code.
• Vendoring (go mod vendor) is a legitimate choice for hermetic builds and regulated environments; it makes dependency diffs reviewable at the cost of repository churn.
• Diagnostics: go mod why -m X, go mod graph, go list -m -u all, go version -m ./bin/app (embedded build info).

Workspaces

go.work lets multiple modules resolve each other locally without replace lines in every go.mod.

go 1.24

use (
	./service
	./libs/logger
)

replace github.com/acme/logger => ./libs/logger

• Use workspaces for multi-module repositories during active development. They are not published and do not affect consumers.
• go work sync aligns workspace requirements back into member go.mod files.
• In CI, either commit go.work deliberately or run with GOWORK=off to test the published dependency graph. Test both states if the workspace is committed.
• Avoid workspaces to patch a dependency you do not own; fork or upstream the fix instead.

Toolchain and tools

Two ways to pin developer tooling, in order of preference:

1. Tool directives (Go 1.24+): go get -tool golang.org/x/tools/cmd/stringer@latest adds a tool block to go.mod; run with go tool stringer, version automatically resolved.
2. Legacy tools.go for older floors:

//go:build tools

package tools

import (
	_ "github.com/golangci/golangci-lint/cmd/golangci-lint"
	_ "golang.org/x/vuln/cmd/govulncheck"
)

go run golang.org/x/vuln/cmd/govulncheck@latest ./...

• Pin exact versions of linters and generators in the repo; "latest" drift is a supply-chain and reproducibility risk.
• gopls is the standard language server; enable staticcheck diagnostics and format-on-save. Ship .gopls/gopls settings in the repo so editors converge.

Linting and static analysis

| Tool | Catches | Notes | |---|---|---| | gofmt / gofumpt | formatting | mandatory, zero discussion | | go vet | real correctness bugs (printf, copylocks, struct tags, lostcancel) | run in CI, not just editor | | staticcheck (SA/S/ST/QF) | dead code, misuse, simplifications | high signal; wire into gopls | | golangci-lint v2 | aggregates linters | config schema changed in v2; verify upstream and migrate configs | | govulncheck | reachable known vulnerabilities | run in CI and on release; also ./08-observability-security.md | | deadcode | unreachable exported/unused code | useful when trimming APIs | | go fix | modernizes idioms | modernizer set grows; verify against current toolchain | | gosec | security-oriented patterns | complements, never replaces review |

CI gate: gofmt -l ., go vet ./..., staticcheck ./... (or golangci-lint), go test -race ./..., govulncheck ./....

Code generation

• Directives: //go:generate go tool stringer -type=State, //go:generate go tool sqlc generate, //go:generate go tool buf generate.
• Generated files start with // Code generated by <tool>. DO NOT EDIT. and are committed; CI regenerates and fails if git diff is non-empty. That keeps builds working without codegen tools installed.
• Common generators: stringer, mockgen/moq, sqlc (./05-data.md), oapi-codegen, protoc-gen-go/buf (./04-web-services.md), wire for DI.
• Keep generation hermetic: pin tool versions and input paths; never generate from network state.
• Use //go:build ignore on generator driver programs rather than dumping them in main.

Build tags

• Use the modern //go:build syntax; // +build lines are obsolete. A blank line must separate the constraint from package and from doc comments.
• Standard axes: GOOS, GOARCH, cgo, unix, release tags (go1.24). Custom tags express build variants: //go:build integration, //go:build e2e.
• Filename conventions: foo_linux.go, foo_amd64.go, foo_linux_amd64.go, foo_test.go. Prefer filename suffixes for OS/arch; use explicit tags for variants.
• Test-only tags must pair with a CI job that actually runs them; otherwise integration tests become dead code.
• Beware !cgo builds silently omitting functionality (DNS resolvers, netgo); assert capabilities in tests.

Cross-compilation and build flags

CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -trimpath \
  -ldflags="-s -w -X main.version=$VERSION -X main.commit=$COMMIT" \
  -o bin/service-linux-arm64 ./cmd/api

• CGO_ENABLED=0 yields static binaries; keep a cgo build only if you need system libraries (libsqlite, libgit2), and then test it separately.
• -trimpath removes local paths for reproducible builds. -buildvcs=true (default) stamps VCS info; disable only for hermetic reproducibility.
• go build is cached; go install pkg@version for tools. GOFLAGS=-mod=readonly prevents accidental go.mod edits in CI.
• -gcflags=-m for escape analysis (./07-performance-profiling.md); Profile-Guided Optimization uses a committed default.pgo.
• Reproducibility: same toolchain, -trimpath, pinned modules, SOURCE_DATE_EPOCH for archive timestamps. Verify with a second build and go version -m.

Release

• Version with semver git tags: v1.4.2. For libraries the tag is the release. For applications, inject version/commit into main via -ldflags -X.
• Use GoReleaser (or an equivalent) for cross-platform archives, checksums, SBOM (syft), and signatures (cosign). Verify the current plugin set upstream.
• Containers: multi-stage build, final stage distroless/static or scratch with a non-root user, -trimpath binary, no shell.
• Publishing a module: push a tag, then go list -m pkg@version from a clean module to confirm resolution through the checksum database.
• Keep a CHANGELOG generated from commits or release notes; consumers need to know about retract and behavior changes.
• See ./08-observability-security.md for signing and supply-chain controls, and ./09-ecosystem-2026.md for library landscape.

Anti-patterns

• A util package that imports half the repo and is imported by everything.
• internal used as a synonym for "ugly"; it is an API boundary, not a style label.
• Committing go.work with absolute local paths (or committing one at all without a reason).
• Bumping the go directive to get a feature without checking OS/toolchain consumers.
• Running go mod tidy with the network unavailable, silently dropping requirements.
• Linters pinned to @latest; generated code edited by hand; generator versions not pinned.
• Build-tag combinations that are never compiled in CI (dead branches).
• -ldflags "-s -w" without retaining debug symbols for crash triage.

Review checklist

• [ ] Layout matches actual needs; no speculative pkg/ or empty layers.
• [ ] go.mod directive floor is deliberate; toolchain pinned; go mod tidy -diff clean.
• [ ] Workspaces handled explicitly in CI (GOWORK=off where published graph must be tested).
• [ ] gofmt, go vet, staticcheck, go test -race, govulncheck in CI.
• [ ] Tools pinned via tool directives or tools.go; generation reproducible and committed.
• [ ] Build tags used for variant and integration coverage with matching CI jobs.
• [ ] Release artifacts reproducible, versioned, signed, SBOM-attached.
• [ ] No hand edits to generated files; no secrets in build args or tags.

Web Services

Scope: net/http routing and middleware, framework selection, JSON and streaming transports, gRPC, graceful shutdown, server/client timeouts, and request hardening.

Routing with net/http (1.22+)

The standard ServeMux supports method and wildcard patterns, removing the main reason to adopt a router for straightforward APIs.

mux := http.NewServeMux()
mux.HandleFunc("GET /healthz", health)
mux.HandleFunc("GET /v1/orders/{id}", getOrder)
mux.HandleFunc("POST /v1/orders", createOrder)
mux.HandleFunc("GET /v1/files/{path...}", serveFile)

func getOrder(w http.ResponseWriter, r *http.Request) {
	id := r.PathValue("id")
	...
}

• Pattern grammar: METHOD /path/{name} and {name...} for trailing segments; a pattern without a method matches all methods.
• Precedence is by specificity: literal segments beat wildcards, and {$} matches only the exact path (distinguishing /x from /x/). Panics at registration time on conflicts — a good thing.
• PathValue returns "" for missing names; validate before use.
• Middleware is plain func(http.Handler) http.Handler; chain in main, order matters (recovery outermost, then request ID, logging, auth, rate limit, handler).

func recoverer(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		defer func() {
			if rec := recover(); rec != nil {
				slog.ErrorContext(r.Context(), "panic", "err", rec,
					"stack", string(debug.Stack()))
				http.Error(w, "internal error", http.StatusInternalServerError)
			}
		}()
		next.ServeHTTP(w, r)
	})
}

Framework decision table

| Situation | Choice | Rationale | |---|---|---| | Straightforward JSON API, no deps | stdlib net/http + 1.22 routing | smallest audit surface, zero upgrade treadmill | | Need middleware ecosystem, sub-routers, URL params today | chi | thin, stdlib-compatible http.Handler | | Large team wants built-in binding/validation | echo or gin | be consistent org-wide; do not mix routers across services | | Typed service-to-service RPC | grpc-go (or Connect) | protobuf contract, streaming, codegen | | GraphQL | gqlgen | schema-first codegen | | Realtime browser updates | SSE; WebSocket only when bidirectional | SSE survives proxies, uses plain HTTP |

The framework is a small part of the service. Ports/adapters around the router keep handlers testable and make framework swaps rare. Avoid context in framework-specific types leaking into domain code.

Middleware mechanics

• Wrap http.ResponseWriter carefully: hiding http.Flusher, http.Hijacker, io.ReaderFrom, or Unwrap() breaks SSE, WebSockets, and io.Copy fast paths. Use http.NewResponseController(w) (1.20+) to reach deadlines/flush/hijack through wrappers.
• Panic recovery must run outermost so no request escapes unlogged; log with request ID and stack, then return a generic body.
• Request ID: accept inbound X-Request-ID only after validation, else generate; put it in context and in slog via a handler attribute (./08-observability-security.md).
• Auth middleware should attach an immutable principal to context and fail closed. Never trust headers alone from the public internet.
• CSRF: for cookie-based browser sessions, use SameSite cookies plus token validation; Go 1.25+ offers http.CrossOriginProtection (verify upstream) as origin-based defense.
• Rate limiting belongs in middleware with bounded memory (token bucket per key) or at the gateway; an unbounded map[string]*limiter is a memory leak.
• CORS: explicit allowlists; wildcard plus credentials is invalid and dangerous.

JSON handling

type createReq struct {
	Name string `json:"name"`
}

func createOrder(w http.ResponseWriter, r *http.Request) {
	r.Body = http.MaxBytesReader(w, r.Body, 1<<20)
	dec := json.NewDecoder(r.Body)
	dec.DisallowUnknownFields()

	var req createReq
	if err := dec.Decode(&req); err != nil {
		writeProblem(w, http.StatusBadRequest, "invalid json", err)
		return
	}
	if dec.More() {
		writeProblem(w, http.StatusBadRequest, "trailing data", nil)
		return
	}
	...
}

• DisallowUnknownFields for first-party clients; leave it off for forward compatibility with third parties, and always reject trailing data for single-object endpoints.
• Bound bodies everywhere (MaxBytesReader), and set Content-Type: application/json on responses.
• Prefer json.Decoder streaming for large payloads; decode timestamps as time.Time with RFC 3339 conventions, not strings.
• Error responses: stable machine-readable shape ({"error":{"code":"...","message":"..."}}), no internal error text; log the detailed error server-side.
• encoding/json/v2 is under experiment in recent toolchains (GOEXPERIMENT=jsonv2); do not depend on it in production until stable — verify upstream. json.RawMessage and json.Number remain the escape hatches.
• Avoid map[string]any for domain data; define structs for contracts and validate them explicitly.

Streaming: SSE and WebSockets

SSE for one-way server push:

w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
rc := http.NewResponseController(w)
for ev := range events {
	if _, err := fmt.Fprintf(w, "data: %s\n\n", ev); err != nil {
		return
	}
	if err := rc.Flush(); err != nil {
		return
	}
}

• SSE needs a write deadline strategy: WriteTimeout on the server kills long streams, so set WriteTimeout: 0 and use ResponseController.SetWriteDeadline per write or heartbeat.
• Send periodic comments (: ping) to keep intermediaries from closing idle connections; watch for client disconnects via r.Context().Done().

WebSockets:

• Prefer github.com/coder/websocket (maintained successor to nhooyr.io/websocket) for context-aware APIs and simpler concurrency rules; github.com/gorilla/websocket is maintained again and widely deployed. Verify current status upstream; see ./09-ecosystem-2026.md.
• Enforce SetReadLimit, origin checks (OriginPatterns/CheckOrigin), and ping/pong deadlines. Exactly one concurrent reader and one writer per connection; serialize writes.
• Treat WebSocket messages as untrusted input: validate size, schema, and rate. Close with proper codes on auth expiry.

gRPC

• Define .proto contracts, generate with buf (linting/breaking-change detection) and protoc-gen-go/protoc-gen-go-grpc. Generated code is committed per ./03-project-layout-tooling.md.
• Server options: grpc.MaxRecvMsgSize/MaxSendMsgSize bounded, keepalive enforcement (keepalive.EnforcementPolicy) to reject abusive clients, and health service (grpc_health_v1) plus reflection for tooling.
• Interceptors mirror HTTP middleware: recovery, logging, auth, metrics, and deadline propagation (the client's deadline is in the incoming context; respect it).
• Map domain errors to codes.NotFound/InvalidArgument/Internal; never leak internal messages. status.Error/status.FromError.
• Plaintext vs TLS: use credentials.NewTLS or mTLS inside the cluster; insecure only on loopback/dev.
• Connect (connectrpc.com/connect) is a strong alternative when you want HTTP/JSON compatibility and gRPC semantics without the gRPC transport stack; evaluate before committing.
• Client-side: set per-call timeouts (context.WithTimeout), retry only idempotent methods with backoff and jitter, and configure the resolver/balancer (round_robin for headless services).

Graceful shutdown

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

srv := &http.Server{
	Addr:              ":8080",
	Handler:           mux,
	ReadHeaderTimeout: 5 * time.Second,
	ReadTimeout:       30 * time.Second,
	WriteTimeout:      30 * time.Second,
	IdleTimeout:       120 * time.Second,
	MaxHeaderBytes:    1 << 20,
}

go func() {
	if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
		slog.Error("serve", "err", err)
		stop()
	}
}()

<-ctx.Done()
stop()

shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
	slog.Error("shutdown", "err", err)
}

• Shutdown stops listeners, closes idle connections, and waits for active requests until the context expires; then Close can force-terminate.
• Drain order: stop accepting, finish in-flight requests, stop background workers, flush telemetry, close DB pools. srv.RegisterOnShutdown and sync.WaitGroup for worker drain.
• Kubernetes: readiness fails first (pod removed from endpoints), liveness stays up while draining; terminationGracePeriodSeconds must exceed the shutdown deadline.

Timeouts and limits

Server:

| Setting | Purpose | Guidance | |---|---|---| | ReadHeaderTimeout | Slowloris protection | always set; a few seconds | | ReadTimeout | Whole request including body | set; raise for uploads via ResponseController.SetReadDeadline | | WriteTimeout | Whole response | set for typical APIs; disable for SSE/streaming and use per-write deadlines | | IdleTimeout | Keep-alive reuse | > load balancer idle timeout | | MaxHeaderBytes | Header bomb defense | default 1 MB; lower if proxies bound it | | http.MaxBytesReader | Body size | per endpoint | | handler context timeout | Dependency budget | shorter than client deadline |

Client hardening:

client := &http.Client{
	Timeout: 10 * time.Second,
	Transport: &http.Transport{
		MaxIdleConns:        100,
		MaxIdleConnsPerHost: 10,
		IdleConnTimeout:     90 * time.Second,
	},
}

• Never use http.DefaultClient in production: no timeout, shared mutable state, easy to hijack via http.DefaultTransport tweaks from a dependency.
• Always close and drain response bodies (io.Copy(io.Discard, resp.Body) before Close when reusing the connection is desired).
• Retries: only idempotent calls, capped attempts, exponential backoff with jitter, and respect Retry-After. Do not retry on context cancellation.
• SSRF: do not fetch user-supplied URLs without allowlisting hosts/schemes and blocking link-local/metadata ranges (./08-observability-security.md).

Anti-patterns

• Business logic in handlers; handlers should decode, validate, call a service, and encode.
• http.Error with internal error strings; stack traces in responses.
• Missing timeouts on either side; WriteTimeout set while serving SSE.
• Global mutable http.DefaultClient/DefaultServeMux shared across packages.
• Reading unbounded bodies or ignoring r.Context() cancellation in long queries.
• CORS wildcard with credentials, permissive WebSocket origins, unvalidated Host/X-Forwarded-* headers.
• Retry loops without backoff or idempotency; logging every request twice (middleware plus handler).
• Health endpoints that check downstream dependencies deeply and flap under partial outages.

Review checklist

• [ ] Routing uses method patterns with explicit validation of PathValue.
• [ ] Middleware order documented; recovery outermost; response wrapper preserves interfaces via ResponseController.
• [ ] All server timeouts set and compatible with streaming endpoints.
• [ ] Body size, header size, and per-handler context deadlines bounded.
• [ ] Structured error responses with stable codes; internal details only in logs.
• [ ] Graceful shutdown drains HTTP, workers, and pools within the platform grace period.
• [ ] Client has timeout, pooled transport, retry policy, and body handling.
• [ ] Transports (SSE/WebSocket/gRPC) enforce limits, heartbeats, and auth.

Data

Scope: SQL access with database/sql and pgx, generated queries with sqlc, migrations, transactions, caching with Valkey/Redis, pool sizing, and database test strategy.

database/sql fundamentals

sql.Open validates arguments and returns immediately; it does not dial. Verify connectivity with PingContext at startup and fail fast.

db, err := sql.Open("pgx", dsn) // pgx stdlib adapter
if err != nil {
	return fmt.Errorf("open db: %w", err)
}
db.SetMaxOpenConns(20)
db.SetMaxIdleConns(20)              // match open to avoid churn
db.SetConnMaxLifetime(30 * time.Minute)
db.SetConnMaxIdleTime(5 * time.Minute)

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := db.PingContext(ctx); err != nil {
	return fmt.Errorf("ping db: %w", err)
}

• Always pass ctx: QueryContext, ExecContext, QueryRowContext. A missing deadline turns a slow database into a hung service.
• QueryRow.Scan returns sql.ErrNoRows for zero rows — a normal condition, map it explicitly. Query requires rows.Close() even on error paths; check rows.Err() after the loop.
• Scan into *sql.Null[T] (generics since 1.22) or *T pointers for nullable columns; never into sql.NullString when the SQL type is not string.
• Prepared statements: pgx caches per-connection automatically; with database/sql use Stmt only when the same connection affinity matters. Re-preparing per request is usually unnecessary.
• Batch inserts: multi-row INSERT ... VALUES or COPY via pgx beats N round trips by orders of magnitude. Batch sizes of 500-5000 rows; bound the total statement size.
• Never build SQL with fmt.Sprintf on user input. Parameters only. Identifiers (table/column names) cannot be parameters — allowlist them.
• Connection pool knobs interact with the database's own max_connections and any PgBouncer in between; size pools from the sum of all service replicas, not per pod in isolation. See ./04-web-services.md for request budgets.

Transactions

func (r *Repo) Transfer(ctx context.Context, from, to string, amount int64) error {
	tx, err := r.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
	if err != nil {
		return fmt.Errorf("begin: %w", err)
	}
	defer func() { _ = tx.Rollback() }() // no-op after commit

	if _, err := tx.ExecContext(ctx, `UPDATE accounts SET balance = balance - $1 WHERE id = $2`, amount, from); err != nil {
		return fmt.Errorf("debit: %w", err)
	}
	if _, err := tx.ExecContext(ctx, `UPDATE accounts SET balance = balance + $1 WHERE id = $2`, amount, to); err != nil {
		return fmt.Errorf("credit: %w", err)
	}
	if err := tx.Commit(); err != nil {
		return fmt.Errorf("commit: %w", err)
	}
	return nil
}

• defer tx.Rollback() is the standard guard; Rollback after Commit returns sql.ErrTxDone and is safely ignorable. If you must surface rollback failures, errors.Join them in the deferred function.
• Canceling the context rolls back the transaction and returns the context error; wrap it so callers can errors.Is(err, context.DeadlineExceeded).
• Keep transactions short: no HTTP calls, no user think-time, no time.Sleep, no logging I/O. Long transactions hold locks and consume a pooled connection.
• Serialization failures (40001) and deadlocks (40P01) are retryable in the database, not the Go runtime. Wrap the whole tx in a retry helper with jitter, bounded attempts, and idempotent statements.
• Nested transactions become savepoints; prefer explicit SAVEPOINT when partial rollback is genuinely needed.
• A Tx is bound to one connection: never capture it in goroutines.

pgx v5

For PostgreSQL, pgx v5 is the default driver. lib/pq is in maintenance mode; its use is legacy. database/sql remains useful for portability and for libraries that require it — pgx provides stdlib registration for that case.

cfg, err := pgxpool.ParseConfig(dsn)
if err != nil {
	return err
}
cfg.MaxConns = 20
cfg.MinConns = 2
cfg.MaxConnLifetime = 30 * time.Minute
cfg.HealthCheckPeriod = time.Minute

pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
	return err
}
defer pool.Close()

rows, err := pool.Query(ctx, `SELECT id, name FROM users WHERE tenant = $1`, tenant)
if err != nil {
	return err
}
defer rows.Close()
for rows.Next() {
	...
}
return rows.Err()

| Need | pgx API | |---|---| | Single query | pool.QueryRow/Query | | Many statements per round trip | pgx.Batch | | Bulk load | CopyFrom (COPY protocol) | | Notifications | Conn.WaitForNotification (dedicated connection) | | Custom types | pgtype mappings; RegisterType | | LISTEN/NOTIFY consumers | one goroutine per dedicated pgx.Conn |

• pgx.Rows are single-use; CollectRows/RowToStructByName helpers reduce boilerplate but hide streaming — use them with bounded result sets.
• Default pool size is the greater of 4 and GOMAXPROCS; treat it as a starting point, not a tuning.
• pgx.ErrNoRows from QueryRow.Scan; pgconn.PgError carries SQLSTATE — branch on codes (unique_violation = 23505) rather than message text.
• Context cancellation sends a cancel request to the server; verified with pgx.Conn internals. Avoid context.Background() in queries.

sqlc and query generation

sqlc compiles SQL into typed Go. It fits read-heavy services where SQL is the source of truth and the team wants compile-time safety without an ORM.

version: "2"
sql:
  - engine: "postgresql"
    queries: "internal/storage/queries"
    schema: "migrations"
    gen:
      go:
        package: "db"
        out: "internal/storage/db"
        sql_package: "pgx/v5"

• Pros: no runtime reflection, explicit SQL, generated structs for rows/params, caught schema drift at build time.
• Cons: dynamic filters become awkward; complex queries still need hand tuning; regeneration churn in diffs. The config schema (version: "2" at time of writing) changes between major sqlc releases — verify the current schema upstream before pinning.
• Alternatives: hand-written pgx for small surfaces; ent for graph-like domains with generated hooks; gorm when pragmatism dominates and query control is secondary. Justify the abstraction; ORMs hide N+1 and unbounded queries.
• Generated code is committed and never edited by hand (./03-project-layout-tooling.md).

Migrations

• Tools: goose and golang-migrate are the common embeddable choices; Atlas adds declarative schema diffing and linting. Pick one, run it from CI, and never apply migrations by hand in production.
• Name migrations with timestamps and descriptions; one logical change per file; always write the down migration even if you never roll back.
• Make DDL transactional where the engine supports it (PostgreSQL does); avoid mixing DDL and long data backfills in one migration.
• Zero-downtime pattern: expand/contract. Add nullable column/index with CONCURRENTLY → deploy code writing both → backfill in batches → switch reads → drop old column in a later release.
• Backfills: batch by primary key with LIMIT, sleep/throttle, and make them idempotent and restartable.
• Locking: CREATE INDEX CONCURRENTLY avoids long locks; adding a NOT NULL column with a default historically rewrote tables — check current engine behavior (modern PostgreSQL handles constant defaults efficiently; verify the version).
• Apply migrations at deploy time from a single job, not from every application replica at startup, unless you use an advisory lock.
• Migration state must be observable: log version, duration, and failures; alert on pending migrations.

Caching with Valkey/Redis

Valkey is the BSD-licensed Redis fork and the safe default for new infrastructure; both speak the Redis protocol and are compatible with Go clients. go-redis v9 is the mainstream client; rueidis offers higher throughput with pipelining and client-side caching. Verify current maintenance status upstream.

func (s *Store) GetUser(ctx context.Context, id string) (*User, error) {
	key := "user:v3:" + id
	if raw, err := s.rdb.Get(ctx, key).Bytes(); err == nil {
		var u User
		if err := json.Unmarshal(raw, &u); err == nil {
			return &u, nil
		}
	} else if !errors.Is(err, redis.Nil) {
		return nil, fmt.Errorf("cache get: %w", err)
	}

	u, err := s.db.LoadUser(ctx, id)
	if err != nil {
		return nil, err
	}
	b, _ := json.Marshal(u)
	ttl := 10*time.Minute + time.Duration(rand.IntN(60))*time.Second // jitter
	if err := s.rdb.Set(ctx, key, b, ttl).Err(); err != nil {
		slog.WarnContext(ctx, "cache set failed", "err", err) // degrade, don't fail
	}
	return u, nil
}

• Cache-aside is the default; write-through/write-behind only with a strong reason.
• TTL jitter prevents synchronized expiry stampedes. For hot keys, add singleflight so one caller recomputes while others wait (./02-concurrency.md).
• Version keys (user:v3:) so schema changes are a cache flush, not a migration.
• Negative caching needs shorter TTLs and explicit invalidation on create.
• Never cache authorization decisions without an invalidation path; stale permissions are a security bug.
• In-process caches (ristretto, otter, bigcache) remove network hops but add coherence problems; use only for immutable or slowly changing data with clear bounds.
• Serialization: prefer a stable format (JSON or protobuf); version the payload; do not cache Go gob across binary versions.
• Treat the cache as optional: all cache errors degrade to the source of truth. If it is not optional, it is a database.

Connection pools and resource math

Let replicas × maxConnsPerReplica + adminReserve ≤ database max_connections (or PgBouncer max_client_conn). Exceeding it produces connection storms and mysterious 500s.

• Symptoms of pool exhaustion: latency spikes that grow linearly, context deadline exceeded on Begin, DB-side "too many clients".
• Long transactions and LISTEN connections hold slots; size them separately.
• PgBouncer in transaction mode breaks session-level features (prepared statement names, advisory locks, LISTEN); configure clients accordingly (pgx has a special setting and uses SQL-level prepared statements by default).
• Monitor: open/in-use/idle connections, wait duration, acquisition errors.

Testing database code

| Approach | Use when | Trade-off | |---|---|---| | Testcontainers (testcontainers-go) | real engine semantics, CI has Docker | startup cost; cache images | | Template database / snapshot restore | many tests on one instance | isolation discipline required | | Transaction-per-test rollback | tests touching one connection | breaks code that commits or uses multiple connections | | sqlmock | unit-testing query construction only | does not validate SQL; brittleness; often net negative | | Pure functions over rows | parsing/mapping logic | easiest and fastest — extract logic from SQL |

• Run migrations against the container at suite start; never maintain a parallel test schema.
• Use t.Cleanup for container termination and per-test truncation; isolate tests that can run in parallel with schemas or tenant IDs.
• Time-bound every test DB call; a hung test is worse than a failing one.
• Test the failure paths: constraint violations map to domain errors, serialization retries actually retry, context cancellation rolls back.
• See ./06-testing.md for Testcontainers lifecycle patterns and CI caching.

Anti-patterns

• SELECT * in application code; schema changes silently alter scan behavior.
• N+1 queries from looped ORM loads; batch with IN or joins.
• Unbounded WHERE scans; every query that grows with data needs an index and a LIMIT.
• Missing rows.Err() checks; leaked rows keep the connection checked out.
• Transactions wrapping HTTP calls, file I/O, or logging.
• Retrying non-idempotent writes blindly.
• Cache used as primary storage; no TTL; invalidation only in tests.
• Dynamic SQL built by string concatenation.
• Migrations applied by every pod on startup.

Review checklist

• [ ] Every query uses a context with a deadline; no background contexts in request paths.
• [ ] Rows closed, rows.Err() checked, ErrNoRows mapped to a domain error.
• [ ] Transactions short, with defer Rollback and retry-on-serialization where needed.
• [ ] Pool sizes derived from database limits and replica count; monitored.
• [ ] Migrations reviewed for locks and rollback; expand/contract for breaking changes.
• [ ] Cache keys versioned, TTLs jittered, failures degrade, invalidation defined.
• [ ] DB tests use the real engine (Testcontainers) for semantics; mocks limited to query-shape tests.
• [ ] No SQL string interpolation of values; identifiers allowlisted.

Testing

Scope: table-driven tests, helpers and cleanup, fuzzing, benchmarks, golden files, HTTP testing, Testcontainers, deterministic concurrency tests, and flake elimination.

Table-driven tests

The default shape for any function with more than two cases.

func TestNormalize(t *testing.T) {
	t.Parallel()

	tests := []struct {
		name    string
		in      string
		want    string
		wantErr error
	}{
		{name: "empty", in: "", want: ""},
		{name: "trims", in: "  a  ", want: "a"},
		{name: "rejects control", in: "a\x00", wantErr: ErrInvalid},
	}

	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			t.Parallel()
			got, err := Normalize(tt.in)
			if !errors.Is(err, tt.wantErr) {
				t.Fatalf("err = %v, want %v", err, tt.wantErr)
			}
			if got != tt.want {
				t.Errorf("got %q, want %q", got, tt.want)
			}
		})
	}
}

• t.Run gives isolation, filtering (-run 'TestNormalize/trims'), and parallel subtests. Loop variables are per-iteration since 1.22; older language versions need tt := tt.
• t.Fatal stops the current subtest; use it for setup failures, t.Errorf for accumulating assertion failures.
• Table fields should be explicit (name, in, want, wantErr); avoid positional anonymous structs that become unreadable.
• Prefer errors.Is/errors.As over string comparison. For deep structures, go-cmp:

if diff := cmp.Diff(want, got); diff != "" {
	t.Errorf("mismatch (-want +got):\n%s", diff)
}

• cmp.Diff handles unexported fields via cmpopts.IgnoreUnexported; never add Equal methods just for tests.
• Fakes over mocks by default: implement the small consumer interface by hand. When you need a mock framework, testify/mock + generated mocks (mockery) or go.uber.org/mock are the mainstream options; keep expectations behavioral, not call-count theater.

Helpers, cleanup, fixtures

• Mark helpers with t.Helper() so failures point at the caller.
• t.Cleanup(fn) replaces hand-rolled defers in helpers; it runs LIFO even when the test fails.
• t.TempDir() provides auto-removed unique directories; t.Setenv sets env vars and restores them (incompatible with t.Parallel).
• t.Context() (1.24+) is canceled when the test ends — use it as the parent for code under test and Testcontainers.
• Share read-only fixtures via sync.OnceValues at package level; never share mutable state between parallel tests.
• Test data lives in testdata/; the toolchain ignores that directory when building.

Testify

• require.* fails fast (FailNow) — use for preconditions. assert.* records and continues — use when you want all failures in one run.
• Avoid testify's suite unless the lifecycle genuinely helps; t.Run plus helpers is simpler and plays better with t.Parallel.
• testify makes error messages concise, but the standard library's if got != want needs no dependency; keep assertion choices consistent within a repo.

Golden files

For large outputs (rendered templates, codecs, CLI output), compare against checked-in golden files instead of inline strings.

var update = flag.Bool("update", false, "update golden files")

func TestRender(t *testing.T) {
	got, err := Render(input)
	if err != nil {
		t.Fatal(err)
	}
	golden := filepath.Join("testdata", "render.golden")
	if *update {
		if err := os.WriteFile(golden, got, 0o644); err != nil {
			t.Fatal(err)
		}
	}
	want, err := os.ReadFile(golden)
	if err != nil {
		t.Fatal(err)
	}
	if diff := cmp.Diff(string(want), string(got)); diff != "" {
		t.Errorf("render mismatch (-want +got):\n%s", diff)
	}
}

• Review golden diffs in PRs; a blind -update that hides a regression defeats the purpose.
• testscript/txtar is the standard-library-friendly format for scripted CLI/end-to-end tests with files.
• Normalize nondeterminism (timestamps, map order, absolute paths) before comparison.

httptest

func TestCreateOrderHandler(t *testing.T) {
	srv := httptest.NewServer(newRouter())
	defer srv.Close()

	resp, err := srv.Client().Post(srv.URL+"/v1/orders", "application/json",
		strings.NewReader(`{"name":"a"}`))
	if err != nil {
		t.Fatal(err)
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusCreated {
		t.Fatalf("status = %d", resp.StatusCode)
	}
}

• httptest.NewRecorder tests a single handler without a network; NewServer exercises middleware, routing, and real HTTP semantics.
• NewServerTLS for TLS paths; srv.Client() is preconfigured with the test certificate.
• Assert on status, headers, and decoded bodies; test auth failures and malformed JSON, not just the happy path.
• Close every response body; leaked bodies break keep-alive and hide resource bugs.

Fuzzing

func FuzzParse(f *testing.F) {
	f.Add("name=value")
	f.Add("")
	f.Fuzz(func(t *testing.T, input string) {
		v, err := Parse(input)
		if err == nil && v == nil {
			t.Fatal("nil value without error")
		}
	})
}

• go test -fuzz=FuzzParse -fuzztime=30s ./pkg explores; discovered crashers are minimized into testdata/fuzz/ and become regression seeds.
• Seed with realistic and adversarial inputs. Add invariants (round-trip, no panic, bounded output), not just "does not crash".
• Fuzz parsers, validators, decoders, and anything consuming untrusted bytes. Keep targets fast and allocation-light.
• CI runs seed corpus (go test ./...) on every change and a short -fuzztime nightly; long campaigns run on dedicated machines. Verify current corpus-cache behavior upstream.

Benchmarks

func BenchmarkEncode(b *testing.B) {
	data := makePayload()
	b.ReportAllocs()
	for b.Loop() { // 1.24+
		if _, err := Encode(data); err != nil {
			b.Fatal(err)
		}
	}
}

func BenchmarkEncodeSize(b *testing.B) {
	for _, n := range []int{100, 1000, 10000} {
		b.Run(fmt.Sprintf("n=%d", n), func(b *testing.B) {
			data := makePayloadSized(n)
			b.ReportAllocs()
			for b.Loop() {
				_, _ = Encode(data)
			}
		})
	}
}

• b.Loop (1.24+) keeps the loop body live, resets the timer once, and prevents dead-code elimination. Before 1.24 use for i := 0; i < b.N; i++ with a package-level sink to defeat elimination.
• b.ReportAllocs() plus -benchmem exposes allocations per op.
• Use benchstat over -count=10 runs for comparisons; a single run is noise. Keep the machine idle and pinned, and cache binaries/inputs outside the timed loop.
• Benchmark one function per scenario; keep setup in b.StopTimer/b.StartTimer or testing.B-level setup.
• Profile while benchmarking: -cpuprofile/-memprofile (./07-performance-profiling.md).

Deterministic concurrency and leak tests

• testing/synctest (stable 1.25+): run concurrent code in a bubble with a fake clock; synctest.Wait() blocks until every goroutine is durably blocked, making timeouts and retries deterministic. Verify the exact API shape upstream.
• go.uber.org/goleak: goleak.VerifyTestMain(m) in TestMain or defer goleak.VerifyNone(t)() catches leaked goroutines after tests.
• Replace sleeps with channel handshakes or synctest; time.Sleep in tests is a flake generator.
• Run go test -race -count=100 for synchronization-heavy packages; -shuffle=on randomizes test order to expose hidden dependencies.
• See ./02-concurrency.md for the concurrency primitives under test.

Integration tests with Testcontainers

• testcontainers-go starts real dependencies (PostgreSQL, Valkey/Redis, Kafka, MinIO) from tests.
• Lifecycle: testcontainers.GenericContainer or module helpers; pass t.Context() to Run so cleanup is automatic; use t.Cleanup for termination.
• Wait strategies (wait.ForLog, wait.ForListeningPort, wait.ForSQL) instead of sleeps; module helpers provide sensible defaults.
• Reuse one container per package with sync.OnceValues and truncate state between tests; reuse across packages is opt-in (TESTCONTAINERS_RYUK_DISABLED/reuse settings — verify upstream).
• Tag integration tests with //go:build integration and run them in a dedicated CI job so go test ./... stays fast; see ./03-project-layout-tooling.md.
• Database-specific patterns: migrations at suite start, template databases or per-test transactions — ./05-data.md.

Organization, coverage, flakiness

| Concern | Practice | |---|---| | Package boundary | external _test package for public API tests; internal for white-box | | Build variants | integration/e2e tags with matching CI jobs | | Coverage | -coverpkg=./... for cross-package impact; go tool cover -html; never a target in itself | | Caching | go test caches success; use -count=1 when you suspect stale results | | Parallelism | t.Parallel + -parallel=N; ensure port/tempfile isolation | | Ordering | -shuffle=on in CI to catch inter-test coupling |

Flake causes and fixes:

• Time: fake clocks (synctest) or injected Clock interfaces.
• Ports: :0 and read the assigned port; httptest does this.
• Shared state: per-test fixtures, unique DB schemas, no package-level mutable globals.
• Network: Testcontainers or fakes; never hit production in tests.
• Sleeps and retries: replace with eventual assertions (assert.Eventually with short interval and deadline) used sparingly and only for external systems.
• Botched cleanup: t.Cleanup over defer in helpers.

Anti-patterns

• One giant TestEverything function with shared state.
• Asserting implementation details (call counts, private fields) instead of behavior.
• Testing obvious getters and generated code; no tests where a regression actually hurts.
• time.Sleep(100 * time.Millisecond) as synchronization.
• Mocking what you own instead of designing for testability; deep mock chains.
• Golden files regenerated on every failing run.
• Skipping -race because "it's slow"; marking flaky tests as skipped forever.
• Coverage theater: 100% coverage with no assertions.

Review checklist

• [ ] Table-driven where cases multiply; subtests named and parallel-safe.
• [ ] Assertions use errors.Is/cmp.Diff; failure messages include got/want and inputs.
• [ ] Every test cleans up via t.Cleanup; no leaked goroutines or files.
• [ ] Fuzz targets exist for parsers/decoders; corpus committed.
• [ ] Benchmarks use b.Loop (or b.N + sink), report allocs, and are compared with benchstat.
• [ ] Integration tests tagged and run in CI with real dependencies.
• [ ] No sleeps; concurrency tested with synctest/goleak where applicable.
• [ ] -race, -shuffle=on and repeated -count runs are part of CI.

Performance and Profiling

Scope: measuring before optimizing, pprof profiles, execution traces, escape analysis, allocation reduction, GC tuning, sync.Pool, and benchmark discipline for Go 1.22-1.26.

Method

1. Define the metric — latency percentiles, throughput, allocations per op, RSS, or GC pause. "Slow" is not a metric.
2. Reproduce with a benchmark, load test, or a profile from production. Never optimize from code reading alone.
3. Profile to find where time/memory actually goes.
4. Change one thing and re-measure with the same harness and benchstat.
5. Guard the regression with a benchmark or budget test in CI.

Most production latency is waiting (I/O, locks, queueing), not CPU. Start with traces/logs for latency and CPU profiles for throughput.

pprof

Enable in servers via net/http/pprof (import for side effects) on an internal-only listener or admin mux. Never expose it publicly: profiles leak memory contents and function arguments, and CPU profiling is a DoS vector.

admin := http.NewServeMux()
admin.HandleFunc("/debug/pprof/", pprof.Index)
admin.HandleFunc("/debug/pprof/profile", pprof.Profile)
admin.HandleFunc("/debug/pprof/trace", pprof.Trace)
go http.ListenAndServe("127.0.0.1:6060", admin)

| Profile | Collect | Shows | Use for | |---|---|---|---| | CPU | /debug/pprof/profile?seconds=30 | sampled stack time at 100 Hz | hot functions, cycles | | Heap | /debug/pprof/heap | sampled allocations (alloc_space/alloc_objects) and live (inuse_space) | allocation sources vs retained memory | | Goroutine | /debug/pprof/goroutine?debug=2 | all goroutine stacks | leaks, blocked workers | | Block | runtime.SetBlockProfileRate | time blocked on sync primitives | channel/mutex contention | | Mutex | runtime.SetMutexProfileFraction | lock contention | hot locks | | Threadcreate | /debug/pprof/threadcreate | OS thread creation | cgo/syscall storms |

Block and mutex profiles are off by default because they add overhead; enable with rates (for example SetMutexProfileFraction(100) samples 1/100 events) in staging, not permanently in production.

Interpreting:

go tool pprof -http=:8081 http://localhost:6060/debug/pprof/profile?seconds=30
go tool pprof -top -cum ./bin/app cpu.prof
go tool pprof -list=MyFunc ./bin/app cpu.prof
go tool pprof -diff_base=before.prof after.prof

• -cum shows cumulative cost including callees; -flat shows self time. Read both.
• -list=Func maps samples to source lines; the answer is usually a specific line, not an architecture.
• Compare profiles with -diff_base to prove a change helped.
• For tests: go test -bench=. -cpuprofile=cpu.prof -memprofile=mem.prof, then go tool pprof -http=:8080 cpu.prof.
• For CLIs: runtime/pprof StartCPUProfile/WriteHeapProfile behind a flag.
• Continuous profiling (Pyroscope, Grafana, cloud profilers) is the production-grade alternative; it samples many instances over time and shows trends. Overhead is usually low single-digit percent; verify per vendor.

Execution trace

Use runtime/trace when the question is about scheduling, blocking, or latency, not CPU cost.

f, _ := os.Create("trace.out")
defer f.Close()
if err := trace.Start(f); err != nil {
	return err
}
defer trace.Stop()

ctx, task := trace.NewTask(ctx, "request")
defer task.End()
trace.WithRegion(ctx, "db-query", func() { ... })

• go tool trace trace.out opens the web UI: goroutine states, syscalls, GC, network waits, scheduler latency.
• Tasks/regions give semantic context in the trace; without them you see addresses of functions, not user journeys.
• The flight recorder (1.25+) keeps a rolling window and writes it on trigger, which is the right tool for rare latency spikes; verify the current API upstream.
• Trace overhead is higher than pprof; record short windows (seconds), not hours.

Escape analysis and allocation discipline

go build -gcflags='-m=2' ./... 2>&1 | grep -E 'escapes to heap|moved to heap'

Common escape causes:

• Returning a pointer to a local: legal and usually necessary, but it allocates; returning a value can keep it on the stack for small types.
• Interface boxing: passing a concrete value to any (fmt, errors, slog args) allocates unless the value fits an internal cache.
• Closures capturing variables that outlive the function; goroutines always capture.
• Sending a pointer through a channel or storing it in a slice of interfaces.
• defer with a closure over loop state (open-coded defers avoid the allocation for simple cases).
• Converting between []byte and string around I/O; use unsafe only with a documented, measured reason.

High-yield allocation fixes:

| Fix | Gain | |---|---| | strconv instead of fmt.Sprintf/Sprintf in hot paths | removes boxing + parsing | | strings.Builder with Grow | linear instead of quadratic concat | | Presize make([]T, 0, n) and make(map[K]V, n) | removes growth copies/rehashes | | slices.Grow/bytes.Buffer.Grow | predictable buffers | | io.CopyBuffer with a pooled buffer | avoids default 32 KiB alloc per call | | Return values instead of pointers for small structs | fewer heap objects | | Avoid []any/map[string]any for row data | fewer allocations, better type safety | | Struct field ordering largest→smallest | less padding (measure first) |

Inlining: small functions (roughly under the compiler's cost budget) inline automatically; -gcflags=-m reports decisions. Mid-stack inlining has widened what inlines. For hot paths, avoid making wrapper functions too large to inline, but do not contort code for it.

Profile-Guided Optimization (PGO, 1.21+): commit default.pgo (from production samples) and the build applies it; typical gains are modest (a few percent) and workload-dependent. Verify current tooling support upstream.

GC tuning

Go's GC is a concurrent mark-sweep with a pacing controller. Two knobs matter:

• GOGC (default 100) — target heap growth between collections. Lower = less memory, more CPU; higher = the reverse.
• GOMEMLIMIT (1.19+) — soft memory limit; the GC works harder to respect it. This is the correct response to container memory limits.

debug.SetMemoryLimit(512 << 20)
debug.SetGCPercent(200) // when CPU-bound and memory is abundant

• GOMEMLIMIT is soft: it cannot stop allocation, so a leak still OOMs. Leave headroom above it for non-heap memory (stacks, OS buffers, cgo).
• Setting GOMEMLIMIT slightly below the container limit lets the runtime throttle instead of the kernel killing the process.
• The old "ballast" trick is obsolete; GOMEMLIMIT supersedes it.
• Container awareness: recent toolchains derive GOMAXPROCS from cgroup CPU limits (1.25+, verify), reducing scheduler oversubscription. If you pin CPU via GOMAXPROCS, do not exceed the cgroup quota.
• Diagnose GC problems with GODEBUG=gctrace=1 (pause times, heap growth) and heap profiles, then adjust.
• runtime.ReadMemStats/runtime/metrics for programmatic telemetry; export key GC metrics via Prometheus (./08-observability-security.md).

sync.Pool

Use for short-lived temporary buffers where allocation is a measured hotspot; not as a general cache.

var bufPool = sync.Pool{
	New: func() any { b := make([]byte, 0, 32<<10); return &b },
}

func handle(w io.Writer, data []byte) {
	bp := bufPool.Get().(*[]byte)
	defer bufPool.Put(bp)
	b := (*bp)[:0]
	b = append(b, data...)
	w.Write(b)
}

• Pooled values may be dropped at any GC; never rely on state or count.
• Reset before use and before returning; do not return huge buffers, or the pool becomes a memory retention bug (cap them).
• sync.Pool is safe for concurrent use and cheap; bytes.Buffer values can be pooled directly.
• Avoid pooling small structs; the pool overhead exceeds the allocation savings. Measure.
• Go 1.24+ runtime has improved small-object allocation paths (for example, small non-pointer allocations); still measure on your workload, not from release notes.

Latency and throughput work

• Track percentiles (p50/p95/p99) via histograms, not averages.
• Queueing is the usual culprit: bounded pools, connection limits, and single-threaded locks create wait time that CPU profiles cannot see. Use the block/mutex profiles and execution traces.
• Batching and pipelining beat micro-optimizations for I/O-bound code: fewer round trips, fewer syscalls (bufio, bulk writes).
• Avoid N+1 queries and per-request client construction (./04-web-services.md, ./05-data.md).
• Beware coordinated omission when load testing: a closed-loop generator hides stalls; use open-loop arrival with long enough duration.
• Watch for thundering herds after deploy: jitter TTLs, backoff, and warm caches.

Benchmark hygiene

• Compare only identical workloads: same CPU governor, same machine class, same build flags; avoid running other heavy processes.
• Use -count=10 and benchstat old.txt new.txt for decisions; single runs are anecdotes.
• Benchmark realistic data sizes and include setup outside the timed region (b.ResetTimer or b.Loop).
• Prevent dead-code elimination with b.Loop (1.24+) or a package-level sink; b.ReportAllocs for allocations.
• Profile the benchmark (-cpuprofile, -memprofile) before optimizing; see ./06-testing.md.
• Never optimize from an unrepresentative microbenchmark alone; validate with production-like load.

Anti-patterns

• Optimizing without a profile; "this looks slow".
• Adding caches to fix CPU instead of fixing algorithmic complexity.
• Growing goroutine/connection pools without measuring downstream saturation.
• Tuning GOGC/GOMEMLIMIT before fixing leaks and allocations.
• Using sync.Pool to hide unbounded allocation rates.
• Ignoring p99 because p50 looks fine; ignoring GC because CPU looks fine.
• Public pprof endpoints; profiling with seconds=300 in production loops.
• Microbenchmarks with constant inputs the compiler can fold.

Review checklist

• [ ] A metric and a reproduction exist before optimization claims.
• [ ] CPU/heap/block/mutex profiles (or traces) captured and compared, not guessed.
• [ ] Allocations per request/op measured; hot paths avoid fmt and repeated conversions.
• [ ] Heap retained (inuse_space) vs allocated (alloc_space) distinguished when chasing memory.
• [ ] GOMEMLIMIT set below container limit; GOGC justified; GC telemetry exported.
• [ ] sync.Pool used only with measured benefit and reset/cap discipline.
• [ ] Benchmarks reproducible, alloc-reporting, and regression-checked in CI.
• [ ] Latency measured with percentiles and load realistic enough to expose queueing.

Observability and Security

Scope: structured logging with slog, OpenTelemetry traces/metrics/logs, Prometheus instrumentation, secret handling, applied crypto, input validation at trust boundaries, and supply-chain controls.

Structured logging with slog

log/slog (1.21+) is the standard. Configure one handler at startup and pass *slog.Logger explicitly; do not use the global default in library code.

handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
	Level: slog.LevelInfo,
})
logger := slog.New(handler)
slog.SetDefault(logger)

logger.InfoContext(ctx, "order created",
	slog.String("order_id", id),
	slog.Int("items", n),
	slog.Duration("latency", d),
)

• Levels: Debug for diagnostics, Info for business events, Warn for degraded-but-working, Error for failed operations needing action. Avoid Fatal (it calls os.Exit); let main decide.
• Use LogAttrs/typed attrs on hot paths to avoid any boxing; Info("msg", "k", v) is fine for cold paths.
• Groups (logger.WithGroup("http")) structure nested payloads; logger.With(...) creates request-scoped child loggers.
• Dynamic levels: slog.LevelVar lets a config endpoint or SIGHUP change verbosity at runtime (handler.Level behind LevelVar).
• Request correlation: put a request ID (and trace ID once OTel is wired) in context and attach it in a handler wrapper so every line carries it. Extract with a helper that tolerates a missing value.
• Redaction: secrets, tokens, passwords, PII must never reach logs. Add a ReplaceAttr hook that drops/renames sensitive keys, and never log raw request bodies.
• Log destinations are append-only streams: no multi-line stack traces in JSON fields without escaping; empty error values (err=nil) are noise.
• slog bridges: slog.SetDefault propagates to log; OTel's otelslog bridge sends records to the collector if you centralize there.

OpenTelemetry

OTel is the vendor-neutral standard; the Go SDK plus OTLP exporter is the default pipeline.

res, _ := resource.Merge(resource.Default(),
	resource.NewWithAttributes(semconv.SchemaURL,
		semconv.ServiceName("orders"),
		semconv.DeploymentEnvironmentName("prod"),
	))

tp := trace.NewTracerProvider(
	trace.WithBatcher(otlptracehttp.NewClient()),
	trace.WithResource(res),
	trace.WithSampler(trace.ParentBased(trace.TraceIDRatioBased(0.05))),
)
defer tp.Shutdown(ctx)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
	propagation.TraceContext{}, propagation.Baggage{},
))

• Traces: create spans with meaningful names (GET /orders/{id}, db.query orders) and attributes (semantic conventions); avoid %v user data in span names — it explodes cardinality.
• Propagation: W3C traceparent/baggage for HTTP; make sure every outbound client and inbound server is instrumented, or the trace breaks at service boundaries.
• Instrumentation: otelhttp for http.Handler/http.Client, otelgrpc interceptors, otelsql/pgx tracing, and manual spans around domain operations.
• Sampling: head-based (parent-based ratio) is cheap; tail-based sampling belongs in the collector when you need error/latency-biased retention. Never sample away all errors in a low-traffic service.
• Metrics: OTel metrics API for portable signals; export OTLP to the collector. Keep label sets bounded.
• Collector is the right aggregation point: application sends OTLP to a local sidecar/daemonset; the collector handles batching, retries, sampling, and vendor routing.
• Shutdown must flush: Shutdown with a timeout during graceful stop (./04-web-services.md), or you lose the last spans.

Prometheus

prometheus/client_golang remains the ecosystem standard for Go metrics; many teams run both OTel (traces/logs) and Prometheus (metrics) with a collector bridge.

var (
	requests = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "http_requests_total",
		Help: "HTTP requests processed.",
	}, []string{"method", "route", "status"})

	latency = prometheus.NewHistogramVec(prometheus.HistogramOpts{
		Name:    "http_request_duration_seconds",
		Buckets: prometheus.DefBuckets,
	}, []string{"method", "route"})
)

func init() {
	prometheus.MustRegister(requests, latency)
}

| Type | Use for | Watch out | |---|---|---| | Counter | monotonically increasing events | never reset; use _total suffix | | Gauge | current value (connections, queue depth) | can go up and down; don't fake a counter | | Histogram | latency/size distributions | bucket choice drives accuracy; _bucket/_sum/_count | | Summary | quantiles without aggregation across instances | client-side quantiles are not aggregatable — prefer histograms | | Native histograms | high-resolution, low cardinality | support varies; verify upstream |

• Cardinality is the number one operational risk: label values come from a bounded set (route template, status class, not raw URL or user ID). Never label with unbounded IDs.
• Use the default registry only if you control the process; custom registries are cleaner for libraries. promhttp.Handler() mounts /metrics.
• Export the standard runtime collectors (goroutine count, GC, memory) plus process_*; these catch leaks and GC issues.
• Avoid the Pushgateway except for batch jobs; it breaks counter semantics and stale-marking.
• Instrument at the boundary (middleware/interceptor) and at dependency edges (DB, cache, queue) with the same label vocabulary.
• See ./07-performance-profiling.md for GC/alloc metrics that belong on a dashboard.

Correlation

• One ID to bind them all: accept traceparent (W3C) and a human-readable request ID; log both; put the trace ID in error responses for support.
• Logs at boundaries: inbound request (after auth, never the body), outbound call with target and latency, state transitions, and errors with operation + wrapped context.
• Do not double-log: middleware logs completion; handlers log business events; no fmt.Println anywhere.
• Metrics answer "how much/how many", traces answer "where", logs answer "what exactly". Do not try to get traces from logs or vice versa.

Secrets

• Source: environment injected by the platform secret manager, or a secrets API (Vault, cloud KMS/Secrets Manager). Files are acceptable when mounted read-only and permissioned (0400).
• Never in: source, go.mod/sum, Docker images, build args, CI logs, committed .env, test fixtures, or error messages.
• Read once at startup into a dedicated config type; pass explicitly; do not read env vars at request time.
• Rotation must not require code changes: support two active credentials (current + previous) during rollout, and reload on signal where the secret manager allows.
• Redaction: central slog ReplaceAttr; scrub URLs (query params carry tokens), headers, and stack traces before external reporting.
• Scanning: gitleaks/trufflehog in pre-commit and CI; govulncheck for code; secret scanning of container layers in CI.
• Least privilege: one credential per service and environment; database users scoped to the schema and operations actually used.

Crypto in Go

• Randomness: crypto/rand for anything security-related. math/rand (and math/rand/v2) is not cryptographic; using it for tokens is a vulnerability.
• Symmetric encryption: AES-GCM (crypto/aes + crypto/cipher) or ChaCha20-Poly1305 (golang.org/x/crypto/chacha20poly1305). Always unique nonces; never reuse a nonce with the same key. Prefer an AEAD with associated data for context binding.
• Signatures: Ed25519 for new designs (crypto/ed25519), ECDSA P-256 for ecosystem compatibility, RSA only where required. Verify with crypto/x509.
• Password storage: Argon2id (golang.org/x/crypto/argon2), bcrypt, or scrypt with tuned cost; store parameters and salt with the hash; migrate on successful login. Never hash with MD5/SHA1/SHA256 alone.
• HMAC: crypto/hmac with sha256/sha512; compare with hmac.Equal or crypto/subtle.ConstantTimeCompare.
• Deprecated/banned: MD5, SHA1 for signatures, DES/3DES, RC4, ECB mode, custom crypto, crypto/cipher stream ciphers without authentication.
• TLS: MinVersion: tls.VersionTLS12 (prefer 1.3), proper certificate verification, RootCAs from system pools, client certificates for mTLS. Never InsecureSkipVerify: true outside local tests.
• JWT: validate signature and alg (reject none/algorithm confusion), iss, aud, exp/nbf with small leeway. Use a maintained library (github.com/golang-jwt/jwt/v5) and pin the expected algorithm explicitly.
• Key management: keys from KMS/secret manager, per-environment, versioned; support rotation; encrypt-then-sign only for specific protocols that require it.

Input validation at trust boundaries

Every byte from a network, file, or queue is untrusted. Validate once at the boundary, convert to domain types, then trust internally.

| Threat | Defense | |---|---| | SQL injection | parameterized queries only; identifier allowlists (./05-data.md) | | Command injection | exec.CommandContext with argument slices; never a shell; allowlist binaries | | Path traversal | filepath.Clean + prefix check, or os.Root (1.24+) for constrained file access | | SSRF | allowlist schemes/hosts/ports; block link-local/metadata IPs; disable redirects or re-validate each hop | | XXE / XML bombs | encoding/xml with Decoder settings; prefer JSON; cap entity/decode sizes | | Decompression bombs | io.LimitReader around gzip/zlib with a hard cap | | Template injection | html/template for HTML output; never text/template with user data into HTML | | JSON depth/size abuse | MaxBytesReader, decode depth/size limits, streaming decode | | HTML/JS injection | contextual escaping (html/template), strict CSP, no template.HTML on user input | | Log injection | structured logging; escape newlines; never concatenate raw input into log messages | | Deserialization | no gob/encoding/gob across trust boundaries; validate protobuf/JSON explicitly | | File uploads | size caps, content sniffing, store outside webroot, random names, virus scan where applicable | | Race-based TOCTOU | operate on handles, not re-resolved paths; os.Root mitigates | | Unbounded work | limits on regex complexity, iteration counts, page sizes, and request fan-out |

Validation rules:

• Validate type, length, range, format, and cross-field consistency; reject rather than sanitize silently.
• Use explicit allowlists. Do not rely on blocklists or "escape everything" as a strategy.
• Reject unknown fields on internal APIs; bounds-check integers before conversion (overflow is silently wrapping).
• Return 4xx for client errors without echoing internal state; log details server-side with the request ID.
• Encode output for its destination: JSON encoder for JSON, html/template for HTML, parameterized statements for SQL.

Supply chain

• govulncheck ./... in CI and before release — it reports only vulnerabilities reachable from your code.
• Keep modules on the checksum database (GOSUMDB=sum.golang.org default). GOFLAGS=-mod=readonly; verify with go mod verify.
• Review dependency additions: go mod why, license, maintenance status, release cadence, and transitive weight. Prefer golang.org/x/* and stdlib.
• Generate an SBOM (syft) per release and attach it; sign artifacts and images with cosign; publish checksums.
• Renovate/Dependabot for version updates with go test gates; group stdlib-adjacent updates.
• CI hardening: pin action versions by SHA, least-privilege tokens, no long-lived cloud credentials (OIDC), protect the module proxy from cache poisoning by using the public checksum DB.
• Do not GONOSUMDB=* or GOFLAGS=-insecure to "make it work"; fix the credentials.

Hardening checklist

• [ ] pprof/debug endpoints bound to localhost/admin port; not exposed by ingress.
• [ ] Server timeouts and body/header limits set (./04-web-services.md).
• [ ] Structured logs with request/trace correlation; redaction hook active; no secrets or bodies logged.
• [ ] OTel provider configured with resource attributes, propagation, flushing on shutdown.
• [ ] Prometheus metrics bounded in cardinality; runtime/GC collectors exported.
• [ ] Secrets injected, rotated, and never committed; scanners in CI.
• [ ] crypto/rand for tokens; AEADs with unique nonces; quotas on password hashing cost.
• [ ] All queries parameterized; paths/commands/URLs validated at the boundary.
• [ ] govulncheck, SBOM, signed artifacts, pinned CI actions and tool versions.
• [ ] Error responses generic; panics recovered and logged with context; no stack traces to clients.

Ecosystem 2026

Scope: the state of the Go toolchain and library ecosystem in the 1.22-1.26 window, migration guidance, and the senior review checklist. Version-specific claims are floors/ranges — verify current point releases upstream.

Toolchain timeline

Go ships two releases per year (February/August) and supports the two most recent. A module's go directive is now a hard minimum, so the practical floor for most services is the oldest release still receiving security fixes.

| Release | Headline changes for production Go | |---|---| | 1.21 | log/slog, min/max/clear, slices/maps/cmp, sync.OnceFunc, toolchain directive, WASI preview 1 | | 1.22 | per-iteration loop vars, net/http method+wildcard routing, math/rand/v2, range-over-int, PGO default support continued | | 1.23 | range-over-func and iter.Seq/Seq2, unique, structs, weak, timer changes (unstopped timers collectable), net/netip additions | | 1.24 | generic type aliases, tool directives (go get -tool), os.Root, testing.T.Context/B.Loop, weak references, crypto/mlkem (verify), improved allocation paths | | 1.25 | testing/synctest stable, http.CrossOriginProtection, container-aware GOMAXPROCS, flight recorder in runtime/trace, encoding/json/v2 experiment, slog additions | | 1.26 | GC/toolchain evolution, go fix modernizers, leak/telemetry improvements — verify upstream |

Ecosystem consequences:

• math/rand is auto-seeded (1.20+); global rand.Seed is a no-op. Use math/rand/v2 for non-crypto randomness and crypto/rand for anything security-related.
• io/ioutil has been gone since 1.16; os/io equivalents are the only acceptable imports.
• encoding/json/v2 is still experimental in the 1.25/1.26 window under GOEXPERIMENT=jsonv2. Build on v1 APIs until the v2 API is declared stable; keep JSON handling behind small functions to ease the eventual switch (./04-web-services.md).
• GC direction is lower-latency collection for small-heap services and better container awareness; GOMEMLIMIT remains the operator-facing control (./07-performance-profiling.md).
• go fix modernizers grow each release; run it on major upgrades and review the diff — it is also an excellent teaching tool for current idioms (./03-project-layout-tooling.md).

Standard library direction

• Iterators everywhere: slices, maps, strings, bytes producer helpers make lazy pipelines first-class. Prefer them to ad-hoc slices when the consumer may stop early (./01-language-core.md).
• Testing: synctest stabilized deterministic concurrency tests; B.Loop simplified benchmarks; T.Context standardized cancellation. Expect more in testing rather than third-party frameworks (./06-testing.md).
• HTTP: stdlib routing now covers method and path parameters; CrossOriginProtection addresses CSRF for cookie sessions. The stdlib is a serious default again (./04-web-services.md).
• Security: os.Root removes whole classes of path traversal; ML-KEM and crypto updates follow the post-quantum migration; verify exact package availability per release (./08-observability-security.md).
• Runtime: container-aware CPU limits reduce the classic "GOMAXPROCS = host cores" pathology; unstopped timers no longer leak; execution tracing gained a flight recorder for rare events.
• Compatibility: GODEBUG settings encode per-release behavior changes; read them during upgrades. Never set broad GODEBUG overrides as a permanent workaround.

Library landscape

Web and RPC:

| Purpose | Mainstream choice | Alternatives / notes | |---|---|---| | Router/middleware | chi (v5) | stdlib ServeMux, echo v4, gin v1 | | RPC | grpc-go | connectrpc.com/connect (HTTP-friendly), Twirp | | GraphQL | gqlgen | hand-rolled only for tiny surfaces | | WebSocket | github.com/coder/websocket | gorilla/websocket (maintained again) | | SSE | stdlib | no dependency needed | | Validation | go-playground/validator v10 | hand-written validators often clearer | | Sessions/auth | golang.org/x/oauth2, OIDC libs, golang-jwt/jwt/v5 | do not hand-roll JWT validation |

Data and messaging:

| Purpose | Mainstream choice | Notes | |---|---|---| | Postgres driver | pgx v5 | lib/pq is maintenance-mode legacy | | Query generation | sqlc | ent for graph-heavy domains, gorm for pragmatism | | Migrations | goose, golang-migrate | Atlas adds declarative diff/lint | | Cache/client | go-redis v9, rueidis | Valkey is the license-safe backend | | Kafka | franz-go | segmentio/kafka-go also current | | NATS | nats.go | built-in JetStream support | | Object storage | cloud SDKs (AWS v2, GCS) | minio-go for S3-compatible |

Infrastructure and tooling:

| Purpose | Mainstream choice | Notes | |---|---|---| | Logging | log/slog | zap/zerolog for legacy perf-critical paths | | Traces/metrics | OpenTelemetry SDK, client_golang | collector pipeline | | HTTP client | stdlib + small retry helper | go-retryablehttp, resty for convenience | | Retry/backoff | cenkalti/backoff, avast/retry-go | hand-rolled jitter also fine | | CLI | cobra, urfave/cli, kong | stdlib flag for small tools | | Config | env parsing + koanf | Viper is heavy; 12-factor env first | | DI | manual constructors | wire (codegen) or fx for large apps | | Testing | testify, go-cmp, testcontainers-go, goleak | prefer stdlib where sufficient | | Mocks | go.uber.org/mock, mockery | hand-written fakes often best | | Concurrency | golang.org/x/sync | semaphore, errgroup, singleflight | | Errors | stdlib (%w, errors.Join) | pkg/errors deprecated |

Selection rules: prefer stdlib and golang.org/x/*; one library per concern per repo; check maintenance activity (commits within the last year, responsive issues, tagged releases); read the license; count transitive dependencies before adding.

Migration notes

• From pkg/errors: replace Wrap/Wrapf with fmt.Errorf("%w"), Cause with errors.Is/As, MultiError with errors.Join.
• From lib/pq: move to pgx v5 (native or stdlib adapter). Watch $1 placeholder compatibility (both support it) and pq type specifics.
• From nhooyr.io/websocket: the project moved to github.com/coder/websocket; import path is the only change.
• From dep/vendor-only: modules with go.mod; vendor remains supported but is a deliberate policy.
• Loop variables: code with a go directive below 1.22 keeps the old capture semantics. Upgrading the directive silently changes behavior — audit goroutine closures and &loopVar uses.
• math/rand: replace rand.Seed calls (no-ops) and migrate hot paths to math/rand/v2; never use either for tokens.
• GODEBUG: each toolchain documents behavior changes; after upgrades, test with defaults first, set overrides only with an issue tracking removal.
• Build tags: // +build is gone; use //go:build.
• io/ioutil, bytes.Title, strings.Title, and old net/... helpers are deprecated or removed; go fix and staticcheck flag them.
• Containers: rebuild images with the current toolchain; verify GOMAXPROCS behavior under CPU quotas and set GOMEMLIMIT for memory limits (./07-performance-profiling.md).

Upgrade checklist

• [ ] Read the release notes and GODEBUG history for every skipped version.
• [ ] Bump the go directive in a dedicated PR; run go fix/go vet/staticcheck and review.
• [ ] Run go test -race ./..., integration tests, and a load test before rollout.
• [ ] Rebuild all Docker images and CI tool pins with the new toolchain.
• [ ] Re-measure p99 latency, allocations, and GC after the upgrade; publish a profile diff.
• [ ] Revisit tool/tools.go versions and regenerate code.
• [ ] Re-run govulncheck and update the SBOM.

Senior review checklist

Language and API:

• [ ] Errors wrapped with operation context; callers branch with errors.Is/As.
• [ ] Context first parameter, canceled on all paths, never stored.
• [ ] Generics used only where they reduce real duplication; interfaces small and consumer-defined.
• [ ] Public API documented; zero values sane; no typed-nil interface returns.
• [ ] No goroutine without owner/stop/join; bounded concurrency for external calls.

Architecture and tooling:

• [ ] Package layout reflects actual boundaries; no util dumping ground.
• [ ] Module floors and toolchain pinning deliberate; go mod tidy -diff clean; tools pinned.
• [ ] CI runs format, vet, staticcheck/lint, race tests, govulncheck; generated code verified.
• [ ] Build tags and cross-compilation matrix covered by CI.

Services and data:

• [ ] Server/client timeouts, body limits, graceful shutdown, connection reuse.
• [ ] Queries parameterized, pooled, context-bound; migrations reversible and lock-aware.
• [ ] Caches bounded, versioned, jittered, and non-authoritative.
• [ ] Structs tested at memory boundaries: fakes where practical, real engines for semantics.

Operations:

• [ ] Structured logs with correlation; redaction; no secrets in logs or errors.
• [ ] Traces/metrics exported with bounded cardinality and shutdown flush.
• [ ] GOMEMLIMIT set below container limit; pools and queues monitored.
• [ ] Supply chain: SBOM, signatures, pinned actions, minimal dependencies.
• [ ] Runbooks for the failure modes this service actually has (pool exhaustion, dependency outage, queue backlog).

Common pitfalls, one line each

• Typed nil in an interface, checked with == nil.
• %v on an error where %w was needed, breaking errors.Is.
• Goroutine leak via an unread result channel.
• time.After in a loop.
• WriteTimeout killing SSE and long downloads.
• SELECT * into structs; schema drift at runtime.
• Unbounded label values in metrics.
• math/rand for tokens; InsecureSkipVerify in a service client.
• Unbounded request bodies and header sizes.
• Caching authorization decisions without invalidation.
• go.mod go directive bumped without testing capture semantics.
• replace directives relied on for published behavior.
• Linters and generators pinned to @latest.
• Golden files regenerated without review.
• Profiling guessed instead of measured.