Language Core: Ownership, Borrowing, Lifetimes, Data Modeling

Scope: the ownership/borrowing/lifetime mental model, smart pointers, enums, pattern matching, closures, iterators, and concrete fixes for borrow-checker fights.

Ownership Mental Model

• Every value has exactly one owner. When the owner goes out of scope, the value is dropped.
• Assignment, passing by value, and returning move the value unless the type is Copy.
• Copy is a property of the type (no heap, no destructor semantics enforced); Clone is an explicit deep or shallow copy.
• Drop order: locals drop in reverse declaration order; struct fields drop in declaration order; tuple elements drop in order. Temporaries drop at the end of the enclosing statement.
• Edition 2024 changed the scope of temporaries in tail expressions and if let; code that relies on temporary drop timing should be tested on both editions. Verify upstream for exact rules.
• Use std::mem::drop(x) to end a borrow early; use std::mem::take/replace to move a value out through &mut.

struct Conn { name: String }
impl Drop for Conn { fn drop(&mut self) { /* close */ } }

fn take_ownership(c: Conn) -> String { c.name } // c dropped at end of fn
fn borrow(c: &Conn) -> &str { &c.name }          // no move, no drop

Copy vs Clone decision table

| Situation | Use | |---|---| | Small scalar/identifier (u64, f64, [u8; 16], enums of scalars) | derive Copy | | Owns heap or resource (String, Vec, file handle) | Clone only, explicit | | Type will be serialized/shared and is cheap | derive both | | Contains Drop behavior | never Copy |

Borrowing Rules

• At any time: any number of shared borrows (&T) OR exactly one mutable borrow (&mut T) — never both.
• Borrow ends at its last use under non-lexical lifetimes (NLL). Holding a &mut through an unrelated call still conflicts.
• Reborrowing (&mut *x) is implicit when passing &mut to functions; the original can be reused after the call returns.
• Returning a reference from a function requires the lifetime to come from an input (see elision below).
• Interior mutability (Cell, RefCell, Mutex, atomics) is the escape hatch when aliasing plus mutation is genuinely required.

Lifetime elision rules

1. Each elided input lifetime becomes a distinct parameter lifetime.
2. If there is exactly one input lifetime, it is assigned to all elided outputs.
3. If there is &self or &mut self, its lifetime is assigned to all elided outputs.

fn first_word(s: &str) -> &str { /* ties output to s */ }
fn get<'a>(&'a self) -> &'a str { &self.name }

• 'static as a bound means "may live for the whole program", not "must". &'static str is a reference to static storage; T: 'static on a generic means "contains no non-static borrows", which includes owned types like String.
• Structs holding borrows need explicit lifetimes: struct Parser<'a> { src: &'a str }. If that becomes viral, own the data (String, Arc<str>) at API boundaries.
• Lifetime elision does not apply inside closures or across async blocks; keep borrows short before .await.
• Higher-ranked trait bounds (for<'a> Fn(&'a str)) are occasionally needed; prefer Fn(&str) -> ... sugar.

Smart Pointers

| Type | Sharing | Mutation | Thread-safe | Use when | |---|---|---|---|---| | Box<T> | single | via &mut | if T: Send | heap indirection, recursive types, trait objects | | Rc<T> | shared | no (or with RefCell) | no | single-thread shared ownership, cheap clones | | Arc<T> | shared | no (or with Mutex) | yes (atomics) | cross-thread shared ownership | | Weak<T> | shared | no | matches parent | break reference cycles, caches, parent pointers | | Cell<T> | single | copy-in/copy-out | no | small Copy state behind &self | | RefCell<T> | single | runtime-checked | no | complex mutation behind &self | | Mutex<T>/RwLock<T> | shared | lock | yes | shared mutable state across threads | | OnceLock<T> | shared | set once | yes | lazily initialized global/config; LazyLock for init expressions (Rust 1.80+, verify upstream) | | Cow<'a, T> | clone-on-write | via to_mut | varies | accept borrowed or owned, avoid allocations |

• RefCell panics on conflicting borrows at runtime. Use try_borrow/try_borrow_mut where a panic is unacceptable, and keep borrows short.
• Mutex poisoning: .lock() returns Result; unwrap() is defensible only when a panic while holding the lock already means the process should die. Otherwise recover or use parking_lot::Mutex (no poisoning).
• Never hold a std::sync::MutexGuard across .await. Use tokio::sync::Mutex only if the critical section must await; prefer restructuring to avoid async locks entirely. See ./03-async-concurrency.md.
• Rc cycles leak. Store Weak for back/observer links and upgrade with .upgrade().
• Prefer Arc<str>/Arc<[T]> over Arc<String>/Arc<Vec<T>> for immutable shared data (one indirection, immutable).
• Atomics (AtomicUsize, AtomicBool, AtomicPtr) belong in the standard library; use Ordering::Relaxed only for counters where no synchronization is implied. Verify the necessary ordering against the C++ memory model.

Interior mutability hierarchy

&mut T for free, else
  AtomicBool/AtomicUsize  (Copy, lock-free, cross-thread)
  Cell<T: Copy>           (no borrow checking, single thread)
  RefCell<T>              (runtime borrow checking, single thread)
  Mutex<T>/RwLock<T>      (blocking, cross-thread)
  tokio::sync::Mutex      (async-aware, cross-thread)
  channels                (ownership transfer instead of sharing)

Rule of thumb: if you reach for Rc<RefCell<...>> frequently, redesign around message passing or explicit state structs.

Enums and Pattern Matching

• Prefer enums over boolean flags and over "nullable fields"; model the domain so invalid states do not compile.
• match must be exhaustive. Use #[non_exhaustive] on public enums so downstream matches can add a wildcard.
• let ... else for early exit without deep nesting.
• matches!(x, Pat if guard) for boolean tests.
• if let chains (if let Some(x) = a && let Some(y) = b) stabilize with edition 2024; verify upstream before relying on them in older editions.

#[derive(Debug)]
enum Job { Queued, Running { pid: u32 }, Done(ExitStatus), Failed(String) }

fn label(job: &Job) -> &'static str {
    match job {
        Job::Queued => "queued",
        Job::Running { .. } => "running",
        Job::Done(_) => "done",
        Job::Failed(_) => "failed",
    }
}

let Some(cfg) = load_config() else { return Err(Error::NoConfig) };

Option/Result combinator quick table

| Need | Use | |---|---| | default | unwrap_or, unwrap_or_else, unwrap_or_default | | convert to Result | ok_or, ok_or_else | | chain fallible | and_then, Result::and_then | | branch on both | match, map_or, map_or_else | | swap Result<Option<_>> | .transpose() | | replace with owned | take() | | merge Option<Result<_>> | and_then(|r| r.ok()) |

Closures

• Closure traits: Fn (callable via &), FnMut (via &mut), FnOnce (consumes captured values). Functions implement all three.
• Capture modes follow usage; move forces ownership, which is what 'static spawns need.
• Edition 2021+ captures disjoint fields of a struct instead of the whole struct (closures capture precision). This can change drop timing; be explicit when it matters.
• Pass closures as impl Fn(...), generic parameters, or Box<dyn Fn(...)> when naming/routing is needed.
• A closure that mutates captured state cannot be called through &self. If a struct stores a callback, decide up front whether it is FnMut and expose &mut self.

fn retry<F: FnMut() -> Result<(), E>, E>(attempts: usize, mut f: F) -> Result<(), E> {
    for _ in 0..attempts {
        match f() { Err(e) => last = Some(e), Ok(()) => return Ok(()) }
    }
    Err(last.expect("attempts > 0"))
}

Iterators

• Iterators are lazy; adapters (map, filter) do nothing until a consumer (collect, sum, for_each, count, fold) runs.
• collect::<Result<Vec<_>, _>>() short-circuits on first error — the idiomatic way to parse a sequence.
• iter() yields &T; iter_mut() yields &mut T; into_iter() yields T. On a &Vec<T>, into_iter() already yields &T.
• Prefer iterator chains over index loops: bounds checks usually elide and intent is clearer.
• Watch for: intermediate collect() followed by another iteration, Vec growth without with_capacity, and clone() inside adapters.
• Vec<T> specifics: drain(..).for_each(...) removes and consumes; retain mutates in place; split_at_mut yields two disjoint mutable slices (a standard borrow-checker fix).

let nums: Vec<i32> = lines.iter().map(|l| l.parse::<i32>()).collect::<Result<_, _>>()?;
let total: i32 = nums.iter().filter(|n| **n > 0).sum();

Common Borrow-Checker Fights and Fixes

| Error | Typical cause | Fix | |---|---|---| | E0382 use of moved value | value used after pass/return | borrow it, clone() deliberately, or restructure ownership | | E0499 two mutable borrows | two &mut into same container | split via destructuring, split_at_mut, iterate over fields | | E0502 mutable + shared borrow | read while mutating | compute the read into a local first; scope the borrow | | E0505 moved while borrowed | value dropped/moved while borrowed | clone, reorder drop, mem::take | | E0515 returning local reference | returned temporary/local | return owned (String, Vec) or Cow | | E0597 borrowed value does not live long enough | reference outlives source | own the data or shorten the borrow | | E0716 temporary dropped while borrowed | binding a reference to a temporary | bind the temporary first with let | | E0507 cannot move out of borrowed content | moving a field out of &self | mem::take, Option::take, clone, or match by reference | | E0596 cannot borrow as mutable | binding not mut, or shared &self | add mut, take &mut, or use interior mutability | | E0277 Send/Sync not satisfied | non-thread-safe value crosses threads | see ./03-async-concurrency.md | | E0106 missing lifetime | struct/fn returns borrow without source | add lifetime parameter tied to input |

// E0499 fix: split one &mut Vec into two disjoint slices
let (left, right) = data.split_at_mut(mid);
left[i] += right[j];

// E0507 fix: move a field out of &mut self
let task = self.queue.pop_front().take(); // no clone

Diagnosis strategy: read the compiler's "first borrow occurs here / second borrow occurs here" span pair; the fix is usually to shorten one borrow, not to fight the compiler. Add clone() only after you can justify why sharing is wrong.

Anti-Patterns

• Rc<RefCell<T>> graphs everywhere instead of ownership boundaries or message passing.
• clone() to silence E0382 without understanding sharing.
• unwrap() on lock/parse results in library code (see ./04-error-handling.md).
• Public structs with pub fields that encode invariants impossible to maintain; use constructors and accessors.
• Returning &str/&T from a function backed by an internal cache without a lifetime story.
• Overusing 'static bounds on async traits to dodge lifetime errors.
• Index loops where iterators are clearer and safer.

Checklist

• [ ] Types encode valid states; invalid states do not compile where practical.
• [ ] Borrows are short and scoped; no clone() without a stated reason.
• [ ] Rc/Arc used only for genuine shared ownership; cycles use Weak.
• [ ] No std::sync lock guard held across .await.
• [ ] Public enums are #[non_exhaustive] when extension is expected.
• [ ] Iterators replace manual index loops; no accidental intermediate collections.
• [ ] Every borrow-checker workaround is explained in code or review notes.

Type System and Traits

Scope: traits, associated types, generics versus dyn, blanket impls, marker traits, newtypes, conversions, operator traits, and std::error::Error integration.

Trait Design Fundamentals

• A trait is a contract plus optional default methods. Keep required methods minimal; implement defaults in terms of them.
• Supertraits (trait Conn: Send + Sync) express requirements, not inheritance. Prefer composing small traits over deep supertrait chains.
• Methods taking &self vs &mut self vs self define the object-safety and usability surface. Returning Self or generic methods makes a trait non-object-safe.
• Coherence: a downstream crate cannot implement a foreign trait for a foreign type. Design extension points with local traits or newtypes when downstream impls are expected.

pub trait Store {
    type Error;
    fn get(&self, key: &str) -> Result<Option<Vec<u8>>, Self::Error>;
    fn put(&mut self, key: &str, value: &[u8]) -> Result<(), Self::Error>;
    fn contains(&self, key: &str) -> Result<bool, Self::Error> {
        Ok(self.get(key)?.is_some()) // default method
    }
}

Associated Types vs Generic Parameters

| Question | Associated type (trait Trait { type Out; }) | Generic parameter (trait Trait<T>) | |---|---|---| | One impl per type? | yes, by construction | multiple impls possible | | Caller names the type? | no; inferred from impl | yes, turbofish | | Object safety | usually keeps dyn possible | rarely dyn-compatible | | Typical use | Iterator::Item, Store::Error | From<T>, PartialEq<Rhs> | | Ergonomics | no type annotation at call sites | annotation/noise at call sites |

Rule: if the trait conceptually has exactly one output per implementor, use an associated type. If a type may implement the trait for several parameter types (Add<Meters> and Add<Feet>), use generics.

Generics vs dyn (Static vs Dynamic Dispatch)

| Criterion | Generics (impl Trait/<T: Trait>) | dyn Trait | |---|---|---| | Dispatch | static, inlinable | vtable, indirect | | Binary size | monomorphized per type | one copy | | Heterogeneous collections | no | yes (Vec<Box<dyn Trait>>) | | Compile time | worse as combinations grow | better | | Object safety required | no | yes | | Plugin/runtime selection | no | yes |

• Prefer generics in hot paths and library APIs; prefer dyn at process boundaries (plugins, heterogeneous lists, avoiding combinatorial explosion).
• impl Trait in argument position is sugar for a generic parameter; in return position it hides a concrete (possibly unnameable) type.
• RPITIT (-> impl Trait in trait methods) is stable on modern Rust (1.75+, verify upstream). It makes traits non-object-safe unless dyn support is designed.
• Boxed trait objects allocate: Box<dyn Error + Send + Sync> costs one allocation. Accept it at boundaries, avoid in tight loops.

fn total<I: Iterator<Item = u64>>(it: I) -> u64 { it.sum() }          // static
fn total_boxed(it: Box<dyn Iterator<Item = u64>>) -> u64 { it.sum() } // dynamic

Object safety quick check

A trait can be used as dyn Trait if it has no generic methods, no Self-returning methods (except where Self: Sized), no associated constants, and Self: Sized is not required. Add where Self: Sized on non-object-safe methods to keep the rest dyn-compatible.

Blanket Impls and Coherence

• Blanket impl: impl<T: Display> Loggable for T. Powerful, but any downstream type automatically gets it and overlap becomes likely.
• The orphan rule allows impl LocalTrait for ForeignType and impl ForeignTrait for LocalType.
• Overlap checks are conservative; two blanket impls that "obviously" do not overlap still conflict. Use newtypes or sealed traits to disambiguate.
• Common std blanket impls to know: From<T> for T, From<T> for Option<T>, Into<T> for T, Borrow<T> for T, AsRef<T> for T, Any for 'static types.

pub trait Encode { fn encode(&self, out: &mut Vec<u8>); }
impl<T: serde::Serialize> Encode for T { /* blanket */ }
// Now a manual impl for a local type would conflict; use a newtype instead.

Marker Traits and Auto Traits

| Trait | Meaning | Opt-out | |---|---|---| | Send | can move across threads | PhantomData<*const ()> or Rc field | | Sync | can be shared by reference across threads | Cell/RefCell field | | Unpin | safe to move after pinning | PhantomPinned | | Sized | known size at compile time | ?Sized bound | | Copy | bitwise duplicate, no drop | manual Clone | | UnwindSafe | safe to observe after panic | AssertUnwindSafe wrapper |

• Send/Sync are auto traits: composed structurally from fields. unsafe impl Send for MyType {} is an invariant you now own; document why.
• Negative impls (impl !Send) remain unstable for user types; use marker fields to remove auto traits.
• PhantomData<T> communicates ownership, variance, and auto-trait behavior for types with no field of that type.

Newtypes and Type-State

• Newtype: struct UserId(u64); avoids primitive obsession, enables trait impls on foreign types, and centralizes validation.
• Add #[repr(transparent)] when the wrapper must have the same ABI as the inner type (FFI).
• Implement From<Inner> only when the conversion is infallible and identity-preserving; use a try_new constructor for validation.
• Type-state: encode lifecycle in a type parameter (Request<Draft>, Request<Sent>) so invalid transitions fail to compile. Cost: more generics and sometimes harder dyn integration.

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct Email(String);

impl Email {
    pub fn parse(raw: &str) -> Result<Self, EmailError> {
        let raw = raw.trim();
        if raw.contains('@') && !raw.contains(char::is_whitespace) {
            Ok(Self(raw.to_ascii_lowercase()))
        } else {
            Err(EmailError::Invalid)
        }
    }
}
impl std::fmt::Display for Email { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { f.write_str(&self.0) } }

Deref anti-pattern

Do not implement Deref to gain method forwarding on non-pointer types. It leaks the inner API, confuses inference, and breaks dyn bounds. Use explicit methods or AsRef. Deref is for smart pointers and wrapper types whose entire purpose is delegation.

Conversions

| Trait | Direction | Failure | Convention | |---|---|---|---| | From<T> | infallible | no | also gives Into | | TryFrom<T> | fallible | Error assoc type | also gives TryInto | | AsRef<T> | borrowed view | no | cheap, non-consuming | | Borrow<T> | borrowed view with semantic equality | no | map keys/lookups | | FromStr | parse from &str | Err | "x".parse::<T>() | | Serialize/Deserialize | data mapping | varies | serde |

• Implement From, get Into free. Never implement both.
• Use AsRef<Path> for path-like parameters and impl Into<String> for owned string parameters at API boundaries.
• Borrow differs from AsRef: String: Borrow<str> promises Hash/Eq agree with str, which is what makes HashMap<String, _>::get("key") work.
• Keep conversions lossless. If a conversion truncates (u64 -> u32), make it TryFrom or name it truncate_to_u32.

impl From<io::Error> for AppError { fn from(e: io::Error) -> Self { Self::Io(e) } }
impl TryFrom<&str> for Port { type Error = PortError; fn try_from(s: &str) -> Result<Self, _> { /* ... */ } }

Operator Traits and Core Contracts

| Trait | Operator | Notes | |---|---|---| | Add, Sub, Mul, Div | + - * / | take self/&self; define AddAssign for mutation | | Neg, Not | - ! | Not on booleans is logical | | Index, IndexMut | [] | may panic; document bounds behavior | | PartialEq/Eq | == != | Eq requires reflexivity; floats are only PartialEq | | PartialOrd/Ord | < <= > >= | implement Ord with a total, consistent order | | Hash | hashing | must agree with Eq; derive both together | | Display/Debug | {}/{:?} | Display is user-facing, Debug is diagnostics | | Deref | * | see anti-pattern above | | Drop | scope exit | cannot be called directly; use mem::drop |

• Derive Debug on every public type; include it in deny(missing_debug_implementations) policies.
• Manual PartialEq without Hash consistency silently breaks HashMap.
• For newtypes used as keys, derive PartialEq, Eq, Hash or implement all three with the same field set.

Error Trait Integration

• A custom error type should implement std::error::Error (via thiserror or manually), Display, and Debug.
• Error::source() exposes the causal chain; anyhow and tracing walk it. Provide source via #[from]/#[source] attributes.
• Box<dyn std::error::Error + Send + Sync + 'static> is the universal erased error; use it at API boundaries, not internally.
• Never implement Error for a type whose Display is empty or whose message duplicates the source without context.

#[derive(Debug, thiserror::Error)]
pub enum StoreError {
    #[error("key not found: {key}")]
    NotFound { key: String },
    #[error("backend unavailable")]
    Backend(#[from] tokio::io::Error),
}

See ./04-error-handling.md for the full policy on thiserror vs anyhow and boundary conversion.

Advanced Patterns

• Sealed traits: pub trait Foo: private::Sealed prevents downstream impls while keeping the API public. Use for extension points you intend to grow.
• Extension traits: define a local trait with methods for a foreign type to add behavior (trait VecExt { ... } impl<T> VecExt for Vec<T>).
• GATs (generic associated types, type Item<'a>) are stable on modern Rust; use for lending iterators and view types. They add complexity — verify need first.
• #[non_exhaustive] on public structs/enums reserves the right to add fields/variants.
• Hygiene of derive: #[derive(Clone)] requires all fields Clone; a derive failure names the field, so fix the field rather than writing a manual impl that panics.

Anti-Patterns

• Trait-per-type instead of functions/structs (UserServiceTrait for a single impl, never mocked): only introduce traits at seams you substitute or mock.
• Deref for code reuse.
• Generic explosion: 4+ type parameters on public types instead of boxing or enum dispatch.
• impl Trait in return position hiding a type callers must name for traits (e.g., returning impl Iterator when callers need ExactSizeIterator).
• Blanket impls over foreign traits that create coherence conflicts downstream.
• Clone derived on handle/resource types that should be moved, not duplicated.
• anyhow::Error in a public library API (see ./04-error-handling.md).

Checklist

• [ ] Traits exist at real seams; no speculative abstraction.
• [ ] Associated types used for one-output contracts; generics for multi-impl contracts.
• [ ] dyn used only where heterogeneity or runtime selection is required.
• [ ] Newtypes validate in constructors; no public field that breaks invariants.
• [ ] From/TryFrom used instead of ad-hoc to_* conversion methods.
• [ ] Error types implement Error, Display, Debug with a useful source chain.
• [ ] Every unsafe impl Send/Sync has a written justification.
• [ ] Derived PartialEq/Hash/Ord are mutually consistent.

Async and Concurrency

Scope: tokio runtime tuning, tasks, cancellation and cancellation safety, structured concurrency, channels, select!, backpressure, Send/Sync for futures, and async traits.

Runtime Selection and Tuning

| Runtime | Use when | Notes | |---|---|---| | #[tokio::main] multi-thread | servers, CPU+IO mix | default worker threads = logical CPUs; good default | | #[tokio::main(flavor = "current_thread")] | CLI tools, tests, single-threaded embedders | lower memory, deterministic polling order | | Runtime::new manual | library owns runtime, custom config | build once, never per request | | spawn_blocking pool | filesystem, CPU-bound, FFI, DNS via blocking libs | bounded; default cap 512 (verify upstream) | | LocalSet | non-Send futures | single-thread only | | Other (smol, async-std, embassy) | niche/embedded | see ./07-embedded.md |

let rt = tokio::runtime::Builder::new_multi_thread()
    .worker_threads(4)
    .max_blocking_threads(64)
    .thread_name("app-worker")
    .enable_all()
    .build()?;

• Never call block_on inside async context. Use spawn_blocking or an async equivalent.
• Blocking syscalls on worker threads starve timers and IO. Audit std::fs, std::net, std::process, and synchronous DB drivers.
• tokio::time::pause() in tests makes time deterministic; requires test-util.
• Enable only needed features (rt-multi-thread, net, time, sync, macros) to reduce build time and binary size.
• Tokio Console (console-subscriber) is the tool for diagnosing task stalls and waker storms; keep it behind a dev feature.
• Graceful shutdown belongs to the application: a CancellationToken plus drained JoinSet/TaskTracker. See ./06-web-data.md for HTTP wiring.

Tasks

• tokio::spawn requires F: Future + Send + 'static and returns JoinHandle<T>. The future must own everything it uses.
• JoinHandle returns Result<T, JoinError>; JoinError::is_panic() distinguishes panic from cancellation.
• JoinSet manages a dynamic group: spawn, join_next().await, and abort remaining tasks on drop (verify exact behavior for your toolchain upstream).
• TaskTracker (tokio-util) plus CancellationToken is the canonical structured-concurrency pair.
• AbortHandle/JoinHandle::abort stops at the next await point; it does not run cleanup. Design for drop-based cleanup and cancellation tokens.
• Task-local state (tokio::task_local!) is per-task context that follows spawned work only if re-scoped inside spawn. Prefer passing an Arc<Context> explicitly.

let mut set = JoinSet::new();
for shard in shards {
    set.spawn(async move { process(shard).await });
}
while let Some(res) = set.join_next().await {
    match res { Ok(Ok(())) => {}, Ok(Err(e)) => error!(?e, "shard failed"), Err(je) => warn!(?je, "join failed") }
}

Cancellation and Cancellation Safety

• Dropping a future cancels it. Cancellation happens at .await points; code between awaits runs to completion or not at all.
• A future is cancellation-safe if dropping it cannot lose data that was already committed from its perspective. tokio::io::AsyncReadExt::read is not safe to cancel midway if you must not lose bytes; read_exact is not; AsyncWriteExt::write_all is not.
• Cancellation-safety table (common ops):

| Operation | Safe to cancel? | Guidance | |---|---|---| | TcpStream::connect | yes | retry is free | | read/read_buf | no | bytes may be lost; keep buffer/state | | read_exact/read_to_end | no | wrap in a task or buffer | | write/write_all/flush | no | partial writes | | sleep/timeout | yes | no data | | mpsc::Sender::send | no (may be cancelled after reserve) | use reserve explicitly if needed | | Semaphore::acquire | no | permit may be acquired and dropped | | JoinHandle | yes | abort semantics | | Receiver::recv | yes (for mpsc), but messages may be unreceived | design idempotent handling |

• select! polls branches in random order by default; each unselected branch is dropped, so all branches must be cancellation-safe. Use biased; for priority, and always add a default path or else branch when needed.
• Wrap whole critical sequences in tokio::spawn if they must survive a select! loser drop.
• CancellationToken (tokio-util) composes: child_token, cancelled().await, run_until_cancelled. Cancel is idempotent and wakes all waiters.

tokio::select! {
    _ = token.cancelled() => return Ok(()),
    res = do_work(&mut state) => res?,
}

• timeout(d, fut) drops fut on expiry: only use around cancellation-safe futures; otherwise spawn it and await the handle with a timeout.
• Graceful shutdown sequence: stop accepting (drop listener) -> cancel token -> await tasks with a deadline -> run cleanup -> exit. Never process::exit before drains unless aborts are acceptable.

Structured Concurrency

• Own every task: store handles, join them, and propagate errors. Fire-and-forget tasks are a leak and an observability hole.
• A supervision pattern: one coordinator owns a JoinSet, restarts or records failed children, and exposes a single completion future to the caller.
• Backpressure must be end-to-end: bounded channels + Semaphore + connection limits. Unbounded queues convert overload into OOM.
• Prefer scoped APIs (JoinSet, TaskTracker, tokio::task::scope when available/futures scope) so no task outlives its data.

Channels

| Channel | Producers | Consumers | Semantics | Use | |---|---|---|---|---| | mpsc | many | one | ordered queue, bounded/unbounded | work queues, actor inboxes | | oneshot | one | one | single value | request/response, task results | | broadcast | many | many | every receiver sees every message (lagging drops oldest) | events, shutdown fan-out | | watch | one | many | latest value only | config updates, state snapshots | | Notify | many | many | wake-up signal, no payload | condition-style coordination | | Semaphore | n/a | n/a | permit pool | concurrency limits |

• Always prefer bounded mpsc for work queues. send().await applies backpressure; try_send surfaces Full for load shedding.
• broadcast::Receiver treats lag as RecvError::Lagged(n); decide whether to resync or fail. Default capacity must be sized for burst rate, not steady state.
• watch requires T: Clone and stores one value; good for "current config" and shutdown flags, bad for event streams.
• Cross-process coordination needs an external broker (NATS, Redis streams, Kafka); in-process channels do not survive restart.
• Avoid sharing Arc<Mutex<HashMap>> as a "channel"; a task owning the map plus an mpsc inbox removes lock contention and makes ownership obvious.

Send/Sync for Futures and Tasks

• A future is Send iff all values held across .await points are Send. The compiler points at the offending binding.
• Typical offenders: Rc, RefCell, MutexGuard, non-Send trait objects, *const T, and &T where T: !Sync.
• Holding a std::sync::MutexGuard across .await produces a non-Send future and potential deadlocks. Scope the guard before awaiting, or use tokio::sync::Mutex when the section genuinely awaits.
• Spawning non-Send work requires LocalSet on a current-thread runtime.
• Send bounds propagate: an async fn generic over T may only be Send if T: Send; add bounds at the public boundary, not everywhere.
• Beware async blocks capturing &mut self across awaits; split the method or move owned state in.

// BAD: guard held across await (non-Send, deadlock risk)
let guard = self.inner.lock().unwrap();
some_async_call().await;
drop(guard);

// GOOD
{ let mut g = self.inner.lock().unwrap(); g.update(); }
some_async_call().await;

Blocking, Timeouts, and Retries

• spawn_blocking for anything that can take >100us of CPU or uses blocking IO. Its pool is bounded; a saturated pool queues.
• CPU-bound parallelism should be a rayon pool, not many blocking tasks (see ./10-performance-security-observability.md).
• Timeouts everywhere at boundaries: connect, request handling, DB queries, shutdown. Track remaining deadline (tokio::time::Instant) through retries.
• Retries: exponential backoff with jitter (backoff, tower::retry, or hand-rolled). Only retry idempotent operations; propagate the original error after N attempts.
• Circuit breakers (tower middleware or a custom state machine) prevent retry storms.

Async Traits (AFIT and dyn)

• AFIT: async fn in traits is stable (Rust 1.75+, verify upstream) and lowers to RPITIT. It is not dyn-compatible by default.
• Making async traits dyn-compatible:
  1. #[async_trait] (crate async-trait) boxes every method future — allocation per call, works on all targets, supports dyn.
  2. Return-position impl Future plus manual Pin<Box<dyn Future>> signatures.
  3. trait-variant generates a Send-bounded twin via a macro.
• async fn in a public trait is fine for static dispatch; add Send bounds at the trait or impl level when tasks spawn the returned future.
• Async closures (async || {}, AsyncFn/AsyncFnMut/AsyncFnOnce) are stable in edition 2024 (Rust 1.85+, verify upstream); they simplify callback APIs that spawn.
• Cancellation safety of returned futures is part of the contract; document it for each trait method.

#[async_trait::async_trait]
pub trait Fetcher: Send + Sync {
    async fn fetch(&self, id: u64) -> Result<Vec<u8>, FetchError>;
}

Anti-Patterns

• Blocking calls in async context (std::fs, std::thread::sleep, synchronous HTTP/DB clients).
• tokio::spawn with a detached handle and no error logging; failures vanish.
• Unbounded channels hiding overload.
• select! over cancellation-unsafe branches (partial reads/writes).
• Locking an async mutex for short CPU-only sections (use std mutex, no await).
• Runtime per request (Runtime::new in a handler).
• Using tokio::time::sleep to "fix" races instead of proper synchronization.
• Spawning inside loops without a semaphore or join tracking.

Checklist

• [ ] Every spawned task has an owner, a join path, and error handling.
• [ ] Cancellation behavior is understood for each select!/timeout branch.
• [ ] Shutdown is token-driven and drains with a deadline.
• [ ] All queues are bounded; overload behavior (block vs shed) is explicit.
• [ ] No blocking calls or lock guards across .await.
• [ ] Send/Sync bounds appear where tasks require them; no gratuitous 'static boxing.
• [ ] Timeouts and retries with jitter are applied at every external boundary.

Error Handling

Scope: Result/Option idioms, thiserror versus anyhow policy, error context and chains, panic and unwrap/expect discipline, and conversion at architectural boundaries.

The Error Decision Table

| Situation | Representation | |---|---| | Library public API | custom enum, thiserror or hand-written Error impl | | Binary/application internals | anyhow::Result<T> (context-rich, type-erased) | | Binary top level (main) | anyhow::Result<()> or ExitCode | | Boundary between layers | convert library error into app error (From/map_err) | | Recoverable, expected absence | Option<T> | | Infallible by construction | plain T, not Result | | Erased trait-object boundary | Box<dyn Error + Send + Sync + 'static> | | FFI/panic boundary | no error type; status code / catch_unwind (see ./05-unsafe-ffi.md) |

Policy in one line: libraries use typed errors; applications use anyhow; every conversion at a boundary preserves the cause.

Result and Option Idioms

• ? propagates and converts via From when the function's error type can be built from the inner error.
• ok_or_else(|| ...) maps Option to Result without allocating on the happy path; ok_or is fine for non-allocating errors.
• transpose() converts Result<Option<T>, E> and back; useful with fallible fetch-or-none APIs.
• map_err adds context or maps to a domain error; avoid map_err(|_| MyError::Unknown) which drops the cause.
• Result::and_then chains fallible steps without nesting; inspect/inspect_err (Rust 1.76+, verify upstream) for logging without altering the value.
• try blocks remain unstable on stable Rust; do not design APIs that need them.
• main can return Result<(), E: Debug>; for custom exit codes return ExitCode from std::process.

fn load(path: &Path) -> anyhow::Result<Config> {
    let text = std::fs::read_to_string(path)
        .with_context(|| format!("reading config {}", path.display()))?;
    let cfg = toml::from_str(&text)
        .with_context(|| format!("parsing config {}", path.display()))?;
    Ok(cfg)
}

thiserror vs anyhow

| Aspect | thiserror | anyhow | |---|---|---| | Purpose | define error types | consume/erase errors | | Location | library crates | binaries, app internals, tests | | Downcastable | yes, pattern-match variants | yes via chain/downcast, but discouraged for control flow | | Context | fields + #[source] | .context() / .with_context() | | Public API | recommended | forbidden: leaks opaque type, breaks versioning and matching | | Conversion boilerplate | #[from] generates | not applicable |

• Never expose anyhow::Error from a published crate. It changes type identity with versions and provides no stable contract.
• Library error enums: one variant per actionable failure mode, plus a catch-all #[error(transparent)] Other(#[from] ...) only where genuinely opaque.
• Derive Debug; implement Display via thiserror messages that are lowercase, no trailing period, and name the operation and subject.
• Mark constructors and source fields with #[from], #[source], or #[error(transparent)] correctly: transparent excludes the source from being printed twice.
• anyhow!("...") / bail!(...) for ad-hoc failures inside binaries; still attach chains to boundary errors.

#[derive(Debug, thiserror::Error)]
pub enum AuthError {
    #[error("invalid credentials for {user}")]
    Invalid { user: String },
    #[error("token expired")]
    Expired,
    #[error(transparent)]
    Store(#[from] StoreError),
}

Error Context and Chains

• Context answers "what was the program doing" at the error, not just "what failed". Include the operation and the resource id/path, never secrets.
• Build context lazily (with_context(|| ...)) so the allocation is skipped on success.
• Print full chains: {:#} for anyhow prints the whole chain on one line; {:?} prints source chain plus backtrace when enabled.
• std::backtrace::Backtrace is stable (1.65+); anyhow captures a backtrace when RUST_BACKTRACE=1 (or full) is set at compile/run time. Verify behavior for your toolchain.
• Do not log and return the same error at every level; log once at the boundary where it is handled, with the chain intact.
• Preserve typed errors for programmatic handling (e.g., HTTP 404 vs 500); use downcast_ref only at the edge.

Panic Policy

| Context | Policy | |---|---| | Library code | never panic on user input; return Result | | main/startup (bad config) | panic acceptable: fail fast, no partial service | | Tests | panics are assertions | | FFI boundary | catch (catch_unwind) or document unwind across extern "C" as forbidden | | Task/thread worker | catch at task boundary, log, keep supervisor alive | | Holding a lock | avoid panics; a panic poisons std::sync::Mutex | | panic = "abort" builds | process dies; use only if crash-only design is intentional |

• panic!, assert!, assert_eq!, unreachable!, todo!, unimplemented!, slicing/indexing, integer overflow (debug), .unwrap(), .expect(), division by zero, and RefCell double-borrow all panic.
• Do not use catch_unwind for control flow or expected errors; it is for quarantine/supervision, and requires the boundary to be UnwindSafe or wrapped in AssertUnwindSafe.
• Edition 2024/extern "C-unwind" and "C" unwind behavior differ; pick the ABI deliberately when panics may propagate. Verify upstream for your targets.
• Log panics: install a panic hook that records the message, location, and backtrace (tracing or human-panic for CLIs).

unwrap/expect Discipline

• expect("reason") is allowed when the invariant is local and provable: literals parsed at startup, states just checked, Mutex::lock when a poisoned lock should crash anyway.
• .unwrap() without a message is banned by policy in production code. Enforce with clippy lints:

# clippy.toml / Cargo lints
[workspace.lints.clippy]
unwrap_used = "deny"
expect_used = "warn"
panic = "warn"
indexing_slicing = "warn"

• Allow locally where justified: #[allow(clippy::unwrap_used, reason = "compile-time regex")] (lint reason attributes supported on modern Rust; verify upstream).
• Never unwrap() on: parsing external input, environment variables, file/network operations, JoinHandle in production without logging, Mutex in libraries, or SystemTime arithmetic.
• Prefer let else, ?, unwrap_or_default, or explicit match over is_some() + unwrap() (a fragile double lookup).
• Treat Option::unwrap in tests as acceptable; tests should fail loudly.

Conversion at Boundaries

• Convert at the edge: each layer owns its error type; adapt with From implementations or a single map_err at the call site.
• HTTP boundary: map domain errors to IntoResponse/StatusCode centrally (one impl IntoResponse for AppError), not per handler.
• DB boundary: distinguish constraint violation, not-found, and connectivity at the repository layer; expose domain variants upward.
• gRPC/IPC boundary: map to tonic::Status codes with messages safe for clients; log the full chain internally.
• Task boundary: JoinError -> application error; include the task name.
• Serialization boundary: serde_json::Error with line/column goes into context; do not expose raw parser messages to end users.

impl axum::response::IntoResponse for AppError {
    fn into_response(self) -> axum::response::Response {
        let status = match &self {
            AppError::NotFound(_) => StatusCode::NOT_FOUND,
            AppError::Validation(_) => StatusCode::UNPROCESSABLE_ENTITY,
            _ => StatusCode::INTERNAL_SERVER_ERROR,
        };
        if status.is_server_error() { tracing::error!(error = ?self, "request failed"); }
        (status, status.canonical_reason().unwrap_or("error")).into_response()
    }
}

Anti-Patterns

• String-only errors: Result<T, String> loses structure, source, and downcasting.
• map_err(|_| ...) discarding the cause.
• Logging an error and also returning it, causing duplicate logs (or none at the top).
• unwrap() in request paths; one malformed input crashes a worker.
• Catch-all enum variant that swallows every error type (Other(String)).
• Using panics for validation; returning a Result is the contract.
• expect("this should never happen") without naming the invariant.
• Treating Option and Result interchangeably by smuggling None as an error with no context.

Checklist

• [ ] Libraries expose typed errors; applications use anyhow internally.
• [ ] Every ? conversion has a From impl or explicit map_err with context.
• [ ] Error messages name the operation and subject, contain no secrets, and are lowercase.
• [ ] No unwrap() in production paths; expect messages state invariants.
• [ ] Panics are impossible on parsed user input; FFI boundaries never unwind.
• [ ] One place per boundary maps errors to wire formats and logs once.
• [ ] RUST_BACKTRACE/chain printing is available in staging and production logs.
• [ ] Clippy unwrap_used/panic lints enabled with documented allowances.

Unsafe, Soundness, and FFI

Scope: when unsafe is justified, raw pointers and aliasing rules, Pin, building sound safe abstractions, Miri, C ABI and bindgen, pyo3/wasm-bindgen interop, and auditing unsafe code.

The Soundness Contract

• unsafe does not disable the borrow checker; it grants five extra powers:
  1. Dereference raw pointers.
  2. Call unsafe functions (including foreign functions).
  3. Access fields of a union.
  4. Mutate a mutable static (edition 2024 requires &raw mut for references; verify upstream).
  5. Implement an unsafe trait (unsafe impl Send/Sync).
• Soundness means: no safe code using this API can trigger undefined behavior, for all inputs and all interleavings, including other threads.
• UB examples: data races, dangling/unaligned pointers, out-of-bounds reads/writes, aliasing violations (two &mut to the same memory), invalid values for a type, calling a function with the wrong ABI.
• Rules: keep unsafe blocks minimal, document each with a // SAFETY: comment stating the invariant, and expose a safe wrapper whose type system prevents misuse.
• Enable unsafe_op_in_unsafe_fn (warn by default, deny in edition 2024): every unsafe operation gets its own explicit unsafe {} block inside unsafe fn.
• #[unsafe(no_mangle)]/#[unsafe(export_name = "...")] in edition 2024 make the risky part explicit; older editions accept bare attributes with a lint.

/// # Safety
/// `ptr` must be valid for reads of `len` initialized bytes and aligned.
unsafe fn checksum(ptr: *const u8, len: usize) -> u32 {
    assert!(!ptr.is_null());
    let bytes = unsafe { std::slice::from_raw_parts(ptr, len) };
    bytes.iter().fold(0u32, |acc, &b| acc.wrapping_add(b as u32))
}

Raw Pointers and Provenance

• *const T/*mut T carry provenance in the abstract machine (Stacked Borrows/Tree Borrows); casting through integers (ptr as usize) strips it and is only sound if provenance is restored correctly. Prefer ptr::with_exposed_provenance/expose_provenance (1.84+, verify upstream) over raw integer casts.
• Create with &raw const x / &raw mut x (stable since 1.82, verify upstream) or std::ptr::addr_of!/addr_of_mut! to avoid creating intermediate references.
• NonNull<T> is covariant, null-optimized, and documents non-null invariants; use it for internal pointers.
• .as_ref()/.as_mut() on pointers are unsafe and require validity; &*ptr needs alignment, non-null, dereferenceable, and initialized (unless the type permits uninit, e.g., MaybeUninit).
• Never derive references to uninitialized memory. Use MaybeUninit<T> and only assume_init after all fields are written.
• Prefer safe alternatives first: slices with get, split_at, Vec::spare_capacity_mut, bytemuck/zerocopy for casts, bytemuck::Pod for plain-old-data.

Aliasing quick rules

| Reference | Rule | |---|---| | &T | no writes anywhere in the referent for its lifetime; many may coexist | | &mut T | exclusive; no other &/&mut alias for its lifetime | | UnsafeCell<T> | the only legal path to interior mutation; raw access must be synchronized | | &Cell<T> | fine to mutate, but T: Copy and no reentrancy through the same &Cell | | reborrow &mut *p | creates a fresh exclusive borrow; the old one is invalid until it ends |

Miri detects most violations. Aliasing rules are not a finalized spec; write code that satisfies both Stacked Borrows and Tree Borrows where possible.

Pin and Self-Referential Types

• Pin<P> is a guarantee that the pointee will not move again, until it is dropped. Unpin types ignore the guarantee.
• Pin<&mut T>: safe &mut T access via get_mut() only if T: Unpin; otherwise unsafe { get_unchecked_mut() } with the structural-pinning invariant.
• Never move a !Unpin value after pinning. std::pin::pin! and Box::pin are safe constructors; Pin::new requires T: Unpin.
• Async blocks are !Unpin when they hold references across await; executors pin them in place.
• Structural pinning: if a wrapper exposes a field through Pin<&mut Field> and relies on field stability, it cannot implement Drop that moves the field, and unsafe projection code must uphold "never move" for that field. Use pin-project-lite/pin-project instead of hand-rolling projections.
• PhantomPinned makes a type !Unpin; it costs nothing.

use std::pin::{pin, Pin};
async fn demo() { let fut = pin!(some_async()); fut.await; }

Building Sound Safe Abstractions

• Pattern: a private owned buffer (Vec, Box<[MaybeUninit<T>]>) plus a safe API that maintains initialization and aliasing invariants; unsafe is confined to a few methods.
• Encapsulate: struct TinyVec<T> { buf: Vec<T>, len: usize } internals are private; public methods take &self/&mut self and uphold invariants.
• Use debug_assert! for invariants that safe callers cannot violate but bugs might.
• Drop correctly: partially initialized values require dropping only initialized elements (drop_in_place on slices), and panics during drop need guards (DropGuard/ManuallyDrop patterns).
• unsafe impl Send/Sync needs a proof: describe which fields, which synchronization makes cross-thread access safe, and why no aliasing rule is broken. If the proof needs more than three sentences, redesign.
• Panics must not cross an unsafe abstraction boundary in a half-updated state; restore invariants before propagating panics.

Miri

• cargo +nightly miri test interprets MIR and checks UB, leaks, data races, and invalid values. Most useful flags: -Zmiri-strict-provenance, -Zmiri-symbolic-alignment-check, -Zmiri-many-seeds, and Tree Borrows (MIRIFLAGS=-Zmiri-tree-borrows; verify defaults as they change).
• Limitations: no FFI (unless shims exist), only supported targets (host-like), much slower than native, and non-deterministic timing does not exist (races are modeled). Unsupported code produces "unsupported operation" errors rather than proof of soundness.
• Run Miri on the unsafe-heavy crate or module, not necessarily the whole workspace; use #[cfg(miri)] shims for syscalls and time.
• A clean Miri run is necessary but not sufficient: concurrency bugs under real schedulers still need loom (see ./08-testing.md) and stress tests.
• CI: nightly toolchain plus rustup component add miri, cargo miri setup, cache the sysroot.

C ABI and bindgen

• extern "C" is the stable interchange ABI. Use repr(C) for structs crossing the boundary and repr(transparent) for newtypes. Rust's default repr(Rust) layout is unspecified.
• Edition 2024: extern blocks require unsafe extern "C" { ... }; choose "C-unwind" when panics or exceptions may propagate across the boundary and that behavior is intended.
• Generation: bindgen (C headers to Rust) in a build.rs; cbindgen (Rust to C headers); keep generated files checked in or cached for reproducible builds.
• Ownership rules must be explicit in the header docs: who allocates, who frees, which allocator. Freeing a Rust allocation with free(3) (or vice versa) is UB with different allocators.
• Strings: CStr::from_ptr is unsafe and requires NUL-termination; CString::new rejects interior NUL. Use std::ffi::{c_char, c_int} rather than libc types where possible.
• Callbacks: never let unwinding cross extern "C" (abort on panic or catch and translate); document threading expectations; pass an opaque void* context back to Rust.
• Optional/error conventions: return status codes, use out-parameters for values, and translate errors into a stable C error enum. Never return Rust references that Rust may free later.
• Structs returned by value must be #[repr(C)] and contain only C-compatible fields; avoid bool (use u8) and Rust enums (use c_uint constants).
• Cargo: [lib] crate-type = ["cdylib", "staticlib"]; panic = "abort" in the FFI profile if unwinding cannot be contained.

#[repr(C)]
pub struct Span { pub ptr: *const u8, pub len: usize }

#[unsafe(no_mangle)]
pub extern "C" fn span_free(s: Span) {
    if !s.ptr.is_null() {
        // SAFETY: ptr/len came from Box<[u8]> with the same allocator in this library.
        drop(unsafe { Box::from_raw(std::ptr::slice_from_raw_parts_mut(s.ptr as *mut u8, s.len)) });
    }
}

Python and WebAssembly Interop

• pyo3 (0.2x line, verify upstream):
  • Prefer the Bound<'py, T> API; #[pyfunction]/#[pymodule]; Python::attach/with_gil has evolved — verify current naming upstream.
  • The GIL protects Python objects but not Rust data; do not block holding the GIL, release it for long Rust work (py.allow_threads or equivalent).
  • Map Result<T, E> to PyErr with From<E> for PyErr; keep error translation at the boundary.
  • Build with maturin; use abi3 wheels for forward compatibility with a version floor; profile release with lto = "thin".
  • Panics in #[pyfunction] are converted to Python exceptions by the runtime, but prefer returning Result.
• wasm-bindgen:
  • #[wasm_bindgen] for exported functions/structs; JsValue for JS interop, web_sys/js_sys bindings.
  • Wasm linear memory is not shared: JS strings are copied; use typed arrays (Uint8Array) for bulk data to avoid conversion overhead.
  • wasm32-unknown-unknown + wasm-bindgen-cli (version must match the crate; verify) or wasm-pack for bundling.
  • Panics abort the wasm instance by default; use console_error_panic_hook in dev for stack traces.
  • Test in Node/browsers with wasm-bindgen-test; keep JS glue generated, not hand-edited.

Auditing Unsafe Code

• Inventory: rg "unsafe", cargo geiger for dependencies; track unsafe surface per crate and require review for increases.
• Documentation: each public unsafe fn must have a # Safety doc section; every unsafe block a // SAFETY: comment referencing the invariant it upholds.
• Encapsulation audit: verify every public safe function keeps invariants (check constructor paths, Default, Clone, Deserialize derived impls — derive can smuggle invalid states past constructors).
• Fuzzing: cargo-fuzz targets for parsers and unsafe-heavy code paths; run under sanitizers (ASan/TSan) where Miri cannot (FFI).
• Sanitizers: -Zsanitizer=address / thread on nightly for FFI and cross-thread code; not a substitute for Miri's aliasing checks.
• Keep unsafe code in a dedicated module or crate with tests that exercise edge cases, alignment, zero-length slices, and null handling.
• Dependencies: cargo-deny/cargo-audit in CI; prefer well-maintained crates with documented safety posture for crypto and parsing.

Anti-Patterns

• unsafe to defeat the borrow checker for convenience.
• "I tested it once" as a soundness argument; no Miri, no fuzzing, no review.
• Returning raw pointers to stack data or references derived from temporaries.
• mem::transmute for type punning; prefer bytemuck, zerocopy, or explicit conversion.
• unsafe impl Send/Sync on a type containing Rc, RefCell, raw pointers, or interior mutability without synchronization.
• Freeing memory across allocator or language boundaries.
• Letting panics unwind across FFI boundaries.
• Multiple &mut derived from the same UnsafeCell without synchronization.
• Ignoring MaybeUninit initialization before assume_init.

Checklist

• [ ] Every unsafe block has a // SAFETY: comment and a test that would fail on violation.
• [ ] unsafe is minimized and wrapped in a safe API whose misuse cannot compile.
• [ ] Miri passes for unsafe modules; tree-borrows mode checked.
• [ ] FFI types are #[repr(C)]/#[repr(transparent)], ABI (including unwind) is explicit.
• [ ] Ownership and allocator rules documented for both sides of FFI.
• [ ] unsafe impl Send/Sync has a written proof and required synchronization.
• [ ] Pinned types never move after pinning; projections use a vetted crate.
• [ ] CI tracks unsafe surface (cargo geiger) and denies increases without review.

Web Services and Data Layer

Scope: axum versus actix-web, tower middleware, serde, database access (sqlx/diesel/sea-orm), migrations, configuration, auth patterns, and graceful shutdown.

Framework Decision Table

| Criterion | axum | actix-web | |---|---|---| | Ecosystem | tokio + tower + hyper, broad middleware reuse | custom runtime (actix-rt), mature plugin set | | Extractors | trait-based, composable, typed | custom extractors via FromRequest | | Middleware | tower::Layer (tower-http, tower limits, tracing) | Transform/Service, actix-specific | | Learning curve | moderate, explicit types | moderate, macros (#[get("/")]) | | WebSocket/gRPC | axum + tokio-tungstenite, tonic in the same server | actix actors, actix-ws; tonic separate | | Performance | high (hyper-based) | high; benchmarks are workload-dependent — measure, do not trust microbenchmarks | | Best fit | services already on tokio/tower; library-friendly | teams already on actix; heavy actor-style state |

Default for new projects in 2026: axum, because middleware, HTTP clients, gRPC (tonic), and tracing all share the tower/hyper stack. Choose actix-web when the team knows it and the application fits its model; do not mix both in one service.

axum Essentials

• Router composition: Router::new().route("/x", get(h).post(h2)).nest("/api", api).layer(TraceLayer::new_for_http()).
• State: Router::with_state(state); extract with State<T> where T: Clone + Send + Sync + 'static. Wrap in Arc when non-trivial; Arc the whole state once rather than per-field.
• Extractors run in order: body-consuming extractors (Json, Form) must be last. Extractors that fail return non-2xx by default via IntoResponse.
• Handlers return impl IntoResponse; use Result<Json<T>, AppError> and one IntoResponse for AppError mapping.
• Path/query: Path<(u64, String)>, Path<Struct>, Query<Pagination> with serde Deserialize.
• Custom extractor: implement FromRequestParts (no body) or FromRequest (consumes body) for auth/api-key.
• Sharing HTTP clients: reqwest::Client is Clone and pools connections; build once in state with timeouts.
• axum 0.7/0.8 era changed path syntax (/{id} instead of /:id) and removed async_trait requirements; verify exact API upstream.

#[derive(Clone)]
struct AppState { db: PgPool, http: reqwest::Client }

async fn get_user(State(s): State<AppState>, Path(id): Path<i64>) -> Result<Json<User>, AppError> {
    let user = queries::user_by_id(&s.db, id).await?.ok_or(AppError::NotFound)?;
    Ok(Json(user))
}

tower Middleware

• Layers wrap services; order matters: outermost runs first on request, last on response. ServiceBuilder::new().layer(A).layer(B).service(svc) results in A(B(svc)).
• Recommended stack order (outer to inner): request id -> tracing -> timeout -> concurrency limit -> compression -> catch panic.
• tower-http: TraceLayer, TimeoutLayer, CompressionLayer, CorsLayer, RequestIdLayer, SetRequestIdLayer, RequestBodyLimitLayer, ServeDir/ServeFile.
• tower::limit::ConcurrencyLimitLayer and tower::load_shed::LoadShedLayer protect the service; combine with tower::buffer::Buffer carefully (buffer adds a task and queue).
• Rate limiting: tower-governor or custom keyed limiter; token buckets need shared state (governor).
• ServiceBuilder vs Router::layer: Router::layer applies to routes added before it; route_layer only runs on matched routes (so 404s skip auth). Use route_layer for auth.
• Middleware errors must map to responses; a layer returning a body must set content type and length consistently.
• Test layers by calling router.oneshot(request) (tower ServiceExt).

Serde

• #[derive(Serialize, Deserialize)]; #[serde(rename_all = "camelCase")], #[serde(deny_unknown_fields)] for strict external input, #[serde(default)] for optional fields.
• skip_serializing_if = "Option::is_none" for sparse payloads; flatten sparingly (breaks deny_unknown_fields and can hide errors).
• Use serde_with for base64, hex, timestamps, and duration formats; chrono/time serializers must be explicit.
• Custom types: implement Serialize/Deserialize for newtypes to validate at parse time (TryFrom inside Deserialize).
• JSON: serde_json is the baseline. simd-json/sonic-rs accelerate hot paths with compatibility caveats (verify upstream); profile before switching.
• Enum representations: #[serde(tag = "type", rename_all = "snake_case")] internally tagged is common; untagged is last resort (poor errors, quadratic in variants).
• Do not use serde_json::Value for domain data; parse into typed structs at the boundary.
• Version payloads explicitly (#[serde(rename = "...")] legacy aliases) instead of relying on field-order.

Database Access

| Criterion | sqlx | diesel (+ diesel-async) | sea-orm | |---|---|---|---| | Style | async, SQL-first, compile-time checked macros | typed query builder + async in diesel-async | async ORM over sea-query | | Compile-time SQL check | yes (query! against live DB or offline .sqlx cache) | yes via schema types | no (runtime), but strong typing | | Migrations | sqlx migrate | diesel migration | sea-orm-cli migrate | | Best fit | existing SQL, precise queries, Postgres/MySQL/SQLite | teams wanting ORM-like safety with Rust types | entity modeling, CRUD-heavy apps | | Async | native | requires diesel-async | native |

• Pooling: sqlx::postgres::PgPoolOptions — set max_connections from DB limits and concurrency budget, acquire_timeout, max_lifetime, idle_timeout. Never max_connections = number of tasks.
• Transactions: let mut tx = pool.begin().await?; ...; tx.commit().await?; — dropping rolls back. Hold transactions short; never across external HTTP calls.
• sqlx compile-time macros need DATABASE_URL at build time or cargo sqlx prepare (SQLX_OFFLINE=true in CI). Commit .sqlx/ for reproducible builds.
• Use query_as!/FromRow for row mapping; add #[sqlx(rename = "...")] for column mismatches; prefer explicit SELECT columns over SELECT *.
• Migrations: forward-only, immutable once applied; additive changes (new nullable column, backfill, then constraint) to allow rolling deploys. Never edit an applied migration.
• N+1 detection: log query counts in dev; batch with WHERE id = ANY($1) or INSERT ... ON CONFLICT.
• SQLite for embedded/small services with WAL mode and a single writer or a connection-pool bound of 1 for writes; libsql/turso for remote SQLite if needed.
• SeaORM entities generated from schema drift; re-run generation in CI to catch drift.
• Connection health: set test_before_acquire/before_acquire hooks; handle transient failures with retries at the repository layer.

let rows = sqlx::query_as!(User,
    "SELECT id, email, created_at FROM users WHERE id = $1", id)
    .fetch_optional(&pool).await?;

Configuration

• Precedence: defaults < config file < environment < CLI flags. Document it once and test each layer.
• figment (layered providers: TOML + env) or the simpler config crate; envy for env-to-struct only. dotenvy in dev only — never load .env in production.
• Parse config into a typed struct at startup with serde; fail fast with context (anyhow with file/key names). No std::env::var scattered through the codebase.
• Secrets: inject via environment or a secret manager (Vault, AWS Secrets Manager, SOPS); never commit them, never log them. Use secrecy::SecretString/zeroize for sensitive fields.
• Validate config ranges at startup (port, pool size > 0, URL schemes); reject unknown keys in config files.
• Feature flags/tenancy config that changes at runtime belongs in a watch channel or the DB, not OnceLock reads.

Authentication and Authorization Patterns

• Password hashing: Argon2id (argon2 crate) with sane parameters (memory >= 19 MiB, iterations >= 2, parallelism 1 as OWASP baseline; verify current recommendations upstream). Never SHA/MD5, never unsalted.
• Sessions: opaque random tokens stored server-side (hashed) with expiry and rotation; or signed cookies. Cookies: HttpOnly, Secure, SameSite=Lax/Strict, __Host- prefix.
• JWT (jsonwebtoken): short-lived access tokens (5-15 min), refresh tokens server-side revocable, validate alg explicitly (never none/unknown), aud, iss, exp, nbf. Public key set rotation via JWKS; cache keys.
• API keys: store a hash, show once, prefix for identification, per-key scopes and rate limits.
• Authorization: enforce in the service layer (fn can(user, action, resource) -> bool), not only in middleware; middleware handles authentication, handlers enforce per-resource access. Never trust client-supplied tenant/user ids.
• OAuth/OIDC: use openidconnect crate or a gateway; implement PKCE; validate state and nonce.
• Rate-limit auth endpoints and return uniform errors/timings to avoid user enumeration.
• mTLS/service auth inside the mesh or with signed service tokens; do not rely on network location.

Graceful Shutdown

• Sequence: receive signal (SIGTERM/SIGINT) -> stop accepting -> drain in-flight requests with deadline -> close DB pools -> flush telemetry -> exit non-zero on timeout.
• axum: axum::serve(listener, app).with_graceful_shutdown(shutdown_signal()). shutdown_signal awaits tokio::signal::ctrl_c() and unix SIGTERM.
• Background tasks: CancellationToken + TaskTracker; cancel after HTTP drain or in parallel depending on semantics (e.g., stop consumers after in-flight writes complete).
• Readiness: flip to not-ready before drain so load balancers stop routing; keep liveness green until exit.
• Pools: pool.close().await after handlers stop.
• Timeouts: hard deadline (e.g., 30s) then abort; log remaining tasks. Kubernetes terminationGracePeriodSeconds must exceed the deadline.
• Deserialization/queue workers: finish the current message, then cancel; make handlers idempotent for redelivery.

let token = CancellationToken::new();
let mut app = app.with_state(state);
axum::serve(listener, app)
    .with_graceful_shutdown(shutdown_signal(token.clone()))
    .await?;
token.cancel();
tracker.close().await;

Anti-Patterns

• Business logic in handlers; handlers should parse, call a service, map the result.
• Connection pool per request or unbounded pool size.
• Letting serde_json::Value leak into domain code.
• SELECT * plus struct field order assumptions.
• Middleware ordering that runs auth after body parsing or skips 404s unintentionally.
• JWT alg accepted from the token header; missing audience/issuer validation.
• Editing applied migrations; mixing schema DDL into request handlers.
• Blocking shutdown: dropping the future before drain; no deadline.
• Storing secrets in config.toml committed to the repo.

Checklist

• [ ] Framework chosen deliberately; one HTTP stack in the service.
• [ ] State is Clone + Send + Sync; no per-request clients.
• [ ] tower layers ordered (trace -> timeout -> limits -> body) and tested.
• [ ] All external input parsed into typed structs with deny_unknown_fields where strictness helps.
• [ ] Pool sizes and acquire timeouts configured from DB limits; queries prepared/offline-checked.
• [ ] Migrations forward-only, applied in CI before deploy; rollback plan documented as a new migration.
• [ ] Authn at middleware, authz in services; tokens short-lived and revocable.
• [ ] Graceful shutdown drains with a deadline and closes pools; readiness flips first.
• [ ] Config validated at startup; secrets from environment/secret manager only.

Embedded and no_std

Scope: no_std fundamentals, embedded-hal traits, the embassy async ecosystem, defmt/probe-rs debugging, memory constraints, and target/build configuration.

no_std Fundamentals

• #![no_std] links core (and optionally alloc); you lose std types, std::io, panics-with-unwind, and the default global allocator.
• #![no_main] plus a runtime crate's entry macro (#[cortex_m_rt::entry], #[esp_hal::main], #[embassy_executor::main]) replaces the standard start code.
• Every binary needs a panic handler and (on bare metal) a memory layout. Pick:
  • panic-halt: stop execution; smallest.
  • panic-probe (defmt): prints the panic over RTT; best for debugging.
  • panic-reset / custom #[panic_handler].
• default-features = false everywhere in the dependency tree; verify each crate supports no_std (many do behind a std feature).
• alloc needs a #[global_allocator]; on MCUs prefer embedded-alloc or linked_list_allocator, but treat dynamic allocation as exceptional.
• core::fmt is available but pulls formatting code (~10s of KiB); avoid format! in hot or size-critical paths.
• Replace std tooling with core equivalents: core::fmt::Write, core::cell::RefCell/Cell, core::sync::atomic, spin::Mutex or critical-section for locks.

#![no_std]
#![no_main]
use panic_probe as _;
use cortex_m_rt::entry;

#[entry]
fn main() -> ! {
    loop { core::hint::spin_loop(); }
}

Target and Build Configuration

• Common targets: thumbv6m-none-eabi (Cortex-M0), thumbv7em-none-eabihf (M4F/M7), thumbv8m.main-none-eabihf (M33+), riscv32imac-unknown-none-elf, riscv32imc-unknown-none-elf (ESP32-C3), xtensa-esp32-none-elf.
• Pin the toolchain in rust-toolchain.toml with the target and components; commit it.
• build-std (rebuilding core/alloc) is still nightly-only in practice; use it for custom targets or when a dependency requires it. Verify upstream for stable support.

# .cargo/config.toml
[build]
target = "thumbv7em-none-eabihf"
rustflags = ["-C", "link-arg=-Tlink.x"]

[target.thumbv7em-none-eabihf]
runner = "probe-rs run --chip STM32F411CEUx"

[env]
DEFMT_LOG = "info"

• memory.x declares FLASH/RAM origins and lengths for cortex-m-rt; a wrong value silently corrupts links. Custom memory layouts need matching linker scripts and stack placement.
• build.rs copies memory.x into OUT_DIR (typical cortex-m-rt boilerplate) or sets link search paths.
• Probe/runner setup: probe-rs run, OpenOCD, or vendor tools; cargo embed wraps flashing + RTT.
• For RP2040/ESP, use the vendor HAL's build flow (embassy-rp, esp-hal + espflash); bootloader and partition concerns differ from Cortex-M.
• Size checks in CI: cargo size --release -- -A, cargo bloat, cargo nm for symbol sizes; fail on regressions.

embedded-hal (1.0 line)

• Traits are the portability layer between drivers and HALs. Implement HAL traits; write drivers against traits.
• Core traits (verify exact signatures upstream for the 1.0 line):
  • OutputPin (set_high, set_low), InputPin (is_high).
  • SpiDevice/SpiBus (transaction-based), I2c (write, read, write_read).
  • DelayNs (delay_ns/delay_ms/delay_us), Pwm, Adc, Serial.
• Async twins in embedded-hal-async (SpiDevice, I2c, DelayNs as Future) enable interrupt-driven drivers without blocking.
• embedded-io/embedded-io-async for byte-stream abstractions; embedded-can, embedded-graphics, embedded-nal for other domains.
• Driver crates should be generic over traits (pub struct Mpu6050<I: I2c> { i2c: I }) and return Result<_, I::Error> or a driver-defined error.
• Do not implement embedded-hal traits for types whose errors are unrepresentable; use Infallible where no failure exists.

use embedded_hal::digital::OutputPin;
use embedded_hal::delay::DelayNs;

fn blink<P: OutputPin, D: DelayNs>(pin: &mut P, delay: &mut D) {
    loop {
        let _ = pin.set_high();
        delay.delay_ms(500);
        let _ = pin.set_low();
        delay.delay_ms(500);
    }
}

Async Embedded (embassy and friends)

• embassy-executor is the de-facto async executor: cooperative, interrupt-driven wakers, static task allocation via #[embassy_executor::task] and Spawner.
• embassy-time provides timers (Timer::after, Ticker); configure a time driver from the HAL.
• embassy-sync mirrors async sync primitives for no_std: Mutex, Signal, Channel, Watch, PubSubChannel.
• embassy-stm32/embassy-nrf/embassy-rp combine HAL + interrupt integration; esp-hal has its own async executor support.
• Tasks are statically allocated with pool sizes; the executor is a future scheduler, not a preemptive RTOS. Reentrancy safety comes from interrupt priorities and critical sections.
• Interrupt handlers must be cooperative: no blocking waits, minimal work, wake a task instead.
• Cancellation and drop semantics match std async: dropping a future cancels at await points. See ./03-async-concurrency.md for the mental model.
• Priorities: hardware interrupts preempt tasks; use critical-section to protect shared state shared with ISRs.

#[embassy_executor::task]
async fn heartbeat(mut led: Output<'static>) {
    loop {
        led.toggle();
        embassy_time::Timer::after_millis(500).await;
    }
}

Memory Constraints

• RAM is measured in tens/hundreds of KiB; stack overflow is silent corruption, not a clean panic. Estimate the deepest call chain and async state sizes (core::mem::size_of_val on futures at compile time via tests).
• Static allocation first: static buffers, heapless::{Vec, String, Deque, Map} with fixed capacities.
• If using alloc: single-threaded allocator, set aside a fixed arena, and either make allocation failure fatal or avoid it in steady state.
• Watch sizes: size_of on futures, cargo bloat, nm --size-sort; a single format! can add kilobytes.
• defmt strings live in flash, not RAM; still avoid formatting in hot loops.
• Use #[inline] selectively; flash is finite too. opt-level = "s" or "z", lto = true, codegen-units = 1, panic = "abort" for release.
• Stack placement and interrupt stack: on some targets interrupts use the main stack; on others (e.g., RP2040) there are per-core stacks. Know your chip.
• Data in flash: static immutable data, #[link_section = ".rodata"]; DMA buffers must be in RAM and often aligned.
• Avoid recursion; use explicit state machines or heapless::Vec-based stacks.

Debugging and Logging (defmt + probe-rs)

• defmt compresses log strings into flash and sends binary frames over RTT/SWO; far cheaper than log+UART.
• Setup: defmt = "0.3" (verify current line), defmt-rtt transport, panic-probe with the defmt feature, and DEFMT_LOG=info in .cargo/config.toml.
• defmt::info!, warn!, error!, debug!, trace!, assert!; derive defmt::Format for custom types (or use {=[u8]:a} hints for slices).
• Global logger: depend on defmt-rtt; only one global logger per firmware.
• probe-rs reads RTT: probe-rs run --chip <CHIP>, probe-rs attach, probe-rs gdb; defmt-print decodes captured streams.
• Timestamps/sequences: defmt supports timestamping (elapsed time) when the HAL provides a timer; verify integration for your HAL.
• Alternatives: rtt-target for plain RTT, UART logging when a probe is unavailable (costly), tracing on host-emulated targets.
• Debugging: probe-rs gdb or arm-none-eabi-gdb plus openocd; hardware breakpoints are limited (typically 2-6); flash breakpoints can be used.
• embedded-test runs unit tests on hardware; keep a #[cfg(test)] harness that doesn't require std::test.

One Professional Embedded Crate

firmware/
  Cargo.toml
  build.rs            # memory.x
  memory.x
  .cargo/config.toml  # target, runner, defmt env
  rust-toolchain.toml # pinned nightly/channel + target
  src/main.rs         # entry, tasks
  src/net.rs          # ...
crates/
  sensor-driver/      # no_std driver over embedded-hal traits

• Split hardware-independent logic (state machines, protocols) into portable crates tested on the host; keep HAL coupling in the firmware crate.
• Run host tests on the portable crates; hardware tests stay minimal and manual or on a bench rig.

Anti-Patterns

• std-only crates pulled in by accident (often via default features).
• Overflowing the stack silently by recursion or deep async chains.
• Blocking Delay inside async tasks (embassy-time or embedded-hal-async instead).
• Using a heap in interrupt context or from multiple cores without synchronization.
• Ignoring critical-section around shared state accessed by ISRs.
• Shipping with panic-halt and no boot-time watchdog when field resets are required.
• Building for the wrong target/features and only noticing a crash in the field.
• Logging through a blocking UART in a timing-critical loop.

Checklist

• [ ] no_std gate verified (cargo check --target <t>) in CI; no accidental std dependency.
• [ ] Panic handler chosen and documented; watchdog strategy defined for field devices.
• [ ] Drivers written against embedded-hal traits; async twins where appropriate.
• [ ] Memory budget measured (.bss/.data/.text, stack high-water) and enforced in CI.
• [ ] defmt used with RTT; log level configurable per build.
• [ ] Shared state with ISRs protected (critical-section or atomics with correct ordering).
• [ ] Toolchain, target, and runner pinned in-repo.
• [ ] Portable logic host-tested; hardware tests documented.

Testing

Scope: unit/integration/doc tests, property testing, criterion benchmarks, cargo-nextest, Miri, loom, snapshot testing, fuzzing, coverage, and test doubles.

Test Taxonomy

| Kind | Location | Compiles with crate? | Access | Use for | |---|---|---|---|---| | Unit | src/**/tests.rs under #[cfg(test)] mod tests | yes | private items | internal invariants, edge cases | | Integration | tests/*.rs | links public API | public only | public contract, end-to-end of a crate | | Doc test | ```rust in doc comments | compiled as separate crate | public, with extern crate | API examples that must keep compiling | | Example | examples/*.rs | built by cargo test/cargo build --examples | public | usage demos, smoke binaries | | Benchmark | benches/*.rs (criterion) | yes | varies | perf regression detection | | Fuzz | fuzz/fuzz_targets/*.rs | separate, cargo-fuzz | varies | parsers, untrusted input, unsafe | | Property | wherever | yes | varies | invariants over generated input |

• Unit tests may use private APIs; integration tests must not. If an integration test needs internals, extract a testable module or add a #[doc(hidden)] pub seam.
• #[cfg(test)] code is not compiled for doc tests or downstream users — do not depend on it from pub generics.
• Keep tests deterministic: no real clock/network/random; inject Clock, Rng, and clients.

Writing Useful Tests

• Name tests after behavior: rejects_expired_token, not test_validate_2.
• One logical assertion per test where practical; use table tests for many cases.
• Test the contract, not the implementation: public API behavior, invariants, error variants, and boundary values (empty, max, off-by-one, unicode, zero-length).
• Assert on error types/variants and messages that are part of the contract; use matches! or assert_matches to avoid brittle formatting.
• Use #[should_panic(expected = "...")] only for genuinely panicking contracts (or remove the panic and return Result instead).
• #[ignore] marks tests that need external resources; run them explicitly in CI with --ignored.
• #[cfg_attr(miri, ignore)] for tests Miri cannot execute (FFI, filesystem unsupported ops).

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn parses_empty_input_as_none() {
        assert_eq!(parse(""), None);
    }

    #[test]
    fn rejects_oversized_frame() {
        assert!(matches!(decode(&[0xFF; 64]), Err(DecodeError::TooLarge { .. })));
    }
}

Doc Tests

• Doc tests are executed by cargo test; they are the best regression net for README-level examples.
• Attributes: no_run (compile but do not run), ignore (skip; avoid), compile_fail (must not compile), should_panic, text.
• Hidden setup lines start with # and are compiled but not rendered — use for fn main, imports, or unwraps.
• Mark non-runnable examples (no_run) when they need a server/DB; still compile them in CI.
• cargo test --doc isolates runtime; rustdoc linkcheck is not a thing — verify intra-doc links with RUSTDOCFLAGS="-D warnings" and cargo doc.
• Set [lib] doctest = false only when doctests are impossible (e.g., no_std crates); otherwise keep them on.
• Add #![doc = include_str!("../README.md")] to test README examples.

Property Testing

• proptest (or quickcheck): generate inputs, shrink failures to minimal counterexamples.
• Define strategies near the type: fn arb_email() -> impl Strategy<Value = Email>; compose with prop_map, prop_flat_map.
• Assert properties, not examples: round-trip (decode(encode(x)) == x), idempotence, commutativity, invariants (len() <= capacity), and no-panic.
• Persist regressions: proptest writes failures to proptest-regressions/; commit those files.
• Bound the search space (prop_filter sparingly, prop_assume for preconditions) and set PROPTEST_CASES in CI for longer runs.
• Combine with fuzzing: proptest for structured inputs, cargo-fuzz for raw bytes and long campaigns.

use proptest::prelude::*;

proptest! {
    #[test]
    fn roundtrip(input in any::<Vec<u8>>()) {
        let frame = encode(&input);
        prop_assert_eq!(decode(&frame).unwrap(), input);
    }
}

Continuous Benchmarking (criterion)

• criterion measures wall time with statistical analysis: warmup, sample size, outliers, and change detection against a baseline.
• Always black_box inputs and outputs so the optimizer cannot elide work.
• Benchmark at the right granularity: micro (function), meso (request handling), and end-to-end (process) — only micro-benchmarks in-repo by default.
• cargo bench compares to the previous saved baseline; use --save-baseline/--baseline and store target/criterion in CI artifacts.
• Throughput::Bytes/Elements reports throughput instead of raw time for IO/parsing benchmarks.
• For instruction-count benchmarks (immune to CI noise), use iai-callgrind (Valgrind-based) — verify the current crate/function names upstream.
• Do not gate PRs on wall-time microbenchmarks without noise controls (dedicated runner, instruction counts, or median of many runs).
• cargo-flamegraph and perf for finding what to benchmark (see ./10-performance-security-observability.md).

cargo-nextest

• Nextest runs each test in its own process, defaulting to parallelism with per-test timeouts; output is clearer and faster than cargo test for large suites.
• Config nextest.toml: profiles (default, ci), retries for known-flaky tests (retries = { backoff = "exponential", count = 2 }), test groups for serial tests.
• --partition count:1/4 splits suites across CI shards; --profile ci adds retries and strict output.
• cargo nextest run --all-features --workspace; doc tests still run via cargo test --doc (nextest does not run them).
• cargo nextest list for auditing; --no-fail-fast to see all failures.
• JUnit output: --profile ci --message-format libtest-json piped through cargo-nextest tooling or nextest-rs converters; many CIs accept nextest's native JUnit.

Miri and Loom

• Miri catches UB in unsafe code and some logic errors (invalid values, leaks, races) by interpreting MIR. See ./05-unsafe-ffi.md for flags and setup.
• Run Miri on unsafe modules with strict provenance; expect it to be 10-100x slower — keep the Miri test set focused.
• Loom model-checks concurrent code by exploring interleavings under a configurable bound (LOOM_MAX_PREEMPTIONS).
  • Gate code on #[cfg(loom)] use loom::sync::{Mutex, Arc}; and mirror the types in a small sync module.
  • Test: loom::model(|| { ... }); exhaustively explores schedules for small thread counts.
  • Loom substitutes atomics and Arc; loom::thread::spawn instead of std::thread.
  • Loom does not model tokio task scheduling; test the synchronous primitives (locks, queues) under loom, then test task orchestration separately.
• Sanitizers (ASan/TSan on nightly or -Zsanitizer) complement Miri where FFI/unsupported ops block it.

Snapshots

• insta for snapshot assertions: assert_snapshot!, assert_debug_snapshot!, assert_json_snapshot!.
• Workflow: run tests -> pending .snap.new files -> cargo insta review or cargo insta accept; commit .snap files.
• Use inline snapshots (assert_snapshot!(value, @"...")) for small, local expectations; file snapshots for larger output.
• Redact nondeterminism (timestamps, ids, paths) with insta::with_settings!({filters => ...}) rather than post-processing.
• Snapshot tests are excellent for API responses, CLI output, error messages, and generated code; they are poor for behavior assertions (they fail on any formatting change).
• expect-test is the lighter alternative for inline expected strings.

Fuzzing

• cargo-fuzz (libFuzzer) is the baseline: cargo fuzz init, cargo fuzz add target, cargo fuzz run target.
• Feed &[u8] through arbitrary to structured types where a grammar helps; keep a raw-bytes target for parser robustness.
• Corpus management: commit seed corpus, download OSS-Fuzz corpus in CI; crashes are written to artifacts/ — turn each into a regression test.
• Run short fuzz jobs on PRs (time-boxed) and long campaigns nightly; use -max_total_time=..., -rss_limit_mb, -jobs/-workers.
• For integration with Rust test harnesses, cargo afl/afl.rs is an alternative; verify maintenance status upstream.
• Fuzz unsafe code under ASan (cargo fuzz run uses sanitizers by default on supported targets).

Coverage

• cargo-llvm-cov is the current standard: cargo llvm-cov --workspace --all-features --html, producing source-based coverage.
• Branch coverage: cargo llvm-cov --branch on modern LLVM; line-only coverage can hide untested match arms.
• CI: upload lcov/cobertura artifacts; gate on coverage of changed lines rather than a global percentage.
• grcov is an older alternative; both are acceptable — pick one and keep it stable.
• Coverage of tests themselves is not useful; exclude test modules and generated files via --ignore-filename-regex.
• Coverage cannot prove correctness; combine with property tests and Miri for high-risk code.

Test Doubles and Integration Fixtures

| Need | Tool | |---|---| | Mock traits | mockall (auto-mock with #[automock]), hand-written fakes | | HTTP clients | wiremock, httpmock, mockito | | HTTP servers | axum router + tower::ServiceExt::oneshot (no port needed) | | Databases | testcontainers, sqlx::test (per-test DB), pg_embed/libsql for SQLite | | Time | tokio::time::pause, injected Clock trait, time fakes | | Randomness | seeded rand::rngs::StdRng, injected Rng | | Filesystem | tempfile::TempDir | | Env vars | temp-env or a config struct built in tests (never mutate process env in parallel tests) |

• Prefer hand-written fakes over generated mocks; they exercise behavior and stay readable.
• sqlx::test creates an isolated database per test when given a database URL; migrations applied automatically (verify behavior upstream).
• Testcontainers require Docker in CI; keep a fast unit suite that runs without Docker and a slower integration suite that can be skipped.
• Avoid global mutable state between tests; nextest's process-per-test makes leaks less likely but not harmless.

Anti-Patterns

• Tests coupled to private implementation details; refactors break tests without behavior changes.
• sleep to synchronize: use channels, barriers, or paused time.
• Testing only happy paths and no error variants.
• Snapshot tests accepted blindly via cargo insta accept without review.
• Gating on flaky wall-clock microbenchmarks.
• Ignoring Miri/loom results because "it's probably fine".
• A single giant integration test that exercises everything and fails with no localization.
• Tests that require network access or a shared database instance.

Checklist

• [ ] Public APIs have unit and integration coverage; error variants asserted.
• [ ] Doc examples compile and run (or are no_run with a reason).
• [ ] Properties tested with proptest for parsers/codecs/state machines.
• [ ] Benchmarks run in a stable environment and track regressions.
• [ ] cargo-nextest in CI with timeouts; flaky tests quarantined with retries, then fixed.
• [ ] Miri on unsafe modules; loom on lock/queue implementations.
• [ ] Fuzz targets for every untrusted-input parser; corpora committed.
• [ ] Coverage tracked per changed lines, not as a vanity metric.
• [ ] Integration fixtures are hermetic (containers/temp dirs), no shared external state.

Tooling and Build

Scope: cargo workspaces, feature flags and unification, profiles, task runners, lints, cross-compilation, MSRV, release automation, and supply-chain hygiene.

Workspaces

• One workspace for related crates: a root Cargo.toml with [workspace] members = ["crates/*"], one shared Cargo.lock, one target/.
• Resolver: edition 2024 defaults to resolver v3 (MSRV-aware feature/dependency resolution); edition 2021 defaults to v2. Set [workspace] resolver = "3" explicitly if members differ. Verify upstream for resolver behavior changes.
• [workspace.dependencies] centralizes versions; members use dep = { workspace = true } plus local features.
• [workspace.package] centralizes version, edition, license, repository, rust-version.
• [workspace.lints] (stabilized 1.74) with [lints] workspace = true per crate is the standard lint policy mechanism.
• Keep Cargo.lock committed for binaries and applications; libraries generally commit it too for reproducible CI (policy choice — document it).
• cargo metadata --format-version 1 for tooling; cargo tree -d to find duplicate versions; cargo tree -e features -i serde to see who enables features.
• Publishing order matters; cargo-release, release-plz, or cargo-workspaces automate version bumps and ordered publishes.

[workspace]
members = ["crates/*"]
resolver = "3"

[workspace.package]
edition = "2024"
rust-version = "1.85"
license = "MIT OR Apache-2.0"

[workspace.dependencies]
serde = { version = "1", features = ["derive"] }
tokio = { version = "1", default-features = false }
thiserror = "2"

[workspace.lints.rust]
unsafe_op_in_unsafe_fn = "deny"

[workspace.lints.clippy]
unwrap_used = "deny"

Feature Flags

• Features are additive and must never be mutually exclusive; enabling a feature must not disable another. This rule is non-negotiable for published crates.
• Feature unification: with a shared lockfile, features enabled by any member are visible to all members using that dependency. Resolver v2 avoids unifying build-dependencies/proc-macros/target-specific deps; v3 additionally prefers MSRV-compatible versions.
• Use dep: syntax to create features that enable optional dependencies without exposing implicit features (serde = { version = "1", optional = true } plus derive = ["dep:serde"]). cargo no longer creates implicit features for optional deps when dep: is used (verify upstream).
• Default features should be minimal and safe. default-features = false in libraries that want to opt in.
• Feature design table:

| Kind | Naming | Example | |---|---|---| | Additive capability | noun | tls, http2, json | | Backend selection | backend-<name> | backend-postgres | | Runtime selection | rt-<name> | rt-tokio | | Unstable/experimental | unstable-<x> | unstable-gats | | Test-only helpers | _test-util (underscore = internal) | _test-util |

• Test feature combinations with cargo hack --feature-powerset --depth 2 test (feature powerset can explode; limit depth) or a targeted cargo-hack job per combination.
• cargo hack check --each-feature --workspace catches features that do not compile alone.
• docs.rs: [package.metadata.docs.rs] all-features = true plus rustdoc-args = ["--cfg", "docsrs"]; crates conditionally document features with #[cfg_attr(docsrs, doc(cfg(feature = "...")))].
• Optional dependencies that create feature unification hazards (e.g., a runtime) should be kept small and never enabled transitively.

Profiles

| Profile | Defaults | Common overrides | |---|---|---| | dev | opt-level=0, debug info, overflow-checks=true | add [profile.dev.package."*"] opt-level = 2 to speed deps | | release | opt-level=3, no debug info, lto=false, codegen-units=16 | lto="thin", codegen-units=1, debug=1, strip="debuginfo" | | test | dev-like; cargo test --release uses bench? no, use --profile | [profile.test] opt-level=1 for faster integration tests | | bench | release-like | debug=true, lto=true for accurate flamegraphs |

[profile.release]
lto = "thin"
codegen-units = 1
panic = "abort"      # only if no unwind boundary is required
strip = "debuginfo"  # keep symbols for profiling builds via a custom profile
overflow-checks = true

[profile.profiling]
inherits = "release"
debug = true
strip = "none"

• Custom profiles inherit from dev or release; use them for profiling, fuzzing, and sanitizer builds.
• panic = "abort" reduces size and removes unwinding; it also makes catch_unwind ineffective and changes FFI panic behavior. Decide per binary.
• build-override for build scripts and proc-macros: [profile.release.build-override] opt-level = 3.
• Overriding dependency optimization ([profile.release.package."*"]) trades build time for runtime.
• -Zbuild-std for custom targets stays nightly-gated in practice; verify upstream.
• Debug symbols in release: debug = "line-tables-only" (or 1) balances symbol size and actionable profiles; strip = "symbols" for shipping binaries.

Build Acceleration and Caching

• sccache: RUSTC_WRAPPER=sccache; configure a local or remote (S3/GCS) backend in CI.
• Linkers: mold or lld cut link time substantially on Linux: RUSTFLAGS="-C link-arg=-fuse-ld=mold" (or rustflags in .cargo/config.toml).
• cranelift backend (-Zcodegen-backend=cranelift) speeds debug builds on nightly; not for release artifacts.
• cargo build --timings and cargo nextest reports for build hotspots.
• Docker: cargo-chef to cache dependency compilation in a separate layer; build with --locked.
• Separate target dirs per profile/target in CI caches; cache keyed by Cargo.lock hash and toolchain.
• cargo build --workspace --all-targets once before parallel test jobs avoids recompilation across jobs.
• cargo udeps (nightly) and cargo machete to remove unused dependencies; both reduce compile time.

Task Runners

| Approach | Pros | Cons | |---|---|---| | just | simple recipes, shell-friendly, widely installed | no Rust integration | | cargo-make | TOML tasks, cross-platform, conditionals | verbose; plugin surface | | xtask | plain Rust, typed, debuggable, no extra config | requires a workspace crate |

• xtask is the preferred approach for complex automation (codegen, bundling, release steps): a crates/xtask binary invoked as cargo run -p xtask -- <cmd> or aliased in .cargo/config.toml.
• Keep a thin justfile/Makefile as the discoverable entry point calling cargo/cargo xtask.
• CI should call the same commands developers run; no divergent CI-only scripts where avoidable.

# .cargo/config.toml
[alias]
xtask = "run --package xtask --"
ci = "clippy --workspace --all-targets --all-features -- -D warnings"

Lints: rustfmt and clippy

• rustfmt.toml: edition, max_width, use_small_heuristics, newline_style. Note: imports_granularity, group_imports, and style_edition are nightly-only options; if you want them, run cargo +nightly fmt or avoid them.
• clippy.toml: msrv, cognitive-complexity-threshold, too-many-arguments-threshold, disallowed-types, disallowed-methods, avoid-breaking-msg (verify current option names upstream).
• Gate CI: cargo clippy --workspace --all-targets --all-features -- -D warnings and cargo fmt --all --check.
• Set baseline lints in the crate root (#![deny(missing_docs, missing_debug_implementations, rust_2018_idioms)]) and policy lints in [workspace.lints].
• Common library lint set: clippy::all, clippy::pedantic (allow noisy ones explicitly), unwrap_used, expect_used (warn), panic, indexing_slicing (warn), arithmetic_side_effects (selective).
• clippy::correctness is deny-by-default; never blanket-allow it.
• Allow lints at the narrowest scope with a reason: #[expect(clippy::cast_possible_truncation, reason = "checked above")] (#[expect] supported on modern Rust; verify upstream).
• Test code may relax policy lints via [lints] on the test target or #![cfg_attr(test, allow(...))].

Cross-Compilation

• Install targets: rustup target add aarch64-unknown-linux-musl x86_64-pc-windows-gnu wasm32-wasip2.
• Static Linux binaries: *-unknown-linux-musl + -C target-feature=+crt-static (default for musl on most targets).
• Linker configuration in .cargo/config.toml per target; cross (Docker images with toolchains) or cargo-zigbuild (zig as cross linker) removes most sysroot pain.
• Apple targets require the Xcode SDK and license acceptance; aarch64 macOS cross from Linux is not officially supported without an SDK — use a macOS runner.
• Windows: x86_64-pc-windows-gnu (MinGW) cross-compiles from Linux; MSVC should run on Windows.
• WASM: wasm32-unknown-unknown + wasm-bindgen, or wasm32-wasip1/wasip2 for the component model; wasm-pack/cargo-component for bundling. Verify target names and component-model status upstream.
• Test cross outputs with QEMU (cross test --target ..., qemu-<arch> binfmt) or run on matching runners.
• Check cargo tree --target <triple> because target-specific dependencies change feature graphs.

MSRV (Minimum Supported Rust Version)

• Declare rust-version in [package]/[workspace.package]; cargo uses it for resolver v3 compatibility hints and toolchain checks.
• Test the MSRV in CI with a pinned toolchain (actions-rs/toolchain-style matrix or rustup toolchain install 1.85.0).
• cargo-msrv can bisect the minimal version; use it when bumping dependencies floods the MSRV.
• MSRV policy: bumping requires a semver-minor release of your library (or is a breaking change under some policies) — document the policy in the README.
• cargo update -p <dep> --precise <version> in CI keeps the MSRV lockfile coherent; or maintain a Cargo.lock.msrv.

Release and Supply Chain

| Tool | Purpose | When | |---|---|---| | cargo-deny | advisories, bans, licenses, source allowlist | every CI run | | cargo-audit | RustSec advisory DB | every CI run (or rely on cargo-deny) | | cargo-vet | audit trail for dependency reviews | org policy for high-assurance | | cargo-udeps | unused dependencies (nightly) | periodic | | cargo-machete | unused dependencies without nightly | every CI run | | cargo-semver-checks | API break detection | before release | | cargo-sbom / cyclonedx | SBOM generation | release artifacts |

# deny.toml (abridged)
[advisories]
version = 2
yanked = "warn"

[licenses]
allow = ["MIT", "Apache-2.0", "BSD-3-Clause", "ISC", "Unicode-3.0"]
confidence-threshold = 0.93

[bans]
multiple-versions = "warn"
wildcards = "deny"

[sources]
unknown-registry = "deny"
unknown-git = "deny"

• Lockfile: cargo build --locked in CI and release; cargo update is a reviewed PR, never silent.
• Vendoring for hermetic builds: cargo vendor and [source.crates-io] replace-with = "vendored-sources".
• cargo publish --dry-run before release; publish with --locked (supported by modern cargo; verify upstream).
• crates.io trusted publishing (OIDC-based CI publishing) exists on the modern registry; verify upstream before adopting, and prefer it to long-lived tokens.
• Tag releases, attach checksums/SBOM, and sign when the org requires it (Sigstore/cosign is common).
• Reproducibility: pin toolchain (rust-toolchain.toml), commit lockfile, use --locked, set SOURCE_DATE_EPOCH where applicable, and avoid path-dependent build scripts.
• Security response: watch RustSec advisories, have an update path for transitive deps, and CI-fail on new advisories (cargo-deny check advisories).

Anti-Patterns

• Always-on [features] default = ["full"] that pulls TLS, DB drivers, and runtimes into every consumer.
• Mutually exclusive features ("enable exactly one backend") that only work by convention.
• Cargo.lock ignored in a binary repository.
• -D warnings on rustc but not clippy, or clamping the toolchain without pinning it.
• Cross-compiling without testing the artifact.
• Committing cargo vendor trees without an update process.
• Long-lived crates.io tokens in CI.
• Relying on cargo update to silently fix advisories in production releases.

Checklist

• [ ] Workspace resolver matches the edition; shared deps/lints centralized.
• [ ] Features are additive and documented; powerset check runs in CI.
• [ ] Release/profile settings deliberate (lto, codegen-units, panic, strip, debug).
• [ ] Lint policy in [workspace.lints]; CI runs fmt + clippy with -D warnings.
• [ ] Cross targets built and smoke-tested where shipped.
• [ ] MSRV declared, pinned, and tested.
• [ ] cargo-deny/cargo-audit and cargo-semver-checks in CI; release uses --locked.
• [ ] Build cache configured; clean-checkout build time measured.
• [ ] Release process automated (version bump, changelog, publish order, artifacts).

Performance, Security, Observability

Scope: profiling tools, allocation and CPU optimization, rayon, SIMD basics, crypto and secrets hygiene, and tracing/metrics instrumentation policy.

Performance Method

1. Define the metric and budget (p99 latency, throughput, RSS, binary size, energy).
2. Measure with a representative workload; record a baseline in version control or CI.
3. Profile to find the hotspot; fix the largest cost first.
4. Verify with the same measurement; add a regression guard (benchmark, size check).
5. Repeat. Never optimize from intuition.

| Symptom | First tool | Then | |---|---|---| | CPU-bound slowness | perf record --call-graph dwarf | cargo-flamegraph, samply | | Allocation pressure | DHAT/heaptrack | cargo-bloat, massif | | Long compile times | cargo build --timings | cargo-llvm-lines | | Large binary | cargo bloat | cargo size, nm --size-sort | | Lock contention | perf lock? use samply/perf with thread views | loom, redesigned ownership | | High syscall overhead | strace -c, perf trace | buffering, batching |

• cargo-flamegraph (CARGO_PROFILE_RELEASE_DEBUG=true or the profiling profile) gives a quick live view; samply provides a nicer timeline and is widely used in 2026 — verify current project status upstream.
• perf needs symbols and frame pointers for good stacks: build with debug = true and prefer --call-graph dwarf when frame pointers are unavailable.
• DHAT (dhat-rs), heaptrack, and bytehound find allocation hot spots; dhat integrates as an allocator behind a feature flag and writes a JSON report.
• cargo-bloat --release ranks functions by size; cargo-bloat --crates splits by crate.
• For I/O-bound services, measure queue times and connection pools, not CPU.
• Profile release-like builds: optimizations change hotspots. Use a custom profiling profile with debug symbols (see ./09-tooling-build.md).

Allocation Reduction

• Vec::with_capacity, String::with_capacity when sizes are known; reserve before loop growth.
• Reuse buffers across requests (pool them) instead of allocating per request; bytes::Bytes/BytesMut for zero-copy slicing of shared buffers.
• Replace small Vec/String with SmallVec/ArrayVec/smallstr to avoid heap for typical sizes; measure the inline size tradeoff.
• Cow<'_, str> (or Cow<'_, [T]>) in parsing pipelines to avoid copying unchanged input.
• Write into buffers with std::fmt::Write (write!(&mut String, ...)) instead of format! when the destination is reused.
• Avoid collect() in hot paths; chain iterators or use fold/for_each with preallocated output.
• boxed trait objects allocate; hoist to setup, or use enums for a fixed set of variants.
• Arena allocation (bumpalo) for batch processing with a single lifetime; reset the arena per batch.
• Global allocator choice: jemalloc/tikv-jemallocator or mimalloc can help multi-threaded alloc-heavy workloads; measure RSS and latency — they are not automatic wins, especially for short-lived processes.
• Vec::shrink_to_fit/into_boxed_slice after final growth when memory is tight.
• Beware hidden allocations: format! in Display impls, to_string() in logs, Path::to_path_buf, and closures capturing owned Strings.

let mut out = String::with_capacity(input.len() + 16);
write!(out, "{prefix}:{input}")?;

CPU and Codegen

| Technique | When | Caveat | |---|---|---| | Iterator chains instead of indexing | loops over slices | usually enables bounds-check elision | | #[inline] | tiny, cross-crate hot functions | can bloat; rely on LTO first | | #[inline(always)] | proven hotspots only | hurts icache and compile time | | lto = "thin"/"fat" | release binaries | build-time cost; fat maximizes cross-crate inlining | | codegen-units = 1 | release | slower builds | | target-cpu=native | controlled deployment | illegal-instruction risk on heterogeneous fleets; prefer runtime dispatch | | PGO/BOLT | high-value binaries | tooling-heavy; nightly -Cprofile-generate / BOLT on ELF | | #[cold] and #[inline(never)] | error paths | keep hot path small | | Branch hints | rarely needed | std::hint::likely is unstable; do not use on stable |

• Bounds checks: use iter(), windows(), chunks(), split_at, and get with explicit handling; keep -C debug-assertions off in release.
• Prefer &[T] parameters over &Vec<T> to accept more callers without conversion.
• Avoid dyn in inner loops; hoist dynamic dispatch to the outer layer.
• Use #[repr(C)]/#[repr(align(N))] only with a concrete reason (ABI, cache-line separation for atomics).
• parking_lot locks are faster than std and have no poisoning; measure contention before switching.

Rayon and Data Parallelism

• rayon provides work-stealing par_iter, join, and scope; use it for CPU-bound data processing, not for I/O.
• data.par_iter().map(f).collect::<Vec<_>>() preserves order; for_each/reduce for unordered.
• Custom tasks: rayon::join(|| a(), || b()) for divide-and-conquer; scope for borrowing local data.
• Do not block worker threads on I/O or locks; rayon's pool is sized to CPUs. Blocking a rayon thread stalls the whole pool.
• Nested parallelism with tokio: never call rayon inside async on every item; either spawn_blocking a rayon batch, or use tokio::task::spawn_blocking per batch, or bridge with pollster/block_in_place deliberately.
• Global pool configuration: rayon::ThreadPoolBuilder::new().num_threads(n); build a dedicated pool when the library must not hijack the global one.
• False sharing: pad per-thread accumulators (CachePadded from crossbeam) for high-contention counters.
• Determinism: reduce with an associative operation; avoid floating-point nondeterminism when a stable result is required.

use rayon::prelude::*;
let sum: u64 = data.par_chunks(4096).map(|c| c.iter().map(|&x| x as u64).sum::<u64>()).sum();

SIMD Basics

• Start with autovectorization: contiguous slices, simple loops, no data-dependent branches per element, and -C target-feature or target-cpu where safe.
• std::simd (portable SIMD) remains nightly-only in practice; do not put it in a stable crate without a gate. Verify upstream.
• Stable options: std::arch intrinsics behind #[target_feature(enable = "avx2")] unsafe fn, plus runtime detection with is_x86_feature_detected!/is_aarch64_feature_detected!.
• Cross-platform wrappers: wide (portable SIMD types) and multiversion (compiles feature-specialized versions and dispatches at runtime).
• Keep SIMD kernels isolated in a module with scalar fallbacks, tests comparing scalar vs SIMD outputs, and Miri-friendly scalar paths (Miri does not support many intrinsics).
• Alignment: unaligned loads (loadu) are fine on modern x86 but cost on some ARM; align when easy.
• Crypto and compression libraries already ship tuned SIMD; use them rather than hand-rolling.
• Measure with perf stat (IPC, cache misses) and check that vectorization actually happened (cargo asm, llvm-mca).

Crypto and Secrets

| Need | Recommended | Notes | |---|---|---| | TLS | rustls (+ aws-lc-rs or ring provider) | pure-Rust, no OpenSSL linkage; native-tls when OS trust store integration is required | | Symmetric AEAD | aes-gcm, chacha20poly1305 (RustCrypto) | use well-reviewed, actively maintained crates; verify audit status upstream | | Hashing | sha2, blake3 | blake3 for speed, sha2 for compatibility | | Password hashing | argon2, password-hash | Argon2id; store parameters with the hash | | KDF | hkdf, pbkdf2 | derive distinct keys per purpose | | Randomness | getrandom (OS CSPRNG), rand for non-crypto | never seed crypto from rand::thread_rng state you control; use OsRng | | Constant-time compare | subtle | never compare secrets with == | | Secret storage | secrecy, zeroize | SecretString, zeroize on drop, avoid Debug leakage | | JWT | jsonwebtoken | validate alg/aud/iss explicitly (see ./06-web-data.md) |

• Do not roll your own crypto, padding, or protocol; compose audited primitives.
• RustCrypto crates are widely used but many carry "no formal audit" disclaimers; check the current audit status of each crate before high-assurance use and verify upstream.
• Constant-time behavior is not guaranteed by the type system: subtle::ConstantTimeEq, avoid branching on secrets, avoid secret-dependent indexing (table lookups), and avoid secrets in error messages/timing.
• Use zeroize/Zeroizing for keys and passwords; note that Rust may copy values — zeroize best-effort and do not rely on it for safety guarantees.
• Secrets in process: environment variables are visible in /proc for the same user and in crash dumps; prefer a secret manager fetched at startup, or workload identity. Never bake into images.
• Logging policy: never log tokens, keys, passwords, or PII; run a lint/review check for Debug on types containing secrets (derive a redacting Debug manually).
• Dependencies: pin crypto crates tightly, vendor or lock, and track advisories (see ./09-tooling-build.md).
• Platform RNG failures: handle getrandom errors (early boot, restricted sandboxes) by failing closed, not by falling back to weak randomness.

Observability with tracing

• tracing replaces log for structured, async-aware diagnostics; tracing-subscriber is the runtime side.
• Create spans at request/job boundaries: #[instrument(skip(state), fields(user_id = %id))]; keep span fields high-cardinality limited (ids are fine; unbounded values are not).
• Events carry the detail: info!, debug!, warn!, error! with structured fields (error = ?err, latency_ms = elapsed.as_millis()), not string interpolation.
• Subscriber stack: registry().with(EnvFilter::from_default_env()).with(fmt::layer().json()); add console-subscriber in dev for task stalls.
• Log levels: error = page-worthy; warn = degraded; info = lifecycle; debug/trace = development. Never ship trace in production defaults.
• JSON logging in production, pretty in development; include service name, version, trace/span ids.
• OpenTelemetry: tracing-opentelemetry + OTLP exporter; propagate context through HTTP/gRPC with opentelemetry propagators. Sample at the collector when possible; head sampling in-process for cost control.
• Avoid per-request format! in hot paths; tracing is cheap when the level is disabled, but field formatting still costs when enabled.
• Libraries depend on tracing (or log with a feature) and emit spans; applications configure filtering and exporters. Never install a global subscriber in a library.
• Flush telemetry on shutdown: subscriber guards + opentelemetry::global::shutdown_tracer_provider (verify current API).
• Correlation: inject/propagate traceparent (W3C) through clients and servers; include request id in error responses without leaking internals.

Metrics

• Use the metrics facade plus an exporter (metrics-exporter-prometheus) or the OpenTelemetry metrics API; keep metric names stable, and use units in the name (http_request_duration_seconds).
• Counters for totals, gauges for current values, histograms for latency/size; avoid high-cardinality labels (user ids) on histograms.
• RED (rate, errors, duration) for services; USE (utilization, saturation, errors) for resources.
• Export scrape-friendly endpoints on a separate port or internal router, not the public listener.
• Instrument queues with depth and age; queued age is the leading indicator of overload.
• Define SLOs and alert on symptoms (latency, error rate), not causes (CPU).

Checklist

• [ ] A release-like build is profiled before optimizing; baseline recorded.
• [ ] Perf budget (latency/throughput/memory/size) exists and regressions are caught.
• [ ] Allocations in hot paths measured; buffers reused; hidden format!/to_string() removed.
• [ ] Rayon used only for CPU work; no blocking of its pool; no nested oversubscription with tokio.
• [ ] SIMD kernels have scalar fallbacks, dispatch, and equivalence tests; Miri path exists.
• [ ] TLS, hashing, password hashing, and RNG come from maintained crates with tracked advisories.
• [ ] Secrets never logged, zeroized where practical, delivered by a secret manager.
• [ ] Tracing spans at boundaries; metrics for RED/USE; logs structured with levels.
• [ ] Telemetry flushed on graceful shutdown; no global subscriber installed by libraries.