OpenTelemetry

Scope: OTel architecture, the API/SDK split, semantic conventions, collector deployment patterns, auto-instrumentation per language, and migration off vendor agents.

Architecture in One Picture

app process                          collector (agent and/or gateway)        backend
+----------------------+            +-----------------------------+         +---------+
| OTel API (no-op safe)|            | receivers  (otlp, prometheus|         | traces  |
| OTel SDK             |--OTLP----->|   filelog, hostmetrics,     |--OTLP-->| metrics |
|  resource, sampler,  |            |   k8s_cluster, jaeger, ...) |         | logs    |
|  processors, exporter|            | processors (batch, memory,  |         | profiles|
+----------------------+            |   filter, transform, tail)  |         +---------+
                                    | connectors (spanmetrics,    |
                                    |   servicegraph, routing)    |
                                    | exporters (otlp, kafka, ...)|
                                    +-----------------------------+

• Signals: traces, metrics, logs, and profiles; baggage travels with traces. Traces, metrics, logs, and baggage are stable; the profiles signal is stabilizing — verify upstream for your SDK and backend.
• Components: the API you call, the SDK that implements it, OTLP as the wire protocol, the Collector for pipeline policy, semantic conventions for naming, and the Operator for Kubernetes packaging.
• The protocol is the contract. Instrument against OTel and OTLP so backends remain replaceable.

API vs SDK Split

| Layer | Responsibility | Who configures it | |---|---|---| | API | Instrumentation calls: tracers, meters, loggers, propagators | Library authors; app code at boundaries | | SDK | Sampling, processing, export, resource, limits | Application entrypoint or bootstrap | | Exporter | Wire format and destination (OTLP, Prometheus, vendor) | Application or collector | | Propagator | Trace context and baggage over the wire | Application or framework integration | | Resource | Identity of the emitting entity | Application or auto-instrumentation |

• Libraries SHOULD depend only on the API (or the SDK-free facade), never install global providers or exporters. If no SDK is configured, the API must behave as a no-op with negligible cost.
• Applications own provider setup exactly once at startup, before serving traffic.
• Pin the semantic-convention version you emit; conventions rename attributes across minor releases. Use the SDK's stability opt-in (for example OTEL_SEMCONV_STABILITY_OPT_IN) during migrations and check the upstream compatibility notes.
• Environment variables are the portable configuration surface: OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_TRACES_SAMPLER, OTEL_SDK_DISABLED. Prefer them over bespoke config so collector-side and SDK-side behavior stay aligned.

OTLP Specifics

• OTLP/gRPC on port 4317 and OTLP/HTTP on 4318; HTTP supports protobuf and JSON. gRPC is the usual inside-cluster choice; HTTP/protobuf travels through HTTP-only proxies.
• Signal paths: /v1/traces, /v1/metrics, /v1/logs, and /v1/profiles where supported. Read endpoints from env vars, not from code defaults.
• Compression is per-exporter (gzip is common); enable it before increasing batch size to fix throughput.
• Partial success is a first-class response: the SDK must surface rejected spans/metrics rather than silently dropping them.
• Queueing and retry: exporters ship with bounded queues and backoff. Configure queue size and timeouts deliberately; a full queue drops data, and an unbounded queue eats memory.
• On shutdown, flush the SDK and shut down providers with a deadline. Container termination that skips this loses the last batch.

Resource and Identity

| Attribute | Purpose | Notes | |---|---|---| | service.name | Primary service identity | Stable, match dashboards/alerts; set explicitly | | service.namespace | Grouping for large orgs | Optional but useful for multi-tenant platforms | | service.version | Deploy correlation | Commit SHA, semver, or build ID | | service.instance.id | Unique instance | Pod UID or hostname; never reused across restarts | | deployment.environment.name | Environment | Prefer this over the older deployment.environment; verify SDK support | | cloud.provider, cloud.region, cloud.availability_zone | Cloud topology | Resource detection or collector enrichment | | k8s.cluster.name, k8s.namespace.name, k8s.pod.name, k8s.deployment.name | Kubernetes topology | Added by the Operator, k8sattributes, or SDK resource detectors | | host.name, host.arch, os.type | Host identity | Useful in bare-metal and VM fleets | | telemetry.sdk.language, telemetry.sdk.name, telemetry.sdk.version | SDK provenance | Automatic; use for SDK-version inventories |

• Identity drift is the most common observability defect: traces, metrics, and logs from one process must carry the same service.* values. Assert it in review and in tests.
• Keep high-cardinality identity in resource attributes (one time series per process is still too many for some backends) and off metric labels. Put instance identity on spans/logs, not on counters.

Semantic Conventions

| Domain | Representative attributes | Notes | |---|---|---| | HTTP server | http.request.method, http.route, url.path, http.response.status_code | http.route is the templated route, never the raw path | | HTTP client | Same server attributes plus server.address, server.port | Old net.peer.* names are deprecated; migrate | | RPC | rpc.system, rpc.service, rpc.method, rpc.grpc.status_code | gRPC status is not HTTP status | | Database | db.system, db.namespace, db.operation.name, db.query.summary | Never record full statements with literals; use parameterized summaries | | Messaging | messaging.system, messaging.destination.name, messaging.operation.type, messaging.message.id | Kafka/RabbitMQ/SQS/NATS conventions differ in detail | | Kubernetes | k8s.* | Prefer collector or Operator enrichment over hand-set values | | GenAI | gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.operation.name | Rapidly evolving; pin the semconv version and verify upstream | | Errors | exception.type, exception.message, exception.stacktrace, span status ERROR | Log the exception once, not at every layer | | Identity | enduser.id, user.id | Treat as PII; high cardinality; never metric labels |

• Convention migrations are mechanical but wide-reaching: net.peer.name to server.address, http.method to http.request.method, http.status_code to http.response.status_code, db.statement usage tightened. Run a grep for deprecated names during upgrades.
• Stable conventions are the contract; stabilizing and development conventions may change without a major version. Check the stability badge before relying on an attribute in SLO queries.

Collector Deployment Patterns

| Pattern | Shape | Use when | Watch out for | |---|---|---|---| | Gateway | Deployment, one or few replicas | Central policy, simple clusters | Single point of failure; scale and PDB required | | Agent | DaemonSet per node | Host/k8s enrichment, local buffering | Config drift; per-node resource overhead | | Agent plus gateway | DaemonSet forwards to Deployment | Production default at scale | Two configs to maintain; test both | | Sidecar | Per-pod container | Hard isolation, per-tenant limits | Cost, pod churn, restart gaps | | Direct export | App to backend/collector-less | Serverless, tiny fleets | No central filtering or redaction |

• Collector config has four sections: receivers, processors, exporters, and service.pipelines, plus extensions and connectors. Pipelines are per signal and explicitly listed.
• Standard order inside a pipeline: memory limiter first, then enrichment (resource, k8sattributes, attributes), then filtering/transform, then sampling, then batching, then export.
• The load-balancing exporter plus tail sampling requires sticky trace routing: hash by trace ID so all spans of a trace reach the same sampling instance. Scale tail-sampling collectors horizontally only with this pattern.
• Connectors turn a pipeline into an input: spanmetrics derives metrics from spans, servicegraph derives dependency graphs, routing fans out by attribute, forward links two pipelines.
• The Kubernetes Operator manages collectors, injects SDK env vars via annotations, and runs the target allocator for Prometheus-style scraping. It is the recommended packaging when you already run Operators.
• Self-telemetry matters: expose collector internal metrics (accepted/refused/sent, queue length, batch size) and alert on refused data and dropped queues. A blind collector loses data silently.
• Backpressure is normal during backend incidents: memory limiter refuses, queues fill, retries back off. Decide per pipeline whether to drop logs first and traces last.

Zero-Code and Auto-Instrumentation

| Language | Mechanism | Maturity notes | |---|---|---| | Java | -javaagent with the OTel Java agent; JVM and framework instrumentations | Most mature; supports many frameworks and manual API alongside | | Python | opentelemetry-distro plus opentelemetry-instrument wrapper | Broad coverage; watch WSGI/ASGI and celery/queue coverage | | Node.js | --require preload of auto-instrumentations (or ESM loader hook) | Requires preload; ESM support improved but verify your runtime | | .NET | Auto-instrumentation startup hook or NuGet-based | Good ASP.NET Core, HttpClient, gRPC coverage | | Go | Compile-time instrumentation is the primary path; eBPF-based zero-code agents exist | Pure-Go needs import-time hooks; eBPF gives no-code coverage with gaps | | Ruby, PHP | Framework gems/extension plus auto-instrumentation | Coverage varies by framework version; verify upstream | | eBPF platforms | Kernel-level probes for HTTP/gRPC/DB and profiles | No code changes, useful for legacy/LTO binaries; check TLS and runtime support |

• Rule of thumb: use zero-code to get coverage fast, then add manual spans for business-critical paths. Zero-code spans often miss domain semantics (order ID, tenant), which are exactly what you need during incidents.
• Auto-instrumentation still needs the resource attributes and sampler configured; injecting the agent without OTEL_SERVICE_NAME and an endpoint produces unnamed or lost telemetry.
• Redaction and cardinality control do not come for free with zero-code. Inspect what the agent emits (DB statements, URLs, headers) before production.
• eBPF agents are kernel- and runtime-version dependent; pin the agent and test after kernel upgrades. Verify upstream support for your Go/Node TLS stack.

Migration From Vendor Agents

1. Inventory every producer: agents, SDKs, log shippers, dashboards, alerts, runbooks, and saved queries. The dashboards and alerts are usually the hard part.
2. Land OTel telemetry in parallel (dual-ship). Keep the vendor agent running until parity is measured per signal, not per service.
3. Map vendor-specific concepts to OTel: host tags to resource attributes, APM service names to service.name, custom metrics to OTel instruments, trace tags to span attributes.
4. Normalize through the collector, not in application code, so migrations do not require redeploys.
5. Rebuild alerts and dashboards on the new backend, then decommission agents signal by signal; remove credentials and old pipelines in a final pass.
6. Expect semantic gaps: vendor agents often capture process metrics, profiles, and database query plans that OTel only approximates. Document what is lost.

SDK Bootstrap Skeleton

# Application entrypoint: the only place providers, exporters, and resources are configured.
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource.create({
    "service.name": "order-api",
    "service.version": "2f9c1ab",
    "deployment.environment.name": "prod",
})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
# On shutdown: provider.shutdown() to flush with a deadline.

• The shape is the same in every language: build a resource, install one provider, attach a batch processor and OTLP exporter, register propagators. Keep it in a bootstrap module that tests can replace.
• Configure via environment variables (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_SAMPLER, OTEL_RESOURCE_ATTRIBUTES) so the same image works across environments.
• For Kubernetes, the Operator can inject these settings from pod annotations, which keeps app code free of endpoints and credentials.

Versioning and Compatibility

• Semantic conventions, SDKs, and the collector version independently. Record all three in an inventory; upgrades in one component can change emitted data for the others.
• Collector upgrades: processors and receivers can move between core and contrib, change default behavior, or be deprecated. Pin image digests, read release notes, and stage upgrades.
• SDK upgrades: attribute renames follow semconv stability levels. Use the stability opt-in during migration and keep a deprecation grep in CI.
• Instrumentation package versions must match supported framework versions; a framework major upgrade without an instrumentation bump is a common source of missing spans.
• Exporters should tolerate a backend version skew; OTLP is the compatibility boundary. Keep a dual-export path during backend migrations.

Anti-Patterns

• Configuring exporters inside libraries or multiple conflicting global providers in one process.
• Setting service.name differently per signal, or leaving it as unknown_service.
• Raw URL paths, user IDs, or error messages as metric labels or span names.
• Running a tail-sampling collector fleet without sticky routing, so traces are sampled per span and lose completeness.
• Sending everything to the backend and filtering "later"; later never comes and the bill does.
• Treating collector refusal metrics as noise instead of data loss.
• Assuming the profiles signal and GenAI conventions are frozen; they are not.

Checklist

• [ ] One global SDK configured at startup; libraries use the API only.
• [ ] service.name, service.version, and deployment environment consistent across signals.
• [ ] Semantic-convention version pinned; deprecated attribute names migrated.
• [ ] Collector topology matches scale: gateway or agent-plus-gateway, with memory limiter and batching.
• [ ] Tail sampling sessions are trace-sticky and horizontally testable.
• [ ] Zero-code agents configured with resource and sampler, and audited for emitted cardinality and PII.
• [ ] Collector self-telemetry exported and alerted on.
• [ ] Migration dual-ships until parity is proven, then decommissions in stages.

Logging

Scope: structured logging, correlation IDs, level policy, sampling, PII redaction, log pipelines, and the OTel logs model versus legacy agents.

Logs as Data, Not Prose

| Property | Good | Bad | |---|---|---| | Format | One JSON object per line, newline-delimited | Multi-line stack traces as separate events | | Timestamp | RFC3339 with offset, UTC | Local time without offset | | Severity | Explicit level/severity_text field | "ERROR:" buried in a message string | | Message | Stable, event-like (order created) | Interpolated prose (Order 123 for bob created in 45ms) | | Detail | Structured fields (order_id, duration_ms) | Values only embedded in the message | | Identity | service.name, service.version, environment | Missing or inconsistent | | Correlation | trace_id, span_id, request ID | None |

{"timestamp":"2026-04-02T10:30:00.412Z","severity_text":"INFO","severity_number":9,
 "service.name":"order-api","service.version":"2f9c1ab","deployment.environment.name":"prod",
 "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7",
 "http.request.method":"POST","http.route":"/orders","order_id":"ord_789","duration_ms":45}

• Prefer event-style messages that are stable across releases; every variable belongs in a field. Then a log line is queryable and a dashboard never breaks because a message was reworded.
• Use the OpenTelemetry log data model as the target shape: timestamp, observed timestamp, severity number and text, body, attributes, resource, trace ID, span ID. OTLP log exporters map native logging libraries into this model.
• Do not log an object's repr, String(), or toString() blindly: it leaks secrets, creates unstable schemas, and bloats volume. Log selected fields.

Correlation IDs

| Layer | Identifier | Propagation | |---|---|---| | Edge / ingress | Request ID | Header (X-Request-ID or W3C traceparent) | | Service calls | trace_id / span_id | W3C Trace Context headers | | Async / queues | trace_id plus message attributes | Producer injects, consumer extracts | | Batch jobs | Job run ID | Job context, carried into every log | | User journey | Session or correlation ID | Low-trust identifier; treat as PII |

• The best correlation ID is the trace ID. Bridge your logging library to the OTel context so every log record during a span automatically carries trace_id and span_id.
• If a request has no incoming trace context and tracing is enabled, start a span at the edge rather than inventing a parallel correlation scheme. If tracing is not enabled, generate a request ID once at ingress, store it in request-scoped context, and include it on every log and outbound call.
• Log the identifiers, not the payloads. Correlation IDs unlock the raw data in other signals; they are not a reason to copy request bodies into logs.
• Downstream services should log the upstream request ID only if it is needed for causality; the trace ID already provides end-to-end linkage.

Severity Policy

| Level | Meaning | Production default | Examples | |---|---|---|---| | FATAL | Process cannot continue | Always on; page | Data corruption detected, failed to bind | | ERROR | Operation failed, not self-healed | Always on; page or ticket by SLO | Payment charge failed, handler returned 5xx | | WARN | Unexpected but handled | On | Retry succeeded, queue backlog high | | INFO | State change or business event | On, low volume | Service started, order shipped, config reloaded | | DEBUG | Diagnostic detail for development | Off by default | Cache miss reason, branch selection | | TRACE | Per-step internals | Off | Loop iterations, wire-level detail |

• Severity must be policy, not taste. Write it down: what pages, what tickets, what is noise. Review the policy every time an on-call rotation changes.
• No sensitive data at any level. WARN and ERROR are often shipped at higher retention, which makes leaks permanent.
• Use structured severity numbers where OTLP requires them; map native levels carefully (WARN often maps to 13, ERROR to 17) and verify the SDK mapping upstream.
• Errors belong at the boundary that owns recovery. A retry that succeeds at layer 2 should not also be logged as ERROR at layers 3 and 4; that is duplicate noise.

Log Sampling

| Technique | Where | Effect | |---|---|---| | Drop DEBUG/TRACE in production | Logging config | Removes the largest, lowest-value volume | | Per-level ratio sampling | App or collector | Bounds volume while preserving counters | | Rate-limit per message template | App | Kills log storms from one broken path | | Deduplicate repeated messages | Collector | Collapses "retrying and failing" floods | | Tail sampling by trace | Collector | Keeps all logs for errors/slow traces, drops healthy verbose traces | | Sampling logs only, keep metrics | Metrics-first design | Preserves detection while cutting text volume |

• Sample by severity, never uniformly: dropping 90 percent of random lines destroys the ability to reconstruct an incident but keeps the cost. Drop DEBUG first, then sample INFO per template, and keep errors complete.
• Log storms are an availability risk: a failing dependency can generate gigabytes per minute and saturate disk or the collector. Rate limits per logger and per template are mandatory in production.
• When tail sampling traces, correlate log sampling to the same policy so a kept trace also has its logs. Independent sampling creates traces with missing narratives.
• Count what you drop. Export a counter for dropped/sampled-out log records so "no logs" is distinguishable from "logs dropped".

PII and Secret Redaction

| Data class | Examples | Handling | |---|---|---| | Authentication secrets | Passwords, tokens, API keys, cookies, Authorization headers | Never log; redact at source and in collector | | Direct PII | Email, phone, address, national ID, precise location | Drop or tokenize; avoid hash-only if reversible lookup is required | | Indirect PII | User agent, IP address, device IDs, free-text support notes | Treat as PII; consider truncation or hashing per privacy policy | | Payment data | PAN, CVV, bank accounts | Prohibited; never enters logs | | Request/response bodies | JSON payloads, form fields | Do not log by default; allowlist fields if needed | | Internal identifiers | Account IDs, order IDs | Safe in logs, unsafe as metric labels |

• Redact at the source first (the code knows field semantics), then again in the collector as defense in depth. A collector-only strategy leaks the moment someone ships a new service or bypasses the gateway.
• Prefer allowlisting known-safe fields over denylisting known-bad patterns. Denylists miss user_email, emailAddress, contact.mail, and the next synonym.
• Redaction must survive encoding: parse structured logs before redacting, and test nested, array, and URL-encoded fields. A regex over a JSON string misses %40 and JSON escapes.
• Secrets in exception messages are a classic leak. Wrap errors so driver/DSN strings are not rendered; rotate anything that may have been logged.
• Keep a documented data classification and a test that fails when a new field is classified PII but appears unredacted.

Log Pipelines

| Stage | Options | Notes | |---|---|---| | Emit | stdout JSON, OTLP logs from SDK | Prefer stdout for containers; OTLP when the SDK owns correlation | | Collect | Collector filelog/otlp, Fluent Bit, Vector, node agents | One log agent per node is enough; avoid stacking two | | Process | Collector transform (OTTL), filter, attributes, redaction | Parse, normalize, redact, enrich, drop | | Route | Collector routing connector | Split security logs, audit logs, and app logs | | Store | Loki, OpenSearch/Elasticsearch, ClickHouse-based, cloud logging | Choose by query style, retention, and cost | | Query | LogQL, KQL, Lucene, SQL | Correlate to traces via IDs |

• Normalize to one schema at the pipeline boundary: same field names, same severity vocabulary, same timestamp precision. Divergent schemas make cross-service queries impossible.
• Enrich centrally (k8s metadata, environment, service version) in the collector rather than in every application.
• Parse multiline stack traces before storage (multiline in the agent, or a collector operator) so one exception is one record. Otherwise every line is a separate, useless event.
• Keep label/index cardinality low. In Loki-like systems the index fields must be low-cardinality (service.name, environment, level); high-cardinality values belong in the body or structured metadata.
• Budget retention per stream: audit and security logs may need a year, debug logs hours. Tiered retention is the main cost lever after volume reduction.

OTel Logs vs Legacy Agents

| Aspect | OTel logs | Legacy/agent-based logging | |---|---|---| | Schema | OTel log data model, resource attributes | Vendor-specific tags and attributes | | Correlation | Native trace_id/span_id on records | Often manual or partial | | Transport | OTLP plus collector processing | Proprietary protocol and agents | | Vendor lock-in | Low; swap exporter or backend | High; retool dashboards and queries | | Coverage | SDK bridges for mainstream logging libraries | Mature, framework-specific integrations | | Processing | Collector processors, OTTL | Agent pipelines with vendor DSLs |

• Migrate by bridging the existing logging library into OTel, not by replacing the library. Keep structured logging in the app; let the SDK/bridge attach context and export.
• Dual-ship during migration: legacy backend plus OTLP. Compare record counts, severity distribution, and a few known incidents before cutting over.
• Watch out for duplicated logs when both stdout collection and OTLP export are enabled for the same process. Pick one path per service.
• Legacy agents often add host metadata automatically; in OTLP you must supply it via resource attributes or collector enrichment.

Example Collector Pipeline

receivers:
  filelog:
    include: [/var/log/pods/*/*/*.log]
    operators:
      - type: json_parser
      - type: multiline
        firstline: '^\d{4}-\d{2}-\d{2}'
  otlp:
    protocols: { grpc: {}, http: {} }
processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80 }
  k8sattributes: {}
  transform:
    log_statements:
      - context: log
        statements:
          - set(attributes["env"], resource.attributes["deployment.environment.name"])
  batch: {}
exporters:
  otlphttp/loki: { endpoint: "http://loki:3100/otlp" }
service:
  pipelines:
    logs:
      receivers: [filelog, otlp]
      processors: [memory_limiter, k8sattributes, transform, batch]
      exporters: [otlphttp/loki]

• Parse, enrich, and normalize at the collector so applications stay simple and the schema is uniform across languages.
• Keep one collection path per workload. Running a node log agent and OTLP log export for the same service doubles volume and confuses operators.
• Validate pipeline changes with the collector's config validation and a staging run; a malformed transform statement can silently drop fields or records.

Schema and Query Hygiene

• Keep a versioned schema document: field name, type, meaning, sensitivity class, and example. New fields are reviewed against it, and renaming is a breaking change.
• Use a small set of common fields across services (service.name, level, message, trace_id, span_id, request_id, error.type) and namespace service-specific fields (orders.order_id).
• Query by field, not by message text. Text search is the fallback for unexplored data, not the primary interface.
• Avoid scanning unbounded time ranges in log queries. Constrain by label, then field, then text, and set limits on result size.
• Keep error messages stable enough to group: normalize variable values out of the message and into fields so identical failures cluster.
• Retain a small "canary" query set per service (errors, slow requests, security events) and run it after schema or pipeline changes to catch regressions.

Anti-Patterns

• Unstructured printf logging in production code paths.
• Logging inside tight loops or per row of a batch job; log once per batch with counts.
• Copying full request/response bodies "for debugging" into production logs.
• Different field names for the same concept across services (user_id, userId, uid).
• Alerting on log text patterns as the primary detection path when metrics would be cheaper and more stable.
• Indexing high-cardinality fields (trace ID, user ID) as log labels.
• Logging secrets or tokens during error handling and never rotating them.
• Retaining DEBUG at production volume because "storage is cheap" — it is not, and it hides signal.

Checklist

• [ ] Newline-delimited structured logs with stable field names and a documented schema.
• [ ] trace_id and span_id automatically attached through the logging bridge.
• [ ] Severity policy documented: what pages, what tickets, what is noise.
• [ ] Per-template rate limits and level-based sampling in production.
• [ ] Source-level redaction plus collector-level redaction, with tests for nested and encoded fields.
• [ ] Multiline stack traces parsed before storage.
• [ ] One collection path per service (stdout or OTLP), no duplicates.
• [ ] Low-cardinality index fields; high-cardinality values in body or structured metadata.
• [ ] Retention tiers by log class; dropped-record counters exported.

Metrics

Scope: Prometheus/OpenMetrics data model, instrument types, exemplars, cardinality control, recording rules, native histograms, and PromQL patterns.

Data Model

• A metric family has a name, type, unit, and help text; it is identified by name plus label set. One unique label set is one time series.
• Samples are (timestamp, value) pairs. Counters are cumulative; gauges float; histograms carry bucket counts and a sum; summaries carry quantiles computed in-process.
• Exposition is pull over HTTP (/metrics), with push gateways and remote write for special cases. OTLP is the push-native alternative and is also accepted by Prometheus-compatible systems (verify upstream support and feature flags).
• Naming: namespace_subsystem_name_unit_suffix. Units are base units: _seconds, _bytes, _total for counters, _info for metadata, _ratio for dimensionless. http_requests_total, http_request_duration_seconds, node_memory_used_bytes.
• Label names are snake_case; label values are low-cardinality and bounded. Every label you add multiplies series count.
• Label values must never be unbounded: no user IDs, order IDs, raw paths, UUIDs, error strings, or timestamps.

Instrument Types

| Type | Semantics | Use for | Do not use for | |---|---|---|---| | Counter | Monotonic, resettable on restart | Requests, errors, bytes, events | Values that can decrease | | Gauge | Current value, up or down | Queue depth, connections, temperature, memory | Event counts | | Histogram | Distribution with configurable buckets | Latency, request size, payload size | Values that need exact quantiles per instance | | GaugeHistogram | Histogram of a current distribution | Snapshot distributions | Cumulative events | | Summary | Client-side quantiles | Legacy or single-process p99 | Aggregating across instances (quantiles are not aggregatable) | | Info | Constant metadata as a series | Build info, config version | Anything changing often |

• Histograms are aggregatable; summaries are not. For fleet-wide p99, use histograms and histogram_quantile, or native histograms. Never average quantiles.
• Instrument selection checklist: does it only go up (counter), does it go up and down (gauge), does the shape matter (histogram/gauge histogram)?
• Bucket boundaries are a compatibility surface. Changing buckets changes query results; version them and document the change. Choose buckets around SLO thresholds so the ratio of good to total is directly computable.
• OpenTelemetry instruments map onto this model: Counter to counter, UpDownCounter to gauge, Histogram to histogram, Gauge to gauge. Decide temporality (cumulative vs delta) once and keep it consistent; the collector can convert but conversion has edge cases.

Cardinality Control

Series count is approximately the product of label cardinalities. It is the main driver of memory, query time, and cost.

| Source | Typical explosion | Control | |---|---|---| | Raw URL path | One series per ID | Use templated routes (/orders/{id}) | | User/tenant ID | Millions | Aggregate; move to traces/logs | | Error message | One per unique string | Normalize to an error class | | Status code grouping | Acceptable (2xx, 5xx) | Keep raw codes when bounded | | Pod/host identity | Churns on every deploy | Drop from metrics used in aggregation; keep only where needed | | UUIDs, request IDs | Unbounded | Never a label | | Build metadata | Churns per release | Use a single _info series with a version label |

• Enforce at three layers: instrumentation review, collector limits (metric_relabel_configs, keep/drop actions, cardinality guards), and backend limits (per-tenant series limits). One layer will be bypassed eventually.
• Audit cardinality continuously: report top series counts by metric name and label set. The scrape_series_added metric and backend cardinality dashboards are the usual starting points.
• When a label is useful for debugging but not for aggregation, push it to traces or logs. Explore there, aggregate here.
• Relabeling can drop labels, replace values, hash, or keep only matching series. Do it centrally so every team benefits and no one has to rebuild dashboards.
• Avoid le collisions when creating histograms with custom buckets; duplicate bucket bounds corrupt the quantile calculation.

Exemplars

• Exemplars attach a trace ID (and often a span ID) to a specific bucket or counter observation. They are the bridge from an aggregate spike to a concrete request.
• Enable exemplars in the histogram instrument and in the scrape path (OpenMetrics exposes them; Prometheus needs exemplar storage enabled and OTLP carries them natively).
• Use them for latency histograms and error counters. When a p99 alert fires, the exemplar links straight to a slow trace instead of a log hunt.
• Keep exemplar volume bounded; they are sampled, but they still travel. Redact anything sensitive in the trace ID itself.
• Grafana-style exemplar-to-trace drilldown depends on the datasource supporting exemplar queries and the trace backend sharing the trace ID. Verify the full path in staging.

Recording Rules and Aggregation

• Recording rules precompute expensive or high-cardinality aggregations into stable series. Rules should produce series that dashboards and alerts consume directly; that is why rule names and labels are an interface.
• Rule hygiene: one logical aggregation per rule, descriptive names (job:http_requests:rate5m), labels only for dimensions users filter on (job, service, cluster), and a versioned file in git.
• Pre-aggregate high-cardinality raw metrics (per-route, per-instance) before storing long retention. The expensive detail can stay short-lived.
• Rule evaluation cost is real: wide by (...) clauses with high-cardinality labels re-create the cardinality you were avoiding. Cap group_left/group_right joins and prefer sum by over sum.
• Test rules with promtool test rules (or the Prometheus-compatible equivalent) in CI; rule errors are outages in alerting.
• for: durations, keep_firing_for, and staleness handling belong in alerting rules, not recording rules.

Query Patterns

# Request rate by service over 5 minutes
sum by (service) (rate(http_requests_total[5m]))

# Error ratio against total, guarding against division by zero
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / clamp_min(sum(rate(http_requests_total[5m])), 1)

# p99 from classic histogram buckets
histogram_quantile(0.99,
  sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

# p99 from native histograms
histogram_quantile(0.99, sum by (service) (rate(http_request_duration_seconds[5m])))

# Saturation: queue wait time
rate(queue_wait_seconds_total[5m]) / clamp_min(rate(queue_items_total[5m]), 1)

# Absence detection
absent(up{job="order-api"} == 1)

• Use rate on counters (it handles resets), never on gauges. increase answers "how many in this window" but is extrapolation-sensitive; prefer rate * window_seconds for consistent math.
• $__rate_interval (Grafana) or a fixed [5m] larger than the scrape interval prevents gaps and counter reset artifacts.
• Use subqueries and @ modifiers when you must align windows; keep them rare and documented, because they are expensive.
• topk, bottomk, and quantile operate per evaluation step and can flap; use them in dashboards with max_over_time or last_over_time where stability matters.
• Prefer sum by over sum only for the dims you need; every extra by dimension multiplies output series.
• absent() and up == 0 are the two ways to detect missing telemetry; a dashboard that cannot distinguish "zero" from "no data" is lying.

Push, Pull, and Storage

| Path | Mechanism | Use when | |---|---|---| | Scrape | Prometheus pulls /metrics | Kubernetes and VM fleets; standard for infrastructure | | Pushgateway | Short-lived jobs push finals | Batch jobs; never for long-running services | | Remote write | Prometheus forwards to a store | Long-term retention, multi-cluster aggregation | | OTLP | SDK pushes to collector/backend | OTel-native stacks and push-only networks | | Agent mode | Prometheus scrapes and remote-writes | Large fleets needing local aggregation |

• Retention tiers: high-resolution raw data hot (days), downsampled or recording-rule aggregates warm (weeks), rollups cold (quarters). Choose per signal value, not per default.
• Long-term stores (Mimir, Thanos, VictoriaMetrics, cloud-managed) add multi-tenancy, object storage, and global query. Costs track active series and query load more than raw bytes; measure both.
• Prometheus 3.x era notes: UTF-8 metric name support, OTLP ingestion improvements, native histogram maturity across the ecosystem — verify feature flags and backend support upstream before standardizing.
• Native histograms remove the bucket-boundary problem and reduce series count for high-cardinality distributions, but tooling, federation, and query ergonomics are still catching up. Run them alongside classic histograms during transition.
• Exporters: node exporter for hosts, kube-state-metrics for object state, language-specific exporters for runtime metrics, and application /metrics for domain metrics. One exporter per concern; avoid duplicate scrape targets.

Scrape Configuration and Relabeling

scrape_configs:
  - job_name: order-api
    kubernetes_sd_configs: [{ role: pod }]
    metrics_path: /metrics
    scrape_interval: 15s
    scrape_timeout: 10s
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: service
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
        action: replace
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "(go_gc_duration_seconds|process_open_fds)"
        action: drop

• Scrape intervals follow detection needs: 10-15 s for user-facing services, 30-60 s for infrastructure fleets, and longer for slow-moving capacity metrics. Align the for: durations in alerts with the interval.
• scrape_timeout must be lower than scrape_interval, or scrapes overlap and pile up. A timeout is a data-quality incident, not a retry opportunity.
• Target limits: set sample and series limits per scrape; a single target with millions of series can take down the server.
• Use honor_labels deliberately, and avoid labelless collisions when merging multiple exporters. Two targets producing the same series are silently deduplicated or rejected.
• Prefer ServiceMonitor/PodMonitor or annotation-based discovery for Kubernetes so onboarding is declarative and reviewable.
• Scrape via the collector or an agent (agent mode, remote write) when you need local filtering or many clusters; scrape directly for small, trusted fleets.

Native Histogram Migration

| Step | Action | Check | |---|---|---| | 1 | Measure classic bucket fit for key services | Are SLO thresholds inside existing buckets? | | 2 | Enable native histograms on a subset of exporters | Backend and remote-write support confirmed | | 3 | Dual-emit classic plus native during transition | Both query paths return consistent results | | 4 | Update recording rules and dashboards to native queries | histogram_quantile without le grouping | | 5 | Retire classic buckets when no downstream depends on them | Alerts and dashboards audited |

• Native histograms shrink series count dramatically for high-dimension latency metrics: one series instead of le times the labels.
• They support higher resolution and cheaper quantile queries, but federation, downsampling, and some managed services lag. Verify every hop in your path upstream.
• Mixed deployments are normal: some exporters and remote-write receivers support them, others do not. Dual-emit until the whole path is native.
• Query differences are subtle: native quantile functions do not use the le label, and rate semantics differ. Test against classic results before migrating alerts.
• Keep buckets for SLO thresholds even in a native world: converting a ratio of counts to a direct good/total computation is often simpler and easier to reason about than quantiles.

Anti-Patterns

• Averaging quantiles or averages across instances instead of aggregating histograms.
• One histogram per route with 30 custom buckets per route per instance per status code.
• Using a summary because it is easy, then discovering it cannot be aggregated fleet-wide.
• Adding a user_id or request_id label "temporarily".
• Recording rules that do not correspond to a dashboard or alert, burning CPU for nothing.
• Alerting on raw rate() thresholds when an SLO burn-rate alert is the correct abstraction.
• Scraping every instance every 5 seconds by default; pick intervals from detection needs.
• Leaving up and scrape error metrics unmonitored, so missing data looks like health.

Checklist

• [ ] Naming follows namespace_subsystem_name_unit_suffix; units are base units.
• [ ] Counters only increase; gauges used for fluctuating values; histograms for distributions.
• [ ] No unbounded labels; cardinality budget documented per service.
• [ ] Histogram buckets chosen around SLO thresholds; changes versioned.
• [ ] Exemplars enabled on latency histograms and drilldown verified end to end.
• [ ] Recording rules named, versioned, tested in CI, and consumed by dashboards/alerts.
• [ ] Alerting and dashboards distinguish zero from no data (absent/up).
• [ ] Retention tiers and long-term storage path defined before volume grows.
• [ ] Cardinality dashboards reviewed regularly by service owners.

Tracing

Scope: spans and trace structure, context propagation, head versus tail sampling, span links and events, cost control, and debugging with traces.

Span Model

| Field | Meaning | Rules | |---|---|---| | Trace ID | 16-byte identifier shared by all spans in a trace | Generated once at the root; propagated everywhere | | Span ID | 8-byte identifier for one operation | Unique within the trace | | Parent span ID | Causal parent | Empty for the root; sets the tree shape | | Name | Operation label | Low cardinality: GET /orders/{id}, db.query orders, not raw values | | Kind | server, client, producer, consumer, internal | Determines conventions and topology reconstruction | | Start/end time | Duration | Clock skew across hosts is expected; do not over-trust absolute order | | Attributes | Key/value metadata | Semantic conventions first; bounded values | | Status | UNSET, OK, ERROR | ERROR means the operation failed, not that an HTTP 4xx happened | | Events | Timestamped annotations on a span | Exceptions, retries, state transitions | | Links | Causality to other spans/traces | Batch fan-in, async handoffs, retries |

trace 4bf92f3577b34da6a3ce929d0e0e4736
├─ server GET /checkout              [frontend]        210 ms  OK
│  ├─ client POST /payments          [frontend]        120 ms  OK
│  │  └─ server POST /payments       [payments-api]    118 ms  OK
│  │     ├─ client INSERT payments   [payments-api]     34 ms  OK
│  │     └─ event payment.captured   [payments-api]        0 s OK (link -> settlement)
│  └─ client GET /cart               [frontend]         40 ms  ERROR
└─ internal cache.warm               [frontend]          5 ms  OK

• One trace per user-visible transaction. A trace that spans an async queue boundary is fine; a trace that spans unrelated requests is not.
• Prefer attributes over span names for detail. Renaming spans because an ID changed breaks dashboards and alert queries.
• Span events are cheaper than child spans for instantaneous facts; child spans are correct when the work has duration and its own causality.
• Span links express "related to" without parenting: batch consumers linking to producer traces, fan-in aggregations, and retried attempts. Use them whenever time order does not equal causality.
• Exception handling: record the exception event once at the deepest layer that understands it, set the span status to ERROR at the boundary that owns the failure, and avoid duplicate stack traces across every span in the path.

Context Propagation

| Transport | Mechanism | Notes | |---|---|---| | HTTP | W3C traceparent and tracestate headers | Default; verify proxies and CDNs forward them | | gRPC | Metadata key traceparent | Interceptors inject/extract | | Messaging | Attributes on the message (headers) | Producer injects, consumer extracts; trace continues or links | | Database | Not standard | Do not try to propagate through SQL; link instead | | Async runtimes | Context propagation across await/task boundaries | The instrumented library must carry context; lost context creates orphan spans | | Thread pools | Explicit context capture and restore | Always capture the context object before submitting work |

• W3C Trace Context is the interoperable baseline. Keep B3 or vendor formats only for legacy interop, and configure multi-format propagation during migration.
• Baggage carries small, cross-cutting values (tenant, release channel) to all downstream services. It is a bandwidth and privacy risk: keep it small, redact secrets, and never put PII in it.
• The three propagation failures to hunt: broken header forwarding at a proxy or gateway, an uninstrumented async hop (queue, cron, worker), and a client library that creates a new root instead of using ambient context.
• Testing propagation: send one request through the full path and assert that every hop shares one trace ID. "Spans exist" is not the test; "one trace exists" is.
• For background jobs, start a root span (or link to the triggering trace) so batch and scheduled work is observable rather than invisible.

Head Sampling

| Sampler | Behavior | Use when | |---|---|---| | always_on | Keep everything | Dev, low-volume, or tail sampling downstream | | always_off | Drop everything | Temporarily muting a noisy producer | | traceidratio | Deterministic ratio by trace ID | Uniform baseline; stable across instances | | parentbased_* | Respect the upstream decision | Default in service meshes; avoids broken traces | | Rate-limited | Adaptive target traces/sec | Unpredictable volume with a hard cap |

• Head sampling decides before the trace is complete, so it cannot keep "all errors". It is the cheap, predictable layer.
• Determinism matters: traceidratio keeps a trace consistent across all services, which random samplers may not. Prefer trace-ID-based sampling.
• Set the sampler once, at the SDK or collector entry, and preserve the decision downstream (parentbased). Mixing samplers mid-trace produces incomplete traces that look like bugs.
• For baseline volume, start with a small ratio (single-digit percent) plus tail sampling at the collector to recover errors and slow requests.

Tail Sampling

• Tail sampling buffers complete (or nearly complete) traces and decides based on outcome: errors, latency, attribute values, rates, or probabilistic fallback.
• Policies compose: keep all errors, keep latency above a threshold, keep a percentage of the rest, and cap volume with a rate limit. Order matters; the first matching keep-policy usually wins by configuration.
• Decision wait: the collector must hold spans for a window (decision_wait) that covers the slowest expected trace. Too short produces partial traces; too long costs memory and delays visibility.
• Sticky routing is mandatory at scale: all spans of a trace must reach the same collector instance, or each instance decides on a fragment. Use a load-balancing exporter keyed by trace ID between an agent tier and a tail-sampling gateway tier.
• Memory is proportional to spans per second times wait window. Size the gateway, set num_traces limits, and monitor dropped spans.
• Correlate with logs: if logs are sampled independently, kept traces may have no logs. Align log sampling with trace outcomes where incidents demand full narratives.
• Tail sampling belongs in the collector, not in the SDK, unless a single-instance deployment makes in-process buffering safe.

Cost Control

| Lever | Effect | Where | |---|---|---| | Head ratio | Linear volume reduction | SDK or agent collector | | Tail policies | Keeps interesting traces, drops healthy volume | Gateway collector | | Drop health checks/probes | Removes pure noise | Agent collector filter | | Attribute truncation | Reduces bytes per span | Collector or SDK | | Span limits | Caps attributes/events/links per span | SDK or collector | | Drop static asset/client spans | Removes low-value spans | Collector | | Route by service or tenant | Different budgets per tenant | Collector routing | | Batch tuning | Fewer, larger exports | Exporter config |

• Span limits (max attributes, events, links, attribute value length) are guardrails against a single runaway instrumentation. Set them explicitly; defaults vary by SDK.
• Span name cardinality also costs: a span name with an ID creates a metric-like explosion downstream in span-metrics connectors. Templatize.
• Compute cost per service before and after sampling: spans/sec, bytes/sec, and backend retention. A trace budget per tenant is the sustainable model.
• Never drop all spans of a trace selectively at the tail; keep complete traces, or link the remaining spans so the story is still reconstructable.

Debugging With Traces

1. Start from a symptom: a latency alert, an SLO burn, or a slow dashboard panel. Get a trace ID from an exemplar if available.
2. Read the critical path: the deepest branch on the slowest path, not the whole tree. Span duration plus self-time (duration minus child durations) finds where time is actually spent.
3. Check for missing spans: an uninstrumented hop or lost context often explains "the service is slow but no span shows it".
4. Read errors in context: status ERROR, exception events, and the attributes at the failing boundary. Compare a failing trace to a successful one of the same route.
5. Correlate outward: logs filtered by trace_id, profiles filtered by service and time, metrics by service and route.
6. Confirm with a controlled experiment: change one thing, reproduce, compare traces. Traces show causality, not proof; combine with metrics for impact.

| Symptom | Likely trace evidence | |---|---| | Tail latency spike | One slow child span, often a DB or external call | | Errors without logs | Status ERROR set but exception not recorded; or 4xx confusion | | Missing trace | Broken propagation at a hop or a new root span | | Duplicated work | Repeated sibling spans (retries, fan-out without dedup) | | Queue delay | Large gap between producer and consumer spans; link the traces | | Clock anomalies | Child starts before parent; check NTP and host clocks |

Propagation Code Pattern

# Inbound: extract context from the carrier (framework integrations do this for you).
from opentelemetry import trace, propagate, context
carrier = {"traceparent": request.headers.get("traceparent", "")}
ctx = propagate.extract(carrier)
token = context.attach(ctx)
try:
    with tracer.start_as_current_span("handle request", kind=SpanKind.SERVER):
        response = handle(request)
finally:
    context.detach(token)

# Outbound: inject ambient context into the outgoing carrier.
headers = {}
propagate.inject(headers)
http_client.post(url, headers=headers)

• Do not attach and detach context across long-lived tasks; capture the context object at submit time and restore it inside the worker.
• Async hops that are easy to miss: thread pools, setTimeout-style callbacks, queue consumers, stream processors, and retry wrappers. Verify each one with a trace assertion.
• Test propagation with a two-service integration test: one root trace ID, correct parent-child across the boundary, and no orphan root spans.

Tail Sampling Configuration Sketch

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
      - name: rate-cap
        type: rate_limiting
        rate_limiting: { spans_per_second: 5000 }

• Order matters and semantics vary across collector versions: verify whether the first matching policy decides the outcome or policies are evaluated as a set, and test with synthetic traffic.
• Keep a probabilistic policy so healthy traffic is still observable; an errors-and-latency-only policy makes baselines invisible.
• Monitor tail-sampling memory and eviction. Evicted traces are silently partial; export the eviction metric and alert on sustained eviction.
• Correlate sampled spans with log sampling so kept traces retain their logs (./02-logging.md).

Anti-Patterns

• Span names containing raw URLs, IDs, or SQL text.
• High-cardinality attributes on every span, including request bodies and headers.
• Installing a new global tracer provider per library; only the application owns the provider.
• Sampling independently at every hop without parentbased, creating partial traces.
• Tail sampling across multiple instances without sticky routing.
• Treating traces as a complete audit log; sampling means they are a sample by design.
• Recording exceptions on every span as the error propagates.
• Ignoring orphan/root-span rates as a health metric for instrumentation coverage.

Checklist

• [ ] One root span per user-visible transaction; span kinds set correctly.
• [ ] W3C Trace Context propagation verified end to end, including queues and cron.
• [ ] Span names low-cardinality; attributes follow semantic conventions and have bounded values.
• [ ] Head sampler deterministic and parent-based; baseline ratio documented.
• [ ] Tail sampling policies cover errors, latency, and a probabilistic remainder, with sticky routing.
• [ ] Span limits configured; health checks and static assets dropped.
• [ ] Trace-to-logs and trace-to-metrics drilldowns verified in the UI.
• [ ] Orphan-span rate and dropped-span counters monitored.
• [ ] Trace budget per service/tenant defined and reviewed with cost data.

Profiling

Scope: continuous profiling, CPU/heap/goroutine profiles, pprof, eBPF-based profilers, and correlating profiles with traces.

Why Continuous Profiling

• Metrics tell you something is slow; traces tell you which service and span; profiles tell you which function, line, and allocation. Profiles are the code-level layer of observability.
• Continuous profiling samples production at low overhead (typically low single-digit percent CPU with sane settings) and aggregates over time, replacing ad-hoc "SSH and run perf" sessions.
• Profiles are aggregated by labels (service, version, host, region, and ideally span or trace) into flame graphs. Aggregation across time makes small costs visible and attributable.
• Use continuous profiling when you need to answer: what consumes CPU after a deploy, what allocates, what leaks, what blocks, and which thread/goroutine holds resources.

Profile Types

| Profile | Measures | Question it answers | Typical signal | |---|---|---|---| | CPU (on-CPU) | Sampled stack traces of running code | Where is CPU time spent? | process_cpu / wall-clock samples | | Wall clock | Stacks by elapsed time including waits | Where is latency spent? | Blocking, I/O waits, scheduler | | Allocation | Bytes and objects allocated | What allocates and churns? | GC pressure, hot allocations | | Heap (in-use) | Live objects at snapshot | What holds memory? | Leaks, growth over time | | Goroutine/threads | Count and stacks of live goroutines/threads | Why are threads piling up? | Leaks, blocked workers | | Block/mutex | Blocked and contended time | What serializes work? | Lock contention, pool exhaustion | | Off-CPU | Stacks while not running | Where does time go while idle? | I/O, locks, page faults | | Lock/contention | Per-lock wait profiles | Which lock hurts? | Contended hot locks |

• On-CPU profiles miss everything waiting. If latency grows while CPU is flat, look at wall-clock or off-CPU profiles.
• Allocation and heap profiles answer different questions: allocation finds the churn, heap finds the retention. Use both for memory incidents.
• Goroutine/thread leaks look like slow memory growth with steady CPU; profile goroutine counts over time, not a single snapshot.
• Sampling profilers assume some regularity; very short-lived processes and functions may be underrepresented. Correlate with traces and metrics before concluding.

Formats and Tooling

| Tool | Profiles | Notes | |---|---|---| | pprof | CPU, heap, goroutine, block, mutex, threadcreate | Ubiquitous Go; also consumed by many profilers | | Linux perf | CPU, off-CPU, call graphs | System-wide to container-level; needs symbols | | async-profiler / JFR | CPU, allocation, lock, wall | JVM standard in production | | py-spy | CPU, wall, native stacks | Python without code changes | | dotnet-trace / dotnet-gcdump | CPU, GC, allocations | .NET production profiling | | --cpu-prof / --heap-prof (Node) | CPU, heap | Built into the runtime; snapshots | | Pyroscope, Parca, Polar Signals, Coroot | Continuous, aggregated | Server components plus language agents | | Cloud APM profilers | Continuous, vendor-hosted | Convenient, less portable; watch cost and egress |

• The pprof wire format is the lingua franca: language agents, eBPF agents, and backends move pprof over HTTP for ingestion and query. OTLP profiles exist as a signal but are stabilizing — verify upstream support.
• Keep the profiling agent's ingestion path separate from the request path; a blocked profiler upload must never add latency to the application.
• Symbols matter: build with debug info or a symbol server and keep a mapping from build ID to symbols. Profiles without symbols are just stack addresses.
• Frame pointers or DWARF unwinding: some runtimes and eBPF profilers assume frame pointers for clean stacks. Decide the build flag per language and test unwinding in staging.

eBPF-Based Profiling

• eBPF profilers sample stacks in the kernel without linking a language agent, which covers any runtime, including legacy binaries and services you cannot redeploy.
• Typical coverage: on-CPU stacks, off-CPU and I/O waits, some allocation/network signals, and language-aware unwinding for common runtimes.
• Constraints: kernel version and build flags, container privilege model (CAP_BPF or privileged), symbol availability, stripped binaries, and JIT runtimes (JVM, Node, .NET) needing per-runtime unwinders. Verify support upstream for your exact versions.
• Security review: eBPF agents see process memory and syscalls. Run with least privilege, restrict the agent's load surface, and treat the profiling data as sensitive (function names, file paths, sometimes string arguments).
• In clusters, run one profiling agent per node (DaemonSet) and aggregate centrally; per-pod sidecars are usually not worth the overhead.

Correlating Profiles With Traces

| Level | Mechanism | Precision | |---|---|---| | Service + time window | Labels shared by all signals | Coarse; good for regressions | | Version/deploy | service.version on profiles | Attributes a regression to a release | | Span | Span ID attached to samples during a span | Pinpoints the span that was slow | | Trace | Trace ID on samples | Reconstructs the full request cost | | Function | Stack labels plus trace attributes | Direct code-to-request link |

• Span profiles (sampling labels while a span is active) connect a slow trace to the exact stack consuming its time. This is the highest-value correlation; verify what your profiler supports upstream.
• Grafana-style "profiles for this service" drilldown from a trace depends on shared service.name and time ranges. Ensure clock sync (NTP) across nodes or correlation windows will be off.
• When a latency alert fires, the workflow is: exemplar or trace to find the slow request, then span profile or service profile to find the function, then a fix. Without correlation, profiling becomes guesswork.
• Keep profiling labels low-cardinality except span/trace IDs, which are intentionally high-cardinality but short-lived. Retention for trace-correlated samples can be shorter than for aggregate profiles.

Workflow

1. Establish the question and the profile type: CPU, allocation, heap, locks, goroutines, or off-CPU.
2. Capture a baseline before the change, ideally at the same load level and time of day.
3. Collect with symbols, correct labels, and enough duration to cover the workload cycle (at least several minutes; hours for leaks).
4. Read top-down for call hierarchy and bottom-up for self-time. Ignore functions that are cheap; fix the widest bar.
5. Check for measurement artifacts: sampling bias, missing symbols, one host dominating the aggregate, or a single outlier trace.
6. Fix, redeploy, recapture, and compare the same view. Record the before/after in the change description.
7. Guard with a benchmark or budget where possible; continuous profiling then serves as the regression detector.

Overhead and Safety

• Sampling frequency is the main overhead knob. Defaults are usually sane; raising frequency for a short investigation is fine, keeping it raised forever is not.
• Heap profiles can stop the world or force a GC depending on the runtime and options; do not run them continuously at full fidelity in latency-sensitive services.
• Profiling endpoints (/debug/pprof, JMX, runtime diagnostics) expose code paths, memory, and sometimes secrets. Bind them to localhost or an internal port, require auth, and never expose them publicly.
• Profiling agents add CPU, memory, and network egress. Budget them like telemetry: measure the agent's own cost and cap upload rate.
• PII risk: stack traces include file paths and function names; some profilers capture string arguments or allocation call sites. Treat profile data under the same privacy policy as logs.
• Test the profiler in staging after kernel, runtime, and orchestrator upgrades; unwinding and privilege models are the usual breakage points.

Reading a Flame Graph

• Width is cost or time, not call order. A wide frame is expensive; a tall stack is deep, not necessarily slow.
• Top-down view: follow the widest branch from the root to understand which subsystem dominates. Bottom-up view: sort by self-time to find the function that actually burns the samples.
• Self-time versus total time: wrapper and dispatcher frames look expensive in total time but cheap in self-time. Optimize self-time first.
• Look for known landmarks: JSON/marshal, regex compilation, lock waits, GC, syscalls, and allocation paths. Recurring shapes across services usually mean shared library costs.
• Compare, do not eyeball: side-by-side before/after or version-over-version flame graphs make regressions obvious. A single graph in isolation rarely proves anything.
• Watch for sampling artifacts: a new function appearing only at the top of a broken stack, missing frames from inlined code, or one host dominating the aggregate.
• Inverted and differential views: inverted flame graphs (icicle) are easier for deep stacks; differential graphs color added versus removed cost per frame.

Profile Storage and Labels

| Label | Purpose | Cardinality | |---|---|---| | service.name | Primary grouping | Low | | service.version | Regression attribution | Medium; churns per deploy | | deployment.environment.name | Environment split | Low | | k8s.pod.name / host.name | Instance comparison | High but bounded by fleet size | | region, zone | Locality comparison | Low | | span_id, trace_id | Request correlation | Very high; short retention |

• Keep label sets identical to traces and metrics so all three overlay in one UI (./01-opentelemetry.md). Drift makes correlation impossible.
• High-cardinality labels (pod, span) are useful for spot checks but expensive for long retention. Aggregate profiles by service and version for the long term; keep per-instance and per-span data short.
• Storage format: pprof over HTTP is the common ingest and query format; some backends store columnar representations for large-scale aggregation. Verify the query and ingest formats your backend supports upstream.
• Query language is usually label selector plus profile type plus time range, with optional function filters. Learn the top queries: top functions by self-time, cost by version, and compare two windows.
• Access control: profiles reveal internal structure. Restrict who can query per-environment data and audit exports.

Language Notes

| Runtime | Practical profiler | Notes | |---|---|---| | Go | net/http/pprof, runtime/pprof | CPU, heap, goroutine, block, mutex; sample rate configurable; goroutine profiles are cheap and excellent for leaks | | JVM | async-profiler, JFR | Wall-clock and allocation modes are the differentiators; JFR is built in and low overhead | | Python | py-spy, OTel profiling agents | GIL means wall-clock and native stacks matter; pure-Python CPU profiles can mislead | | Node.js | --cpu-prof, inspector, profiling agents | Event-loop lag metrics pair well with profiles; async stacks need runtime cooperation | | .NET | dotnet-trace, dotnet-gcdump | GC and allocation profiles are first-class; EventPipe is the mechanism | | Native (C/C++/Rust) | perf, eBPF agents | Symbols and frame pointers dominate profile quality; stripped release builds need a symbol store |

• Always check whether the profile is CPU-time or wall-clock. For I/O-bound and lock-bound services, CPU profiles systematically understate the problem.
• For managed runtimes, separate GC time from application time; a "CPU" profile that is mostly collector work points at allocation reduction, not algorithm changes.
• Keep profiler sampling rates sane in production; investigate with higher rates only for short windows.

Anti-Patterns

• Profiling only in development, where load and data shape rarely match production.
• Chasing the widest bar without checking self-time; a wrapper function is not a hotspot.
• Comparing profiles from different build types (debug vs release) or different optimization flags.
• Leaving pprof endpoints publicly reachable.
• Treating a single heap snapshot as a leak proof; leaks are a trend over time.
• Ignoring GC pauses and off-CPU time because the CPU profile looks clean.
• Running full-fidelity heap profiling continuously in a p99-sensitive service.
• Profiling without symbols and then concluding "the profile is useless".

Checklist

• [ ] Continuous profiling runs in production with documented overhead and an owner.
• [ ] Build artifacts carry symbols or a symbol mapping; build IDs recorded.
• [ ] Profile types selected per question (CPU, allocation, heap, goroutine, locks, off-CPU).
• [ ] Service, version, and environment labels consistent with traces and metrics.
• [ ] Span or trace correlation enabled where supported and verified in staging.
• [ ] Profiling endpoints private, authenticated, and off the public listener.
• [ ] Retention and egress budget defined for profile data.
• [ ] Regression workflow: baseline, fix, recapture, compare, guard.
• [ ] eBPF agent privilege and kernel/runtime compatibility reviewed on upgrades.

Alerting and SLOs

Scope: SLIs/SLOs, error budgets, burn-rate alerts, symptom versus cause alerts, routing, on-call ergonomics, and runbooks.

Definitions

| Term | Definition | Example | |---|---|---| | SLI | Measured service attribute from the user's point of view | Fraction of requests served under 300 ms | | SLO | Target for an SLI over a window | 99.9% of requests under 300 ms over 28 days | | SLA | Contractual commitment with consequences | 99.9% availability, service credits below | | Error budget | Allowed failure: 1 - SLO | 0.1% of requests, about 43 minutes per 30 days | | Burn rate | Speed of budget consumption relative to the SLO window | 1x exhausts in exactly one window |

• SLIs are ratios of good events to valid events. Define all three: what counts as good, what counts as valid, and what is excluded (health checks, bot traffic, client aborts).
• Measure at the closest point to the user: load balancer or ingress for availability and latency; service-side when the load balancer cannot see semantics. State the choice in the SLO document.
• Choose windows by decision horizon: rolling 28 or 30 days for the headline SLO, 1 to 7 days for fast feedback, and calendar months when the business reports monthly.
• Few SLOs, high value: start with availability and latency per user-facing service. Add correctness, freshness, durability, or throughput only where they drive user decisions.

SLI Menu

| SLI type | Good event | Valid event | Watch out for | |---|---|---|---| | Availability | Response not 5xx and completed | All requests | Exclude client errors you do not own, but document it | | Latency | Response under threshold | All successful requests | Pick a threshold users feel, not the current p99 | | Throughput | Work completed on time | Submitted work | Useful for batch and streaming | | Correctness | Result validated | Results produced | Hard to measure; use sampled validation | | Freshness | Data age under threshold | Data reads | ETL and cache-based systems | | Durability | Data retrievable | Data written | Backups and replication health | | Saturation proxy | Queue wait under threshold | Requests | Leading indicator, not an SLI replacement |

# Availability SLI as a ratio over a 28-day window (recording rule input)
sum(rate(http_requests_total{status!~"5.."}[5m]))
  / clamp_min(sum(rate(http_requests_total[28d])), 1)

• Prefer request-based SLIs for request/response services; time-based "uptime" SLIs hide partial failures and are hard to alert on.
• Synthetic probes are valid SLIs for critical entry points when traffic is low; combine with real traffic and label the source.
• Keep the SLI implementation in a recording rule so dashboards, alerts, and reports share one definition. Multiple implementations of the same SLI is a defect.

Error Budget Policy

• Write the policy before the first breach: what happens at 50%, 75%, and 100% budget consumption, who decides, and how the team returns to feature work.
• Typical policy: under 50% consumed, ship normally; 50-75%, focus on reliability work; over 75%, feature freeze except reliability and security; exhausted, mandatory freeze plus a postmortem.
• Exhaustion is not a punishment; it is a signal that reliability work now has higher priority than features. This only works if leadership agrees in advance.
• Track budget by service and by tenant where multi-tenant, and review at a fixed cadence. A budget nobody looks at changes nothing.
• Freeze exceptions must be explicit and time-boxed, recorded in the incident or change log.

Burn-Rate Alerts

Multi-window, multi-burn-rate is the default pattern for SLO alerting. Burn rate B means the budget is consumed B times faster than the SLO window allows; B = 1 exhausts exactly at the end of the window.

| Budget consumed | Long window | Short window | Burn rate | Severity | |---|---|---|---|---| | 2% | 1 h | 5 m | 14.4x | Page | | 5% | 6 h | 30 m | 6x | Page | | 10% | 1 d | 2 h | 3x | Ticket | | 10% | 3 d | 6 h | 1x | Ticket |

• The long window detects, the short window confirms, and both must be above threshold before firing. This kills flapping and long tails without losing fast burns.
• Formula shape: error_ratio > (1 - SLO) * burn_rate over the window, with the long and short conditions ANDed.
• Page only on burns that consume budget fast enough to matter within a shift. Slow burns become tickets.
• Add a "watchdog" always-firing alert that routes to a dead-man's-switch receiver. If alert evaluation or routing breaks, silence is ambiguous; the watchdog makes it explicit.
• For each alert, define the expected consumer action: roll back, scale, fail over, or investigate. No action means no page.

Symptom vs Cause

| Alert type | Example | Route to | Why | |---|---|---|---| | Symptom | p99 latency SLO burn, error rate SLO burn | Page | Directly user-visible | | Saturation | Queue age growing, connection pool near limit | Page or ticket | Leading indicator of symptom | | Cause | CPU high, disk 90%, one pod crash-looping | Dashboard or ticket | Often self-healing or covered by the symptom | | Capacity | Forecasted exhaustion in 30 days | Ticket | Not urgent at 3 a.m. | | Informational | Deploy completed, autoscaler scaled | Chat | No human action |

• Alert on what the user experiences, not on what you think causes it. "CPU is high" is only page-worthy when it is the direct cause of a user symptom and no symptom alert covers it.
• For each page, the question is "what would the on-call do now?" If the answer is "watch it", it is not a page.
• Dependencies: alert on your own SLIs, not on a dependency's internal metrics. If the dependency's failure harms users, your symptom alert fires; their causes are their problem.

Alert Design Rules

• Every alert needs an owner, a severity, a runbook link, and a clear action. Encode owner and runbook in labels so routing is automatic.
• Use for: to require persistence; set it from the detection profile, not habit. A 5-second spike should not page.
• Prefer ratios over absolute counts, and rates over totals. "Error ratio above 5% for 10 minutes" beats "500 errors".
• Group by the dimension that determines action (service, region, tenant), never by pod or instance, to avoid a page per replica.
• Inhibition: suppress downstream alerts when an upstream cause is already paging. Silences are time-boxed and require a reason.
• Test alerts: unit-test rules in CI, and deliberately break a canary in staging to confirm routing, payload, and runbook accuracy.
• Alert payload must include: service, environment, window, current value, threshold, dashboard link, runbook link, and the SLO/budget context.
• Alert on absence: if a service stops reporting, a "no data" alert should fire. Silent producers are worse than noisy ones.

Routing and On-Call

| Route | Criteria | Channel | Response | |---|---|---|---| | Critical | Page-worthy SLO burn, data loss | Pager with escalation | Immediate ack, incident process | | High | Degradation users notice | Pager during business-critical windows | Ack within minutes | | Medium | Risk of future degradation | Chat channel plus ticket | Same business day | | Low | Cleanup, forecast | Ticket queue | Scheduled |

• Route by team ownership labels, not by service name lists maintained by hand. Ownership data belongs in a service catalog and must feed routing.
• Escalation policies: primary, secondary, then engineering manager; define ack timeouts and repeat notifications. Test the escalation path after every rotation change.
• On-call ergonomics: cap pages per shift, ban pages for non-actionable conditions, protect sleep, and rotate consistently. A noisy rotation produces ack-and-ignore behavior.
• Track alert quality: pages per shift, percentage acked, percentage actioned, false-positive rate, mean time to acknowledge, and top noisy alerts. Review monthly and delete or fix the worst offenders.
• Handovers should include ongoing incidents, silences, and burn status; silences that outlive the event are a recurring source of blindness.
• Runbooks: symptom, impact, first three commands, dashboards to open, escalation path, rollback steps. Keep them in version control and link from the alert payload. A runbook nobody tested is a wish.

Example Burn-Rate Rule

groups:
  - name: slo-burn
    rules:
      - record: slo:availability:error_ratio5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / clamp_min(sum(rate(http_requests_total[5m])), 1)
      - alert: AvailabilityBudgetBurnFast
        expr: |
          slo:availability:error_ratio5m > (14.4 * 0.001)
          and
          (sum(rate(http_requests_total{status=~"5.."}[1h]))
            / clamp_min(sum(rate(http_requests_total[1h])), 1)) > (14.4 * 0.001)
        for: 2m
        labels: { severity: critical, team: order-api }
        annotations:
          summary: "Order API burning error budget 14.4x over 1h"
          runbook_url: "https://runbooks/order-api/availability-burn"

• Threshold (1 - SLO) * burn_rate with (1 - SLO) = 0.001 for a 99.9% SLO gives 1.44% error ratio for the 14.4x page. Keep the arithmetic visible in the rule.
• Record the error ratio once and reuse it; the long and short windows are the only difference.
• Add severity, team, service, environment, and runbook_url labels so routing and payload generation need no per-alert special cases.
• Test the rule with a time series that simulates a burn and assert the alert fires with the expected labels.

SLO Document Template

Service:        order-api
Owner:          payments platform team
User journey:   customers place and view orders
SLI:            proportion of valid HTTP requests that succeed and complete under 300 ms
Good event:     status < 500 and duration <= 300 ms
Valid event:    all requests to /orders routes, excluding health checks and bot traffic
Measurement:    ingress proxy metrics, 28-day rolling window
SLO:            99.9% availability, 99.5% latency
Error budget:   0.1% (approx. 40 minutes of full outage per 28 days)
Policy:         <50% ship normally; 50-75% reliability priority; >75% freeze; 100% mandatory review
Alerts:         14.4x/1h page, 6x/6h page, 3x/1d ticket
Dashboards:     Order API / SLO
Runbook:        runbooks/order-api/availability-burn
Review cadence: monthly budget review, quarterly target review

• One page per service. The document is the contract; if the SLI implementation and the document disagree, the document wins and the code is fixed.
• Exclusions must be justified and measurable; "excluding weird traffic" is not a definition.

Anti-Patterns

• Alerting on every metric with a static threshold; the team learns to ignore the pager.
• SLOs invented at the end of the quarter to match current performance rather than user expectations.
• Error budgets with no policy, so exhaustion has no consequence.
• Pages that only say "CPU high" without impact or action.
• One alert per pod or per instance instead of per service.
• Runbooks describing a system that no longer exists.
• Silencing alerts permanently instead of fixing root causes.
• Using burn-rate alerts without the fast/slow window pair, causing flapping or missed fast burns.
• No dead-man's-switch alert, so broken alerting looks like a healthy system.

Checklist

• [ ] SLI definition documented: good event, valid event, exclusions, measurement point.
• [ ] SLO target, window, and error budget agreed with stakeholders.
• [ ] Error budget policy written and signed off, including freeze behavior.
• [ ] Multi-window burn-rate alerts for page and ticket thresholds.
• [ ] Symptom-first alerting; cause metrics on dashboards.
• [ ] Every alert has owner, severity, runbook link, and runbook content.
• [ ] Routing from a service catalog with tested escalation policies.
• [ ] Watchdog/dead-man's-switch alert active.
• [ ] Alert-quality metrics reviewed monthly; noisy alerts fixed or deleted.
• [ ] Alert rules tested in CI and validated with a staging break test.

Dashboards

Scope: Grafana, golden signals, RED/USE methods, dashboard UX, exemplar-to-trace drilldown, and dashboards as code.

Purpose and Audience

| Audience | Question | Time range | Depth | |---|---|---|---| | On-call | Is the service healthy right now? | Minutes to hours | RED/golden signals, SLO burn | | Incident responder | Where is the problem? | Minutes to hours | Traces, logs, dependency health | | Service owner | How is this release behaving? | Days to weeks | Trend by version, budget, saturation | | Capacity planning | What runs out first? | Weeks to quarters | Growth, headroom, forecasts | | Business | Are users getting value? | Days to months | Funnels, transactions, revenue |

• One dashboard, one question. If a panel does not help answer the title question, it belongs on another dashboard.
• Order panels top-down by urgency: health and SLO first, then failure modes, then dependencies, then capacity.
• Time ranges and refresh intervals are part of the design. A 5-second auto-refresh on a 90-day panel is a query-cost anti-pattern.

Methods

| Method | For | Signals | |---|---|---| | Golden signals | User-facing services | Latency, traffic, errors, saturation | | RED | Request-driven services | Rate, errors, duration | | USE | Resources (CPU, memory, disk, network) | Utilization, saturation, errors | | Four golden signals plus dependency health | Distributed systems | Golden signals plus queue age and pool state | | Event/cost views | Platform and FinOps | Deploys, incidents, telemetry spend |

• RED answers "is the service well?"; USE answers "is the resource well?". Most service dashboards need RED plus the saturation signals that predict RED failures.
• Latency panels should show a distribution shape (p50/p90/p99 or a heatmap), not just one number. The mean hides everything that matters.
• Traffic panels should show the unit users care about (requests, jobs, messages), and be labeled with the unit. "Ops" is meaningless.
• Errors: show error ratio, not just counts, and split client versus server errors. A count spike can be traffic growth.
• Saturation: queue depth, queue age, connection pool usage, thread/goroutine pools, memory pressure. Queue age is often the earliest warning of overload.

Grafana Building Blocks

| Concept | Use | |---|---| | Datasources | Prometheus-compatible, Loki, Tempo, Pyroscope, ClickHouse, cloud stores | | Variables | service, environment, region, cluster, version; cascade when possible | | Transformations | Join, reduce, rename, calculate; keep logic in queries if the datasource can do it | | Annotations | Deploys, incidents, feature flags, batch windows | | Referencing | Data links, dashboard links, panel links, trace/log drilldown | | Thresholds and units | Set explicit units and color thresholds tied to SLOs | | Rows and repeats | Repeat by variable to scale without duplicating panels |

• Variables should default to something useful ($service with a sensible default, $env=prod) so the dashboard opens with data.
• Service and environment labels must exactly match the resource attributes defined in ./01-opentelemetry.md; drift between the datasource label and the variable is the most common broken dashboard.
• Color thresholds must come from SLOs: green inside budget, yellow approaching, red burning. Green/red that does not map to an objective is decoration.
• Statistics panels for single numbers (current error budget, deploy version), timeseries for trends, heatmaps for latency distributions, tables for top-N and per-instance detail.
• Keep one dashboard "entrypoint" per service with links to detail dashboards, runbooks, logs, and traces.

Drilldown Paths

dashboard panel (p99 latency spike)
  -> exemplar (trace_id on the bucket)
    -> trace view (critical path, slow span)
      -> span profile (function consuming time)      [./05-profiling.md]
      -> logs filtered by trace_id                    [./02-logging.md]
      -> RED metrics for the downstream service       [./03-metrics.md]

• Exemplars only work when the SDK, scrape/OTLP path, backend, and datasource all carry them. Verify one known-slow request makes the full jump before declaring the drilldown done.
• Trace-to-logs needs the trace ID in the log backend and a consistent field name. Trace-to-metrics needs span-metrics or service-graph data.
• Data links should open in the same time range and with the same service variable. Passing the wrong range is a silent source of confusion.
• For incidents, provide one "triage board" that pulls the golden signals of the affected service and its critical dependencies side by side, plus deploy annotations.
• Avoid dead ends: every panel that can indicate a problem should link somewhere actionable.

Dashboards as Code

| Approach | Fit | Notes | |---|---|---| | Grafana provisioning with JSON | Small teams, stable dashboards | Files in git, applied by Grafana or a sidecar | | Grafonnet / Jsonnet | Many similar dashboards | Programmatic generation; reviewable diffs | | Terraform provider | Platform teams managing many resources | Dashboards alongside alerts and datasources | | Dashboard API in CI | Custom workflows | Validate JSON and apply on merge |

• Version dashboards in the same repository as alerts and collector config. An alert without its dashboard is half a system.
• Review dashboard diffs like code: query changes are behavior changes, especially for SLO panels and thresholds.
• Lint for common defects: missing units, missing datasource variable, hardcoded service names, unbounded .* regexes, missing $__rate_interval.
• Never edit production dashboards by hand without exporting them back to git; drift is guaranteed otherwise.
• Name dashboards with a consistent scheme (Service / Overview, Service / Dependencies, Platform / Cluster) so search works.

Query Cost and Performance

• Every panel runs queries on every refresh. A dashboard with 40 panels and 30-second refresh can generate more load than production traffic.
• Prefer recording rules for expensive aggregations (./03-metrics.md) and query those. Panels should rarely do heavy sum by over raw high-cardinality metrics.
• Use $__rate_interval or a fixed range larger than the scrape interval; too-short ranges produce artifacts and gaps.
• Cap lookback windows on top-N tables and logs panels; unbounded time ranges are the classic cause of query timeouts during incidents.
• Set per-datasource and per-panel timeouts, and design the dashboard to degrade (fewer panels, longer intervals) under load.
• High-cardinality log queries ({} |= "...") can be expensive; constrain by label first, then text.

Reference Layout: Service Overview

| Row | Panels | Query shape | |---|---|---| | Health | SLO budget remaining, burn rate, current version | Recording rules from ./06-alerting-slo.md | | Traffic | Requests/sec by route, active connections | sum by (route) (rate(http_requests_total[5m])) | | Errors | Error ratio, errors by class, top failing routes | sum by (route) (rate(...{status=~"5.."}[5m])) | | Latency | p50/p95/p99, latency heatmap, slowest routes | histogram_quantile over _bucket or native | | Saturation | In-flight, queue depth, queue age, pool usage | Gauges and rate over wait counters | | Dependencies | Per-dependency latency and error ratio, service graph | Client span metrics or servicegraph connector | | Runtime | Memory, GC pauses, goroutines/threads, event-loop lag | Auto-instrumentation runtime metrics | | Deploys | Versions over time, deploy annotations, error ratio by version | _info series and annotations |

• Latency and error panels must share the same route breakdown so a spike in one is immediately comparable in the other.
• Version panels expose regressions that averages hide: plot error ratio and latency by service.version after every deploy.
• Dependencies row is what turns "our service is slow" into "our service waits on payments"; keep it adjacent to latency.
• For batch and queue systems, replace the traffic row with throughput, backlog, and oldest-message-age panels.

Provisioning and Linting

# Grafana provisioning: dashboards loaded from version control
apiVersion: 1
providers:
  - name: observability
    orgId: 1
    folder: services
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    options:
      path: /etc/grafana/dashboards

{
  "title": "Order API / Overview",
  "uid": "order-api-overview",
  "tags": ["service", "red"],
  "templating": {"list": [{"name": "env", "type": "custom", "query": "prod,staging"}]},
  "panels": [{"type": "timeseries", "datasource": {"type": "prometheus", "uid": "${DS}"}}]
}

• uid is the stable identity used in links; never auto-generate it, and keep it unique across environments.
• Provision read-only dashboards in production and export any ad-hoc edits back to code; use folders and tags for discoverability.
• CI lint checklist: valid JSON, resolvable datasource UIDs, explicit units, no hardcoded service names, $__rate_interval used for rates, and thresholds aligned with SLOs.
• Validate queries in CI against a test datasource where possible; a dashboard whose queries error is worse than no dashboard because it hides real data.
• Generate repetitive dashboards (one per service) from a template so an onboarding change is one pull request, not a copy-paste across dozens of files.
• Record a screenshot or rendered snapshot in the pull request for UX changes; dashboard review is visual and diff review alone misses layout regressions.

Anti-Patterns

• Wallpaper dashboards: dozens of panels, no ordering, no owner, no question answered.
• Panels without units or with misleading units (seconds showing milliseconds).
• Averages for latency; percentiles without a distribution view.
• Hardcoded service or environment filters instead of variables.
• Auto-refresh faster than the data resolution.
• SLO panels that do not match the SLO definition used for alerting.
• Exemplar drilldown configured but broken at one hop.
• Dashboards owned by nobody and updated only during incidents.
• Hand-edited production dashboards never committed to git.

Checklist

• [ ] Each dashboard has a stated question, an owner, and a consistent name.
• [ ] Golden signals / RED at the top; saturation and dependencies below.
• [ ] Units, thresholds, and colors tied to SLOs and documented.
• [ ] Variables for service, environment, and cluster; useful defaults.
• [ ] Deploy and incident annotations enabled.
• [ ] Exemplar-to-trace and trace-to-logs drilldowns verified end to end.
• [ ] Expensive panels backed by recording rules; query ranges bounded.
• [ ] Dashboards provisioned from code and reviewed via pull requests.
• [ ] Triage board exists for incidents and links to runbooks.

Pipelines and Cost

Scope: collector pipelines, filtering and transforms, retention tiers, sampling strategy, and telemetry cost governance.

Pipeline Anatomy

receivers -> processors -> exporters        (per signal pipeline)
              |                            connectors bridge pipelines
              +-> alternatives: fan out, route, sample, aggregate

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, filter, tail_sampling, batch]
      exporters: [otlp/backend]

• One pipeline per signal and per destination. Use the routing connector for multi-tenant or multi-region fan-out instead of duplicating processors.
• Processor order is functional: limit memory first, enrich early, filter/transform before sampling, batch last.
• memory_limiter must be first and configured with a hard/soft limit and a check interval; it is the difference between backpressure and OOM.
• batch tuning: larger batches reduce network calls but add latency and memory. For traces, batch by span count and size; for metrics, keep scrape-aligned timing.
• Queueing and retry belong on exporters (sending_queue, retry_on_failure). Enable persistent queues only when durable delivery matters more than disk and throughput.
• Self-telemetry: export collector metrics (accepted, refused, dropped, queue length, batch size) and alert on refusals and queue saturation.

Filtering and Transforms

| Processor | Purpose | Typical use | |---|---|---| | filter | Drop spans/logs/metrics by condition | Drop /healthz, readiness, static assets | | transform (OTTL) | Rename, set, delete, parse attributes | Normalize schemas, redact, add routing keys | | attributes | Add/update/delete attributes | Add environment, delete internal fields | | resource | Modify resource attributes | Force deployment.environment.name | | redaction | Mask sensitive values by pattern | Strip tokens, emails, card-like numbers | | k8sattributes | Attach pod/namespace/deployment metadata | Agent tier enrichment | | resourcedetection | Detect cloud/host metadata | Enrich once at the gateway | | probabilistic_sampler | Sample by trace ID ratio | Baseline volume control for traces/logs | | tail_sampling | Outcome-based trace sampling | Keep errors and slow traces | | metricstransform | Aggregate/rename metrics | Cardinality reduction, unit normalization | | delta/cumulative | Temporality conversion | Match backend expectations |

• OTTL is the expression language for filtering and transforms. Write conditions against attributes and resource, and test with the collector's dry-run or validation mode before rollout.
• Filter as early and as cheaply as possible: at the agent for per-request noise, at the gateway for policy. Filtering after storage is a cost, not a saving.
• Never drop errors or security events with broad filters; scope filters to known-noise routes and require a comment plus an owner for each rule.
• Transformations are an interface: renaming a metric or attribute breaks dashboards, alerts, and saved queries. Treat changes as breaking and version them.
• Validation: reject unknown attributes, enforce enum values, truncate long strings, and drop payload-like fields (request/response bodies) unless explicitly allowlisted.
• Keep one owner per pipeline. Shared processors with per-tenant branches grow into unmaintainable configuration; use routing plus separate config files.

Sampling Strategy by Signal

| Signal | Baseline mechanism | Outcome-based layer | Hard cap | |---|---|---|---| | Traces | Head ratio (deterministic by trace ID) | Tail sampling: errors, latency, attributes | Per-tenant rate limit | | Logs | Level filtering, per-template rate limit | Align with kept traces for incidents | Max bytes per second | | Metrics | Aggregation and relabeling; nothing to sample | N/A | Series limits per metric | | Profiles | Lower sampling frequency | Higher fidelity on demand | Egress rate limit |

• Decide the sampling budget per signal and per tenant, not once globally. One noisy tenant should not exhaust everyone else's budget.
• Tail sampling requires sticky routing (./04-tracing.md); metrics cannot be tail-sampled, they are reduced through aggregation and label dropping.
• Log volume reduction order: remove DEBUG, rate-limit templates, drop duplicate storms, then sample INFO. Keep ERROR/security complete.
• Document what sampling preserves: "all errors, all requests over 1 s, 5% of the rest". That sentence is what on-call needs to interpret dashboards.

Retention Tiers

| Tier | Resolution/Completeness | Typical retention | Storage | |---|---|---|---| | Hot | Full resolution, fast queries | Hours to days | Local SSD, memory-cached stores | | Warm | Downsampled or aggregated | Weeks | Object storage with query cache | | Cold | Rollups, compliance copies | Months to years | Object storage, cheapest class |

• Retention follows value, not signal type. Audit and security logs may need a year; debug logs hours; raw traces days; SLO rollups quarters.
• Downsample deliberately: keep the SLO-relevant aggregates at full fidelity, and let high-cardinality detail expire first.
• Object storage is the cost floor for logs, traces, and metrics blocks (Mimir, Thanos, Loki, Tempo, and similar). Query cost then depends on index design and cache hit rate.
• Egress and cross-region replication are frequently larger than storage costs. Keep queries and storage in the same region; replicate only what compliance requires.
• Document retention per signal and data class; if a regulator or customer asks, the answer must be findable in one place.

Cost Governance

| Cost driver | Where it hits | Lever | |---|---|---| | Active metric series | Memory and query load | Cardinality limits, relabeling, aggregation | | Trace volume | Ingest and storage per span | Head plus tail sampling, drop noise spans | | Log bytes | Ingest, index, storage | Level filtering, rate limits, compress, drop duplicates | | Query load | Backend CPU, concurrency | Recording rules, dashboard hygiene, query limits | | Retention | Storage class and replication | Tiered retention, lifecycle policies | | Network egress | Cross-AZ/region transfer | Co-locate collectors and storage; batch | | Profiles | Egress and storage per sample | Frequency tuning, shorter retention for correlated data |

• Make cost visible per service, per tenant, and per team. Chargeback or showback changes behavior faster than any policy memo.
• Budget per signal and per tenant, and alert on budget consumption before the invoice arrives. Cardinality is a leading indicator; bill is a lagging one.
• Define unit economics: cost per million spans, per GB of logs, per million series, per active service. Track trends over releases and traffic growth.
• Guardrails: per-tenant series limits, ingest rate limits, max attribute sizes, and a review gate for new high-volume telemetry.
• Data-quality gate: before optimizing storage, delete telemetry nobody queries. Unused dashboards, stale exporters, and orphaned pipelines are pure cost.
• Ownership: each service owns its telemetry volume and cost. Central platform teams provide the pipeline, limits, and reporting, not unlimited capacity.
• Procurement: negotiate on ingest plus query plus egress; vendors discount differently for each. Bring real usage data to renewals and test portability with OTLP.

Capacity and Reliability

• Size collectors by throughput per core, not guesswork: measure spans/logs/bytes per second per instance under peak, then leave headroom for retries and bursts.
• Horizontal scaling: agent tier scales with nodes; gateway tier scales with throughput and needs sticky routing for tail sampling.
• Graceful degradation order: refuse new data with clear metrics, drop low-priority signals first, and never fail the application because telemetry is down. The SDK must be non-blocking.
• Backend outage behavior: bounded queues with retries and disk fallback where supported; on sustained outage, drop and count rather than exhaust disk and take down the node.
• Test failure modes: kill the backend, saturate the collector, corrupt a pipeline config; verify alarms fire and applications are unaffected.
• Version pin collector and agents; upgrade staging first. Config schemas change across minor releases, and components can be deprecated or moved between core and contrib.

Cost Estimation Worksheet

| Input | Value | How to measure | |---|---|---| | Request rate | req/s | Ingress metrics | | Spans per request | spans/req | Sampling of production traces | | Average span size | bytes | Export payload size / span count | | Log lines per request | lines/req | Log backend ingest counters | | Average log line size | bytes | Collector or backend ingest | | Active series per service | series | scrape_series_added, backend cardinality API | | Retention hot/warm/cold | days | Storage configuration | | Egress per GB | price | Vendor contract |

span GB/day   = req/s * spans/req * bytes/span * 86400 / 1e9
log GB/day    = req/s * lines/req * bytes/line * 86400 / 1e9
series cost   = active_series * price_per_series_month
monthly total = (span + log + profile GB/day * 30 * ingest+storage price)
              + series cost + egress + query cost

• Do the estimate before onboarding a new service or raising sampling. It takes ten minutes and prevents the quarterly surprise.
• Measure bytes/span and bytes/line with real payloads, not guesses; semantic-convention attributes and long stack traces dominate.
• Cost per million requests is the KPI to track quarter over quarter. Volume growth is fine if unit cost falls; flat unit cost with exploding volume is not.
• Include query cost: a dashboard refresh every 30 seconds on a 40-panel board can exceed the ingest cost in a busy backend.

Collector Config Examples

# Drop health checks and static assets before export
processors:
  filter/denoise:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'
        - 'attributes["url.path"] matches "^/static/.*"'

# Truncate long attribute values and drop payload-like fields
  transform/limits:
    trace_statements:
      - context: span
        statements:
          - truncate_all(attributes, 256)
          - truncate_all(resource.attributes, 256)
          - delete_key(attributes, "http.request.body")
          - delete_key(attributes, "http.response.body")

• Test filter conditions against exported sample data before rollout; a wrong OTTL condition can erase a whole signal class silently.
• Keep filter and transform rules close together and commented with owner and reason, because nobody remembers why a rule exists six months later.
• Apply limits at the agent tier for volume reduction and at the gateway for policy; duplicate rules are acceptable when they serve different owners.
• Export dropped-data counters so filtering is measurable rather than folkloric.

Anti-Patterns

• A single huge pipeline for every signal and tenant with no filtering.
• Memory limiter missing or configured as the last processor.
• Filtering after export "for simplicity", paying for data you then discard.
• Sampling metrics; reducing them only through aggregation and label discipline.
• Keeping raw traces and debug logs at the same retention as SLO rollups.
• One collector replica with no PDB, so an upgrade is an observability outage.
• Tail-sampling gateway without sticky routing, producing partial traces.
• No cost attribution, so the platform team absorbs every team's telemetry growth.
• Committing to a vendor's ingest discount without structuring data for portability via OTLP.

Checklist

• [ ] Pipeline per signal with memory_limiter first and batch last.
• [ ] Enrichment centralized; redaction applied at source and in the pipeline.
• [ ] Filters are specific, owned, and reviewed; errors and security events preserved.
• [ ] Sampling documented per signal and per tenant, with hard caps.
• [ ] Retention tiers mapped to data value and compliance needs.
• [ ] Cost visible per service/tenant with budgets and alerts.
• [ ] Collector self-telemetry exported; refusal and queue metrics alerted.
• [ ] Capacity measured under peak; gateway tier scales horizontally with sticky routing.
• [ ] Backend-outage behavior tested; applications unaffected by telemetry failure.
• [ ] Collector/config versions pinned and upgraded via staging first.

Instrumentation Patterns

Scope: library versus application instrumentation, per-language conventions, span naming, testing telemetry, and a review checklist.

Library vs Application Responsibilities

| Concern | Library | Application | |---|---|---| | API usage | Instrument with the OTel API only | Configure SDK and providers once | | Provider setup | Never install a global provider | Install at startup, before serving | | Exporters | Never configure | Choose OTLP or Prometheus path | | Resource | Do not set service identity | Set service.name, version, environment | | Sampler | Do not force sampling | Set sampler; respect parent context | | Logging | Emit structured records via the bridge | Configure handlers, redaction, sampling | | Attribute values | Semantic conventions; no PII | Add domain attributes at boundaries | | Cost controls | Keep cardinality bounded | Enforce budgets and limits |

• A library that installs a global provider breaks every application that also instruments. Use the API; if no SDK is configured, calls must be no-ops.
• Instrumentation for a library belongs in the library or in a contributed instrumentation package, not copy-pasted into each application.
• Version compatibility: instrumentations track framework versions. Pin the instrumentation package and test after framework upgrades.
• Configuration surface for a library: a feature flag to disable instrumentation is acceptable; defaults must not require application changes.

Instrumentation Points

| Boundary | Traces | Metrics | Logs | |---|---|---|---| | Inbound HTTP/RPC handler | Server span, http.route, status | Request counter, duration histogram, in-flight gauge | Structured request log with trace ID | | Outbound HTTP/RPC call | Client span, target address | Client request counter, duration histogram | Error logs with dependency context | | Database call | Client span with db.* attributes | Query counter, latency histogram | Slow-query logs, no literals | | Message publish | Producer span plus trace context in headers | Publish counter, size histogram | Event published/consumed | | Message consume | Consumer span linked to producer | Consume counter, lag gauge, process duration | Poison-message error logs | | Cache | Client span, hit/miss attribute | Hit/miss counter, latency histogram | Optional | | Background job | Root span or linked span | Job run counter, duration histogram | Start/finish with outcome | | Key business action | Internal span | Domain counter (orders placed) | Business event log |

• Instrument at the edges of the process and at expensive internal steps. Over-instrumentation adds overhead and noise; under-instrumentation leaves blind spots at the exact boundary where incidents live.
• The rule of one: one inbound server span per request, one client span per outbound call. Duplicate spans from layered interceptors create confusing trees.
• Business metrics belong on the same dashboard as technical metrics: orders per second next to error rate.

Span Naming and Attributes

| Element | Convention | Example | |---|---|---| | HTTP server span | {method} {route} | GET /orders/{id} | | HTTP client span | {method} or {method} {host} | POST payments-api | | Database span | {operation} {table} or db.query {summary} | SELECT orders | | Messaging producer | {destination} publish | orders publish | | Messaging consumer | {destination} process | orders process | | Internal | Lowercase domain verb | validate.cart, pricing.calculate | | Job | job {name} | job nightly-reconcile |

• Low cardinality is mandatory: no IDs, no raw paths, no SQL text, no user input in span names. High-cardinality names explode span metrics and search indexes.
• Attributes: use semantic conventions first, namespace custom attributes (app.*, tenant.id), and keep value sizes bounded. Long free-text belongs in logs.
• Error handling: set span status ERROR only for failed operations; record one exception event with type, message, and stack trace at the layer that handles it.
• Sensitive data: never put tokens, credentials, request bodies, or PII in span attributes or baggage.

Metric Instrument Selection

| You want to record | Instrument | Example name | |---|---|---| | Number of events | Counter | orders_created_total | | Current value | Gauge / UpDownCounter | queue_depth | | Size or duration | Histogram | http_request_duration_seconds | | In-flight operations | UpDownCounter | http_requests_in_flight | | Background job outcome | Counter with outcome label | jobs_completed_total | | Cache effectiveness | Two counters or one with result | cache_requests_total{result="hit"} |

• Prefer adding a bounded result/outcome label over multiple metric names. Keep label cardinality under control (./03-metrics.md).
• Set units in instrument metadata and encode them in the name where the ecosystem expects it (_seconds, _bytes, _total).
• Do not emit a metric per request from a high-traffic path without a histogram/counter design; cardinality is the usual failure.
• Runtime metrics (GC, heap, event loop lag, thread pools) come from auto-instrumentation or exporters; enable them, do not reimplement them.

Telemetry Testing

unit test            -> in-memory span exporter, assert span names/attributes/status
integration test     -> collector container, assert exported payload (OTLP file or debug exporter)
contract test        -> assert required resource attributes and metric names exist
CI pipeline          -> telemetry lint (attribute allowlist, naming, no PII patterns)

• Use an in-memory exporter (InMemorySpanExporter, SimpleSpanProcessor) to assert spans in unit tests. Assert names, key attributes, status, and parent-child structure, not every field.
• Assert metrics with a test reader: instrument a code path, collect, and check the value, labels, and type. Histograms: assert bucket boundaries and counts.
• Integration tests run the collector with the debug or file exporter and inspect the emitted payload; this catches pipeline misconfiguration that unit tests cannot.
• Golden tests for log schemas: snapshot a structured log record and fail when fields are added, removed, or renamed unintentionally.
• Test redaction explicitly: pass a payload containing a fake token or email and assert it never appears in exported telemetry.
• Assert resource identity (service.name, version) in at least one contract test; missing identity is invisible until an incident.
• Verify shutdown flushing in a test where feasible; telemetry lost on exit is a silent defect.
• Telemetry linting in CI: forbidden attribute names, PII patterns, missing units, span-name cardinality (regex for IDs and URLs).

Per-Language Conventions

| Language | API/SDK | Structured logs | Typical pitfalls | |---|---|---|---| | Python | opentelemetry-api / -sdk, opentelemetry-instrument | structlog or stdlib logging bridge | WSGI/ASGI context loss; blocking exporter threads | | Node.js | @opentelemetry/api / -sdk-node, preload for auto | pino | AsyncLocalStorage context; ESM loader requirements | | Go | go.opentelemetry.io/otel | log/slog plus trace handler | Context must be passed explicitly; goroutine leaks | | Java/Kotlin | OTel Java API/agent; Micrometer for metrics | Logback/Log4j2 JSON encoders | MDC context vs OTel context; agent ordering | | Rust | tracing plus tracing-opentelemetry | tracing JSON layer | No global subscriber in libraries; async span guards | | .NET | OTel .NET API/SDK, auto-instrumentation hook | Microsoft.Extensions.Logging JSON | Activity vs span naming; HttpClient instrumentation | | Ruby/PHP | OTel gems/extension | Framework loggers | Short-lived processes; per-request overhead |

• Match your language's context mechanism (contextvars, AsyncLocalStorage, context.Context, MDC/thread-local, tracing spans) and verify propagation through the framework's async primitives.
• Framework integrations should be preferred over hand-rolled middleware; they handle header extraction, route templating, and error status correctly.
• Keep one instrumentation style per codebase. Mixing vendor SDKs and OTel APIs in the same path produces duplicate or conflicting telemetry.

Code Review Checklist

• [ ] Resource identity set once at startup; consistent with other signals and other services.
• [ ] Libraries use API only; provider configured by the application.
• [ ] Span names low-cardinality and follow conventions; one span per operation.
• [ ] Attributes use semantic conventions; no PII, no payloads, sizes bounded.
• [ ] Errors recorded once; status set only when the operation failed.
• [ ] Metrics use the right instrument; units in name and metadata; no unbounded labels.
• [ ] Logs structured with severity and trace context; no secrets or PII.
• [ ] Propagation verified across HTTP, gRPC, and queue boundaries.
• [ ] Sampling configured and consistent with the parent decision.
• [ ] Span/metric/log limits set to guard against runaway instrumentation.
• [ ] Telemetry tests present and passing; redaction test included.
• [ ] Collector pipeline accepts the new telemetry; cardinality and volume estimated.
• [ ] Shutdown flushes telemetry with a deadline.
• [ ] Dashboards or alerts updated when metric names or labels change.

Minimal Instrumentation Example

from opentelemetry import trace, metrics
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)
meter = metrics.get_meter(__name__)

requests = meter.create_counter(
    "orders_created_total", unit="{order}", description="Orders created")
latency = meter.create_histogram(
    "order_create_duration_seconds", unit="s", description="Order creation latency")

def create_order(ctx, req):
    with tracer.start_as_current_span("create_order") as span:
        span.set_attribute("app.order.channel", req.channel)
        requests.add(1, {"result": "attempt"})
        start = time.perf_counter()
        try:
            order = persist(ctx, req)
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, "persist failed"))
            requests.add(1, {"result": "error"})
            raise
        finally:
            latency.record(time.perf_counter() - start)
        span.set_attribute("app.order.id", order.id)
        requests.add(1, {"result": "success"})
        logger.info("order created", extra={"order_id": order.id, "channel": req.channel})
        return order

• Labels are bounded (result is an enum); the order ID is a span attribute, not a metric label; the log carries the order ID in a field, not the message.
• Record the histogram on every path, including errors, or latency panels will lie about failure latency.
• The span name is stable and low-cardinality; the route and method belong in attributes for HTTP handlers (http.request.method, http.route).

Framework and Library Boundaries

• Prefer the official instrumentation package for your framework (HTTP servers, ORMs, queue clients, gRPC). It handles context extraction, route templating, and status mapping correctly.
• When no package exists, write a thin adapter that follows semantic conventions rather than a bespoke span shape. Contribute it upstream if it is generic.
• Avoid layered instrumentation that creates duplicate spans: one server span per request, not one per middleware, unless the middleware does real work worth a span.
• For shared internal libraries (clients, repositories), instrument with spans and metrics using the API, and let the application decide the exporter and sampler.
• Watch dependency initialization order: SDK setup must run before the first instrumented client is constructed, or early calls become orphan spans.
• For serverless and short-lived processes, flush telemetry before the handler returns; batch processors may not flush automatically.

Anti-Patterns

• Installing global providers inside libraries or twice at startup.
• Creating a new span per loop iteration or per item in a batch.
• Putting identifiers, SQL, or URLs into span names.
• Recording exceptions at every layer, producing ten copies of one stack trace.
• Metrics with user IDs, tenant-supplied strings, or error messages as labels.
• Logging and tracing the same detail twice at different levels "just in case".
• Testing only that instrumentation does not crash, never that it emits correct data.
• Copying telemetry configuration between services without adapting resource identity.
• Treating instrumentation as a one-time task instead of reviewed, tested code.

Summary

• Instrument at boundaries, with semantic conventions, and keep identity consistent across signals.
• Libraries emit via the API; applications own SDK, sampling, redaction, and budgets.
• Telemetry is tested and reviewed like any other code path.
• Every signal has an owner, a limit, and a reason to exist; anything else is cost and noise.